The state of mainline hardware decoding

[gusarg81]

Hi!

Thanks for reply. Seems I have decoding working with ffmpeg, following steeps here https://github.com/blakeblackshear/friga...-939294837:

sudo ffmpeg -f v4l2 -input_format mjpeg -video_size 1920x1080 -hwaccel drm -loglevel debug -i /dev/video1 -c:v copy output.mp4

...
Splitting the commandline.
Reading option '-f' ... matched as option 'f' (force format) with argument 'v4l2'.
Reading option '-input_format' ... matched as AVOption 'input_format' with argument 'mjpeg'.
Reading option '-video_size' ... matched as AVOption 'video_size' with argument '1920x1080'.
Reading option '-hwaccel' ... matched as option 'hwaccel' (use HW accelerated decoding) with argument 'drm'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Reading option '-i' ... matched as input url with argument '/dev/video1'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'copy'.
Reading option 'output.mp4' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input url /dev/video1.
Applying option f (force format) with argument v4l2.
Applying option hwaccel (use HW accelerated decoding) with argument drm.
Successfully parsed a group of options.
Opening an input file: /dev/video1.
[video4linux2,v4l2 @ 0xaaaadf294d50] fd:3 capabilities:84a00001
[video4linux2,v4l2 @ 0xaaaadf294d50] Current input_channel: 0, input_name: Camera 1, input_std: 0
[mjpeg @ 0xaaaadf2960a0] marker=d8 avail_size_in_buf=182990
[mjpeg @ 0xaaaadf2960a0] marker parser used 0 bytes (0 bits)
[mjpeg @ 0xaaaadf2960a0] marker=c0 avail_size_in_buf=182988
[mjpeg @ 0xaaaadf2960a0] Changing bps from 0 to 8
[mjpeg @ 0xaaaadf2960a0] sof0: picture: 1920x1080
[mjpeg @ 0xaaaadf2960a0] component 0 2:1 id: 0 quant:0
[mjpeg @ 0xaaaadf2960a0] component 1 1:1 id: 1 quant:1
[mjpeg @ 0xaaaadf2960a0] component 2 1:1 id: 2 quant:1
[mjpeg @ 0xaaaadf2960a0] pix fmt id 21111100
[mjpeg @ 0xaaaadf2960a0] Format yuvj422p chosen by get_format().
[mjpeg @ 0xaaaadf2960a0] marker parser used 17 bytes (136 bits)
[mjpeg @ 0xaaaadf2960a0] marker=db avail_size_in_buf=182969
[mjpeg @ 0xaaaadf2960a0] index=0
[mjpeg @ 0xaaaadf2960a0] qscale[0]: 2
[mjpeg @ 0xaaaadf2960a0] index=1
[mjpeg @ 0xaaaadf2960a0] qscale[1]: 4
[mjpeg @ 0xaaaadf2960a0] marker parser used 132 bytes (1056 bits)
[mjpeg @ 0xaaaadf2960a0] marker=c4 avail_size_in_buf=182835
[mjpeg @ 0xaaaadf2960a0] marker parser used 0 bytes (0 bits)
[mjpeg @ 0xaaaadf2960a0] escaping removed 398 bytes
[mjpeg @ 0xaaaadf2960a0] marker=da avail_size_in_buf=182415
[mjpeg @ 0xaaaadf2960a0] marker parser used 182017 bytes (1456136 bits)
[mjpeg @ 0xaaaadf2960a0] marker=d9 avail_size_in_buf=7
[mjpeg @ 0xaaaadf2960a0] decode frame unused 7 bytes
[video4linux2,v4l2 @ 0xaaaadf294d50] All info found
...

Still I am not sure if using hwaccel in decoding, but at least CPU use is close to 2-4%, but I don't know if this is because used copy on encoding. Is there a way to know if using hwaccel in decoding here? I don't know if this part of the debugs says so: "Applying option hwaccel (use HW accelerated decoding) with argument drm."

That being done, I need encoding or some way to stream RTSP with decoding only (I say this because my main idea is to use rtsp-simple-server but needs encoding too for that)

Any thoughts?

[CounterPillow]

The example you posted doesn't decode the stream at all. It stream copies the mjpeg into an mp4 container. This is not a valid way to test whether you are hardware decoding.

What you want is something like

ffmpeg -i video.mp4 -hwaccel drm -f null -

This will decode the intput video.mp4 with the hwaccel, and just throw away the output (decoded) data. (note the - at the end, that's required)

[gusarg81]

What is not clear for me after read al this info: does decoding/encoding works with mainline kernel? (lets say, 5.13) or there is some progress about that subject? I insist yet with this board because is the only ARM one I have, and the only I will have like forever, since everything is way expensive right now in my country (Argentina) and I can't afford for a better board.

I've been able to achieve hardware decoding on mainline kernels and this newer ffmpeg fork: https://github.com/jernejsk/FFmpeg

All you need is the device tree changes to make the decoding nodes show up. I can help you getting that to work if you don't know how.

Hardware encoding isn't a thing yet, but there is work being done on it, though mostly on the rk3399 from what I understand.

Hi! Thanks for reply, I didn't saw this reply before.

I've managed to make decoding work with FFmpeg (https://github.com/jernejsk/FFmpeg) and mailine Kernel (I think, since CPU use is very low in the test). But, I need the encoding part as well, since my idea is to use it as rtsp server as well, for example, using rtsp-simple-server (https://github.com/aler9/rtsp-simple-server), to expose video as HLS (so I can manage it easily on a web platform).

All this because my idea is to develop a Smart Doorbell, having what I have:

- Rock64
- UVC USB Video Camera (https://www.arducam.com/product/b0205-ar...h-ir-leds/)
- A small usb Mic

If someone have and idea to avoid the encoding part, please let me know (because I am kind new to the world of decoding/encoding and all related video processing stuff).

Thanks in advance.

[CounterPillow]

If someone have and idea to avoid the encoding part, please let me know (because I am kind new to the world of decoding/encoding and all related video processing stuff).

You can avoid both the encoding and decoding part if you stream-copy, though whether the MJPEG stream coming from the webcam is suitable for this is an entirely different question.

Here's an example:

Code:

ffmpeg -input_format mjpeg -framerate 30 -video_size 1280x720 -i /dev/video0 -c:v copy -f rtsp rtsp://localhost:8554/live

the -input_format mjpeg -framerate 30 -video_size 1280x720 before the -i /dev/video0 set up some options for our input, namely that we ask video4linux2 for the mjpeg stream of the camera, at a framerate of 30 frames per second, with a video size of 1280x720. -i /dev/video0 sets our input device; in this case /dev/video0 (replace with 1 or 2 if there are other cameras on the system).

-c:v copy tells ffmpeg to stream-copy the video stream, meaning it will not decode or encode it. -f rtsp sets the output format to rtsp, and what follows is the URL for this.

In this case, you won't be needing any hardware decoding or encoding, and your CPU won't be stressing too much as it's essentially doing not much at all.

Back to the topic at hand though: there currently is no mainline driver implementation of the hardware encoder for this particular SoC, as far as I'm aware. But that shouldn't be too big of an issue for mjpeg encoding if you end up having to reencode, as it should be very fast either way.

[mjhammel]

Previously I had H.264 decoding working on a 5.10.17 kernel with patches from Pine64 (pbp-tools) for an rk3399.  We had a need to bump to 5.15.14 so I needed to update the decoding stuff, plus get MJPEG encoding working.

I found that the old patches are no longer valid.  The upstreamed rkvdec driver in 5.15.14 works (unpatched) for H.264 decoding using the ffmpeg for xbmc (tag: 4.4-N-Alpha1) plus the LibreELEC patches.  I can decode 4 streams of RTSP, output to null, at 30fps with about 6% CPU usage.  Without hardware decoding this would have been over 25% CPU usage.  The command I used for this test is as shown.

ffmpeg -y  -hwaccel drm -hwaccel_device /dev/dri/renderD128  -rtsp_transport tcp -i rtsp://192.168.101.47/stream1  -rtsp_transport tcp -i rtsp://192.168.101.47/stream1  -rtsp_transport tcp -i rtsp://192.168.101.47/stream1  -rtsp_transport tcp -i rtsp://192.168.101.47/stream1 -threads 1 -f null -

I have not been able to get MJPEG encoding working.  This is my test command, which may not be correct.

ffmpeg -y \
-hwaccel drm \
-hwaccel_device /dev/dri/renderD128 \
-rtsp_transport tcp \
-i rtsp://192.168.101.47/stream1 \
-c:v mjpeg_vaapi -q:v 10 -an -y -frames 900 -vsync 2 \
-threads 1 -f null -

This command fails.

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (mjpeg_vaapi))
Press [q] to stop, [?] for help
[h264 @ 0x19969a10] v4l2_request_try_format: pixelformat 875967059 not supported for type 10
[h264 @ 0x19969a10] v4l2_request_probe_video_device: try output format failed
[h264 @ 0x19969a10] v4l2_request_probe_video_device: set controls failed, Invalid argument (22)
Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed!

Changing the output codec to mjpeg just gets me a CPU-based encoding, so CPU usage goes up to around 25%.

There is ony one v4l2 JPEG controls on the video1 (and none on video2) device but I'm not sure that implies MJPEG support.

$ v4l2-ctl --list-devices
rockchip,rk3399-vpu-enc (platform: hantro-vpu):
        /dev/video1
        /dev/video2
        /dev/media1

rkvdec (platform:rkvdec):
        /dev/video0
        /dev/media0


$ v4l2-ctl --list-ctrls --device /dev/video1 | grep -i jpeg
JPEG Compression Controls

There is a recent patch for 5.18.x that may improve this but I don't see any additional controls in the patch.  I have not applied the patch or attempted backporting it yet.

If anyone has additional information on the state of mjpeg encoding and/or appropriate tests to use let me know.  Thanks.

[sigmaris]

Code:

-c:v mjpeg_vaapi

This isn't going to get you hardware acceleration on RK3399, this is telling FFmpeg to use the VA-API to talk to the kernel/hardware, but the Hantro encoder driver in the kernel doesn't support this API, it only supports v4l2.

If there was a v4l2-accelerated MJPEG encoder in FFmpeg it would be called "mjpeg_v4l2m2m", but I don't think such an encoder exists, at least I can't see it in my list of ffmpeg -encoders, and I can't find any mention of such a thing online.

I've only seen GStreamer's v4l2jpegenc mentioned as working with the Hantro JPEG encoder, in this thread. I recommend trying the GStreamer encoder (and, the v4l2 stateless codecs e.g. v4l2slh264dec in Gstreamer 1.18 should work for hw-accelerated decoding on RK3399, too)

[mjhammel]

Code:

-c:v mjpeg_vaapi

This isn't going to get you hardware acceleration on RK3399, this is telling FFmpeg to use the VA-API to talk to the kernel/hardware, but the Hantro encoder driver in the kernel doesn't support this API, it only supports v4l2.

If there was a v4l2-accelerated MJPEG encoder in FFmpeg it would be called "mjpeg_v4l2m2m", but I don't think such an encoder exists, at least I can't see it in my list of ffmpeg -encoders, and I can't find any mention of such a thing online.

I've only seen GStreamer's v4l2jpegenc mentioned as working with the Hantro JPEG encoder, in this thread. I recommend trying the GStreamer encoder (and, the v4l2 stateless codecs e.g. v4l2slh264dec in Gstreamer 1.18 should work for hw-accelerated decoding on RK3399, too)

I'll give this a try, just to prove the hw encoding works in general.  However the request from my team was to see if v4l2 itself (and specifically not gstreamer) could be used for encoding.  Would you happen to know of a reference on how to use v4l2 directly (like on a commane line) to show hw encoding, specifically on mjpeg, on rk3399?

Also: I have a patch for ffmpeg 4.3.1 that adds v4l2_m2m_mjpeg encoder support. I don't know if it works, but it was in the pbp-tools collection back when I worked with the 5.10.17 kernel.

[sigmaris]

I'll give this a try, just to prove the hw encoding works in general.  However the request from my team was to see if v4l2 itself (and specifically not gstreamer) could be used for encoding.  Would you happen to know of a reference on how to use v4l2 directly (like on a commane line) to show hw encoding, specifically on mjpeg, on rk3399?

Also: I have a patch for ffmpeg 4.3.1 that adds v4l2_m2m_mjpeg encoder support.  I don't know if it works, but it was in the pbp-tools collection back when I worked with the 5.10.17 kernel.

I don't currently know of an easy way to use v4l2 directly via a CLI tool to encode JPEGs, other than GStreamer's gst-launch with v4l2jpegenc - that's what I'd try first for a CLI tool. Of course if you want to use the v4l2 API directly, you can look at what the GStreamer source code is doing and the API documentation. Section 4.5.2 seems to cover the relevant interface for encoding.