The state of mainline hardware decoding

[mjhammel]

Previously I had H.264 decoding working on a 5.10.17 kernel with patches from Pine64 (pbp-tools) for an rk3399.  We had a need to bump to 5.15.14 so I needed to update the decoding stuff, plus get MJPEG encoding working.

I found that the old patches are no longer valid.  The upstreamed rkvdec driver in 5.15.14 works (unpatched) for H.264 decoding using the ffmpeg for xbmc (tag: 4.4-N-Alpha1) plus the LibreELEC patches.  I can decode 4 streams of RTSP, output to null, at 30fps with about 6% CPU usage.  Without hardware decoding this would have been over 25% CPU usage.  The command I used for this test is as shown.

ffmpeg -y  -hwaccel drm -hwaccel_device /dev/dri/renderD128  -rtsp_transport tcp -i rtsp://192.168.101.47/stream1  -rtsp_transport tcp -i rtsp://192.168.101.47/stream1  -rtsp_transport tcp -i rtsp://192.168.101.47/stream1  -rtsp_transport tcp -i rtsp://192.168.101.47/stream1 -threads 1 -f null -

I have not been able to get MJPEG encoding working.  This is my test command, which may not be correct.

ffmpeg -y \
-hwaccel drm \
-hwaccel_device /dev/dri/renderD128 \
-rtsp_transport tcp \
-i rtsp://192.168.101.47/stream1 \
-c:v mjpeg_vaapi -q:v 10 -an -y -frames 900 -vsync 2 \
-threads 1 -f null -

This command fails.

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (mjpeg_vaapi))
Press [q] to stop, [?] for help
[h264 @ 0x19969a10] v4l2_request_try_format: pixelformat 875967059 not supported for type 10
[h264 @ 0x19969a10] v4l2_request_probe_video_device: try output format failed
[h264 @ 0x19969a10] v4l2_request_probe_video_device: set controls failed, Invalid argument (22)
Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed!

Changing the output codec to mjpeg just gets me a CPU-based encoding, so CPU usage goes up to around 25%.

There is ony one v4l2 JPEG controls on the video1 (and none on video2) device but I'm not sure that implies MJPEG support.

$ v4l2-ctl --list-devices
rockchip,rk3399-vpu-enc (platform: hantro-vpu):
        /dev/video1
        /dev/video2
        /dev/media1

rkvdec (platform:rkvdec):
        /dev/video0
        /dev/media0


$ v4l2-ctl --list-ctrls --device /dev/video1 | grep -i jpeg
JPEG Compression Controls

There is a recent patch for 5.18.x that may improve this but I don't see any additional controls in the patch.  I have not applied the patch or attempted backporting it yet.

If anyone has additional information on the state of mjpeg encoding and/or appropriate tests to use let me know.  Thanks.