The state of mainline hardware decoding

[CounterPillow]

Current situation: stuff just works! As long as things implement the v4l2-requests API. See https://wiki.pine64.org/wiki/Mainline_Hardware_Decoding

Original post (no longer applicable)

Or "everything is forever terrible"

First off: I'm not talking about rkmpp. Do not reply to this thread about anything regarding rkmpp, or non-mainline images using rkmpp.

Basically, a sunk a few afternoons of investigating, tinkering and tearing my hair out into getting any hardware decoding working on this using the mainline kernel. My conclusion is that it doesn't work, and won't ever work unless someone puts some effort into making all the moving bits and pieces align.

Here's the farthest one can get, which is upstream gstreamer telling you that despite advertising vp8 decoding, it can't actually decode a vp8 file:

Code:

$ gst-launch-1.0 filesrc location=/home/fratti/ocean.webm ! matroskademux ! v4l2slvp8dec ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/v4l2slvp8dec:v4l2slvp8dec0: Driver does not support the selected stream.
Additional debug info:
../gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp8dec.c(172): gst_v4l2_codec_vp8_dec_negotiate (): /GstPipeline:pipeline0/v4l2slvp8dec:v4l2slvp8dec0
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

It took me forever to find out how to build this simple gst-launch pipeline because their docs are more about API usage, so this will serve as reference for anyone who does a web search for this. Hello web searcher! Yes this is v4l2codecs related! That's the plugin, it's in gst-plugins-bad, fittingly enough.

What else is there?

There's a patchset for ffmpeg that implements the v4l2-requests API. This patchset no longer works, as it relies on the API as it was on older kernels. Since the API was and still is (as of 5.13) in staging, it's still in flux, though recently there have been talks of moving it to mainline proper. It does not look like the collabora person who posted the patchset is interested in upstreaming it any further at this moment, either because their contract stopped at that point or because it doesn't make sense to include it in ffmpeg while the API is still in staging.

The best you'll get if you do hack the patchset into at least building is some errors while decoding:

Code:

[vp8 @ 0xaaaada6f4ec0] v4l2_request_queue_decode: set controls failed for request 29, Invalid argument (22)                                                                            
[vp8 @ 0xaaaada6ff6e0] Discarding interframe without a prior keyframe!
[vp8 @ 0xaaaada702490] Discarding interframe without a prior keyframe!                     
[vp8 @ 0xaaaada705240] Discarding interframe without a prior keyframe!                     
Error while decoding stream #0:0: Operation not permitted
[vp8 @ 0xaaaada707ff0] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input
[vp8 @ 0xaaaada6f4ec0] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input
[vp8 @ 0xaaaada6ff6e0] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input
[vp8 @ 0xaaaada702490] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input
[vp8 @ 0xaaaada705240] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input

This is marvellous, but it's no video.

This post is basically here to let everyone else know that trying right now is a pointless exercise. What appears to need to be done is:

1. fix the kernel side for rk3328 so that they actually work(?) (I'm so unsure I don't even know if it's broken! Also why does it not say it can decode H.264 in gstreamer?!)
   
2. mainline the v4l2-requests api
   
3. write a mergeable patchset against the mainline API for ffmpeg
   
4. you now get hwdec in mpv for free, thanks to ffmpeg
   

Alternative path to the goal:

1. passive-aggressively write a vaapi implementation for the hantro VPU, with no v4l2-requests bits at all because why introduce yet another userspace API for something like this that several other things already do
   
2. never get it merged anywhere because they'll just point to the broken pile of v4l2-requests bits and tell you to use those
   
3. this will also work on DRM, i.e. a terminal with no display server running, at least my AMD vaapi implementation can do this with mpv.
   

What I'd appreciate is the following info:

• ways to tell v4l2 to list all supported codecs/formats. Somehow, v4l2-dbg just does nothing or straight up ruins my terminal by writing binary to it. Good tool. All I want is "vainfo" but for v4l2-requests
  
• information regarding the v4l2-requests design, or where to find such information
  
• anecdotes of using v4l2-requests on any rk3328 on some upstream kernel
  
• information regarding what part of vaapi lives in the kernel, if any, and where one would get started writing a vaapi implementation for hantro without wrapping around v4l2-requests
  

[sigmaris]

ways to tell v4l2 to list all supported codecs/formats. Somehow, v4l2-dbg just does nothing or straight up ruins my terminal by writing binary to it. Good tool. All I want is "vainfo" but for v4l2-requests

There isn't exactly a list, but try listing the controls on each device with

Code:

v4l2-ctl --list-ctrls --device /dev/video0

You can check if they have controls for MPEG2, VP8, VP9, H.264, H.265.

anecdotes of using v4l2-requests on any rk3328 on some upstream kernel

At the moment not all of v4l2-request in the upstream kernel is stable uAPI yet, and drivers are still in staging. So for the best results, not-yet-upstream patches are useful. I have been using the kernel and ffmpeg patches used in LibreELEC in my media centre RockPRO64. Hardware decoding works OK, though I haven't tried VP8 recently - only MPEG-2, H.264 and H.265. But as I recall when I tried VP8 a few months ago it worked. I know that is RK3399 not RK3328, but they both have the hantro and rkvdec video decoder IP blocks.

The key thing is to get FFmpeg libraries that work with the kernel API of the kernel you're running. Hence the usefulness of LibreELEC as they are both maintained in the same project there: https://github.com/LibreELEC/LibreELEC.tv/

[CounterPillow]

Thanks!

Code:

$ v4l2-ctl --list-ctrls --device /dev/video0
        mpeg_2_slice_parameters 0x009909fa (unknown): type=103 flags=has-payload
   mpeg_2_quantization_matrices 0x009909fb (unknown): type=104 flags=has-payload
           vp8_frame_parameters 0x00a409c8 (unknown): type=240 flags=has-payload

This seems to indicate at least mpeg2 and vp8 are exposed. Not sure where H.264 is, maybe it needs something in rkvdec?

I've since learned from ezequielg that there is a patch to make gstreamer work with the new stable VP8 API, which I am currently attempting to build using gst-build. I'll report back with results.

All in all I'm feeling slightly better about the current status than I did 2 hours ago.

[sigmaris]

...
This seems to indicate at least mpeg2 and vp8 are exposed. Not sure where H.264 is, maybe it needs something in rkvdec?
...

This looks like the Hantro decoder device then. The rkvdec device, which supports H.264, would normally appear as another V4L2 device e.g. /dev/video1, but I just checked mainline Linux's device tree for the RK3328, and it doesn't actually include the rkvdec device even though the SoC has one, so your OS probably doesn't know the rkvdec exists. LibreELEC includes a patch that adds the rkvdec device node. I sort of mentioned this earlier, but if you want H.264 acceleration I'd recommend compiling your own kernel with the patches from LibreELEC (and their kernel config to enable e.g. CONFIG_VIDEO_ROCKCHIP_VDEC).

As a side note, the Hantro VPU on the SoC does support H.264 decoding as well, but the register layout is slightly different from other Hantro implementations and so support for it isn't implemented (there is not much benefit as it's limited to 1920x1080 while the rkvdec can decode 4K resolution H.264).

[CounterPillow]

Some more discoveries, thanks to LibreELEC and pgwipeout: the upstream device tree currently does not have the rkvdec video-codec definition for the rk3328. LibreELEC, however, has a patch for it which applies cleanly to linux 5.13.0-rc3 and I'm currently trying to convince my build scripts to actually use. If I can't, then I've got the LibreELEC dts to fall back on, which should be enough to at least make the device show up.

I'll report back with results

EDIT: I see sigmaris just replied with all of this info too. Thank you! I should've refreshed the thread before replying...

[CounterPillow]

Okay, a little update.

Code:

$ v4l2-ctl --list-ctrls --device /dev/video1

Codec Controls

                     h264_level 0x00990a67 (menu)   : min=0 max=15 default=0 value=0
                   h264_profile 0x00990a6b (menu)   : min=0 max=4 default=2 value=2

Stateless Codec Controls

               h264_decode_mode 0x00a40900 (menu)   : min=1 max=1 default=1 value=1
                h264_start_code 0x00a40901 (menu)   : min=1 max=1 default=1 value=1
    h264_sequence_parameter_set 0x00a40902 (unknown): type=200 flags=has-payload
     h264_picture_parameter_set 0x00a40903 (unknown): type=201 flags=has-payload
            h264_scaling_matrix 0x00a40904 (unknown): type=202 flags=has-payload
         h264_decode_parameters 0x00a40907 (unknown): type=204 flags=has-payload

Code:

$ gst-launch-1.0 filesrc location=/home/fratti/duckwater.mp4 ! qtdemux ! v4l2slh264dec ! fakesink
Setting pipeline to PAUSED ...
ERROR: from element /GstPipeline:pipeline0/v4l2slh264dec:v4l2slh264dec0: Driver did not report framing and start code method.
Additional debug info:
../gst-plugins-bad/sys/v4l2codecs/gstv4l2codech264dec.c(134): gst_v4l2_codec_h264_dec_open (): /GstPipeline:pipeline0/v4l2slh264dec:v4l2slh264dec0:
gst_v4l2_decoder_get_controls() failed: Invalid argument
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/v4l2slh264dec:v4l2slh264dec0: Could not initialize supporting library.
Additional debug info:
../gst-plugins-base/gst-libs/gst/video/gstvideodecoder.c(2666): gst_video_decoder_change_state (): /GstPipeline:pipeline0/v4l2slh264dec:v4l2slh264dec0:
Failed to open decoder
ERROR: pipeline doesn't want to preroll.
Failed to set pipeline to PAUSED.
Setting pipeline to NULL ...
Freeing pipeline ...

With the DTS from LibreELEC, I got /dev/video1 to show up. It still doesn't work, but this is progress! I first tried to have the dts stuff generated from just the patches, but that somehow didn't work (video-codec for rkvdec never showed up in the resulting .dtb). So I simply built a dtb from their dts that diederik helpfully provided for me, and it's showing the device now.

My guess is that some of the illegal argument stuff is entirely down to userspace not keeping up with kernel changes. That's ok, and I expect this to get resolved once it's moved out of staging (presumably during the 5.14 merge window.)

I have not yet been able to build gst-plugins-bad with the vp8 API changes, but I'll try again over the coming days. Actually seeing it decode would be awesome.

Preliminary conclusion: rk3328 needs some upstream device tree love, but things are looking way brighter than I initially thought they looked.

I need to learn about device trees before I go chase down the "why does this dtsi change not result in a different rk3328-rock64.dtb" issue. I've already been supplied with some resources for this.

A big thank you to everyone both on IRC/Discord/Matrix and the forums who provided help to get me to this point.

[CounterPillow]

Final update for the next few days: Figured out why my dtb wasn't changing. "git apply" silently failed to apply a patch. On the bright side, I fixed that, on the less bright side, just as I compiled and installed my kernel which would've probably fixed the things it was having trouble with with regards to decoding, the SD card died. Let this be a reminder to not write lots of 3 gigabyte kernel source trees to any random SD card you have, and also to make backups. So now I might get actual work done for a few days until a high endurance card arrives, as I am unable to play with my ROCKs.

[CounterPillow]

New update: I've managed to seemingly get both vp8 and h264 decoding working with gstreamer. You'll need:

1. gst-build to build a git version of gstreamer and its plugins (you'll no longer need this once a new gstreamer release is cut and trickles down to distributions)
   
2. kernel 5.13 (might work with earlier kernels that also have the new uapi)
   
3. the libreELEC device tree changes linked above
   

It's worth noting that the obscure 10-bit H.264 profile decoding, for which rockchip are the only ones who provide a hardware decoder, is not yet possible.

The next step would be playing around with the ffmpeg patches to port them towards the new uapi and the latest ffmpeg version.

[gusarg81]

Hi,

Seems there is a little light for those who have the rk3328?

In my case I have Rock64, abandoned, because for what I need to achieve, does not work. Which is, hwaccel for decoding and encoding.

What is not clear for me after read al this info: does decoding/encoding works with mainline kernel? (lets say, 5.13) or there is some progress about that subject? I insist yet with this board because is the only ARM one I have, and the only I will have like forever, since everything is way expensive right now in my country (Argentina) and I can't afford for a better board.

I've bought this board because Pine64 website specs about Rock64 said that supporst hwaccel (which even with legacy kernel, does not work right and the board crash all the time).

So, I have this board and plus a UVC USB camera module (Arducam, with night vision, IR-Cut, etc) because I wanted to develop a Smart IP Doorbell... but seems is likely impossible so far, because of this main problem.

What a shame...

[CounterPillow]

What is not clear for me after read al this info: does decoding/encoding works with mainline kernel? (lets say, 5.13) or there is some progress about that subject? I insist yet with this board because is the only ARM one I have, and the only I will have like forever, since everything is way expensive right now in my country (Argentina) and I can't afford for a better board.

I've been able to achieve hardware decoding on mainline kernels and this newer ffmpeg fork: https://github.com/jernejsk/FFmpeg

All you need is the device tree changes to make the decoding nodes show up. I can help you getting that to work if you don't know how.

Hardware encoding isn't a thing yet, but there is work being done on it, though mostly on the rk3399 from what I understand.