The state of mainline hardware decoding
#1
Current situation: stuff just works! As long as things implement the v4l2-requests API. See https://wiki.pine64.org/wiki/Mainline_Hardware_Decoding

Original post (no longer applicable)

Or "everything is forever terrible"

First off: I'm not talking about rkmpp. Do not reply to this thread about anything regarding rkmpp, or non-mainline images using rkmpp.

Basically, a sunk a few afternoons of investigating, tinkering and tearing my hair out into getting any hardware decoding working on this using the mainline kernel. My conclusion is that it doesn't work, and won't ever work unless someone puts some effort into making all the moving bits and pieces align.

Here's the farthest one can get, which is upstream gstreamer telling you that despite advertising vp8 decoding, it can't actually decode a vp8 file:

Code:
$ gst-launch-1.0 filesrc location=/home/fratti/ocean.webm ! matroskademux ! v4l2slvp8dec ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/v4l2slvp8dec:v4l2slvp8dec0: Driver does not support the selected stream.
Additional debug info:
../gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp8dec.c(172): gst_v4l2_codec_vp8_dec_negotiate (): /GstPipeline:pipeline0/v4l2slvp8dec:v4l2slvp8dec0
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

It took me forever to find out how to build this simple gst-launch pipeline because their docs are more about API usage, so this will serve as reference for anyone who does a web search for this. Hello web searcher! Yes this is v4l2codecs related! That's the plugin, it's in gst-plugins-bad, fittingly enough.

What else is there?

There's a patchset for ffmpeg that implements the v4l2-requests API. This patchset no longer works, as it relies on the API as it was on older kernels. Since the API was and still is (as of 5.13) in staging, it's still in flux, though recently there have been talks of moving it to mainline proper. It does not look like the collabora person who posted the patchset is interested in upstreaming it any further at this moment, either because their contract stopped at that point or because it doesn't make sense to include it in ffmpeg while the API is still in staging.

The best you'll get if you do hack the patchset into at least building is some errors while decoding:

Code:
[vp8 @ 0xaaaada6f4ec0] v4l2_request_queue_decode: set controls failed for request 29, Invalid argument (22)                                                                            
[vp8 @ 0xaaaada6ff6e0] Discarding interframe without a prior keyframe!
[vp8 @ 0xaaaada702490] Discarding interframe without a prior keyframe!                     
[vp8 @ 0xaaaada705240] Discarding interframe without a prior keyframe!                     
Error while decoding stream #0:0: Operation not permitted
[vp8 @ 0xaaaada707ff0] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input
[vp8 @ 0xaaaada6f4ec0] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input
[vp8 @ 0xaaaada6ff6e0] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input
[vp8 @ 0xaaaada702490] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input
[vp8 @ 0xaaaada705240] Discarding interframe without a prior keyframe!
Error while decoding stream #0:0: Invalid data found when processing input

This is marvellous, but it's no video.

This post is basically here to let everyone else know that trying right now is a pointless exercise. What appears to need to be done is:
  1. fix the kernel side for rk3328 so that they actually work(?) (I'm so unsure I don't even know if it's broken! Also why does it not say it can decode H.264 in gstreamer?!)
  2. mainline the v4l2-requests api
  3. write a mergeable patchset against the mainline API for ffmpeg
  4. you now get hwdec in mpv for free, thanks to ffmpeg

Alternative path to the goal:
  1. passive-aggressively write a vaapi implementation for the hantro VPU, with no v4l2-requests bits at all because why introduce yet another userspace API for something like this that several other things already do
  2. never get it merged anywhere because they'll just point to the broken pile of v4l2-requests bits and tell you to use those
  3. this will also work on DRM, i.e. a terminal with no display server running, at least my AMD vaapi implementation can do this with mpv.
What I'd appreciate is the following info:
  • ways to tell v4l2 to list all supported codecs/formats. Somehow, v4l2-dbg just does nothing or straight up ruins my terminal by writing binary to it. Good tool. All I want is "vainfo" but for v4l2-requests
  • information regarding the v4l2-requests design, or where to find such information
  • anecdotes of using v4l2-requests on any rk3328 on some upstream kernel
  • information regarding what part of vaapi lives in the kernel, if any, and where one would get started writing a vaapi implementation for hantro without wrapping around v4l2-requests

Occasional Linux Kernel Contributor, Avid Wiki Updater, Ask Me About Quartz64
Open Hardware Quartz64 Model A TOSLink Adapter
Pi-bus GPIO Extender For ROCKPro64 And Quartz64 Model A
Plebian GNU/Linux
  Reply


Messages In This Thread
The state of mainline hardware decoding - by CounterPillow - 05-28-2021, 03:39 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
Information Linux Mainline ayufan 63 90,266 05-14-2021, 10:41 AM
Last Post: Wizzard
  Hardware acceleration using FFmpeg gusarg81 0 2,818 08-21-2020, 01:36 PM
Last Post: gusarg81
  mainline kernel sound support Openwrt lucize 2 3,993 05-01-2020, 05:09 PM
Last Post: PakoSt
  Hardware Status Monitoring? Leapo 7 14,896 03-15-2020, 08:44 AM
Last Post: bendem
  Any advantages to using the mainline kernel dkebler 0 2,082 11-16-2019, 12:17 PM
Last Post: dkebler
  Does anybody run the mainline kernel? CameronNemo 3 4,561 09-09-2019, 07:56 PM
Last Post: CameronNemo
  hardware-accelerated video transcoding (Plex) on Rock64 mdr 2 7,219 02-07-2019, 03:42 PM
Last Post: mdr
  Updates on Mainline Linux Support? hnaguski 0 2,434 09-11-2018, 08:16 PM
Last Post: hnaguski
  How to do hardware decoding of video? SuperSaiyanCaleb 9 15,370 08-28-2018, 01:39 PM
Last Post: mcerveny
  H264 hardware encoder not work sueshieh 3 6,427 11-02-2017, 03:57 AM
Last Post: dalmate

Forum Jump:


Users browsing this thread: 2 Guest(s)