mainline kernel hardware acceleration

[xmixahlx]

fyi you can apply rkvdec h264 and vp9 kernel patches to 5.7-rcX. rk3399 is currently dependent upon improved vaapi and v4l2 solutions for any media decoding to be utilized by chromium or firefox.

firefox 75 (and improved in 76) DOES allow for vaapi on wayland, and this seems to be the most likely early testbed, but dependent on the comment above.

[xmixahlx]

an update:

kwiboo has added hevc/h265 decoding to rkvdec and is available in his linux-rockchip and ffmpeg repositories. with the two rk3399 vpu decoders (hantro, rkvdec) this means we have the following hardware decoding support:

hantro: mpeg2, vp8, h264
rkvdec: h264, h265, vp9

both hantro and rkvdec support h264 decoding, but hantro is limited to 1080p. rkvdec is the better solution.

all hardware decoders roughly require 50% compute power of one of the big cores.

ffmpeg works directly with v4l2-request api i.e. vaapi and libva-v4l2-request are not utilized.

a desktop solution for improved firefox and chrome support requires improved vaapi handling, likely building upon ph5's libva hantro work.

vaapi support already exists in firefox on wayland with v75 (h264) and expanded with v76 (all).

if you want to test this yourself i am using tsys 5.7-rc2 patched to rc5 + LibreElec pull requests for 5.6 support (only including next/list), plus hevc specific patches for linux-rockchip and ffmpeg with something like: ffmpeg -report -re -hwaccel drm -hwaccel_device /dev/dri/card0 -i $FILE -pix_fmt bgra -f null -

[xmixahlx]

@termac from the mainline kernel thread

... i tried my hands on getting hardware decoding working, for that I added the rkvdec kernel patches from https://patchwork.kernel.org/project/lin...ter=179601 and patched libva and libva-v4l2-request as per Your instructions in the other thread. vainfo still would only show support for MPEG2. At that point I did give up. So now it is either waiting for full mainline support or have enough time and motivation to figure out where I went wrong.

vainfo will display a single media decoder module, and it defaults to /dev/video0 or whatever you have set as LIBVA_V4L2_REQUEST_VIDEO_PATH (see first few posts). since rk3399 has two (hantro and rkvdec) you need to select them with vainfo separately. with mpeg2 you are seeing hantro. check devices with v4l2-ctl --list-devices. check each device with v4l2-ctl -d /dev/video# --all.

it's notable that ffmpeg currently doesn't list the v4l2request hwaccel decoders in runtime ((ffmpeg -decoders|grep v4l2), but does list v4l2request codecs as detected during build.

it's also notable that these patches handle drm hwaccel because that is what LibreElec uses for kodi.

kernel patch info:

Kwiboo's LibreElec Rockchip 5.6 patches and pull request:
https://github.com/Kwiboo/LibreELEC.tv/t...ux/default
https://github.com/LibreELEC/LibreELEC.tv/pull/4377

Kwiboo hevc kernel repo branch:
https://github.com/Kwiboo/linux-rockchip...ist-v5.6.6
https://github.com/Kwiboo/linux-rockchip...5.6.6-hevc
make a diff of these two above repositories (exclude=.git exclude=sunxi)

Kwiboo's ffmpeg branch supporting rkvdec:
https://github.com/Kwiboo/FFmpeg/tree/v4...2.2-rkvdec
build: ./configure --prefix=/usr/local --libdir=/usr/local/lib/aarch64-linux-gnu --shlibdir=/usr/local/lib/aarch64-linux-gnu --enable-shared --enable-pic --enable-v4l2-request --enable-libdrm --enable-libudev && make -j6 && sudo make install

pbp linux 5.7-rc5 hwaccel patch sequence:
tsys 5.7.0-rc2 branch
5.7.0-rc3 incremental patch
5.7.0-rc4 incremental patch
5.7.0-rc5 incremental patch
LibreElec 002, 0012, 0013, 0021, 0022, 1000 patches
Kwiboo hevc patch

[termac]

vainfo will display a single media decoder module, and it defaults to /dev/video0 or whatever you have set as LIBVA_V4L2_REQUEST_VIDEO_PATH (see first few posts). since rk3399 has two (hantro and rkvdec) you need to select them with vainfo separately. with mpeg2 you are seeing hantro. check devices with v4l2-ctl --list-devices. check each device with v4l2-ctl -d /dev/video# --all.
...

Thanks for pointing that out. I only checked video0 and video1. With LIBVA_V4L2_REQUEST_VIDEO_PATH=/dev/video4 I get:

Code:

$>vainfo
libva info: VA-API version 1.8.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/aarch64-linux-gnu/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_8
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.8 (libva 2.7.0)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
     VAProfileH264Main               : VAEntrypointVLD
     VAProfileH264High               : VAEntrypointVLD
     VAProfileH264ConstrainedBaseline: VAEntrypointVLD
     VAProfileH264MultiviewHigh      : VAEntrypointVLD
     VAProfileH264StereoHigh         : VAEntrypointVLD

Code:

$>v4l2-ctl -d /dev/video4 --all
Driver Info:
Driver name      : rkvdec
Card type        : rkvdec
Bus info         : platform:rkvdec
Driver version   : 5.7.0
Capabilities     : 0x84204000
Video Memory-to-Memory Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps      : 0x04204000
Video Memory-to-Memory Multiplanar
Streaming
Extended Pix Format
Media Driver Info:
Driver name      : rkvdec
Model            : rkvdec
Serial           :
Bus info         : platform:rkvdec
Media version    : 5.7.0
Hardware revision: 0x00000000 (0)
Driver version   : 5.7.0
Interface Info:
ID               : 0x0300000c
Type             : V4L Video
Entity Info:
ID               : 0x00000001 (1)
Name             : rkvdec-source
Function         : V4L2 I/O
Pad 0x01000002   : 0: Source
 Link 0x02000008: to remote pad 0x1000004 of entity 'rkvdec-proc': Data, Enabled, Immutable
Priority: 2
Format Video Capture Multiplanar:
Width/Height      : 48/48
Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
Field             : None
Number of planes  : 1
Flags             :
Colorspace        : Rec. 709
Transfer Function : Default
YCbCr/HSV Encoding: Default
Quantization      : Default
Plane 0           :
  Bytes per Line : 48
  Size Image     : 4608
Format Video Output Multiplanar:
Width/Height      : 48/48
Pixel Format      : 'S264' (H.264 Parsed Slice Data)
Field             : None
Number of planes  : 1
Flags             :
Colorspace        : Rec. 709
Transfer Function : Default
YCbCr/HSV Encoding: Default
Quantization      : Default
Plane 0           :
  Bytes per Line : 0
  Size Image     : 4608

Codec Controls

                   vp9_profile 0x00990b00 (menu)   : min=0 max=0 default=0 value=0
0: 0
   h264_sequence_parameter_set 0x00990ce8 (unknown): type=110 flags=has-payload
    h264_picture_parameter_set 0x00990ce9 (unknown): type=111 flags=has-payload
           h264_scaling_matrix 0x00990cea (unknown): type=112 flags=has-payload
         h264_slice_parameters 0x00990ceb (unknown): type=113 flags=has-payload
        h264_decode_parameters 0x00990cec (unknown): type=114 flags=has-payload
              h264_decode_mode 0x00990ced (menu)   : min=1 max=1 default=1 value=1
1: Frame-Based
               h264_start_code 0x00990cee (menu)   : min=1 max=1 default=1 value=1
1: Annex B Start Code
           vp9_frame_context_0 0x009918a0 (unknown): type=400 flags=has-payload
           vp9_frame_context_1 0x009918a1 (unknown): type=400 flags=has-payload
           vp9_frame_context_2 0x009918a2 (unknown): type=400 flags=has-payload
           vp9_frame_context_3 0x009918a3 (unknown): type=400 flags=has-payload
   vp9_frame_decode_parameters 0x009918a4 (unknown): type=404 flags=has-payload

To me that seems like from the driver and v4l2 side everything is in place. I tried playing video with mpv (unpatched) and the --hwdec options "vaapi" and "v4l2m2m-copy". In the first case I get no error, but the playback drops quite alot of frames and in the second case I get the error

Code:

[ffmpeg/video] h264_v4l2m2m: Could not find a valid device
[ffmpeg/video] h264_v4l2m2m: can't configure decoder
Could not open codec.

So maybe I am just missing a patched ffmpeg now. I will give that a try next.
If that does not work out, I will try the exact patches You listed, as currently I have gotten here using the patches mentioned in the quote from the other thread. Thank You for looking into it!

pbp linux 5.7-rc5 hwaccel patch sequence:
tsys 5.7.0-rc2 branch
5.7.0-rc3 incremental patch
5.7.0-rc4 incremental patch
5.7.0-rc5 incremental patch
LibreElec 002, 0012, 0013, 0021, 0022, 1000 patches
Kwiboo hevc patch

[xmixahlx]

confirmed kodi-gbm from git supports all hantro and rkvdec codecs flawlessly. significantly less processing power is required even compared to ffmpeg drm hwaccel testing (50% of ~1 core, but multithreaded using small cores, so 1-7% total cpu during playback.

example decoding (total cpu %) of a x265 1080p file:
ffmpeg sw decoding = 60% total cpu
ffmpeg drm hwaccel = 7% total cpu
kodi-gbm drm+gles hwaccel = 2% total cpu

i've added a kodi-git build script in another thread: https://forum.pine64.org/showthread.php?tid=9877

[xmixahlx]

also: @termac vaapi is only working with mpeg2 as far as i can tell. it was one of the first tests months ago.

[AndreVallestero]

@xmixahlx Just a headsup, Armbian has an active thread for trying to get all the patches together and into mainline: https://forum.armbian.com/topic/13622-ma...b=comments. I mentioned this post in that thread so they might be able to learn from your work.

[xmixahlx]

excellent. i'll jump over there, too.

some random updates:
ffmpeg 4.2.3 working with v4l2-request-hwaccel-4.2.2-rkvdec patch (4.2.2 diff)
x265 10bit hwaccel is working
NOT working: vp9 10/12 bit, h264 10bit, h265 12bit
h264 10bit gives kernel hard lock
jellyfin-kodi is working
kodi-youtube is wip
updated the first post with most relevant info
added the armbian thread

[xmixahlx]

h264 10bit hwaccel now working thanks to kwiboo. see his recent linux-rockchip and ffmpeg repo branches.

[termac]

@xmixahlx On the armbian forum thread you wrote:

confirmed x264 10bit hwaccel is working. thanks so much, @Kwiboo
using kodi-git, ffmpeg 4.2.3 + new rkvdec branch work, linux 5.7-rc7 + librelec rockchip patches + linuxtv WIP branch diff + drm-next branch diff

I can't figure out against what to diff the two branches "linuxtv WIP" and "drm-next". Just to make sure: "linuxtv WIP" == "linuxtv-rkvdec-work-in-progress" and "drm-next" == "next-20200501-drm-rockchip"? It seems I would only need the most recent commits. Using the common ancestor of the two branches e51759f56d314d28c25be7606b03791f048e44c7 seems too far back, as I get a diff that does not apply cleanly against 5.7-rc7 with the librelec rochchip patches applied. For the librelec patches I used

Code:

linux-0002-rockchip-from-next.patch        
linux-0012-v4l2-from-next.patch            
linux-0013-v4l2-from-list.patch            
linux-0022-drm-from-next.patch            
linux-1000-export-mm_trace_rss_stat.patch  

from https://github.com/LibreELEC/LibreELEC.t...ux/default. Are these the correct patches? Could you advise me on how to get a proper diff for the two branches from Kwiboo's repo?