mainline kernel hardware acceleration

[xmixahlx]

fyi you can apply rkvdec h264 and vp9 kernel patches to 5.7-rcX. rk3399 is currently dependent upon improved vaapi and v4l2 solutions for any media decoding to be utilized by chromium or firefox.

firefox 75 (and improved in 76) DOES allow for vaapi on wayland, and this seems to be the most likely early testbed, but dependent on the comment above.

[xmixahlx]

an update:

kwiboo has added hevc/h265 decoding to rkvdec and is available in his linux-rockchip and ffmpeg repositories. with the two rk3399 vpu decoders (hantro, rkvdec) this means we have the following hardware decoding support:

hantro: mpeg2, vp8, h264
rkvdec: h264, h265, vp9

both hantro and rkvdec support h264 decoding, but hantro is limited to 1080p. rkvdec is the better solution.

all hardware decoders roughly require 50% compute power of one of the big cores.

ffmpeg works directly with v4l2-request api i.e. vaapi and libva-v4l2-request are not utilized.

a desktop solution for improved firefox and chrome support requires improved vaapi handling, likely building upon ph5's libva hantro work.

vaapi support already exists in firefox on wayland with v75 (h264) and expanded with v76 (all).

if you want to test this yourself i am using tsys 5.7-rc2 patched to rc5 + LibreElec pull requests for 5.6 support (only including next/list), plus hevc specific patches for linux-rockchip and ffmpeg with something like: ffmpeg -report -re -hwaccel drm -hwaccel_device /dev/dri/card0 -i $FILE -pix_fmt bgra -f null -

[xmixahlx]

@termac from the mainline kernel thread

... i tried my hands on getting hardware decoding working, for that I added the rkvdec kernel patches from https://patchwork.kernel.org/project/lin...ter=179601 and patched libva and libva-v4l2-request as per Your instructions in the other thread. vainfo still would only show support for MPEG2. At that point I did give up. So now it is either waiting for full mainline support or have enough time and motivation to figure out where I went wrong.

vainfo will display a single media decoder module, and it defaults to /dev/video0 or whatever you have set as LIBVA_V4L2_REQUEST_VIDEO_PATH (see first few posts). since rk3399 has two (hantro and rkvdec) you need to select them with vainfo separately. with mpeg2 you are seeing hantro. check devices with v4l2-ctl --list-devices. check each device with v4l2-ctl -d /dev/video# --all.

it's notable that ffmpeg currently doesn't list the v4l2request hwaccel decoders in runtime ((ffmpeg -decoders|grep v4l2), but does list v4l2request codecs as detected during build.

it's also notable that these patches handle drm hwaccel because that is what LibreElec uses for kodi.

kernel patch info:

Kwiboo's LibreElec Rockchip 5.6 patches and pull request:
https://github.com/Kwiboo/LibreELEC.tv/t...ux/default
https://github.com/LibreELEC/LibreELEC.tv/pull/4377

Kwiboo hevc kernel repo branch:
https://github.com/Kwiboo/linux-rockchip...ist-v5.6.6
https://github.com/Kwiboo/linux-rockchip...5.6.6-hevc
make a diff of these two above repositories (exclude=.git exclude=sunxi)

Kwiboo's ffmpeg branch supporting rkvdec:
https://github.com/Kwiboo/FFmpeg/tree/v4...2.2-rkvdec
build: ./configure --prefix=/usr/local --libdir=/usr/local/lib/aarch64-linux-gnu --shlibdir=/usr/local/lib/aarch64-linux-gnu --enable-shared --enable-pic --enable-v4l2-request --enable-libdrm --enable-libudev && make -j6 && sudo make install

pbp linux 5.7-rc5 hwaccel patch sequence:
tsys 5.7.0-rc2 branch
5.7.0-rc3 incremental patch
5.7.0-rc4 incremental patch
5.7.0-rc5 incremental patch
LibreElec 002, 0012, 0013, 0021, 0022, 1000 patches
Kwiboo hevc patch

[termac]

vainfo will display a single media decoder module, and it defaults to /dev/video0 or whatever you have set as LIBVA_V4L2_REQUEST_VIDEO_PATH (see first few posts). since rk3399 has two (hantro and rkvdec) you need to select them with vainfo separately. with mpeg2 you are seeing hantro. check devices with v4l2-ctl --list-devices. check each device with v4l2-ctl -d /dev/video# --all.
...

Thanks for pointing that out. I only checked video0 and video1. With LIBVA_V4L2_REQUEST_VIDEO_PATH=/dev/video4 I get:

Code:

$>vainfo libva info: VA-API version 1.8.0 libva info: User environment variable requested driver 'v4l2_request' libva info: Trying to open /usr/lib/aarch64-linux-gnu/dri/v4l2_request_drv_video.so libva info: Found init function __vaDriverInit_1_8 libva info: va_openDriver() returns 0 vainfo: VA-API version: 1.8 (libva 2.7.0) vainfo: Driver version: v4l2-request vainfo: Supported profile and entrypoints      VAProfileH264Main               : VAEntrypointVLD      VAProfileH264High               : VAEntrypointVLD      VAProfileH264ConstrainedBaseline: VAEntrypointVLD      VAProfileH264MultiviewHigh      : VAEntrypointVLD      VAProfileH264StereoHigh         : VAEntrypointVLD

Code:

$>v4l2-ctl -d /dev/video4 --all Driver Info: Driver name      : rkvdec Card type        : rkvdec Bus info         : platform:rkvdec Driver version   : 5.7.0 Capabilities     : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities Device Caps      : 0x04204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Media Driver Info: Driver name      : rkvdec Model            : rkvdec Serial           : Bus info         : platform:rkvdec Media version    : 5.7.0 Hardware revision: 0x00000000 (0) Driver version   : 5.7.0 Interface Info: ID               : 0x0300000c Type             : V4L Video Entity Info: ID               : 0x00000001 (1) Name             : rkvdec-source Function         : V4L2 I/O Pad 0x01000002   : 0: Source  Link 0x02000008: to remote pad 0x1000004 of entity 'rkvdec-proc': Data, Enabled, Immutable Priority: 2 Format Video Capture Multiplanar: Width/Height      : 48/48 Pixel Format      : 'NV12' (Y/CbCr 4:2:0) Field             : None Number of planes  : 1 Flags             : Colorspace        : Rec. 709 Transfer Function : Default YCbCr/HSV Encoding: Default Quantization      : Default Plane 0           :   Bytes per Line : 48   Size Image     : 4608 Format Video Output Multiplanar: Width/Height      : 48/48 Pixel Format      : 'S264' (H.264 Parsed Slice Data) Field             : None Number of planes  : 1 Flags             : Colorspace        : Rec. 709 Transfer Function : Default YCbCr/HSV Encoding: Default Quantization      : Default Plane 0           :   Bytes per Line : 0   Size Image     : 4608 Codec Controls                    vp9_profile 0x00990b00 (menu)   : min=0 max=0 default=0 value=0 0: 0    h264_sequence_parameter_set 0x00990ce8 (unknown): type=110 flags=has-payload     h264_picture_parameter_set 0x00990ce9 (unknown): type=111 flags=has-payload            h264_scaling_matrix 0x00990cea (unknown): type=112 flags=has-payload          h264_slice_parameters 0x00990ceb (unknown): type=113 flags=has-payload         h264_decode_parameters 0x00990cec (unknown): type=114 flags=has-payload               h264_decode_mode 0x00990ced (menu)   : min=1 max=1 default=1 value=1 1: Frame-Based                h264_start_code 0x00990cee (menu)   : min=1 max=1 default=1 value=1 1: Annex B Start Code            vp9_frame_context_0 0x009918a0 (unknown): type=400 flags=has-payload            vp9_frame_context_1 0x009918a1 (unknown): type=400 flags=has-payload            vp9_frame_context_2 0x009918a2 (unknown): type=400 flags=has-payload            vp9_frame_context_3 0x009918a3 (unknown): type=400 flags=has-payload    vp9_frame_decode_parameters 0x009918a4 (unknown): type=404 flags=has-payload

To me that seems like from the driver and v4l2 side everything is in place. I tried playing video with mpv (unpatched) and the --hwdec options "vaapi" and "v4l2m2m-copy". In the first case I get no error, but the playback drops quite alot of frames and in the second case I get the error

Code:

[ffmpeg/video] h264_v4l2m2m: Could not find a valid device [ffmpeg/video] h264_v4l2m2m: can't configure decoder Could not open codec.

So maybe I am just missing a patched ffmpeg now. I will give that a try next.
If that does not work out, I will try the exact patches You listed, as currently I have gotten here using the patches mentioned in the quote from the other thread. Thank You for looking into it!

pbp linux 5.7-rc5 hwaccel patch sequence:
tsys 5.7.0-rc2 branch
5.7.0-rc3 incremental patch
5.7.0-rc4 incremental patch
5.7.0-rc5 incremental patch
LibreElec 002, 0012, 0013, 0021, 0022, 1000 patches
Kwiboo hevc patch

[xmixahlx]

confirmed kodi-gbm from git supports all hantro and rkvdec codecs flawlessly. significantly less processing power is required even compared to ffmpeg drm hwaccel testing (50% of ~1 core, but multithreaded using small cores, so 1-7% total cpu during playback.

example decoding (total cpu %) of a x265 1080p file:
ffmpeg sw decoding = 60% total cpu
ffmpeg drm hwaccel = 7% total cpu
kodi-gbm drm+gles hwaccel = 2% total cpu

i've added a kodi-git build script in another thread: https://forum.pine64.org/showthread.php?tid=9877

[xmixahlx]

also: @termac vaapi is only working with mpeg2 as far as i can tell. it was one of the first tests months ago.

[AndreVallestero]

@xmixahlx Just a headsup, Armbian has an active thread for trying to get all the patches together and into mainline: https://forum.armbian.com/topic/13622-ma...b=comments. I mentioned this post in that thread so they might be able to learn from your work.

[xmixahlx]

excellent. i'll jump over there, too.

some random updates:
ffmpeg 4.2.3 working with v4l2-request-hwaccel-4.2.2-rkvdec patch (4.2.2 diff)
x265 10bit hwaccel is working
NOT working: vp9 10/12 bit, h264 10bit, h265 12bit
h264 10bit gives kernel hard lock
jellyfin-kodi is working
kodi-youtube is wip
updated the first post with most relevant info
added the armbian thread

[xmixahlx]

h264 10bit hwaccel now working thanks to kwiboo. see his recent linux-rockchip and ffmpeg repo branches.

[termac]

@xmixahlx On the armbian forum thread you wrote:

confirmed x264 10bit hwaccel is working. thanks so much, @Kwiboo
using kodi-git, ffmpeg 4.2.3 + new rkvdec branch work, linux 5.7-rc7 + librelec rockchip patches + linuxtv WIP branch diff + drm-next branch diff

I can't figure out against what to diff the two branches "linuxtv WIP" and "drm-next". Just to make sure: "linuxtv WIP" == "linuxtv-rkvdec-work-in-progress" and "drm-next" == "next-20200501-drm-rockchip"? It seems I would only need the most recent commits. Using the common ancestor of the two branches e51759f56d314d28c25be7606b03791f048e44c7 seems too far back, as I get a diff that does not apply cleanly against 5.7-rc7 with the librelec rochchip patches applied. For the librelec patches I used

Code:

linux-0002-rockchip-from-next.patch         linux-0012-v4l2-from-next.patch             linux-0013-v4l2-from-list.patch             linux-0022-drm-from-next.patch             linux-1000-export-mm_trace_rss_stat.patch  

from https://github.com/LibreELEC/LibreELEC.t...ux/default. Are these the correct patches? Could you advise me on how to get a proper diff for the two branches from Kwiboo's repo?