mainline kernel hardware acceleration

[xmixahlx]

collecting information and projects regarding hardware acceleration in mainline kernel, ffmpeg, and vaapi.

STATUS:
drm hwaccel working with hantro (mpeg2, vp8, h264) and rkvdec (vp9, h264 8+10bit, h265 8+10bit)
ffmpeg 4.2.3 with v4l2-request-hwaccel-4.2.2-rkvdec patch working well, except decoders listing is broken (i.e. mpv and others will be broken)
kodi supports all hantro and rkvdec hwaccel, including plugins (youtube and jellyfin confirmed) *see kodi-git thread*
gstreamer supports vp8 and h264 via v4l2codecs in gstreamer-plugins-bad
vaapi hwaccel working with hantro (mpeg2)
firefox 75 supports initial vaapi on wayland, v76 expands vaapi support to all supported codecs, no X11 support
chrome supports vaapi with unofficial patch that will never be upstreamed, but included in some linux distros (fedora, etc.)

PBP-TOOLS

pinebook pro tools, including installation scripts, patches, and configs, now on git. also see forum thread.
https://github.com/xmixahlx/pbp-tools
https://forum.pine64.org/showthread.php?tid=10190

REFERENCES

linux patchwork - linux-media:
https://patchwork.kernel.org/project/linux-media/list/

linux patchwork - linux-rockchip:
https://patchwork.kernel.org/project/lin...chip/list/

libreelec/kwiboo mainline kernel patches:
https://github.com/LibreELEC/LibreELEC.t...ux/default

libreelec/kwiboo mainline kernel testing patches:
https://github.com/Kwiboo/LibreELEC.tv

kwiboo's rockchip-linux repos:
https://github.com/Kwiboo/linux-rockchip/

bbrezillon's rockchip-linux repos:
https://github.com/bbrezillon/linux/

ph5's libva patch:
https://github.com/intel/libva/pull/332

libva-v4l2-request;
https://github.com/bootlin/libva-v4l2-request/

ph5's libva-v4l2-request patches:
https://github.com/bootlin/libva-v4l2-request/pulls

kwiboo's ffmpeg v4l2-request branch:
https://github.com/Kwiboo/FFmpeg/tree/v4...ccel-4.2.2
https://github.com/Kwiboo/FFmpeg/tree/v4...2.2-rkvdec

gstreamer v4l2codecs:
https://gitlab.freedesktop.org/gstreamer...uests/1141

gstreamer vaapi:
https://gitlab.freedesktop.org/gstreamer...quests/171

firefox vaapi:
https://bugzilla.mozilla.org/show_bug.cgi?id=1610199

chrome vaapi:
https://chromium-review.googlesource.com...c/+/532294
https://github.com/saiarcot895/chromium-...an/patches
https://aur.archlinux.org/packages/chromium-vaapi/

OTHER

allwinner cedrus hwaccel project:
https://linux-sunxi.org/Sunxi-Cedrus

LibreElec - Early Mainline Images for <...> RK3399:
https://forum.libreelec.tv/thread/17540-...nd-rk3399/

LibreElec - Not Official LE Version for <...> RK3399
https://forum.libreelec.tv/thread/20823-...28-rk3399/

Armbian - early VPU work:
https://forum.armbian.com/topic/4806-the-vpu-driver/

Armbian - Mainline VPU
https://forum.armbian.com/topic/13622-mainline-vpu/

[xmixahlx]

mpeg2video hwaccel is tested working.

testing requirements:
patched kernel (libreelec; optionally kwiboo v4l2 5.5 wip and rkvdec, plus bbrezillon patches)
patched libva (plus pulls 332,340)
patched libva-v4l2-request (plus pulls 28,29,30)
patched ffmpeg (v4l2-request-hwaccel) -- recommend 4.2.2 or -master
> build with --enable-v4l2-request --enable-libdrm --enable-libudev --enable-pic --enable-shared (plus optional config)

determine video decoder:
dmesg | grep rk3399-vpu-dec
note video device (i.e. /dev/video1)

export variables:
export LIBVA_DRIVER_NAME=v4l2_request
export LIBVA_V4L2_REQUEST_VIDEO_PATH=/dev/video1

vainfo output:

libva info: VA-API version 1.6.1
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/local/lib/aarch64-linux-gnu/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_6
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.6 (libva 2.6.0)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
     VAProfileMPEG2Simple            : VAEntrypointVLD
     VAProfileMPEG2Main              : VAEntrypointVLD
     VAProfileH264Main               : VAEntrypointVLD
     VAProfileH264High               : VAEntrypointVLD
     VAProfileH264ConstrainedBaseline: VAEntrypointVLD
     VAProfileH264MultiviewHigh      : VAEntrypointVLD
     VAProfileH264StereoHigh         : VAEntrypointVLD

vlc abbreviated output:
> opengl video, vaapi hwaccel

libva info: VA-API version 1.6.1
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/local/lib/aarch64-linux-gnu/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_6
libva info: va_openDriver() returns 0
[0000ffff5cc19640] avcodec decoder: Using v4l2-request for hardware decoding

ffmpeg -decoders | grep v4l2request output:

V..... h263_v4l2request h263 (v4l2request) (codec h263)
V..... h264_v4l2request h264 (v4l2request) (codec h264)
V..... hevc_v4l2request hevc (v4l2request) (codec hevc)
V..... mpeg1_v4l2request mpeg1 (v4l2request) (codec mpeg1video)
V..... mpeg2_v4l2request mpeg2 (v4l2request) (codec mpeg2video)
V..... mpeg4_v4l2request mpeg4 (v4l2request) (codec mpeg4)
V..... vc1_v4l2request vc1 (v4l2request) (codec vc1)
V..... vp8_v4l2request vp8 (v4l2request) (codec vp8)
V..... vp9_v4l2request vp9 (v4l2request) (codec vp9)

screenshot eye candy

[vrabac]

@xmixahlx  great work, I was looking for something like this to as I was going to test Kodi on my PBP with but with VPU support and using USB-C to DVI (HDMI) on Manjaro KDE. But seems there is no already package for these kinde of support on Manjaro KDE jet.

Do you have some sources for your PKGBUILDs or you are building packages without PKGBUILDs?

From arbian forum I have these information (from 2019), the kwiboo's rockchip-linux repos you already linked.

Code:

The VPU in rk3288 (vpu1) should have VP8 support, same as the VPU in rk3328/rk3399 (vpu2)
rk3288 has a second VPU block that can decode hevc
rk3328/rk3399 has a second VPU block (rkvdec) that can decode h264/hevc/vp9 (they have two different vpu blocks that can decode h264)

[xmixahlx]

@xmixahlx  great work, I was looking for something like this to as I was going to test Kodi on my PBP with but with VPU support and using USB-C to DVI (HDMI) on Manjaro KDE. But seems there is no already package for these kinde of support on Manjaro KDE jet.

Do you have some sources for your PKGBUILDs or you are building packages without PKGBUILDs?

From arbian forum I have these information (from 2019), the kwiboo's rockchip-linux repos you already linked.

Code:

The VPU in rk3288 (vpu1) should have VP8 support, same as the VPU in rk3328/rk3399 (vpu2)
rk3288 has a second VPU block that can decode hevc
rk3328/rk3399 has a second VPU block (rkvdec) that can decode h264/hevc/vp9 (they have two different vpu blocks that can decode h264)

i'm on debian sid arm64 and following guidance from kernel patches and commentary. keep an eye out for av's arch patches. we're collaborating on pine64 irc occasionally.
https://aur.archlinux.org/packages/?O=0&..._Search=Go

[amiraeva]

Fantastic! I haven't been able to replace my Thinkpad just yet because of a lack of video acceleration in Manjaro. Looks like it won't be too long until it'll be plug and play.

[mamboman777]

Fantastic! I haven't been able to replace my Thinkpad just yet because of a lack of video acceleration in Manjaro. Looks like it won't be too long until it'll be plug and play.

The last blog post had a panfrost installation script that got hardware acceleration working on my pine book pro.

[xmixahlx]

panfrost hwaccel and hantro/rkvdec hwaccel are different.

hantro/rkvdec are the rockchip hwaccel kernel modules for video decoding (VPU).

there are numerous capabilities of the rockchip VPU that are not yet utilized, and development is ongoing.

one of the main developers, kwiboo, also is on the libreelec team and their distro is the most advanced in this regard. other distros will use this as a base until accepted upstream to provide similar support.

[vrabac]

is the patched kernel  an must to or optional? I used under Manjaro Arm 20.02 the AndreVallestero PKGBUILDs but having problem to compile ffmpeg but i just compiled and installed these to prior to compile ffmpeg: libva-hantro-h264-git and libva-v4l2-request-hantro-h264-git

Code:

CC      libavcodec/v4l2_m2m.o
In file included from /usr/include/string.h:494,
                from ./libavutil/common.h:40,
                from ./libavutil/avutil.h:296,
                from ./libavutil/samplefmt.h:24,
                from ./libavcodec/avcodec.h:31,
                from libavcodec/v4l2_m2m.c:30:
In function ‘strncpy’,
   inlined from ‘ff_v4l2_m2m_codec_init’ at libavcodec/v4l2_m2m.c:362:9:
/usr/include/bits/string_fortified.h:106:10: warning: ‘__builtin___strncpy_chk’ specified bound depends on the length of the source argument [-Wstringop-overflow=]
 106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
     |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_m2m.c: In function ‘ff_v4l2_m2m_codec_init’:
libavcodec/v4l2_m2m.c:362:35: note: length computed here
 362 |         strncpy(s->devname, node, strlen(node) + 1);
     |                                   ^~~~~~~~~~~~
CC      libavcodec/v4l2_m2m_dec.o
CC      libavcodec/v4l2_m2m_enc.o
CC      libavcodec/v4l2_request.o
libavcodec/v4l2_request.c: In function ‘v4l2_request_queue_decode’:
libavcodec/v4l2_request.c:216:90: error: ‘V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF’ undeclared (first use in this function)
 216 |     ret = v4l2_request_queue_buffer(ctx, req->request_fd, &req->output, last_slice ? 0 : V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF);
     |                                                                                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request.c:216:90: note: each undeclared identifier is reported only once for each function it appears in
libavcodec/v4l2_request.c: In function ‘ff_v4l2_request_decode_slice’:
libavcodec/v4l2_request.c:300:37: error: ‘V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF’ undeclared (first use in this function)
 300 |     if ((req->output.capabilities & V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF) != V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF)
     |                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make: *** [ffbuild/common.mak:59: libavcodec/v4l2_request.o] Error 1
==> ERROR: A failure occurred in build().
   Aborting...

[xmixahlx]

the patched kernel provides necessary kernel headers for ffmpeg-v4l2-request, although mpeg2video hwaccel is now working with mainline kernel. (i tested with 5.6.0-rc3+tsys). i recommend applying the libreelec kernel patches if you want to test.

things are picking up momentum. it looks like 5.7 will include many hwaccel improvements. see:
https://patchwork.kernel.org/patch/11403025/

[xmixahlx]

a good read on firefox wayland:
https://mastransky.wordpress.com/2020/03...n-wayland/