pinebook pro tools

[mjhammel]

I've been following this thread for a short while - thanks for the really great information!  I've been tasked with getting the VPU working on a rk3399-based board with a custom distro.  For user space I think I understand what's required and where to get it.  For the kernel I'm a little confused on what's required.

It seems 5.8 is the place to start.  That seems somewhat stable at this point, at least for some codecs.  Do I need both the hantro and rkvdec drivers or is the hantro sufficient?  We need h264, mp4 and mjpeg decoding and mjpeg encoding at a minimum.  I may be required to backport to 5.4 but I can't tell if that would be harder than forward porting our tweaks of 5.4 instead.

I've started following the linux-media mailing list for updates.  I just need some help on where the best place to start is.

Any pointers would be greatly appreciated. Thanks!

[xmixahlx]

you could do everything with hantro as it also has h264 support, however rkvdec supports h264 also and is favored.

you should be able to review the linux and ffmpeg scripts, files under /resources/, and kwiboo's repositories for linux-rockchip and ffmpeg. see how these are addressed by libreelec - kwiboo prepares patches for matching repositories, and the linuxtv-rkvdec linux-rockchip branch is additional WIP, but usually stable.

if possible, i would rebase your project on 5.8. i will be rebasing the hwaccel patches for 5.9 soon.

i added ADDPATCH and KERNELCONFIG variables to the linux script, so you may be able to use the script with your own patch and kernel for your project. this approach can actually work for any rockchip hardware with some effort, snd i will soon be expanding for rockpro64.

[mjhammel]

you could do everything with hantro as it also has h264 support, however rkvdec supports h264 also and is favored.

you should be able to review the linux and ffmpeg scripts, files under /resources/, and kwiboo's repositories for linux-rockchip and ffmpeg. see how these are addressed by libreelec - kwiboo prepares patches for matching repositories, and the linuxtv-rkvdec linux-rockchip branch is additional WIP, but usually stable.

if possible, i would rebase your project on 5.8. i will be rebasing the hwaccel patches for 5.9  soon.

i added ADDPATCH and KERNELCONFIG variables to the linux script, so you may be able to use the script with your own patch and kernel for your project. this approach can actually work for any rockchip hardware with some effort, snd i will soon be expanding for rockpro64.

Thanks for the tips.  I decided it was easier to forward port our 5.4 changes to 5.8.5.  I spent yesterday doing that (mostly dts and some network mods) and the node boots fine with both the hantro and rkvdec drivers.  I ran some v4l2 commands and it looks far better than the original 4.4 stuff I had.

Next I'll take a look at your ffmpeg script(s) and kwiboo's ffmpeg repo.  My goal is to show ffmpeg doing hwaccel'd encoding and decoding, though I'm not quite sure the best way to do that yet.  I guess I could try a file conversion from h.264 to ... something else? ... and back.  I also need to test motion jpeg encoding somehow.  I don't have video input on the device (at least not yet).  And I don't have a display working yet either (we do on 4.4 - it's dsi, not hdmi - but I haven't ported that forward yet).  

Any additional suggestions are appreciated - and again: thanks for your work and help on this!

[mjhammel]

you could do everything with hantro as it also has h264 support, however rkvdec supports h264 also and is favored.

you should be able to review the linux and ffmpeg scripts, files under /resources/, and kwiboo's repositories for linux-rockchip and ffmpeg. see how these are addressed by libreelec - kwiboo prepares patches for matching repositories, and the linuxtv-rkvdec linux-rockchip branch is additional WIP, but usually stable.

if possible, i would rebase your project on 5.8. i will be rebasing the hwaccel patches for 5.9  soon.

i added ADDPATCH and KERNELCONFIG variables to the linux script, so you may be able to use the script with your own patch and kernel for your project. this approach can actually work for any rockchip hardware with some effort, snd i will soon be expanding for rockpro64.

One more question: why is the rkvdec driver favored?  Is the hantro driver still dependent on MPP?

[jh76]

upgrade system before read this thread  
my current kernel version 5.8.6
how to downgrade kernel version? or you packaged will update soon?

[xmixahlx]

iirc rkvdec supports greater resolution decode than hantro for h264. kwiboo wrote support for h264 on hantro, but then dropped in favor of rkvdec.


still using 5.8. no rush on 5.9. i have other projects in the works and those should be discuswed here soon.

ffmpeg uses drm hwaccel. kodi is the most supported software bevayse that is what libreelec uses as frontend. you can process via cli also. see the "mainline hwaccel" thread for more details.

[spikerguy]

iirc rkvdec supports greater resolution decode than hantro for h264. kwiboo wrote support for h264 on hantro, but then dropped in favor of rkvdec.


still using 5.8. no rush on 5.9. i have other projects in the works and those should be discuswed here soon.

ffmpeg uses drm hwaccel. kodi is the most supported software bevayse that is what libreelec uses as frontend. you can process via cli also. see the "mainline hwaccel" thread for more details.

Hello @xmixahlx 

I have finally got back to trying vpu support for Manjaro.
I patched the following packages

1. Linux Kernel 5.9.11 with vpu patch from your git.
   
2. Ffmpeg with the patched you have in pbp-tools git
   
3. Libva
   
4. libva-utils ( rebuilt)
   
5. libva-v4l2-request
   

After all this I see that hantro and rkvdec is loaded during boot but ffmpeg decoder doesn't list v4l2-request any more

v4l2-ctl --list-devices
rockchip,rk3399-vpu-enc (platform: hantro-vpu):
        /dev/video2
        /dev/video3
        /dev/media1

rockchip-rga (platform:rga):
        /dev/video1

rkvdec (platform:rkvdec):
        /dev/video0
        /dev/media0

dmesg | grep hantro
[    7.618917] hantro_vpu: module is from the staging directory, the quality is unknown, you have been warned.
[    7.620917] hantro-vpu ff650000.video-codec: Adding to iommu group 0
[    7.623875] hantro-vpu ff650000.video-codec: registered rockchip,rk3399-vpu-enc as /dev/video1
[    7.624402] hantro-vpu ff650000.video-codec: registered rockchip,rk3399-vpu-dec as /dev/video2

dmesg | grep rk3399-vpu
[    7.296421] hantro-vpu ff650000.video-codec: registered rockchip,rk3399-vpu-enc as /dev/video2
[    7.297825] hantro-vpu ff650000.video-codec: registered rockchip,rk3399-vpu-dec as /dev/video3

ffplay -decoders | grep v4l2
ffplay version n4.3.1-Kodi Copyright © 2003-2020 the FFmpeg developers
  built with gcc 10.2.0 (GCC)
  configuration: --enable-v4l2-request --enable-libdrm --enable-libudev --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-omx --enable-shared --enable-version3 --host-cflags='"-fPIC"'
  libavutil      56. 51.100 / 56. 51.100
  libavcodec    58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter    7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample  3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
V..... h263_v4l2m2m        V4L2 mem2mem H.263 decoder wrapper (codec h263)
V..... h264_v4l2m2m        V4L2 mem2mem H.264 decoder wrapper (codec h264)
V..... hevc_v4l2m2m        V4L2 mem2mem HEVC decoder wrapper (codec hevc)
V..... mpeg1_v4l2m2m        V4L2 mem2mem MPEG1 decoder wrapper (codec mpeg1video)
V..... mpeg2_v4l2m2m        V4L2 mem2mem MPEG2 decoder wrapper (codec mpeg2video)
V..... mpeg4_v4l2m2m        V4L2 mem2mem MPEG4 decoder wrapper (codec mpeg4)
V..... vc1_v4l2m2m          V4L2 mem2mem VC1 decoder wrapper (codec vc1)
V..... vp8_v4l2m2m          V4L2 mem2mem VP8 decoder wrapper (codec vp8)
V..... vp9_v4l2m2m          V4L2 mem2mem VP9 decoder wrapper (codec vp9)

ffmpeg -re -hwaccel drm -i bbb_1080-30_3M_x265_main_8b_lv51.mkv -f null -
ffmpeg version n4.3.1-Kodi Copyright © 2000-2020 the FFmpeg developers
  built with gcc 10.2.0 (GCC)
  configuration: --enable-v4l2-request --enable-libdrm --enable-libudev --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-omx --enable-shared --enable-version3 --host-cflags='"-fPIC"'
  libavutil      56. 51.100 / 56. 51.100
  libavcodec    58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter    7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample  3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, matroska,webm, from 'bbb_1080-30_3M_x265_main_8b_lv51.mkv':
  Metadata:
    ENCODER        : Lavf58.29.100
  Duration: 00:00:43.80, start: 0.000000, bitrate: 3049 kb/s
    Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1k tbn, 30 tbc (default)
    Metadata:
      ENCODER        : Lavc58.54.100 libx265
      DURATION        : 00:00:43.800000000
    Stream #0:1: Audio: ac3, 48000 Hz, 5.1(side), fltp, 320 kb/s (default)
    Metadata:
      DURATION        : 00:00:43.776000000
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (ac3 (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[hevc @ 0xaaaaca429460] v4l2_request_probe_video_device: missing required mem2mem capability
    Last message repeated 1 times
[hevc @ 0xaaaaca429460] v4l2_request_try_format: pixelformat 892744275 not supported for type 10
[hevc @ 0xaaaaca429460] v4l2_request_probe_video_device: try output format failed
[hevc @ 0xaaaaca429460] v4l2_request_try_format: pixelformat 892744275 not supported for type 10
[hevc @ 0xaaaaca429460] v4l2_request_probe_video_device: try output format failed
Output #0, null, to 'pipe:':
  Metadata:
    encoder        : Lavf58.45.100
    Stream #0:0: Video: wrapped_avframe, nv12, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default)
    Metadata:
      DURATION        : 00:00:43.800000000
      encoder        : Lavc58.91.100 wrapped_avframe
    Stream #0:1: Audio: pcm_s16le, 48000 Hz, 5.1(side), s16, 4608 kb/s (default)
    Metadata:
      DURATION        : 00:00:43.776000000
      encoder        : Lavc58.91.100 pcm_s16le
frame=  590 fps= 30 q=-0.0 Lsize=N/A time=00:00:19.84 bitrate=N/A speed=0.998x

I am not sure what else is needed. It still takes alot of cpu when playing via ffplay as mpv is not patched and as v4l2_request is not detected then no point of patching mpv, for kodi I might have to do a fresh pkgbuild for gbm with patched ffmpeg.

Any advice?

[xmixahlx]

there is/was a bug where v4l2-request was not being properly detected, so solutions depending on detection (vlc, mpv) were also not working. however, ffmpeg directly using drm hwaccel and kodi were/are working. although not ideal, i simply switch into kodi from sway when wanting to watch something.

i need to get back in sync to the current kernel and kwiboo patches. i'm still using 5.8 and have been detached from development for ~2 months. hoping to receive a rockpro64 in the mail shortly to help jumpstart this and a few other projects in the works.

[mjhammel]

there is/was a bug where v4l2-request was not being properly detected, so solutions depending on detection (vlc, mpv) were also not working. however, ffmpeg directly using drm hwaccel and kodi were/are working. although not ideal, i simply switch into kodi from sway when wanting to watch something.

i need to get back in sync to the current kernel and kwiboo patches. i'm still using 5.8 and have been detached from development for ~2 months. hoping to receive a rockpro64 in the mail shortly to help jumpstart this and a few other projects in the works.

Any updates on this? I don't think I'm seeing hw accel with the 5.10.17 kernel, at least not based on CPU load when I start my tests.
My software setup is as follows:

1. kernel 5.10.17, no patches
2. v4l2-utils 1.20.0 (no patches)
3. libva (git master) with patches
  a. url: https://patch-diff.githubusercontent.com...ibva/pull/
  b. pbp-tools patches: 332.patch, 340.patch
  c. patches edited based on https://github.com/xmixahlx/pbp-tools/bl...tall-libva
4. libva-v4l2-request (git master) with patches
  a. url: https://patch-diff.githubusercontent.com...uest/pull/
  b. pbp-tools patches: 30.patch, 32.patch
5. libva-utils (git master, no patches)
6. ffmpeg n4.3.1 with patches
  a. url: https://raw.githubusercontent.com/xmixah...fmpeg/4.3/
  b. pbp-tools patch: ffmpeg-4.3-v4l2request-rkvdec_20200709.diff
7. v4l2-request-test (git master, no patches)

All patches are from pbp-tools.  Does anyone know if all/any of these patches are still needed?
My test script: https://www.graphics-muse.org/archives/s...vputest.sh

vainfo:
vainfo --display drm --device /dev/videoX

  vainfo.log (Size: 1.55 KB / Downloads: 311)

Decode to NULL (-f NULL -):
ffmpeg  -hwaccel drm -vaapi_device /dev/dri/renderD128 -an -i big_buck_bunny_480p_H264_AAC_25fps_1800K_short.MP4 -map 0:0 -threads 8 -aspect 16:9 -y -f mp4  decodeVideos.mk

  vpu.log (Size: 6.24 KB / Downloads: 317)

[mjhammel]

I've fixed some bugs in my test script that now show vainfo correctly.  The updated script is vputests.sh, as in this example.

|----------------------------------------------------
++++ /dev/video0:
|----------------------------------------------------

LIBVA_DRIVER_NAME=v4l2_request vainfo --display drm --device /dev/video0
libva info: VA-API version 1.11.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_11
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.11 (libva 2.11.0.pre1)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
      VAProfileH264Main              : VAEntrypointVLD
      VAProfileH264High              : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264MultiviewHigh      : VAEntrypointVLD
      VAProfileH264StereoHigh        : VAEntrypointVLD
      VAProfileHEVCMain              : VAEntrypointVLD

I also ported the kernel patch to 5.10.17.  Before and after the patch I have only seen what appears to be HW accelleration with mpeg2video files.  H264 doesn't not appear to be getting accelerated based on CPU uage.  The patch has been posted to the pbp site.

I've also created my own sample files using OpenShot to save in H264, mpeg2video and mpeg4 formats (I'm not trying VP9 because its not one of our use cases) as 720p 25fps 1280x720 videos in MP4 and AVI containers.  The videos are longer than most of the samples I've been using to give me a chance to watch CPU usage over time.  That's how I'm testing if HW accel is apparently working.  With mpeg2video I'm seeing about 7% CPU usage.  With the others it varies from about 50% to 80% cpu usage.

I tried kwiboo's ffmpeg but there was no difference in the tests.

At this point I'm convinced that only mpeg2video are HW accelerated with the upstreamed hantro/rkvdec drivers plus the patches available here to both the kernel and user space utilities for VAAPI support.