HW image transitions in Python

[gaggleoxfoggy]

Working on Rock64/Debian Stretch and trying to crossfade two images. I've tried Pygame, Kivy, and just basic SDL, but it seems like none of these are able to do anything other than SW mode. Is there a way of making them use HW or some other method of displaying images through Python where I'd be able to get a smooth realtime transition?

[mcerveny]

It depends:

• alfa channel is sufficient for composition, two frames with scaling and positioning, only on video output -> use DRM (libdrm ...)
  
• more complex compositions -> use OpenGLES 2.0 with shader program (libGLESv2/libEGL ...) (Search for OpenGLES python bindings and examples for RasberryPI.)
  

You must use libraries with support for ROCK64 HW and compatible kernel drivers - eg. libdrm rockchip video output and/or OpenGLES for mali-450 (utgard) for example.

[gaggleoxfoggy]

Bloody hell this looks like some useful stuff that I haven't seen before! Thank you, I'll try this out.

[gaggleoxfoggy]

So I'm still having issues trying to get this set up. When I try building libdrm, it works but says it will not be compiled with Rockchip API. The OpenGLES for mali-450 seems to go on just fine. Is there something I'm supposed to do to make sure it becomes a default or anything like that? I've tested with Glmark2, which I configure for drm-glesv2. It still only shows Mali when it loads. I did find RPIGL, which looks like it will work for me once this is able to use the hardware properly.

EDIT: Just flashed an image back to before any of this so I can start clean and try again. Trying to configure glmark, the first part checks for several things, including gbm, egl, and glesv2. It will fail until I install the Rockchip/Mali stuff, but then when it tries to build glmark I end up getting errors that it cannot find lgbm, legl, and lglesv2. If I put on the mesa versions then it will find them, so it seems like maybe they are getting installed but just not being pointed to properly. Is there a specific way to point to them?

[gaggleoxfoggy]

I'm not getting anywhere with this. The drivers don't seem to be used, I've found this version of xserver and this kernel, both of which are Rockchip specific and say they'll be useful in making HW acceleration work. I think they're both a couple of jerks. They fail to install, and neither of which are for lack of dependencies. Anyone have more insight? I'm no expert on getting these things set up, but this is pretty ridiculous and definitely falls well within the realm of things I think should have just been working when I bought the stupid board.

[mcerveny]

I'm not getting anywhere with this...

Step by step:

1. use supported image - https://github.com/ayufan-rock64/linux-build/releases/ bionic-lxde-rock64-0.7.9-1067-arm64.img.xz
   
2. login to lxde as rock64/rock64 and check installed packages
   
   Code:

   $ sudo bash
# apt list --installed | egrep 'utgard|rockchip'
...
libdrm-rockchip1/bionic,now 2.4.89-1ayufan1 arm64 [installed]
libmali-rk-utgard-450-r7p0/bionic,now 1.6-1ayufan10 arm64 [installed]
librockchip-mpp1/bionic,now 1.4.0-1ayufan3-2 arm64 [installed,automatic]
linux-headers-4.4.132-1075-rockchip-ayufan-ga83beded8524/unknown,now 4.4.132-1075-rockchip-ayufan arm64 [installed,automatic]
linux-image-4.4.132-1075-rockchip-ayufan-ga83beded8524/unknown,now 4.4.132-1075-rockchip-ayufan arm64 [installed,automatic]
...
# dmesg | egrep -i 'drm|mali|utgard'
...
[    1.377772] [drm] Rockchip DRM driver version: v1.0.1
[    2.874168] I : [File] : drivers/gpu/arm/mali400/mali/linux/mali_kernel_linux.c; [Line] : 417; [Func] : mali_module_init(); svn_rev_string_from_arm of this mali_ko is '-a83beded8524', rk_ko_ver is '5', built at '08:21:11', on 'Jul 26 2018'.
...
3. install required development packages
   
   Code:

   # apt install python pkg-config libjpeg-dev libpng-dev xserver-xorg-dev/bionic libmali-rk-dev
# ls -l /usr/lib/aarch64-linux-gnu/ | egrep 'libEGL|libGLESv2'
lrwxrwxrwx  1 root root       10 Mar  4  2018 libEGL.so -> libMali.so
lrwxrwxrwx  1 root root       10 Mar  4  2018 libEGL.so.1 -> libMali.so
lrwxrwxrwx  1 root root       10 Mar  4  2018 libEGL.so.1.0.0 -> libMali.so
lrwxrwxrwx  1 root root       10 Mar  4  2018 libEGL.so.1.4 -> libMali.so
lrwxrwxrwx  1 root root       10 Mar  4  2018 libGLESv2.so -> libMali.so
lrwxrwxrwx  1 root root       10 Mar  4  2018 libGLESv2.so.2 -> libMali.so
lrwxrwxrwx  1 root root       10 Mar  4  2018 libGLESv2.so.2.0 -> libMali.so
lrwxrwxrwx  1 root root       10 Mar  4  2018 libGLESv2.so.2.0.0 -> libMali.so
# ls -l /usr/include/EGL/ /usr/include/GLES2
/usr/include/EGL/:
total 80
-rw-r--r-- 1 root root 15361 Nov 19  2017 egl.h
-rw-r--r-- 1 root root 59487 Nov 19  2017 eglext.h
-rw-r--r-- 1 root root  3839 Nov 19  2017 eglplatform.h
/usr/include/GLES2:
total 256
-rw-r--r-- 1 root root  43972 Nov 19  2017 gl2.h
-rw-r--r-- 1 root root 210702 Nov 19  2017 gl2ext.h
-rw-r--r-- 1 root root   1160 Nov 19  2017 gl2platform.h
4. download, configure, compile, install glmark2 from sources
   
   Code:

   # git clone --depth 1 https://github.com/glmark2/glmark2.git
# cd glmark2
# ./waf configure --with-flavors=x11-glesv2
Setting top to                           : /home/rock64/glmark2
Setting out to                           : /home/rock64/glmark2/build
Checking for 'gcc' (C compiler)          : /usr/bin/gcc
Checking for 'g++' (C++ compiler)        : /usr/bin/g++
Checking for header stdlib.h             : yes
Checking for header string.h             : yes
Checking for header unistd.h             : yes
Checking for header stdint.h             : yes
Checking for header stdio.h              : yes
Checking for header jpeglib.h            : yes
Checking for library m                   : yes
Checking for library jpeg                : yes
Checking for function memset             : yes
Checking for function sqrt               : yes
Checking for program 'pkg-config'        : /usr/bin/pkg-config
Checking for 'libpng12'                  : not found
Checking for 'libpng15'                  : not found
Checking for 'libpng16'                  : yes
Checking for 'x11'                       : yes
Checking for 'gl'                        : not found
Checking for 'egl'                       : yes
Checking for 'glesv2'                    : yes
Checking for 'libdrm'                    : yes
Checking for 'gbm'                       : yes
Checking for 'libudev'                   : not found
Checking for 'mirclient'                 : yes
Checking for 'wayland-client'            : not found
Checking for 'wayland-egl'               : yes
Prefix                                   : /usr/local
Data path                                : /usr/local/share/glmark2
Including extras                         : No
Building flavors                         : ['x11-glesv2']
'configure' finished successfully (5.990s)
# ./waf
Waf: Entering directory `/home/rock64/glmark2/build'
[ 1/67] Compiling src/main.cpp
[ 2/67] Compiling src/canvas-generic.cpp
[ 3/67] Compiling src/native-state-x11.cpp
...
# ./waf install
5. run "glmark2-es2" (performance results are dependend onscreen/offcreen, gpu/memory/cpu clock ...)
   
   Code:

   $ /usr/local/bin/glmark2-es2
=======================================================
    glmark2 2017.07
=======================================================
    OpenGL Information
    GL_VENDOR:     ARM
    GL_RENDERER:   Mali-450 MP
    GL_VERSION:    OpenGL ES 2.0
=======================================================
[build] use-vbo=false: FPS: 50 FrameTime: 20.000 ms
[build] use-vbo=true: FPS: 50 FrameTime: 20.000 ms
...
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 40 FrameTime: 25.000 ms
=======================================================
                                  glmark2 Score: 40
=======================================================
$ /usr/local/bin/glmark2-es2
=======================================================
    glmark2 2017.07
=======================================================
    OpenGL Information
    GL_VENDOR:     ARM
    GL_RENDERER:   Mali-450 MP
    GL_VERSION:    OpenGL ES 2.0
=======================================================
[build] use-vbo=false: FPS: 196 FrameTime: 5.102 ms
[build] use-vbo=true: FPS: 224 FrameTime: 4.464 ms
...
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 175 FrameTime: 5.714 ms
=======================================================
                                  glmark2 Score: 165
=======================================================

Now OpenGLESv2 development platform is verified to be working.
And remember OpenGLES is not OpenGL !

Buy me a Beer.

[mcerveny]

I'm not getting anywhere with this.

You should check MPP (HW accelerated video encoding/decoding) and direct DRM (output image composing). DRM windows for rk3328 are described here (eg. win0 - SDR/HDR (best for video output), win1 - SDR (best for video OSD) and win2 - SDR with limited size (best for cursor)). I prepared working proof of concept example (post). You can use bionic-lxde-rock64-0.7.9-1067-arm64.img.xz.

1. check/prepare developent environment
   
   Code:

   # apt list --installed | egrep 'mpp|libdrm-rockchip'
libdrm-rockchip1/bionic,now 2.4.89-1ayufan1 arm64 [installed]
librockchip-mpp1/bionic,now 1.4.0-1ayufan3-2 arm64 [installed,automatic]
# apt install librockchip-mpp-dev libdrm-dev
# ls -l /usr/lib/aarch64-linux-gnu/libdrm.so /usr/lib/aarch64-linux-gnu/libdrm_rockchip.so /usr/lib/aarch64-linux-gnu/librockchip_mpp.so
lrwxrwxrwx 1 root root 15 Dec 19  2017 /usr/lib/aarch64-linux-gnu/libdrm.so -> libdrm.so.2.4.0
lrwxrwxrwx 1 root root 24 Dec 19  2017 /usr/lib/aarch64-linux-gnu/libdrm_rockchip.so -> libdrm_rockchip.so.1.0.0
lrwxrwxrwx 1 root root 20 Mar  2  2018 /usr/lib/aarch64-linux-gnu/librockchip_mpp.so -> librockchip_mpp.so.1
# ls /usr/include/rockchip/ /usr/include/libdrm/
/usr/include/libdrm/:
... drm_mode.h ... rockchip_rga.h ... drm.h  ... rockchip_drm.h .... drm_fourcc.h  ... rockchip_drmif.h  ...
/usr/include/rockchip/:
mpp_buffer.h  mpp_err.h  mpp_frame.h  mpp_meta.h  mpp_packet.h  mpp_task.h  rk_mpi.h  rk_mpi_cmd.h  rk_type.h  vpu.h  vpu_api.h
2. compile and check MPP/DRM example (beware - working directly to drm framebuffer (eg. working without xserver and replace original FB when running)).
   
   Code:

   # git clone https://github.com/mcerveny/utils.git
# cd utils
# gcc -o mpi_dec -I/usr/include/rockchip -I/usr/include/libdrm -g -Wno-multichar mpi_dec.c -lrockchip_mpp -lpthread -ldrm
# ./mpi_dec 2k60.hevc 16777220

Now there are verified MPP and DRM HW acceleration libraries.

[gaggleoxfoggy]

Thank you so much for your help on this!

I've been able to verify HW working in all examples you provided. I still don't quite understand how to turn this into something usable, as any examples that have used the HW were not in Python, which is the only language I know at all... still just learning that one. Using your example, I could potentially just process my image transitions into videos beforehand and play them back when needed. But so far I've failed to get this running with pogles, kivy, pyqt5, or rpigl. RPIGL causes a segmentation fault as soon as it tries to create the window, and I haven't been able to get pogles, pyqt5, or kivy to even install. They all fail, kivy can't find GL, pyqt5 (specifically installing qt open source version) I can't even make sense of why its failing, and pogles seems to have an issue with my sip. I may have configured sip for the wrong platform, but even still I'd have a very hard time trying to integrate pogles into my program. The RPIGL examples I can make sense of enough to make it do what I need, and I've done a test in kivy before that I could work with. But is this where I'm misunderstanding GL vs GLES? Kivy says it uses OpenGL ES, but still seems hung up on GL. PyQt5 seems to me the most promising if I can make it install, but are these all just lost causes trying to make run?

EDIT: And looking back through the QT make log it looks like it is getting hung up on GL stuff as well. Which comes back to, am I misunderstanding how these work or do they just need to be configured a different way?