Real-time webcam shaders

[yoann.b87]

Hello,

In my project I need to apply shaders on a webcam video flux in real-time and show the result on fullscreen.
Webcam output can be MjPEG or H.264, thus we also need to use hardware for the decoding process.
Maybe there is different way to achieve it.

I tried to use OpenCV with OpenGL. However the OpenCV doesn't use hardware acceleration to decode the camera flux.
I've compiled and installed it on a ayufan Bionic lxde 0.6.41 image. I've also tried to do this on Stretch 0.5.15. Other more recent version or Bionic or Stretch doesn't boot on my Rock64 board.
Hardware acceleration seems to work because rkmpv is able to display nearly real-time webcam video, there is juste a huge latency reducing over the time.



Do you have ideas to make OpenCV use GPU or another app/library etc to to realize it with OpenGL ?

Thank you very much in advance.

[fire219]

The GPU (Mali450) on the Rock64 can't do compute workloads, only graphics rendering. You're going to have to stick with doing that work on the CPU, or find another board with a GPU which does support compute (such as the RockPro64).

However, there is hardware video acceleration, as you have already noticed. That uses a separate part of the video pipeline from the actual GPU core.

[yoann.b87]

Hello,

Thank you very much for your reply.
I well understand that the GPU is a bit limited. I forgot to precise the type of treatment. I thought that the GPU can process very simple ones.
Shaders I want to apply are negative filter or similar. I already tried to apply this filter (negative filter) on an other board with a quite similar CPU and it is not possible to have a good FPS using a CPU process.

I'm ok that the Rockpro64 have a better GPU, however i afraid about having similar difficulties to make working the hardware acceleration with OpenCV or similar libraries.

[fire219]

I would think that it should be capable of doing that, as shaders are well within the graphical realm of work. But most people just answer "no" to using OpenCV on Mali4xx GPUs.

However, it does seem to be possible to do -- this smart camera demo does it, but they have a lot of it abstracted into their own libraries...

[yoann.b87]

Hello,

Thank you for the link.
This project is interesting, However, on this project the resolution processed is 320x240 where I need to process 1920x1080 in my first prototype and 3840x2160 in "production". Due to resolution authors can use a raw video data connection. With 4k resolution we must compress and uncompress data what uses a good part of GPU.

The only way to know it can really work is to test in real conditions on the Rock64 board. For that I need to make OpenCV working on GPU to decode H.264 and send images to OpenGL to apply simple filters (through shaders). OpenCV is not used to process the image like motion detection or tracking etc.

Does someone can help me make OpenCV working on GPU or maybe know an other way to decode H.264 (and MjPEG) and inject images into OpenGL buffer to efficiently applly sharders using OpenGL Library ?
Just for information I develop in C/C++.