Getting started for developers

[lagrang3]

Hello,

I am looking for information about developing applications for the pinephone.

As far as I understand, from the software layer point of view,
the Pinephone should behave like any multipurpose desktop computer.

But I wonder if there is a development kit for accessing the pinephone's devices:
cameras, microphone, gravity sensor, etc.

Thanks for any information,
Lagrang3

[Kevin Kofler]

For cameras and microphone, you are probably best off targeting PipeWire (using libcamera as the backend) or a higher-level interface such as GStreamer on top of PipeWire (or even a wrapper around GStreamer, such as QtMultimedia). Though you will find that the cameras on the original PinePhone only work with libcamera if you use at least version 6.2 of the megi kernel, which most distros are not yet shipping because the camera application Megapixels (which uses low-level camera interfaces) does not yet work with that version on the original PinePhone.

As for the accelerometer (gravity sensor), it is just an IIO device, so you can access it with a file-based interface, see, e.g., https://svn.calcforge.org/viewvc/pinephone-videorec/record.sh?revision=1&view=markup#l8. Though reportedly the "device3" part actually depends on the distro, so you may be better off allowing any "device*" as long as it contains the correct files. (A device is only going to have files like "in_accel_x_raw" in its directory if it is an accelerometer.) But if all you want to know is whether we are in portrait or landscape (or reverse portrait or reverse landscape, there are 4 possible rotation angles) mode, you are better off asking Wayland directly (as done, e.g., in this Megapixels commit) or using a wrapper API like QtGui's QScreen (QGuiApplication::primaryScreen()->orientation()).

[lagrang3]

For cameras and microphone, you are probably best off targeting PipeWire (using libcamera as the backend) or a higher-level interface such as GStreamer on top of PipeWire (or even a wrapper around GStreamer, such as QtMultimedia). Though you will find that the cameras on the original PinePhone only work with libcamera if you use at least version 6.2 of the megi kernel, which most distros are not yet shipping because the camera application Megapixels (which uses low-level camera interfaces) does not yet work with that version on the original PinePhone.

As for the accelerometer (gravity sensor), it is just an IIO device, so you can access it with a file-based interface, see, e.g., https://svn.calcforge.org/viewvc/pinephone-videorec/record.sh?revision=1&view=markup#l8. Though reportedly the "device3" part actually depends on the distro, so you may be better off allowing any "device*" as long as it contains the correct files. (A device is only going to have files like "in_accel_x_raw" in its directory if it is an accelerometer.) But if all you want to know is whether we are in portrait or landscape (or reverse portrait or reverse landscape, there are 4 possible rotation angles) mode, you are better off asking Wayland directly (as done, e.g., in this Megapixels commit) or using a wrapper API like QtGui's QScreen (QGuiApplication::primaryScreen()->orientation()).

Thanks.

I am writing an application in Python, with Gtk. One of my very first tasks is to get a reading from the camera, but the usual lines that would work on my computer fail on the pinephone:
```
import cv2
cap = cv2.VideoCapture(0)
```
I tried passing the path to `VideoCapture`, like `/dev/video0`, `/dev/video1`, etc, but that doesn't work either.

[Kevin Kofler]

So you are using OpenCV. It looks like your OpenCV is trying to use raw legacy V4L2 (or even V4L1) as the backend. That will not work. Since OpenCV does not (currently) support libcamera directly (nor Pipewire), you will need an OpenCV compiled with GStreamer support, which should work with libcamera if OpenCV is using GStreamer correctly. (Unfortunately, GStreamer exposes the implementation details in some APIs, so some applications using GStreamer do not work unchanged with libcamera.)

EDIT: Judging from the errors you quoted in another thread, it looks like OpenCV is already using GStreamer, but in a way that hardcodes the V4L2 source (v4l2src0), so the OpenCV GStreamer backend needs fixing.

[lagrang3]

So you are using OpenCV. It looks like your OpenCV is trying to use raw legacy V4L2 (or even V4L1) as the backend. That will not work. Since OpenCV does not (currently) support libcamera directly (nor Pipewire), you will need an OpenCV compiled with GStreamer support, which should work with libcamera if OpenCV is using GStreamer correctly. (Unfortunately, GStreamer exposes the implementation details in some APIs, so some applications using GStreamer do not work unchanged with libcamera.)

EDIT: Judging from the errors you quoted in another thread, it looks like OpenCV is already using GStreamer, but in a way that hardcodes the V4L2 source (v4l2src0), so the OpenCV GStreamer backend needs fixing.

Thanks.

I did some quick research and I found the identifiers for the backends

Code:

>>> [ (b,cv2.videoio_registry.getBackendName(b)) for b in cv2.videoio_registry.getCameraBackends() ]
[(1800, 'GSTREAMER'), (200, 'V4L2'), (300, 'FIREWIRE'), (2500, 'UEYE'), (1700, 'GPHOTO2')]
>>> assert cv2.CAP_GSTREAMER == 1800
>>> cv2.videoio_registry.getBackendName(cv2.CAP_GSTREAMER)
'GSTREAMER'

But OpenCV stills calls v4l when I call `VideoCapture`

Code:

>>> cap = cv2.VideoCapture(0,cv2.CAP_GSTREAMER)
[ WARN:0@1615.039] global ./modules/videoio/src/cap_gstreamer.cpp (2401) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src9 reported: Device '/dev/video0' is not a capture device.
[ WARN:0@1615.040] global ./modules/videoio/src/cap_gstreamer.cpp (1356) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@1615.040] global ./modules/videoio/src/cap_gstreamer.cpp (862) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

[Kevin Kofler]

As I wrote in the other thread, try using the Python GStreamer bindings (gst-python) instead of OpenCV. OpenCV is an extra layer that is not necessary if all you want is to get an image from the camera. It only makes sense if you want to use its image analysis features.