Pinetime and P8 Integration testing framework

[maiden]

Once upon a time, i found my self in a position where i had to develop a firmware for an IoT device with different hardware revisions, different cellular modems used in several different countries / cellular networks. There was continous delivery of new and custom fw versions pushed over the air. To debug and verify new firmware releases was.. interesting.. 

I tried to set up solutions with jenkins or gitlab runners to automatically or on request deploy code to the targets, but it was hard to get it "right".

To make my life easier (possible), I developed an integration testing framework this framework consisted of  "test nodes" connecting to a "test manager" and then a API that i implementad a commandline interface and a web gui for. 

The test node:
The test node consist of a raspberry pi / orange pi / pine board, a jtag/swd flasher (or gpio flashing), a Ina219 module for current measuring, a servo and a relay.

DUT = Device Under Test, in this case a PineTime

The test node can:

• cause button press / release
  
  
• cause motion / tilt
  
  
• measure current consumtion / charge current
  
  
• turn on / off charging
  
  
• read debug output (RTT and / or UART)
  

The test manager
the manager keeps track of nodes and stores test results 

The API / GUI
lets the user allocate / queue test executions on test nodes. fetch / view logs of previous tests, view test results

The command line tool can be triggered like this:
python3 runOnTestNode.py completeTest node1 test.hex
you can also get tests, nodes, and test results listed as well as real time RTT/Debug output from the DUT.

while the GUI currently looks like this:




old comparison view between two executions:



Everything is written in Python, and some html/javascript for the web interface. 

To make things easy, tests often rely on parsing debug output, this can be an issue in some cases, but in general it makes life easier for both developer and test framwork.

Here is an example of a test script (extract from powerOnPowerOff.py):

Code:

    def run(self):
        self.timeSinceStart = 0
        self.flash()

        # verify that modem was detected
        self.addDelayedParse(20,"IAmNow",1)

        # run, then turn off'
        self.addDelayedEvent(20,self.buttonIn,())
        self.addDelayedEvent(5,self.buttonOut,())
        self.addDelayedEvent(30,self.parse,("WillNowPowerOff",1))

        # verify that we do not boot when off and charging starts
        self.addDelayedEvent(10,self.chargerOn,())
        self.addDelayedEvent(20,self.parse,("OFFcharging",1))

        # wait, then turn on again, verify that we remember our modem type
        self.addDelayedEvent(10,self.buttonIn,())
        self.addDelayedEvent(5,self.buttonOut,())
        self.addDelayedEvent(30,self.parse,("IKnowIAmA",1))
        self.addDelayedEvent(1,self.end,())
        self.start()

I have also used this type of setup in production, where each test node is a test fixture for PCBA testing, you will want to modify the tests and framework a bit for this usecase.

So i will do two things, 1. i will release the test framework as open source, 2. I will leave my own hosting of the framework open for public use (with a pinetime node and a p8 node running)  

But first i must get everything up and running, its working fine right now, but i do only have "mocked" hw interfaces, i am working on wiring everything up and beautifying the user interface.

[JF002]

This test setup looks awesome!

Would you like to provide information on how to build a test node (parts references, maybe a picture of the final result,..) ? It looks simple enough for me to build one by myself

[maiden]

The wiring of the test node is the first image of my first post.

The ina219 board can be found here:
https://www.adafruit.com/product/904
or
https://www.aliexpress.com/item/40009835...web201603_

the servo is any "9g servo"

the relay is something simular to this:
https://www.aliexpress.com/item/40009560...web201603_

for flashing, i have used in the past:
rpi gpio flashing, jlink (nrf dev board), stlinkv2, stlinkv2 (flashed with blackmagicprobe).
Currently i use the nRF52 DK as a flasher, as show in the image above.

If you want to use UART instead of RTT, then hook up rx/tx between target and RPI.

I will CAD and 3d print some mechanics to neatly hold the PCBA while being able to tilt the device and provide some kind of strain relief on the wires.

hopefully i will get some time over this week to get the "real" test nodes up and running. I am expecting my two P8 devices this week.

After my initial release, i will start to add database support (TinySQL). right now all the files are stored as filesystem files / folders. This works well and was good for single / few developers in a controlled environment. But it does not scale well.


The testscript on the node calls some "interface" files, located in the node/interface/ folder. in the past it have been "buttonIn.sh, buttonOut.sh, readSerial.sh, readPower.sh, chargerOn.sh, chargerOff.sh, flash.sh", when i used the .sh files, they where called in new threads, but cleanup of readSerial and readPower threads caused issues. so now i have changed to python files where i can have more flexibility.
Right now, the cleanup of JLinkRTTClient is hardcoded to a "killall JLinkRTTClient" inside the actual test node code. this should be moved to a execute and a cleanup function call inside the interface file.

The communication between testnode / testmanager / testAPI is json structures sent through TCP. most data sent is serialized as hex, (i will probably move to base64 when i get time).

[maiden]

Made a small video to demonstrate the workflow:

[JF002]

Thanks for these additional informations! I ordered some items to try to reproduce a similar setup. If you provide 3D print models, I'll ask a friend to print them for me

At first, I couldn't understand how to connect the battery and charger (the red wires are mixed). Could you confirm this : 
 - The charger is powered by the USB port of the RPI, and enabled/disable by the relay.
 - The relay board is powered by the 5V from the pin header of the RPI.
 - The battery is connected to Vin+ and Vin- of the INA219 board

I'm sure that people would be interested to contribute to the project if you publish it on github/gitlab/whatever !

[maiden]

The code is already on gitlab, in a public repo, 
I wanted to resolve the interface handling before sharing the link.

First part in that process is to bring up a test node to verify everything on real hardware.
I am currently working on that.

Here is the repo: (remember, the code is a horrible mess, and some core stuff will probably change soon-ish) 
https://gitlab.com/maidenOne/trixtestframework/

the wiring for the relay is:

RPI 5V -> Relay 5v
RPI GND -> Relay gnd 
RPI GPIO pin -> Relay pin (even if the relay runs on 5v logic, a 3.3V high should trigger)

USB / 5V source -> Relay in, Relay out -> Pinetime 5v pad 
USB / GND -> Pinetime GND pad


The wiring for the INA219 and battery is:

RPI I2c -> INA I2C
RPI 3.3V -> INA VCC
RPI GND -> INA GND

PINETIME BATTERY VCC WIRE (RED) -> INA V+ 
INA V- -> PINETIME PCB Battery pad
BATTERY GND WIRE (BLACK) -> PINETIME PCB Battery pad

[maiden]

I have improved the styling a bit more, and solved a few bugs in the interface code. I have also rewritten the interface handling.

Now we have the following "structure"

Testfile -> TrixTest -> HAL -> Interface

when you create a test file, you only have to look at the API from TrixTest.
when you add a new test node, you modify the interface files only, or possibly some small changes in HAL.

   

[maiden]

its now live at http://foss.site/

there is not any good tests yet, and the TestAPI is not live yet. but feel free to play around.

issues / feature requests is best put on the project page on gitlab, link above.