Pinetime and P8 Integration testing framework

[maiden]

Once upon a time, i found my self in a position where i had to develop a firmware for an IoT device with different hardware revisions, different cellular modems used in several different countries / cellular networks. There was continous delivery of new and custom fw versions pushed over the air. To debug and verify new firmware releases was.. interesting.. 

I tried to set up solutions with jenkins or gitlab runners to automatically or on request deploy code to the targets, but it was hard to get it "right".

To make my life easier (possible), I developed an integration testing framework this framework consisted of  "test nodes" connecting to a "test manager" and then a API that i implementad a commandline interface and a web gui for. 

The test node:
The test node consist of a raspberry pi / orange pi / pine board, a jtag/swd flasher (or gpio flashing), a Ina219 module for current measuring, a servo and a relay.

DUT = Device Under Test, in this case a PineTime

The test node can:

• cause button press / release
  
  
• cause motion / tilt
  
  
• measure current consumtion / charge current
  
  
• turn on / off charging
  
  
• read debug output (RTT and / or UART)
  

The test manager
the manager keeps track of nodes and stores test results 

The API / GUI
lets the user allocate / queue test executions on test nodes. fetch / view logs of previous tests, view test results

The command line tool can be triggered like this:
python3 runOnTestNode.py completeTest node1 test.hex
you can also get tests, nodes, and test results listed as well as real time RTT/Debug output from the DUT.

while the GUI currently looks like this:




old comparison view between two executions:



Everything is written in Python, and some html/javascript for the web interface. 

To make things easy, tests often rely on parsing debug output, this can be an issue in some cases, but in general it makes life easier for both developer and test framwork.

Here is an example of a test script (extract from powerOnPowerOff.py):

Code:

    def run(self):
        self.timeSinceStart = 0
        self.flash()

        # verify that modem was detected
        self.addDelayedParse(20,"IAmNow",1)

        # run, then turn off'
        self.addDelayedEvent(20,self.buttonIn,())
        self.addDelayedEvent(5,self.buttonOut,())
        self.addDelayedEvent(30,self.parse,("WillNowPowerOff",1))

        # verify that we do not boot when off and charging starts
        self.addDelayedEvent(10,self.chargerOn,())
        self.addDelayedEvent(20,self.parse,("OFFcharging",1))

        # wait, then turn on again, verify that we remember our modem type
        self.addDelayedEvent(10,self.buttonIn,())
        self.addDelayedEvent(5,self.buttonOut,())
        self.addDelayedEvent(30,self.parse,("IKnowIAmA",1))
        self.addDelayedEvent(1,self.end,())
        self.start()

I have also used this type of setup in production, where each test node is a test fixture for PCBA testing, you will want to modify the tests and framework a bit for this usecase.

So i will do two things, 1. i will release the test framework as open source, 2. I will leave my own hosting of the framework open for public use (with a pinetime node and a p8 node running)  

But first i must get everything up and running, its working fine right now, but i do only have "mocked" hw interfaces, i am working on wiring everything up and beautifying the user interface.