New version of MCUBoot bootloader : I need your help!

[JF002]

Hi everyone!
These last few weeks, I've been working on improving the MCUBoot bootloader. This bootloader is the one that is programmed at the factory along with InfiniTime 0.7.1.
I think I've reached a state that allows other developers and users of PineTime devkit (not sealed) to test, debug and raise issues.
But first, a bit of background info:

What is the purpose of a bootloader?
The bootloader is the first software to be ran after a reset. Its main goal is to initialize the MCU and load the application firmware. The bootloader can provides many additional functionalities like firmware upgrade, recovery, firmware authentification, HW diagnostic and selftest,...

As of now, there are 2 major bootloaders for the PineTime:

• The MCUBoot bootloader (used by InfiniTime)
  
• The NRF Bootloader (used by Wasp-os)
  

They provide different functionalities and are based on different software.

The MCUBoot bootloader is 100% open-source and provide  a minimal set of functionalities : firmware upgrade (the upload of the new firmware is done by the application firmware, InfiniTime for example), manual revert to the previous version of the firmware, really basic UI.

The NRF bootloader is based on closed-source software from Nordic Semi and provide more functionalities like OTA (firmware upload via BLE). This bootloader is tightly coupled to the NRF SoftDevice BLE stack (closed source as well).

Both bootloaders are incompatible (a firmware built for one cannot run on the other one).

Why do we need a new version of the MCUBoot bootloader?
To improve it! The version that is currently programmed at the factory is the very first version we released. Since then, we found some bugs and found out we needed a bit more functionalities and reliability.

The most visible additions to this new version are :

• The possibility to force revert to the previous version of the firmware
  
• The possibility to install a recovery firmware from the external flash memory. This firmware can be useful if you installed a firmware that does not support OTA, for example.
  
• Simple UI
  
• The bootloader displays its own version and exports it to the application.
  

In the end, the goal is to be confident enough in this bootloader to advise users of sealed PineTimes to update their device. And maybe apply the new bootloader at production level too.

How to test it?
The code of the new bootloader is available on github. It contains the code of the bootloader extracted from Lup's repo.

The file README.md explains how the bootloader works and provides pictures of the UI.

The release page contains one test release.

If you have a devkit, a SWD debugger and if you know how to restore you device in case of issue, you can try to apply the update as explained on the release page.

If you don't have a SWD setup, don't know how to use it to reflash your device from scratch or if you use a sealed device, please, do not apply this update and wait for an actual release of the bootloader !

What should be tested?

• The update procedure from a device coming out of factory (factory bootloader + InfiniTime 0.7.1) : First update the bootloader, then flash the recovery firmware, then use your PineTime
  
• The OTA procedure (update InfiniTime)
  
• The revert procedure
  
• The recovery procedure
  
• The switching procedure to/from InfiniTime and wasp-os
  

And report issues, comments, questions, feedbacks and successes on this post, or in the github repo!

[EDIT 01/01/2021] : Test degraded cases
The release page now contains 3 test firmwares that can be applied to test degraded cases that could happen during the OTA:

• testfw-random-dfu.zip : this firmware contains only random data instead of executable code. This could happen if the user applied a DFU file that is not intended to be run on a Pinetime. In this case, the bootloader will apply the upgrade, notice that it's not runnable and will automatically revert to the previously running firmware.
  
• testfw-no-validate-no-watchdog.zip : this firmware displays a yellow InfiniTime logo. It does not validate the firmware and does not refresh the watchdog. It simulates a firmware that crashes and that is not able to refresh the firmware. The watchdog will reset the device and MCUBoot will revert to the previously running firmware.
  
• testfw-validate-wdt-no-ota.zip : This firmware does refresh the watchdog AND automatically validate the version. It means that to the point of view of MCUBoot, the firmware is working fine! BUT... this bad firmware does not provide the OTA and the reset functionality. This is one of the worst degraded case : the bootloader cannot do anything! The only workaround is to wait for the battery to drain completely and then charge the PineTime again. This is reset the CPU and offer the possibility to force a firmware revert during the next boot.
  
• The worst of the worst degraded cases is one where the new firmware erase or overwrite the bootloader and/or the recovery firmware with invalid data. In this case, the device is totally bricked and can only be recovered using SWD (meaning the device must be opened).
  

Thanks!

[wibble]

I've not done anything with the PineTime in a while, so this is as good an excuse as any The Pi I was using for building and flashing has been repurposed since, so I might have a bit to do before I can test. Do you have links to the procedures you want testing, or did I miss something in the readme?

[JF002]

I've not done anything with the PineTime in a while, so this is as good an excuse as any The Pi I was using for building and flashing has been repurposed since, so I might have a bit to do before I can test. Do you have links to the procedures you want testing, or did I miss something in the readme?

I'm happy to provide you with an excuse to work on your PineTime !

The installation procedure is in the release note.

To install the bootloader, you'll first need to run on InfiniTime (0.7.1 or 0.9.0, for example). Then, you have 2 options:

• Using SWD : flash bootloader.bin at offset 0x00, reset, done.
  
• Using OTA : use NRFConnect or gadgetbridge to OTA a specific firmware that upgrades the bootloader : reloader-mcuboot.zip. When the transfert is complete, your device will reset, the bootloader will be updated, the device will reset again on the new bootloader.
  

When the new bootloader is installed, you can install the recovery firmware by using pinetime-mcuboot-recovery-loader.zip. The procedure is the same : OTA this file, reset, let the tool run, reset again and it's done.

You can have a look at the video listed in the release note to have an overview of the procedure : https://video.codingfield.com/videos/wat...a466957169

From there, you can test the revert and recovery functionalities.
Revert :

• OTA a new version of InfiniTime and validate it.
  
• Reset the PineTime (long push on the button).
  
• When you see a pinecone, press the button until the cone is drawn in blue.
  
• Wait for the revert to be applied
  
• and voilà, you're running the previous version of your firmware.
  

Recovery:

• Reset the PineTime (long push on the button).
  

• When you see a pinecone, press the button until the cone is drawn in red.
  
• Wait for the recovery firmware to be installed.
  
• Your Pinetime should reboot and display the InfiniTime logo. That's it, you're running the recovery firmware. You can try to OTA any version of InfiniTime you like (>= 0.7.1).
  

[koemdjiev@gmail.com]

Hello JF002! I followed the OTA steps above:
"Using OTA : use NRFConnect or gadgetbridge to OTA a specific firmware that upgrades the bootloader : reloader-mcuboot.zip. When the transfert is complete, your device will reset, the bootloader will be updated, the device will reset again on the new bootloader."

However, I noticed that I never got version 0.9.0 of the firmware (same as in the video).

The procedure I followed is this one - I used nRFConnect app to flash the reloader-mcuboot.zip file and I checked the version which was 0.8.3 (the version I had before I started the steps). Then I flashed the pinetime-mcuboot-recovery-loader.zip file but the version didn't change. It looks strange to me.

[JF002]

Hi @koemdjiev and thanks for testing the update procedure!

This is an expected behavior : the application firmware (InfiniTime), the bootloader and the recovery firmware are 3 different softwares and they all have their own version.
In your case, before the update, you were running InfiniTime 0.8.3, the old bootloader (I can't remember its version), and no recovery software.

When you applied reloader-mcuboot.zip, it only upgraded the bootloader (old -> test version "1.2.3"). But it didn't upgrade the version of InfiniTime that was running on your device.
Then your applied pinetime-mcuboot-recovery-loader.zip. This tool flashes the recovery firmware into the SPI flash memory, but again, it didn't touch the version of InfiniTime you are currently running.
Now, you're still running InfiniTime 0.8.3, and you have the new bootloader (test version "1.2.3") and the new recovery firmware installed on your device.

So, now, if these 2 steps were successful (do you see the new screen of the bootloader when you reset the watch ?), you can update InfiniTime by using the DFU file from the release page (dfu-0.9.0.zip).
[url=https://github.com/JF002/Pinetime/releases/tag/0.9.0][/url]

[koemdjiev@gmail.com]

Hi @koemdjiev and thanks for testing the update procedure!

This is an expected behavior : the application firmware (InfiniTime), the bootloader and the recovery firmware are 3 different softwares and they all have their own version.
In your case, before the update, you were running InfiniTime 0.8.3, the old bootloader (I can't remember its version), and no recovery software.

When you applied reloader-mcuboot.zip, it only upgraded the bootloader (old -> test version "1.2.3"). But it didn't upgrade the version of InfiniTime that was running on your device.
Then your applied pinetime-mcuboot-recovery-loader.zip. This tool flashes the recovery firmware into the SPI flash memory, but again, it didn't touch the version of InfiniTime you are currently running.
Now, you're still running InfiniTime 0.8.3, and you have the new bootloader (test version "1.2.3") and the new recovery firmware installed on your device.

So, now, if these 2 steps were successful (do you see the new screen of the bootloader when you reset the watch ?), you can update InfiniTime by using the DFU file from the release page (dfu-0.9.0.zip).
[url=https://github.com/JF002/Pinetime/releases/tag/0.9.0][/url]

No problem at all, I am glad to help and put my pinetime to use. Thank you very much for the explanation. That makes so much sense now. As for the logo, yes I see the new bootloader screen fine. I also tested the revert function by updating to 0.9.0 and then reverting to the previous 0.8.3 version. It worked fine

[VMMainFrame]

I've not done anything with the PineTime in a while, so this is as good an excuse as any The Pi I was using for building and flashing has been repurposed since, so I might have a bit to do before I can test. Do you have links to the procedures you want testing, or did I miss something in the readme?

I'm happy to provide you with an excuse to work on your PineTime !

The installation procedure is in the release note.

To install the bootloader, you'll first need to run on InfiniTime (0.7.1 or 0.9.0, for example). Then, you have 2 options:

• Using SWD : flash bootloader.bin at offset 0x00, reset, done.
  
• Using OTA : use NRFConnect or gadgetbridge to OTA a specific firmware that upgrades the bootloader : reloader-mcuboot.zip. When the transfert is complete, your device will reset, the bootloader will be updated, the device will reset again on the new bootloader.
  

When the new bootloader is installed, you can install the recovery firmware by using pinetime-mcuboot-recovery-loader.zip. The procedure is the same : OTA this file, reset, let the tool run, reset again and it's done.

You can have a look at the video listed in the release note to have an overview of the procedure : https://video.codingfield.com/videos/wat...a466957169

From there, you can test the revert and recovery functionalities.
Revert :

• OTA a new version of InfiniTime and validate it.
  
• Reset the PineTime (long push on the button).
  
• When you see a pinecone, press the button until the cone is drawn in blue.
  
• Wait for the revert to be applied
  
• and voilà, you're running the previous version of your firmware.
  

Recovery:

• Reset the PineTime (long push on the button).
  

• When you see a pinecone, press the button until the cone is drawn in red.
  
• Wait for the recovery firmware to be installed.
  
• Your Pinetime should reboot and display the InfiniTime logo. That's it, you're running the recovery firmware. You can try to OTA any version of InfiniTime you like (>= 0.7.1).
  

I installed the new bootloader and the recovery firmware OTA using Gadgetbridge (much faster than NRFConnect).  This went well.  I then reset the Pinetime and held down the button until the pinecone was red.  The recovery firmware came up with the blue InfiniTime logo.  I noticed the screen does not turn off when running the recovery firmware.  I then uploaded 0.9.0 OTA using Gadgetbridge and that went well also.

Hugh

[wibble]

Ok - using the Pi for the SWD updates and GadgetBridge 0.51.0 from F-Droid for the OTA. I've not tried OTA before. Note that the phone is somewhat knackered, and has been slightly glitchy with bluetooth recently. This appears unrelated to GadgetBridge and PineTIme.

First used Lup's pinetime-updater to get the PineTime into a known state - it's 'latest' bootloader and application, which is 0.7.1 at present.
Download the RC1 bootloader.bin and flash to 0x0000 using pinetime-updater's from downloaded file option. This gets me the new bootloader working v1.2.3 bootloader, still bringing me into 0.7.1
Download RC1 pinetime-mcuboot-recovery-loader.zip, connect GadgetBridge and use it to OTA. This failed on the first attempt soon after touching the screen to keep it from dimming, but worked on a second attempt when I didn't touch the screen. I don't know if that's an unexpected quirk of GadgetBridge, or just the glitchy bluetooth on this phone.
Download dfu-0.9.0.zip and use GadgetBridge to OTA that - I didn't touch the screen and it worked first time. I now have 0.9.0 and can validate it.
I now tried a few reboot cycles, both green to keep current version, and blue to swap between 0.7.1 and 0.9.0 - it seems smooth and reliable.
Long press to red during the bootloader gets me to the white InfiniTime logo. I then tried OTA to 0.9.0 again. The first time this failed with the progress bar just short of completion, and I had to long-press to reboot and try again. The second attempt failed at ~50% but the third succeeded, and I rebooted into 0.9.0 and validated. I suspect my iffy bluetooth again, but this also shows the process is robust - even with failures during OTA I could still eventually get a successful OTA and a working application without having to use the cable. Nice

I haven't tried the to/from wasp-os yet.

---

On a side note the combination of GadgetBridge and kdeconnect allowed the media controls in 0.9.0 to control the video in the desktop browser, but the scrolling of a title as long as 'WIP: bootloader improvements, factory firmware, OTA - Codingfield' is too fast for comfort.

[VMMainFrame]

Ok - using the Pi for the SWD updates and GadgetBridge 0.51.0 from F-Droid for the OTA. I've not tried OTA before. Note that the phone is somewhat knackered, and has been slightly glitchy with bluetooth recently. This appears unrelated to GadgetBridge and PineTIme.

First used Lup's pinetime-updater to get the PineTime into a known state - it's 'latest' bootloader and application, which is 0.7.1 at present.
Download the RC1 bootloader.bin and flash to 0x0000 using pinetime-updater's from downloaded file option. This gets me the new bootloader working v1.2.3 bootloader, still bringing me into 0.7.1
Download RC1 pinetime-mcuboot-recovery-loader.zip, connect GadgetBridge and use it to OTA. This failed on the first attempt soon after touching the screen to keep it from dimming, but worked on a second attempt when I didn't touch the screen. I don't know if that's an unexpected quirk of GadgetBridge, or just the glitchy bluetooth on this phone.
Download dfu-0.9.0.zip and use GadgetBridge to OTA that - I didn't touch the screen and it worked first time. I now have 0.9.0 and can validate it.
I now tried a few reboot cycles, both green to keep current version, and blue to swap between 0.7.1 and 0.9.0 - it seems smooth and reliable.
Long press to red during the bootloader gets me to the white InfiniTime logo. I then tried OTA to 0.9.0 again. The first time this failed with the progress bar just short of completion, and I had to long-press to reboot and try again. The second attempt failed at ~50% but the third succeeded, and I rebooted into 0.9.0 and validated. I suspect my iffy bluetooth again, but this also shows the process is robust - even with failures during OTA I could still eventually get a successful OTA and a working application without having to use the cable. Nice

I haven't tried the to/from wasp-os yet.

---

On a side note the combination of GadgetBridge and kdeconnect allowed the media controls in 0.9.0 to control the video in the desktop browser, but the scrolling of a title as long as 'WIP: bootloader improvements, factory firmware, OTA - Codingfield' is too fast for comfort.

I have also had problems with Bluetooth connecting to the Pinetime.  My phone is a Pixel 3A.  NRFConnect and Gadgetbridge sometimes seem to interfere with one another.  If Gadgetbridge won't connect to the Pinetime I have to start NRFConnect, scan and connect with it, then exit NRFConnect and go to Gadgetbridge and then it will connect.  I also have the problem with OTA uploads to the Pinetime failing sometimes.  A retry usually works but there was an occasion I had to try five times before it worked.

Hugh

[JF002]

@koemdjiev@gmail.com @VMMainFrame @wibble : Thank you very much for your success reports! It's really nice to see that it runs on other's setup too!

I noticed the screen does not turn off when running the recovery firmware.

Yes, I've disabled this feature in the recovery firmware to make it even more simple.


Regarding BLE connection reliability : Indeed, sometimes, the connection fails, sometimes the OTA fails,... It's not easy to figure out what's failing in these cases : ble on the phone, ble on the Pinetime, RF interference,... We'll try to improve that in the future, but in the meantime, here are some workarounds:

• Most of the time, resetting the watch and/or the phone and retrying works. To reboot the watch running InfiniTime, simply push the button for 5-10s, it'll reset and reboot.
  
• Do not run NRFConnect and Gadgetbridge at the same time. If one of them auto-connects to the PineTime, the other one won't be able to connect. Also, I noticed that disabling the CTS (current time service) into NRFConnect improves the situation with Gadgetbridge, even if you're not using NRFConnect...
  
• The proximity of other WIFI/BLE/USB-C(with low quality wires) devices might cause RF interferences and cause the transfer to fail.
  
• [EDIT] When using Gadgetbridge, the OTA fails if you receive a notification (gadgetbridge will formward it to InfiniTime and it'll conflict with the OTA process). This is a known bug. As a workaround, you can disconnect from the internet to stop receiving notifications.