03-01-2020, 07:09 AM
For a few weeks now, basic functionality is working with my PineTime firmware project. It displays the time, synchronized over Bluetooth, and I'm able to
navigate between simple applications with touchscreen gestures. This last week I started work on getting the hardware scroll support in the display working for my firmware. The goal was to have a smooth scrolling experience between different watch faces. Now that I have it working the way I want it, it is time to write down how I've done it. This post got a bit longer than I initially expected. Scroll down to the bottom for a link to my code and a video of the result.
For those not familiar with my firmware, I've been working on a generic, but fairly modular PineTime firmware for a while now. One of the goals is to make it easy to develop extra watch faces and applications by other developers and extend and tune their version of the firmware to their needs. The firmware is based on RIOT, LittlevGL, Nimble and will use LittleFS for the filesystem. RIOT is an embedded operating systems geared towards IoT devices. It provides threading, power efficiency and the HAL out of the box to support devices like the PineTime.
Integrating smooth scrolling into the LittlevGL graphics was challenging. LittlevGL provides a high level graphics interface, screens are built from elements such as labels, buttons and other high level GUI elements. Setup of the library requires a few callbacks, one to render a rectangular area of pixels on the screen and one to retrieve input events from the touch screen. LittlevGL supports full partial updates where it will only render the rectangular area on the display that requires updating. LittlevGL is able to work with rendering buffers smaller than the display area and will send out multiple render commands to update the full display if this is the case.
However, for the scroll to work it is required to render the full screen of the PineTime and not only the diff between the two different watch faces. This is because of how the hardware scroll behaviour on the ST7789V works. The ST7789V is a 240x320 pixels TFT controller. With the display of the PineTime being only 240x240 pixels big, we have an area of 240x80 pixels that is not visible on the screen, but is available in the chip to write to. So of this 240x320 pixels of display RAM on the chip, an area of 240x240 is used for the screen. The exact area displayed on the screen can be adjusted with the VSCSAD command (0x37, page 218 of the manual). With this command it is possible to configure on which row the top of the screen will start to render.
To achieve vertical scrolling we first write part of the new screen content to the unused display RAM area. Then the scroll start address is modified to show the new part of the new screen content and the remaining part of the old screen content. With the scroll start address adjusted, the next area that is out of the display can be updated after which the scroll start address is adjusted again to show the new area. Repeat this until the old content is completely replaced with the new content. If this is done fast enough, the stutter of the piecewise content updates is not visible to the human eye. The direction of the scroll start address adjustments will determine the direction of the display scroll motion. The catch with this hardware scroll is that it requires a full display content update. Even if both screens are similar in content, the content of the display RAM is shifted by 80 rows compared to the previous content. Furthermore, most of the previous content is overwritten by this operation.
LittlevGL is optimized to efficiently update partial areas of the screen to minimize the amount of data transfers required to the display. With the SPI connection to the ST7789V chip limited at 8Mhz, this is most of the time the limiting factor in the refresh rate of the screen content. The partial updates help here to make screen updates as smooth and fast as possible and minimize the screen render tearing associated with this.But for the smooth scrolling to work, it is LittlevGL needs to be convinced to render the full screen content instead of a partial refresh. To know how to do this we need to know of the refresh behavior of LittlevGL is done.
Internally, LittlevGL tracks which objects are modified between render operations. Modifying the text string of a label, or pressing a button sets a flag on those objects and triggers a re-render of the area occupied by those objects and all child objects. The rendered objects are internally organized within LittlevGL as a tree with a screen type object at the root. Multiple levels are possible, for example a button and a label contained within a small pop-up message. As mentioned, a screen type object is the base object of the tree structure of the GUI elements. It can be replaced by a new screen object for example to render different applications.
My PineTime firmware creates a new screen when switching between applications. The application itself will build a new screen and the GUI library only has to pass the new screen object to LittlevGL and remove the old screen object. The trick is that this in turn can be used to trigger a scroll operation. As a screen object encompasses the full display area, invalidating it by replacing it triggers a refresh of the full 240x240 display area. At this point it hooks into the low level GUI code of my firmware. When the firmware is instructed to switch between two applications, it is passed a scroll direction. It will switch the active screen object for a new one in LittlevGL and store locally the specified scroll direction. As soon as LittlevGL starts calling the rendering callback, this scroll direction is retrieved and used to determine if and how both the render offset and a new display scroll start address should be modified. After a full 240x240 pixels render, the GUI code automatically switch back to non-scroll behaviour to allow partial updates of the current screen by LittlevGL.
Results:
The result of this is visible in a short video I recorded. It shows smooth scrolling between different watch faces. Only the watch face
showing the time is fully functional, the other two are dummies two demonstrate the scroll operation. Switching between the watch faces is triggered by up and down swipe gestures as detected by the touch screen controller. A single render chunk is 240x5 pixels of content on the display. After every pixel chunk transferred, the scroll start address is updated to show the new chunk. This slows down the refresh speed a tiny bit, but this is acceptable to get this result.
Full firmware source (binaries available) on the Github project
navigate between simple applications with touchscreen gestures. This last week I started work on getting the hardware scroll support in the display working for my firmware. The goal was to have a smooth scrolling experience between different watch faces. Now that I have it working the way I want it, it is time to write down how I've done it. This post got a bit longer than I initially expected. Scroll down to the bottom for a link to my code and a video of the result.
For those not familiar with my firmware, I've been working on a generic, but fairly modular PineTime firmware for a while now. One of the goals is to make it easy to develop extra watch faces and applications by other developers and extend and tune their version of the firmware to their needs. The firmware is based on RIOT, LittlevGL, Nimble and will use LittleFS for the filesystem. RIOT is an embedded operating systems geared towards IoT devices. It provides threading, power efficiency and the HAL out of the box to support devices like the PineTime.
Integrating smooth scrolling into the LittlevGL graphics was challenging. LittlevGL provides a high level graphics interface, screens are built from elements such as labels, buttons and other high level GUI elements. Setup of the library requires a few callbacks, one to render a rectangular area of pixels on the screen and one to retrieve input events from the touch screen. LittlevGL supports full partial updates where it will only render the rectangular area on the display that requires updating. LittlevGL is able to work with rendering buffers smaller than the display area and will send out multiple render commands to update the full display if this is the case.
However, for the scroll to work it is required to render the full screen of the PineTime and not only the diff between the two different watch faces. This is because of how the hardware scroll behaviour on the ST7789V works. The ST7789V is a 240x320 pixels TFT controller. With the display of the PineTime being only 240x240 pixels big, we have an area of 240x80 pixels that is not visible on the screen, but is available in the chip to write to. So of this 240x320 pixels of display RAM on the chip, an area of 240x240 is used for the screen. The exact area displayed on the screen can be adjusted with the VSCSAD command (0x37, page 218 of the manual). With this command it is possible to configure on which row the top of the screen will start to render.
To achieve vertical scrolling we first write part of the new screen content to the unused display RAM area. Then the scroll start address is modified to show the new part of the new screen content and the remaining part of the old screen content. With the scroll start address adjusted, the next area that is out of the display can be updated after which the scroll start address is adjusted again to show the new area. Repeat this until the old content is completely replaced with the new content. If this is done fast enough, the stutter of the piecewise content updates is not visible to the human eye. The direction of the scroll start address adjustments will determine the direction of the display scroll motion. The catch with this hardware scroll is that it requires a full display content update. Even if both screens are similar in content, the content of the display RAM is shifted by 80 rows compared to the previous content. Furthermore, most of the previous content is overwritten by this operation.
LittlevGL is optimized to efficiently update partial areas of the screen to minimize the amount of data transfers required to the display. With the SPI connection to the ST7789V chip limited at 8Mhz, this is most of the time the limiting factor in the refresh rate of the screen content. The partial updates help here to make screen updates as smooth and fast as possible and minimize the screen render tearing associated with this.But for the smooth scrolling to work, it is LittlevGL needs to be convinced to render the full screen content instead of a partial refresh. To know how to do this we need to know of the refresh behavior of LittlevGL is done.
Internally, LittlevGL tracks which objects are modified between render operations. Modifying the text string of a label, or pressing a button sets a flag on those objects and triggers a re-render of the area occupied by those objects and all child objects. The rendered objects are internally organized within LittlevGL as a tree with a screen type object at the root. Multiple levels are possible, for example a button and a label contained within a small pop-up message. As mentioned, a screen type object is the base object of the tree structure of the GUI elements. It can be replaced by a new screen object for example to render different applications.
My PineTime firmware creates a new screen when switching between applications. The application itself will build a new screen and the GUI library only has to pass the new screen object to LittlevGL and remove the old screen object. The trick is that this in turn can be used to trigger a scroll operation. As a screen object encompasses the full display area, invalidating it by replacing it triggers a refresh of the full 240x240 display area. At this point it hooks into the low level GUI code of my firmware. When the firmware is instructed to switch between two applications, it is passed a scroll direction. It will switch the active screen object for a new one in LittlevGL and store locally the specified scroll direction. As soon as LittlevGL starts calling the rendering callback, this scroll direction is retrieved and used to determine if and how both the render offset and a new display scroll start address should be modified. After a full 240x240 pixels render, the GUI code automatically switch back to non-scroll behaviour to allow partial updates of the current screen by LittlevGL.
Results:
The result of this is visible in a short video I recorded. It shows smooth scrolling between different watch faces. Only the watch face
showing the time is fully functional, the other two are dummies two demonstrate the scroll operation. Switching between the watch faces is triggered by up and down swipe gestures as detected by the touch screen controller. A single render chunk is 240x5 pixels of content on the display. After every pixel chunk transferred, the scroll start address is updated to show the new chunk. This slows down the refresh speed a tiny bit, but this is acceptable to get this result.
Full firmware source (binaries available) on the Github project
