Bad apple (video) playing on a pinetime

[TT-392]

I put bad apple on the pinetime.

You can also watch it on odysee:
https://odysee.com/@TT-392:b/bad-apple-pinetime:8

Source code:
https://github.com/TT-392/pinetime-bad-apple
https://github.com/TT-392/Pinetime-bad-a...-processor

[barray]

Haha very cool demonstration! Can you share some technical description of how this is working? (I see the source, but I mean some high level description about what you did.)

Depending on the bitrate, I wonder whether whether it could be possible to stream video via BT - that could be a legitimately useful (and uber cool) feature!

Great work

[TT-392]

Haha very cool demonstration! Can you share some technical description of how this is working? (I see the source, but I mean some high level description about what you did.)

Depending on the bitrate, I wonder whether whether it could be possible to stream video via BT - that could be a legitimately useful (and uber cool) feature!

Great work

I might put some more effort into a proper technical description on the github page at some point though not today.

But anyways, I'll give a quick description here:
The main magic happening here, is in the fact that this video is preprocessed beforehand on a powerful desktop pc. This thing first takes out only the pixels that actually change. And then tries to figure out the optimal blocks to write this to the display.

The way you write something to the display is: x1, y1, x2, y2, bitmap. I call these blocks.
Basically, writing a new block takes time, and writing pixels in a bitmap takes less time, so the bigger the blocks the better. So basically, if you have a block of 10x10 of changing pixels, with one non changing pixel in the middle, it is faster to overwrite that pixel, than to write 4 blocks around it. So that is basically what the preprocessor does. These blocks are then put in a file, with coordinates of the blocks (second relative can be encoded in 4 bits), followed by a 1 bit bitmap. This file is then lz4 compressed, and written to the spi flash of the pinetime. The pinetime decompresses, processes the blocks, and writes them to the display.


As for bluetooth streaming, I kinda doubt that would be doable, I haven't looked into bluetooth yet, but if the speed is anything like the time it takes to upload through nrf connect (uploading this video takes about half an hour). I doubt that is gonna happen. But idk, maybe some optimizations are possible.

[Mpoint]

You mention
`The main magic happening here, is in the fact that this video is preprocessed beforehand on a powerful desktop pc. This thing first takes out only the pixels that actually change. And then tries to figure out the optimal blocks to write this to the display.`

What Video preprocessing application are you using, Adobe After Effect ?
And the speed of the video appears smooth because of the powerful Hardware ? So what is the PC like ? i7 Intel or Alienware's that are used for gaming. 

Thanks for the Awesome video

[barray]

I might put some more effort into a proper technical description on the github page at some point though not today.

Please do!

The main magic happening here, is in the fact that this video is preprocessed beforehand on a powerful desktop pc. This thing first takes out only the pixels that actually change. And then tries to figure out the optimal blocks to write this to the display.

Haha I figured there was at least some magic

The way you write something to the display is: x1, y1, x2, y2, bitmap. I call these blocks.
Basically, writing a new block takes time, and writing pixels in a bitmap takes less time, so the bigger the blocks the better. So basically, if you have a block of 10x10 of changing pixels, with one non changing pixel in the middle, it is faster to overwrite that pixel, than to write 4 blocks around it. So that is basically what the preprocessor does. These blocks are then put in a file, with coordinates of the blocks (second relative can be encoded in 4 bits), followed by a 1 bit bitmap.

Is it that both the 'disk' (flash) is limited _and_ the write speed to the display?

Given the nature of the video, it could be worth checking out some vector graphics. The decision of what to send next to the display could possibly even be done real time.

This file is then lz4 compressed, and written to the spi flash of the pinetime. The pinetime decompresses, processes the blocks, and writes them to the display.

That sounds awesome Do you get much savings with lz4? I assume the data you are sending is already pretty high entropy?

As for bluetooth streaming, I kinda doubt that would be doable, I haven't looked into bluetooth yet, but if the speed is anything like the time it takes to upload through nrf connect (uploading this video takes about half an hour). I doubt that is gonna happen. But idk, maybe some optimizations are possible.

Depends how much data you are sending, but BT should be capable of some reasonable bitrate if you can stream audio over it. You'll definitely want some buffering though (as you do with audio).

The benefit to such a system is that you could potentially then stream all kinds of things from say a computer or mobile - it could lead to very interesting capabilities. For example:

* Remote photo taking on a camera
* Stream map data from phone
* Play a high-end graphics game streamed from your phone to watch (and use the watch as a dumb terminal of sorts) - games that could be cool are fruit ninja or angry birds for example

Anyway, if video could be streamed via BT it could add massive functionality!

[TT-392]

You mention
`The main magic happening here, is in the fact that this video is preprocessed beforehand on a powerful desktop pc. This thing first takes out only the pixels that actually change. And then tries to figure out the optimal blocks to write this to the display.`

What Video preprocessing application are you using, Adobe After Effect ?
And the speed of the video appears smooth because of the powerful Hardware ? So what is the PC like ? i7 Intel or Alienware's that are used for gaming. 

Thanks for the Awesome video

The preprocessing is not like you'd do in an application like adobe Adobe After Effect. It is just turned into an image sequence (done with ffmpeg, and some imagemagick) and then ran through what I call the preprocessor, which is just a custom application that I wrote to optimize the format of the video for display performance, and to fit it in the limited space of the watch, it is basically just processing the video into a more practical custom file format, specifically designed for the watch.

The video doesn't get smoother thanks to powerful hardware. I wasn't really clear about this in my comment, but when I mention a powerful pc, I mainly mean powerful compared to the watch, it is optimized on a desktop pc first, which is processor intensive, before it is flashed onto the watch, if this is done on a less powerful pc, the preprocessing is gonna take long, but it isn't gonna change anything about the smoothness of the final video. Also, because it took about 2.5 hours to do the preprocessing on a ryzen 7 1700, it might take really long to process the video on a less powerful machine. (the application does do multithreading, so it is not just singlethreaded performance that counts here)

[TT-392]

The way you write something to the display is: x1, y1, x2, y2, bitmap. I call these blocks.
Basically, writing a new block takes time, and writing pixels in a bitmap takes less time, so the bigger the blocks the better. So basically, if you have a block of 10x10 of changing pixels, with one non changing pixel in the middle, it is faster to overwrite that pixel, than to write 4 blocks around it. So that is basically what the preprocessor does. These blocks are then put in a file, with coordinates of the blocks (second relative can be encoded in 4 bits), followed by a 1 bit bitmap.

Is it that both the 'disk' (flash) is limited _and_ the write speed to the display?

Both the flash and the display share the same 8 MHz spi bus. This spi speed, and the fact that the only real way to send data to the display is in bitmaps (as in, you can't just tell it to draw a line or square, you have to send each pixel individually), in practice, this means that the fastest way to rewrite the entire screen is 125 ms (8 fps). We aren't really limited in terms of flash speed, because there is enough still frames, and frames where not much happens that enough data can be fetched to keep the buffer full.

barray

---

Given the nature of the video, it could be worth checking out some vector graphics. The decision of what to send next to the display could possibly even be done real time.

I don't really see how this would speed things up or would make things more compact, also, you could probably do the processing real time, but stuff would be a lot less optimized.

barray

This file is then lz4 compressed, and written to the spi flash of the pinetime. The pinetime decompresses, processes the blocks, and writes them to the display.

That sounds awesome  Do you get much savings with lz4? I assume the data you are sending is already pretty high entropy?

It goes from 5046208 bytes to 4099181. Which makes it small enough to fit in the 4 MiB of spi flash

As for bluetooth streaming, I kinda doubt that would be doable, I haven't looked into bluetooth yet, but if the speed is anything like the time it takes to upload through nrf connect (uploading this video takes about half an hour). I doubt that is gonna happen. But idk, maybe some optimizations are possible.

Depends how much data you are sending, but BT should be capable of some reasonable bitrate if you can stream audio over it. You'll definitely want some buffering though (as you do with audio).

The benefit to such a system is that you could potentially then stream all kinds of things from say a computer or mobile - it could lead to very interesting capabilities. For example:

* Remote photo taking on a camera
* Stream map data from phone
* Play a high-end graphics game streamed from your phone to watch (and use the watch as a dumb terminal of sorts) - games that could be cool are fruit ninja or angry birds for example

Anyway, if video could be streamed via BT it could add massive functionality!

Full color imagery, where small changes in color tint happen all over the display quite often (like remote photo taking on camera) is gonna be really slow on this display, this video is really only possible because big blobs of the same color pixels are moving around the display, though, still images of the camera, or, if a max speed of 8 fps, are probably possible (depending on bluetooth speed and stuff of course). For this same reason, games streamed from your phone are gonna look horrible, if they aren't specifically optimized for this display. Also, bluetooth as a protocol is definitely capable of high bitrates, but idk about this specific implementation of low power bluetooth hardware, I'd have to look into that.