Product Idea: USB Flash Drives

[dsimic]

I understand the point about GitHub, but it's also quite popular. GitLab is not nice to maintain for sure (as is most large web applications), but the one hosted at https://gitlab.com would be sufficient.

Quite frankly, I'd like the most a simple Git or Subversion repository hosted on our own, together with cgit or ViewVC, and accessed using everyone's favorite tools.  Optionally, there could be a read-only replica of the repository on GitHub.  By the time we get to writing actual code, I should be able to provide a hosted repository.

I don't think it replaces a human, but it for sure ensures that at least the simplest of the bugs can be caught early on. I would see several layers of testing anyway, regression testing (done on build), file transfer testing (a script that can be run) and manual testing (use the stick for a week with some code before declaring a release).

I agree.  Automated testing has its place and should be implemented, there's no doubt about it, but it cannot entrirely replace humans when it comes to testing a hot-plug hardware device and its firmware.

I did link it: https://wiki.pine64.org/index.php?title=...oldid=9632 but then it got yeeted from the front page: https://wiki.pine64.org/index.php?title=...oldid=9634

Apparently they want something more fledged out before adding it and there is no real consensus about exactly how it would be presented.

I'll see what I can do about editing my original comment to point to the wiki.

Yes, I saw your edit to the main wiki page and its later revert.  I agree that an "imaginary product" doesnt't have its place on the main wiki page in its current layout, but the PineFlash wiki page needs to be linked somewhere.  However, at the moment I'm really not sure how to tweak the layout of the main wiki page to achieve that.

The link to the PineFlash wiki page you've added to the inital post in this thread looks good to me.

I think we will cross that bridge when we come to it - I suspect manual testing will be good for quite a while. I imagine most of our bugs will come from more humble origins and will be more easily reproduced.

Absolutely, a lot of work and effort will need to go into putting together something that can actually be tested in any way.

Yeah, we would need something to disconnect all the lines and that doesn't unintentionally add capacitance (which at high speed is a large ask). Let's just not explore this manual disconnect unless we absolutely have to... Again, I think manual disconnect will be sufficient for the time being.

Totally agreed.  Quite frankly, when it comes to automating the disconnect, it would be actually easier to build a robotic arm that would physically plug the device in and out of a USB port instead of a human.

Any memory we are using we have control. If we want, we could fit the device with a beefy cap and continue running despite being powered off - the hardware is entirely in our control. And also there are filesystems that are robust to random disconnects, like ZFS for example. We won't have to re-invent the wheel on this one.

Umh, I'm not sure about that.  Even simple Micro SD cards are rather out of control when it comes to knowing that a write operation reported as successful has actually reached the underlying flash memory, which would make it possible to randomly cut the power or yank the card and lose no data.  Even rebooting the host can be problematic, because in many cases the reboot must include power cycling of the Micro SD card.  If the host uses the card at higher speeds and 1.8 V, the card must be power cycled to become initializable and usable by the boot loader after the reboot.

Letting "smart" flash do its job for a certain period of time (with an additional supercap or anything else) and hoping that data actually reaches the flash still doesn't provide the required level of data reliability.  It would be a hit-and-miss approach, which would only provide a false sense of reliability.  To have reliable storage, in our case we need underlying flash that is dumb, in the sense of being raw and having no integrated logic.  In the case of raw flash, having a write operation that is reported back as successful actually means reaching the flash with the written data, which is what we need.

Having ZFS as an example, one of its requirements to provide data reliability is to have nothing between the filesystem and the actual drives.  In other words, no hardware RAID controllers (although there are actually no hardware RAID controllers) may be used, only raw drives.  Sure, drives also provide write caching on their own, which is a potential issue, but they can also reliably be told to flush the caches or to comply to a write barrier.  It took years to flesh out all kinds of bugs related to the management of write caches on HDDs.