Product Idea: USB Flash Drives

[barray]

Gah, for some reason I don't like GitHub...  IMHO, it's vastly overhyped, and people just don't think too much and use Git for projects that would actually benefit from using Subversion.  Also, GitLab is written in Ruby, which just plain s*cks when it comes to the installation and later maintenance.

I understand the point about GitHub, but it's also quite popular. GitLab is not nice to maintain for sure (as is most large web applications), but the one hosted at https://gitlab.com would be sufficient.

Automated testing, unit tests or however people call it are good, but the trouble with hardware projects is that fully automating the testing is usually not possible.  For example, a lot of trouble with proper operation of a USB device comes from its plugging and unplugging into the host device, which would be really complicated to be automated, if possible at all.

However, automated testing could be implemented to a certain degree.

I don't think it replaces a human, but it for sure ensures that at least the simplest of the bugs can be caught early on. I would see several layers of testing anyway, regression testing (done on build), file transfer testing (a script that can be run) and manual testing (use the stick for a week with some code before declaring a release).

This is a good solution for now, but the new PineFlash wiki page should be linked from somewhere.  I'd suggest that we link it in the first post in this thread, until it's decided where to link it from within the wiki.

I did link it: https://wiki.pine64.org/index.php?title=...oldid=9632 but then it got yeeted from the front page: https://wiki.pine64.org/index.php?title=...oldid=9634

Apparently they want something more fledged out before adding it and there is no real consensus about exactly how it would be presented.

I'll see what I can do about editing my original comment to point to the wiki.

Also, I took some time to clean up the PineFlash page a bit, and to improve the wording and layout of the main wiki page.  The main page looks a bit more professional now, if you agree.

I checked it out earlier, seems mostly good.

In regards to testing, @dsimic you have a good point about insertion. However, perhaps that can be simulated by having the host port's USB power turned off. Something like this;

- Start with USB port power off
- Plug in USB storage device
- Initiate automated testing, which turns power on

During various steps, the testing can disable power, to power cycle / reboot the USB storage device. Whence all those tests pass, perhaps having another set of partially automated testing with interactive USB storage device insertions and removals.

Even if host power can't be removed, (and we can probably create a special test jig that can do so, if we can't find one already available), using interactive tests with the user performing the inserts and removals will still be quite valuable.

In fact, one of my original suggestions is copy on write. Meaning that the internal directory structure remains consistent, even through unexpected power cycles. So with automated power cycle support, we can test that function by writing to the USB flash drive and disabling power at various intervals. Then on power up, make sure that preexisting data is still available.

I think we will cross that bridge when we come to it - I suspect manual testing will be good for quite a while. I imagine most of our bugs will come from more humble origins and will be more easily reproduced.

One of the problems is that using software to cut the power from a plugged-in USB device is actually not possible on regular USB ports.  The available power-related knobs in /sys actually only tell the plugged-in USB devices to go into suspended state, and it's up to the particular USB device to do (or ignore) the requested suspend operation.  In other words, no power cuts are actually performed by the host.

Maybe it's possible to cut power on USB ports that are present on some SBCs, which I'm not sure about at the moment, but the lack of that feature on regular (i.e. found on PC motherboards) USB ports turns the whole thing into being impossible to automate. We must not rely on testing using only the USB ports on a few select SBCs.

Edit #2: I stand corrected. It is actually possible to use software to cut power on regular USB ports, with certain rules and restrictions as described in detail here, but the note on having no delay between connecting power and data lines still remains.

I think it is possible as you say, but not on absolutely every host and turning the USB port power back on can be a pain... Some for example only trigger the power when the resistance between D+ and D- is detected and require a disconnect followed by a reconnect in order to reset this condition.

Moreover, it's not just about cutting the power.  The four USB 2.0 strips inside the USB Type-A plug are intentionally not of the same lenght; the two strips that carry power are longer than the strips that carry data, so the device has some time to wake up before the data lines are connected upon plugging a USB device into a USB receptacle.  In the scenario of cutting and re-establishing power to a USB device, there would be no such delay, which would take the whole setup out of the USB specification and make the whole automated setup rather pointless.

Yeah, we would need something to disconnect all the lines and that doesn't unintentionally add capacitance (which at high speed is a large ask). Let's just not explore this manual disconnect unless we absolutely have to... Again, I think manual disconnect will be sufficient for the time being.

It's even worse with USB 3.0; sometimes, USB 3.0 devices are seen as "ghost" USB 2.0 devices upon unplugging them from a USB 3.0 host receptacle.  All that due to the different positions and lengths of the USB 2.0 and 3.0 pins inside the connectors.

Ouch, wasn't aware of this.

Edit: By the way, flash memory is inherently of a COW nature.  However, the only way we could guarantee data consistency in presence of sudden power cuts is to use raw flash, with no built-in wear leveling, and do everyhing in the device firmware.  If the flash does anything on its own, using some sort of a built-in microcontroller, all bets are off and no data consistency can be guaranteed.

Any memory we are using we have control. If we want, we could fit the device with a beefy cap and continue running despite being powered off - the hardware is entirely in our control. And also there are filesystems that are robust to random disconnects, like ZFS for example. We won't have to re-invent the wheel on this one.