Product Idea: USB Flash Drives

[dsimic]

Given the current trend of things, I have a pessimistic outlook. On the plus side, hopefully it gives the world an opportunity to reorganize their supply chains so that they rely less on shipping stuff all around the world - which is not particularly sustainable.

I agree.  Earning more money is good, but not at the cost of destroying things, environment, nature, jobs, or people's health and lives.  Unfortunately, many aspects of humanity seem to have become pushed to the side or even forgotten in the last couple of decades.

It'll take a while to set something up anyway... I really want to be able to group discussions so that they make more logical sense, I think so far we have the greater subjects of 'software', 'hardware' and 'production'. Then for each of those is several important discussions that need to happen. Long threads tend to result in lost information in the long run.

If Pine64 eventually remains silent, we should move the entire discussion elsewhere.  I would suggest using some of the existing wiki implementations (preferably one written in PHP) instead of setting up a forum, because the wikis provide good page talks and very good revisioning of pretty much everything.  The usual wiki flow is to discuss on a talk page and then update the main page, which is usually very effective: condensed results of talks end up on main pages for future reference.

Moreover, splitting the discussion into multiple threads right now might actually be counterproductive, because that could be good for you and me, but would probably make the whole thing much harder to grasp for someone just having a first look at the entire idea/project.

I hope so. The other option is that we post updates back to this forum to incorporate feedback from the community. People post about significant software updates for example, I don't think it's unreasonable as long as we don't spam and the device still relies on the BL602.

Good idea.  After moving the discussion somewhere else, we should definitely provide periodic updates here, of course provided that the first storage device is a "hat" for one of the BL602 boards made by Pine64.

It's now been almost a month of hoping for some feedback. I'm not going to hold my breath - I have no idea what we have to do to get some attention.

Let's keep the hope present for a while, but expect nothing.  More than enough has been done to attract their attention, so the ball is in their court now.

[barray]

I agree.  Earning more money is good, but not at the cost of destroying things, environment, nature, jobs, or people's health and lives.  Unfortunately, many aspects of humanity seem to have become pushed to the side or even forgotten in the last couple of decades.

Seems like that ship got unblocked.

If Pine64 eventually remains silent, we should move the entire discussion elsewhere.  I would suggest using some of the existing wiki implementations (preferably one written in PHP) instead of setting up a forum, because the wikis provide good page talks and very good revisioning of pretty much everything.  The usual wiki flow is to discuss on a talk page and then update the main page, which is usually very effective: condensed results of talks end up on main pages for future reference.

If we are to go down that road, then it's probably better to use a Wiki associated with something like GitHub/GitLab (https://docs.gitlab.com/ee/user/project/wiki/). We could then even get a webpage out of it (https://docs.gitlab.com/ee/user/project/pages/).

Moreover, splitting the discussion into multiple threads right now might actually be counterproductive, because that could be good for you and me, but would probably make the whole thing much harder to grasp for someone just having a first look at the entire idea/project.

I think it would make sense to start consolidating the ideas here anyway into a more presentable format - who is going to read through so many comments just to find out what the basic idea is?

[xalius]

Have you talked about the project idea in chat, Pine community chat is usually pretty good at getting everyone's attention?

[Arwen]

One thing I saw on another software project, was automated testing. Anytime a feature was changed or added, a test routine was changed or added to specifically test that change or addition.

Perhaps something along that lines should be implemented.

Start with simpler firmware without encryption or UAS / UASP, (USB attached SCSI protocol), and make the test routines verify those features. Then move on to more important features.

[barray]

Have you talked about the project idea in chat, Pine community chat is usually pretty good at getting everyone's attention?

No, but I did and you were in the IRC chat.

@dsimic We have been suggested to make a wiki page, hence: https://wiki.pine64.org/wiki/PineFlash We are still discussing (as I write) how such a thing can be linked to, making it clear it is community driven at this stage. We should use this space to consolidate the ideas from these discussions.

One thing I saw on another software project, was automated testing. Anytime a feature was changed or added, a test routine was changed or added to specifically test that change or addition.

For sure we will need tonnes of testing - losing data is unacceptable and we should guard against that in every way possible. Being open source we can make this effort very clear.

Start with simpler firmware without encryption or UAS / UASP, (USB attached SCSI protocol), and make the test routines verify those features. Then move on to more important features.

That would be the intention - to develop this incrementally

[barray]

@Arwen @Julius_GU I added your suggestions under 'Suggest Features': https://wiki.pine64.org/wiki/PineFlash I've tried to condense them down to their core ideas. I may have mis-attributed or missed some stuff, so please take a read.

The wiki is of course just a first draft, feel free to add some stuff. I feel a lot better now that we are able to document some of the awesome discussions happening in here.

[dsimic]

If we are to go down that road, then it's probably better to use a Wiki associated with something like GitHub/GitLab (https://docs.gitlab.com/ee/user/project/wiki/). We could then even get a webpage out of it (https://docs.gitlab.com/ee/user/project/pages/).

Gah, for some reason I don't like GitHub...  IMHO, it's vastly overhyped, and people just don't think too much and use Git for projects that would actually benefit from using Subversion.  Also, GitLab is written in Ruby, which just plain s*cks when it comes to the installation and later maintenance.

One thing I saw on another software project, was automated testing. Anytime a feature was changed or added, a test routine was changed or added to specifically test that change or addition. Perhaps something along that lines should be implemented.

Start with simpler firmware without encryption or UAS / UASP, (USB attached SCSI protocol), and make the test routines verify those features. Then move on to more important features.

Automated testing, unit tests or however people call it are good, but the trouble with hardware projects is that fully automating the testing is usually not possible.  For example, a lot of trouble with proper operation of a USB device comes from its plugging and unplugging into the host device, which would be really complicated to be automated, if possible at all.

However, automated testing could be implemented to a certain degree.

We have been suggested to make a wiki page, hence: https://wiki.pine64.org/wiki/PineFlash We are still discussing (as I write) how such a thing can be linked to, making it clear it is community driven at this stage. We should use this space to consolidate the ideas from these discussions.

This is a good solution for now, but the new PineFlash wiki page should be linked from somewhere.  I'd suggest that we link it in the first post in this thread, until it's decided where to link it from within the wiki.

Also, I took some time to clean up the PineFlash page a bit, and to improve the wording and layout of the main wiki page.  The main page looks a bit more professional now, if you agree.

[Arwen]

In regards to testing, @dsimic you have a good point about insertion. However, perhaps that can be simulated by having the host port's USB power turned off. Something like this;

- Start with USB port power off
- Plug in USB storage device
- Initiate automated testing, which turns power on

During various steps, the testing can disable power, to power cycle / reboot the USB storage device. Whence all those tests pass, perhaps having another set of partially automated testing with interactive USB storage device insertions and removals.

Even if host power can't be removed, (and we can probably create a special test jig that can do so, if we can't find one already available), using interactive tests with the user performing the inserts and removals will still be quite valuable.

In fact, one of my original suggestions is copy on write. Meaning that the internal directory structure remains consistent, even through unexpected power cycles. So with automated power cycle support, we can test that function by writing to the USB flash drive and disabling power at various intervals. Then on power up, make sure that preexisting data is still available.

[dsimic]

Things may sometimes seem deceptively simple.  Please allow me to explain.

One of the problems is that using software to cut the power from a plugged-in USB device is actually not possible on regular USB ports.  The available power-related knobs in /sys actually only tell the plugged-in USB devices to go into suspended state, and it's up to the particular USB device to do (or ignore) the requested suspend operation.  In other words, no power cuts are actually performed by the host.

Maybe it's possible to cut power on USB ports that are present on some SBCs, which I'm not sure about at the moment, but the lack of that feature on regular (i.e. found on PC motherboards) USB ports turns the whole thing into being impossible to automate.  We must not rely on testing using only the USB ports on a few select SBCs.

Moreover, it's not just about cutting the power.  The four USB 2.0 strips inside the USB Type-A plug are intentionally not of the same lenght; the two strips that carry power are longer than the strips that carry data, so the device has some time to wake up before the data lines are connected upon plugging a USB device into a USB receptacle.  In the scenario of cutting and re-establishing power to a USB device, there would be no such delay, which would take the whole setup out of the USB specification and make the whole automated setup rather pointless.

It's even worse with USB 3.0; sometimes, USB 3.0 devices are seen as "ghost" USB 2.0 devices upon unplugging them from a USB 3.0 host receptacle.  All that due to the different positions and lengths of the USB 2.0 and 3.0 pins inside the connectors.

Edit: By the way, flash memory is inherently of a COW nature.  However, the only way we could guarantee data consistency in presence of sudden power cuts is to use raw flash, with no built-in wear leveling, and do everyhing in the device firmware.  If the flash does anything on its own, using some sort of a built-in microcontroller, all bets are off and no data consistency can be guaranteed.

Edit #2: I stand corrected. It is actually possible to use software to cut power on regular USB ports, with certain rules and restrictions as described in detail here, but the note on having no delay between connecting power and data lines still remains.

[barray]

Gah, for some reason I don't like GitHub...  IMHO, it's vastly overhyped, and people just don't think too much and use Git for projects that would actually benefit from using Subversion.  Also, GitLab is written in Ruby, which just plain s*cks when it comes to the installation and later maintenance.

I understand the point about GitHub, but it's also quite popular. GitLab is not nice to maintain for sure (as is most large web applications), but the one hosted at https://gitlab.com would be sufficient.

Automated testing, unit tests or however people call it are good, but the trouble with hardware projects is that fully automating the testing is usually not possible.  For example, a lot of trouble with proper operation of a USB device comes from its plugging and unplugging into the host device, which would be really complicated to be automated, if possible at all.

However, automated testing could be implemented to a certain degree.

I don't think it replaces a human, but it for sure ensures that at least the simplest of the bugs can be caught early on. I would see several layers of testing anyway, regression testing (done on build), file transfer testing (a script that can be run) and manual testing (use the stick for a week with some code before declaring a release).

This is a good solution for now, but the new PineFlash wiki page should be linked from somewhere.  I'd suggest that we link it in the first post in this thread, until it's decided where to link it from within the wiki.

I did link it: https://wiki.pine64.org/index.php?title=...oldid=9632 but then it got yeeted from the front page: https://wiki.pine64.org/index.php?title=...oldid=9634

Apparently they want something more fledged out before adding it and there is no real consensus about exactly how it would be presented.

I'll see what I can do about editing my original comment to point to the wiki.

Also, I took some time to clean up the PineFlash page a bit, and to improve the wording and layout of the main wiki page.  The main page looks a bit more professional now, if you agree.

I checked it out earlier, seems mostly good.

In regards to testing, @dsimic you have a good point about insertion. However, perhaps that can be simulated by having the host port's USB power turned off. Something like this;

- Start with USB port power off
- Plug in USB storage device
- Initiate automated testing, which turns power on

During various steps, the testing can disable power, to power cycle / reboot the USB storage device. Whence all those tests pass, perhaps having another set of partially automated testing with interactive USB storage device insertions and removals.

Even if host power can't be removed, (and we can probably create a special test jig that can do so, if we can't find one already available), using interactive tests with the user performing the inserts and removals will still be quite valuable.

In fact, one of my original suggestions is copy on write. Meaning that the internal directory structure remains consistent, even through unexpected power cycles. So with automated power cycle support, we can test that function by writing to the USB flash drive and disabling power at various intervals. Then on power up, make sure that preexisting data is still available.

I think we will cross that bridge when we come to it - I suspect manual testing will be good for quite a while. I imagine most of our bugs will come from more humble origins and will be more easily reproduced.

One of the problems is that using software to cut the power from a plugged-in USB device is actually not possible on regular USB ports.  The available power-related knobs in /sys actually only tell the plugged-in USB devices to go into suspended state, and it's up to the particular USB device to do (or ignore) the requested suspend operation.  In other words, no power cuts are actually performed by the host.

Maybe it's possible to cut power on USB ports that are present on some SBCs, which I'm not sure about at the moment, but the lack of that feature on regular (i.e. found on PC motherboards) USB ports turns the whole thing into being impossible to automate. We must not rely on testing using only the USB ports on a few select SBCs.

Edit #2: I stand corrected. It is actually possible to use software to cut power on regular USB ports, with certain rules and restrictions as described in detail here, but the note on having no delay between connecting power and data lines still remains.

I think it is possible as you say, but not on absolutely every host and turning the USB port power back on can be a pain... Some for example only trigger the power when the resistance between D+ and D- is detected and require a disconnect followed by a reconnect in order to reset this condition.

Moreover, it's not just about cutting the power.  The four USB 2.0 strips inside the USB Type-A plug are intentionally not of the same lenght; the two strips that carry power are longer than the strips that carry data, so the device has some time to wake up before the data lines are connected upon plugging a USB device into a USB receptacle.  In the scenario of cutting and re-establishing power to a USB device, there would be no such delay, which would take the whole setup out of the USB specification and make the whole automated setup rather pointless.

Yeah, we would need something to disconnect all the lines and that doesn't unintentionally add capacitance (which at high speed is a large ask). Let's just not explore this manual disconnect unless we absolutely have to... Again, I think manual disconnect will be sufficient for the time being.

It's even worse with USB 3.0; sometimes, USB 3.0 devices are seen as "ghost" USB 2.0 devices upon unplugging them from a USB 3.0 host receptacle.  All that due to the different positions and lengths of the USB 2.0 and 3.0 pins inside the connectors.

Ouch, wasn't aware of this.

Edit: By the way, flash memory is inherently of a COW nature.  However, the only way we could guarantee data consistency in presence of sudden power cuts is to use raw flash, with no built-in wear leveling, and do everyhing in the device firmware.  If the flash does anything on its own, using some sort of a built-in microcontroller, all bets are off and no data consistency can be guaranteed.

Any memory we are using we have control. If we want, we could fit the device with a beefy cap and continue running despite being powered off - the hardware is entirely in our control. And also there are filesystems that are robust to random disconnects, like ZFS for example. We won't have to re-invent the wheel on this one.