Product Idea: USB Flash Drives

[barray]

Too high load on the server would actually be a good problem to have.  I wouldn't worry too much about it for now, though.

I did manage to get cgit up and running, was a little bit of a pain. Seems like it's not quite yet mature for small servers hosting large repositories, but should be good enough for our purpose.

That's an accurate description of the issue at hand.  Maybe we could put the link into the "Community and Support" section on the main wiki page, as an example of the community collaboration and effort?  That might even attract the attention of Pine64.

I was essentially told that anything put on the main page would be removed. I think we would need to ask before adding something, otherwise some wiki admin will just revert it again.

Meh, not even doing a repeated read from "smart" flash would guarantee anything, because the embedded flash controller could simply return the buffered data that is still in transit to the underlying flash memory. 

Some industrial full-size SD cards mitigate the issue by including supercaps that provide the emergency power required to actually complete all write operations that were acknowledged as successful, which is the same approach as in enterprise SSDs.  However, there's simply no space for such supercaps in Micro SD cards or SPI flash chips, which makes them inherently unreliable.

I believe that between the filesystem and the flash control, there should be something that can be done in this space. But I guess this can also be part of the experimentation anyway.

The leakage you mentioned actually belongs to a whole other category of issues, usually called bit rot or silent data corruption.  It is actually unrelated to the acknowledged writes that end up not reaching the underlying flash memory.  I'll try to explain this further.

There are three "players" in this game: the flash memory, the embedded flash controller, and the host.  When the flash "lies" to the controller, by experiencing the effects of cosmic rays or a defective cell, it is the silent data corruption.  In this case, the flash actually doesn't "lie", it's instead simply unaware of the changes to the data.  When the controller "lies" to the host, as a result of performing the wear leveling or as just an attempt to improve performance, it is the write that hasn't actually reached the flash memory.  In this case, the controller really "lies", by being fully aware of the data not reaching the underlying flash.

Eliminating the controller (a true "liar") from the above-described equation leaves the host to deal with low-level flash issues, but eliminates the possibility of having incomplete writes treated as complete.  Of course, the possibility for silent data corruption still remains, as a constraint of the flash memory technology, which can be mitigated using additional ECC data.

I'll have to look at this further. By the way, the chips linked in the wiki-page are automotive grade, so they have ECC in them too.

It would be fun, but receiving a copyright infrigement notice would be much less fun.

Inspired of course - not a direct clone. Also the PCB would be entirely hidden in the final casing for the device.

I've added SMART to the Wiki's suggested features, (with a probable pre-req of UASP).

I saw, looks good

In regards to the copy on write, what I meant was that the flash is not used as simple addressable blocks. Since we have wear leveling, and spares, we have to have an internal "directory" which when I ask for from block 357 it gives me logical block 357, and not the physical block 357 which might be for something else.

My thought is to make sure that this internal "directory" of logical blocks from the host, to physical blocks in flash, is kept consistent. Even if we have to re-write the "directory", (with version number so we know which is more current), into another place. So that the old "directory" is good, UNTIL the new "directory" is completely written.

Perhaps taking a feature from ZFS and having 2 copies of the "directory" at all times may help avoid corruption.

Note: All metadata on ZFS has at least 2 copies, even on a single disk.

I think I get what you mean - but this is wear leveling, not a filesystem?

Last, many software & hardware development projects include test jigs. We may want to create a USB test jig that will remove power. Perhaps by relay to avoid the capacitance issue. Or perhaps use an existing USB controlled relay with a special USB to USB adapter that cuts power.

Here are a few USB controlled relays I found cheaply;
https://www.amazon.com/NOYITO-1-Channel-...B07C3LPH3X
https://www.amazon.com/SMAKN®-LCUS-1-mod...B01CN7E0RQ

That first one has additional models that have 2 or 4 which could in theory cut the other lines too. (Though, I don't know about capacitance issues...)

Those seem to be USB controlled relays, not relays to control USB? The problem with having one of those on each of the lines is the capacitance and resistance they add. Honestly the complexity overhead is not worth it - designing this test rig could end up as large a task as the USB device itself.