Product Idea: USB Flash Drives

[Arwen]

Yes, the USB protocol for the old "Bulk transfer / storage" method should probably be one of the first items.

I'd suggest a higher level language too, C or even Rust. And perhaps selecting an embedded OS, (open source of course), though Linux could be used, it's now pretty heavy weight for an embedded device.

[barray]

I think that the most important thing to tackle first is to do the usb protocoll and raid.

Yes, the USB protocol for the old "Bulk transfer / storage" method should probably be one of the first items.

Right, first step is to get it talking at least something via USB.

Believe it or not, RAID is the easy part in the gran scheme of things once you have basic communication going to speak to each device.

I'd suggest a higher level language too, C or even Rust. And perhaps selecting an embedded OS, (open source of course), though Linux could be used, it's now pretty heavy weight for an embedded device.

How low a level a language were you considering, assembly? We wouldn't get much portability down that path... C would likely be the way to go, simply because there is already software efforts in this area. It's already ambitious enough and reinventing the wheel adds too much risk.

Linux is completely out of the question on this level, you definitely need a small realtime OS. You don't want to get some kind of interrupt down at this level. And from a security perspective, I don't want somebody SSH'ing into my flash drive!

---

What I really need to kick this project off is to have some people committed to the development of it. It would also be great to hear from Pine64 as to whether they are interested in helping with the project. What I don't want is to be stuck completely alone developing this.

[Arwen]

My low level programming skills are decades out of date. And all I touch now days is shell scripting, (as a Sr. Unix SysAdmin). And my skills with schematics are also decades out of date. (I HAVE done programming on embedded boards, and had to have my engineer boos fix things on his boards... multiple times.)

The only thing I can add to the project, is specific suggestions on features and why they would be useful. Like UASP, (USB Attached SCSI Protocol), which will allow SMART, as well as up to 50% faster speeds.

Well, I guess I can afford to pay for a dozen or so development kits. Then send them out to various developers. Gee, kickstarter anyone?

[barray]

My low level programming skills are decades out of date. And all I touch now days is shell scripting, (as a Sr. Unix SysAdmin). And my skills with schematics are also decades out of date. (I HAVE done programming on embedded boards, and had to have my engineer boos fix things on his boards... multiple times.)

Eh, embedded systems are really one of the last places to really change all that much. If your skills were valid decades ago, they likely still hold up today. Besides, I would expect most of the work to be in debugging and testing.

The only thing I can add to the project, is specific suggestions on features and why they would be useful. Like UASP, (USB Attached SCSI Protocol), which will allow SMART, as well as up to 50% faster speeds.

I imagine we will also need testers for these and other features - so there is space even if you weren't to do coding for example. It's okay a developer saying 'works for me', but storage is hardly the sort of device you want to accidentally have fail on you!

Well, I guess I can afford to pay for a dozen or so development kits. Then send them out to various developers. Gee, kickstarter anyone?

I have no idea how much the initial boards would cost, my guestimate is that they will be sub $50 each (low-volume parts ordering and manufacturing). Ideally the whole thing would be sub $20 (not including shipping).

Kickstarter is great in theory, but there is this expectation that you are delivering a product and the project has some definite unmovable deadline. It's not something I would personally want to expose myself to.

[dsimic]

It would be awesome to have open-hardware flash-based storage devices.  The first device wouldn't need to be the fastest and greatest.

I would be willing to take part in this project, but the biggest thing, IMHO, would be to get Pine64 on board.  However, Pine64 seems to be ignoring questions and suggestions lately?  I've reached out to them more than a few times, but I got no response at all.

[barray]

It would be awesome to have open-hardware flash-based storage devices.  The first device wouldn't need to be the fastest and greatest.

Right, and I think adding the security element that @Julius_GU suggested with the fingerprint reader would make this a killer device with no real competition out there, even at lower speeds. The open source community are often quite privacy focused and at the right price point, I'm sure tonnes of people would love such a product.

I would be willing to take part in this project, but the biggest thing, IMHO, would be to get Pine64 on board.  However, Pine64 seems to be ignoring questions and suggestions lately?  I've reached out to them more than a few times, but I got no response at all.

Perhaps it is worth pinging somebody like @Luke ?

Yeah, I would be happy to build out a prototype - but there is absolutely no way I am in a position to take something into production. This would for sure need support from somebody like Pine64. I think I have laid out a relatively viable path in the above to at least start exploring such a device.

In terms of part availability - they are talking about 60 weeks until things start to calm down a bit, which we would likely need to get this project into a usable state anyway.

[dsimic]

Right, and I think adding the security element that @Julius_GU suggested with the fingerprint reader would make this a killer device with no real competition out there, even at lower speeds. The open source community are often quite privacy focused and at the right price point, I'm sure tonnes of people would love such a product.

Absolutely, we could implement numerous features.  Not necessarily in the first open-hardware device, but the key would be adding new features over time.

Perhaps it is worth pinging somebody like @Luke ?

I've tried reaching TL Lim and Luke multiple times, using pings in forum posts, private messages on the forum, and email messages.  Other forum members tried pinging them as well.  No response at all, which is very strange to me.

Yeah, I would be happy to build out a prototype - but there is absolutely no way I am in a position to take something into production. This would for sure need support from somebody like Pine64. I think I have laid out a relatively viable path in the above to at least start exploring such a device.

Even making a dozen of prototypes wouldn't be exactly trivial.  Sourcing various components in very small quantities is much more difficult and expensive than it may seem at first glance.  Adding flash chips into the BOM would only make things more difficult.

Edit: Please, note that I might be able to produce a very small batch of post-prototype devices, but not before the awful COVID-19 pandemic is over, which will hopefully be sooner than later.  Clearly, the emphasis is on "might".

Edit #2: The OpenSSD project looks very interesting and seems to provide useful resources.  Also, their presentation looks rather promising.  Though, the provided source code looks weird and rather useless.  Of course, it could very well be that I'm missing something.

Edit #3: The "pagemap.c" file from the OpenSSD codebase actually looks interesting, but it seemingly contains no "magic".

[barray]

Absolutely, we could implement numerous features.  Not necessarily in the first open-hardware device, but the key would be adding new features over time.

Right, but we could likely add them to the hardware in such a way that we are able to test them if we can get the other parts working correctly. But that's a minor detail anyway.

I've tried reaching TL Lim and Luke multiple times, using pings in forum posts, private messages on the forum, and email messages.  Other forum members tried pinging them as well.  No response at all, which is very strange to me.

Maybe they have given up trying to stay on top of communications whilst there is a parts shortage, I bet 90% of emails are people asking "when will X be back in stock?".

Even making a dozen of prototypes wouldn't be exactly trivial.  Sourcing various components in very small quantities is much more difficult and expensive than it may seem at first glance.  Adding flash chips into the BOM would only make things more difficult.

Well this is why I suggest using BL602 as a base as that would cover the majority of the components as we have a community created build system for the device too. We would just be creating a 'hat' for the device, which should reduce development risk. Regarding the flash chips, luckily the pinout tends to be the same, so we could hot-swap for other chips that happen to be available. As long as we refrain from anything chip specific we should be fine, but worst case we could just have hardware specific builds (something I guess we cannot avoid anyway).

Edit: Please, note that I might be able to produce a very small batch of post-prototype devices, but not before the awful COVID-19 pandemic is over, which will hopefully be sooner than later.  Clearly, the emphasis is on "might".

In any case, things should become much easier once this period of time passes!

Edit #2: The OpenSSD project looks very interesting and seems to provide useful resources.  Also, their presentation looks rather promising.  Though, the provided source code looks weird and rather useless.  Of course, it could very well be that I'm missing something.

The SSD part is still very much "in the future" for now... But I have seen the OpenSSD project before. From what I can tell it's mostly dead? From my understanding, their platform is more about developing a board they can experiment with for research - rather than ever aiming towards anything that would end up in the hands of consumers.

Edit #3: The "pagemap.c" file from the OpenSSD codebase actually looks interesting, but it seemingly contains no "magic".

This would be the part I would be most confident with - the part I wouldn't be is the actual communications protocol they use. NVMe for example is quite complex. An open source implementation of this stuff would be ridiculously valuable.

[dsimic]

Maybe they have given up trying to stay on top of communications whilst there is a parts shortage, I bet 90% of emails are people asking "when will X be back in stock?".

It could be the reason.  Actually, I really hope you're right.  However, their lack of communication with the community is bad for everyone, and nobody knows when the pandemic will be over so the manufacturing and parts availability could start returning to normal.  Thus, simply ignoring the messages isn't a very good solution.

Well this is why I suggest using BL602 as a base as that would cover the majority of the components as we have a community created build system for the device too. We would just be creating a 'hat' for the device, which should reduce development risk. Regarding the flash chips, luckily the pinout tends to be the same, so we could hot-swap for other chips that happen to be available. As long as we refrain from anything chip specific we should be fine, but worst case we could just have hardware specific builds (something I guess we cannot avoid anyway).

I'm not sure that the BL602 (block diagram) has a NAND flash interface?  The BL602 specification states that it contains an Intel IP core for the SPI flash interface, intended for storing firmware (hence the XIP/execute-in-place feature), which would be too slow for a storage device.  However, I think that it shouldn't be a problem to select an appropriate low-cost microcontroller with an ARM Cortex-M0 core, a NAND flash interface, and a USB interface.

Another possibility for the partial source of parts could be to use an appropriate commercially available USB flash drive to harvest a couple of key parts, the flash IC and the case.  Sourcing those two parts in very low quantities would drive their prices very high.

The SSD part is still very much "in the future" for now... But I have seen the OpenSSD project before. From what I can tell it's mostly dead? From my understanding, their platform is more about developing a board they can experiment with for research - rather than ever aiming towards anything that would end up in the hands of consumers.

Exactly, the goal of OpenSSD is to provide a platform for research, development and education, with no intentions of creating an actual product for end users.

[barray]

It could be the reason.  Actually, I really hope you're right.  However, their lack of communication with the community is bad for everyone, and nobody knows when the pandemic will be over so the manufacturing and parts availability could start returning to normal.  Thus, simply ignoring the messages isn't a very good solution.

Well, if people are asking the same question over and over, it seems like it's a good time to invest some time to automate the handling of such queries. I think they now have a Telegram chat or something that advises when products are about to drop, but I guess they really need something else too.

I'm not sure that the BL602 (block diagram) has a NAND flash interface?  The BL602 specification states that it contains an Intel IP core for the SPI flash interface, intended for storing firmware (hence the XIP/execute-in-place feature), which would be too slow for a storage device.  However, I think that it shouldn't be a problem to select an appropriate low-cost microcontroller with an ARM Cortex-M0 core, a NAND flash interface, and a USB interface.

Well I was initially looking at using NAND SPI flash with quad SPI: https://nz.mouser.com/datasheet/2/949/w2...608377.pdf

That gives you up to 50MB/s and you could then pick up some extra speed with the RAID format. I suspect the latency between read request and reply to be constant for example (and most of these devices support artificial delay in the worst case), so you would just need to be smart about chip select and could multiplex whilst each chip is busy. If you were doing mirroring for example, I wonder if you could even chip select two identical flash units at the same time.

These are the sorts of cool things that an open source device allow us to look into anyway!

Another possibility for the partial source of parts could be to use an appropriate commercially available USB flash drive to harvest a couple of key parts, the flash IC and the case.  Sourcing those two parts in very low quantities would drive their prices very high.

I would try to stay away from this, I imagine this will just end up being a headache in the long run. There's no reason why a flash drive company might not run several revisions based on what parts are cheapest at the time, so components may not even match up even if we buy the same device.

Exactly, the goal of OpenSSD is to provide a platform for research, development and education, with no intentions of creating an actual product for end users.

Right, and it's the project that most people point towards when you suggest 'open source storage'. This is why I think this project is so important - because there really isn't anything out there being actively worked on.