03-03-2021, 05:57 AM
(This post was last modified: 04-01-2021, 05:33 PM by barray.
Edit Reason: Add link to wiki
)
Hi Pine64'ers 
Update: There is now a wiki where ideas from this thread are consolidated: https://wiki.pine64.org/wiki/PineFlash
I had a crazy random idea, but I think it is worth exploring. Open to some feedback.
I propose Pine64 looks into creating custom open-source flash drives. I think these are devices we all use regularly and I think there is some great opportunity to improve on them!
# Why?
Currently, flash drives are almost exclusively closed-source. This poses a few major problems:
1. Security: You have zero idea what is running on the drive. It would be exceptionally easy for example to change a file at will. Imagine this happening to an SSH key for example, which is quite easily recognizable.
2. Corruption: We've all been in this scenario where a flash drive suddenly fails without warning.
3. Encryption: Some drives offer "encryption", but this rarely includes the memory itself and if it does, it's all done client side. And then you have to trust this is actually robust and trust the implementation.
4. Planned Obsolescence: In theory a drive could purposely give out after so many writes or reads, forcing you to go out and buy another.
# Opportunities
I think there are tonnes of opportunities to improve on existing designs:
1. RAID: If you were to use several lower-capacity flash chips (which could also end up cheaper), the community could explore RAID to increase either speed, redundancy or both! This would need a minimum of two chips, ideally four.
2. Cache: It could be possible to do some intelligent caching - for example if you know a particular format is being used, you could cache the file table.
3. Encryption: Of course the big one could be some form of encryption. This could make this an attractive option for people who travel or want some protection! You may even want the drive to display something entirely different until some password is provided to reveal the real contents.
One way this could be done is to have it first present a small blank file system, where either the user or some script copies a key (possibly in response to some content already there) to the drive. The drive could then unmount and remount as the full drive and use this to decrypt the rest of the contents.
4. Wear Leveling: Just because companies claim to have really good wear-leveling, doesn't mean they actually do.
5. Compression: It might be possible to compress certain types of data. There are a few algorithms, such as LZ4 that don't really add too much overhead if they are unable to compress some data, but could yield large returns on compress-able data.
6. SMART: The drive could provide a SMART table to provide information about disk wear (the same thing SSDs do). This could even be calibrated to temperature and time for example.
7. Type A & Type C USB: Type C is likely the future, but there is also tonnes of devices still using type A. It would be cool to have a device that is robust for both.
8. Casing: A nice little piece of positive PR with the Pine64 logo
# Expectations
I imagine the first version will realistically only support USB 2 speeds at most, possibly USB 1, but I expect this community to be capable of getting up to USB 3 speeds after a few iterations.
There are some flash chips and USB controllers that already do most of the work, so this could greatly reduce the risk of v1 not working at all and to work on one problem at a time.
# Long-term
I suggest this could lay some ground-work for a custom M.2. drive, something some current Pine64 devices support and something I think future Pine64 devices will make more use of.
Open to ideas!
Update: There is now a wiki where ideas from this thread are consolidated: https://wiki.pine64.org/wiki/PineFlash
I had a crazy random idea, but I think it is worth exploring. Open to some feedback.
I propose Pine64 looks into creating custom open-source flash drives. I think these are devices we all use regularly and I think there is some great opportunity to improve on them!
# Why?
Currently, flash drives are almost exclusively closed-source. This poses a few major problems:
1. Security: You have zero idea what is running on the drive. It would be exceptionally easy for example to change a file at will. Imagine this happening to an SSH key for example, which is quite easily recognizable.
2. Corruption: We've all been in this scenario where a flash drive suddenly fails without warning.
3. Encryption: Some drives offer "encryption", but this rarely includes the memory itself and if it does, it's all done client side. And then you have to trust this is actually robust and trust the implementation.
4. Planned Obsolescence: In theory a drive could purposely give out after so many writes or reads, forcing you to go out and buy another.
# Opportunities
I think there are tonnes of opportunities to improve on existing designs:
1. RAID: If you were to use several lower-capacity flash chips (which could also end up cheaper), the community could explore RAID to increase either speed, redundancy or both! This would need a minimum of two chips, ideally four.
2. Cache: It could be possible to do some intelligent caching - for example if you know a particular format is being used, you could cache the file table.
3. Encryption: Of course the big one could be some form of encryption. This could make this an attractive option for people who travel or want some protection! You may even want the drive to display something entirely different until some password is provided to reveal the real contents.
One way this could be done is to have it first present a small blank file system, where either the user or some script copies a key (possibly in response to some content already there) to the drive. The drive could then unmount and remount as the full drive and use this to decrypt the rest of the contents.
4. Wear Leveling: Just because companies claim to have really good wear-leveling, doesn't mean they actually do.
5. Compression: It might be possible to compress certain types of data. There are a few algorithms, such as LZ4 that don't really add too much overhead if they are unable to compress some data, but could yield large returns on compress-able data.
6. SMART: The drive could provide a SMART table to provide information about disk wear (the same thing SSDs do). This could even be calibrated to temperature and time for example.
7. Type A & Type C USB: Type C is likely the future, but there is also tonnes of devices still using type A. It would be cool to have a device that is robust for both.
8. Casing: A nice little piece of positive PR with the Pine64 logo
# Expectations
I imagine the first version will realistically only support USB 2 speeds at most, possibly USB 1, but I expect this community to be capable of getting up to USB 3 speeds after a few iterations.
There are some flash chips and USB controllers that already do most of the work, so this could greatly reduce the risk of v1 not working at all and to work on one problem at a time.
# Long-term
I suggest this could lay some ground-work for a custom M.2. drive, something some current Pine64 devices support and something I think future Pine64 devices will make more use of.
Open to ideas!
