I think this is really an attempt at user feedback, rather than user discussion. But there's no such thing as a user-feedback mail list. Nevertheless, others may find this pertinent: why doesn't mke2fs handle USB's competently? And if it does, why doesn't it reassure me so? And how can I handle a linux-formatted USB flash drive in the absence of my system giving me any guidance? I cannot accept the usual advice you see in the forums: simply format it FAT16 or whatever. I need it to use a linux file system. But linux may have problems, apparently, with the device: http://lwn.net/Articles/428584/ Plenty of people say in the forums that ext3 and ext4 will wear your USB out quickly, effectively thrashing it with their journaling. But the above-linked article suggests it may depend on the device. Some people say ext4 is simply okay, some that it is actually advisable to use it. I'm none the wiser. If it is true that ext4 will thrash your USB then it could important if it is also true that, for example, the life-expectancy of a USB can really as low as 10,000 writes: http://linux.koolsolutions.com/2009/01/29/installing-linux-on-usb-part-3-which-linux-filesystem-for-usb-devices/. I spoke to Integral Memory, the manufacturer of my USB drive. Their technical support line said their device had a life expectancy of 1000 writes, and came with a two-year warranty. He didn't know what he was talking about. None of Integral's publically available spec-sheets tell you anything useful. They give an approximate write-speed and that's about it. Nothing about block size, whether optimized for linear write or not, what method of wear distribution they use - all the things you are led to believe may be important in deciding whether you should format your USB ext4 or not. Of course, I shouldn't have to know about any of these things as a user anyway. But I'm left with no choice but to lower myself into this arcane realm and get in a tangle. One of the things I'm led to believe I must consider when formatting a USB drive is its physical block size. This appears to be something of a mystery, but someone reckons he has worked out a simple method of discerning it: http://kim.oyhus.no/FlashBlockSize.html. Using that method, and assuming Kim G. S. ?yhus is correct, my drive seems to have 4k block size. Now perhaps if I format my drive with a filesystem that uses a 4k block size I might stop it being mashed by ext4. Perhaps. It does seem strange to me though that mke2fs doesn't already do this for me: discerns what's going on underneath the casing of my USB, adjust its formatting parameters accordingly - and tell me what its doing so I can be reassured. Or that the help pages at least tell me definitively whether and in what circumstances it can be used with USB so I don't have to waste so much time trying to unravel the mystery by going like a pinball from one contradictory, incomplete or well-meaning nincompoop forum message to another; and so I don't mash my USB, and lose my data, because I should have listened to the pessimists and not put an ext4 filesystem on my USB. I actually don't want to believe them. But they shout loudest. mb.
On Thu, Aug 29, 2013 at 12:56:35PM +0100, Mark Ballard wrote:> I think this is really an attempt at user feedback, rather than user > discussion. But there's no such thing as a user-feedback mail list. > > Nevertheless, others may find this pertinent: why doesn't mke2fs > handle USB's competently? And if it does, why doesn't it reassure me > so? And how can I handle a linux-formatted USB flash drive in the > absence of my system giving me any guidance?The problem is that it really depends on the quality of the flash drive. Two main things drive this: 1) Whether the flash used in the drive is SLC. MLC, or TLC. This will tell you how many write cycles the flash can support. This is where you will see numbers such as 10,000 write cycles, all the way down to 1,000 write depending on the quality of the flash used. 2) The quality of the Flash Translation Layer (FTL). The important thing to remember here is that not all parts of the storage device will get used to the same amount. Some files will rarely change once they are installed on the device; others will change a huge amount. So a good Flash Translation Layer will spread out the wear so that if you have a 120GB device, with flash cells that support 10,000 write cycles, that ideally you can support up to 10,000 times 120GB == 120 TB worth of writes. Of course, that assumes 100% efficiency, which is unrealistic. But some FTL's are incredibly bad, and in the worst case, if they aren't spreading the wear around at all, if you write a single block 1,000 times, and the flash only supports 1,000 write cycles, then that block will go bad. Unfortunately, the cheapest USB flash devices that you find in the bargain bin at the checkout counter of the Micro Center tend to use the crappiest flash media possible, and the crappist FTL's. Some even will only have 512MB of flash even though it is advertised as having 8GB of flash, such that if you ever write more than 512MB worth of files on the flash, you will start losing data. (Fortunately most people don't actually use the full capacity of their USB sticks, which is why the manufacturers can get away with this kind of fraud.) So this is fundamentally a problem with the quality of the hardware, and that's not something the file system can really compensate for. And there's no way to tell whether a particular USB device has has a high quality flash device, or is a craptastic flash device. It's not like we can query the device for "I ripped off the purchaser" bit. :-) Regards, - Ted
On 8/29/13 10:46 AM, Theodore Ts'o wrote:> So this is fundamentally a problem with the quality of the hardware, > and that's not something the file system can really compensate for. > And there's no way to tell whether a particular USB device has has a > high quality flash device, or is a craptastic flash device. It's not > like we can query the device for "I ripped off the purchaser" bit. :-)Yes, this is the fundamental problem. What's the best setting for the random blob of silicon on the end of your usb plug? e2fsprogs can't call tech support, even to get bad answers. We'd like to help in software, but we can't; we have no reliable way of knowing most of the necessary details of this class of hardware. It's not exported in any way. So unfortunately we are as in the dark as you are in this case. -Eric
Agh. This is great to have some of this clarified, Ted. It does unfortunately reinorce my cynicism. But it also fills my heart. That is, though it might be bad news to hear that I have most likely bought a piece of crap, and that there's no way I can really tell what's inside the cover unless either I do some kind of low-level alchemy of a kind that mere users would normally be well advised to avoid, or the manufacturer fesses up to its scam by publishing the specs; it is nevertheless the case that this news has come from the filesystem maintainer himself, and my faith in humanity is thus restored by this simple proof of floss's egalitarian creed. Amen. That said, the user forums are full of lost souls, despairing for want of an answer to the simple question of whether they could well or should not format their USB in ext3/4. Are you implying that it cannot be known unless the manuf gives you the specs and if this is the case and you've bought a bargain bucket flash drive (and assuming that without a decent flash h/w controller it is true that ext3/4 will thrash your drive) then you should stick to ext2? What then are the features a USB flash spec you should look for as the minimum required before installing ext4? cheers mb. On 29 August 2013 16:46, Theodore Ts'o <tytso@mit.edu> wrote:> On Thu, Aug 29, 2013 at 12:56:35PM +0100, Mark Ballard wrote: >> I think this is really an attempt at user feedback, rather than user >> discussion. But there's no such thing as a user-feedback mail list. >> >> Nevertheless, others may find this pertinent: why doesn't mke2fs >> handle USB's competently? And if it does, why doesn't it reassure me >> so? And how can I handle a linux-formatted USB flash drive in the >> absence of my system giving me any guidance? > > The problem is that it really depends on the quality of the flash > drive. Two main things drive this: > > 1) Whether the flash used in the drive is SLC. MLC, or TLC. This > will tell you how many write cycles the flash can support. This is > where you will see numbers such as 10,000 write cycles, all the way > down to 1,000 write depending on the quality of the flash used. > > 2) The quality of the Flash Translation Layer (FTL). The important > thing to remember here is that not all parts of the storage device > will get used to the same amount. Some files will rarely change once > they are installed on the device; others will change a huge amount. > So a good Flash Translation Layer will spread out the wear so that if > you have a 120GB device, with flash cells that support 10,000 write > cycles, that ideally you can support up to 10,000 times 120GB == 120 > TB worth of writes. Of course, that assumes 100% efficiency, which is > unrealistic. But some FTL's are incredibly bad, and in the worst > case, if they aren't spreading the wear around at all, if you write a > single block 1,000 times, and the flash only supports 1,000 write > cycles, then that block will go bad. > > Unfortunately, the cheapest USB flash devices that you find in the > bargain bin at the checkout counter of the Micro Center tend to use > the crappiest flash media possible, and the crappist FTL's. Some even > will only have 512MB of flash even though it is advertised as having > 8GB of flash, such that if you ever write more than 512MB worth of > files on the flash, you will start losing data. (Fortunately most > people don't actually use the full capacity of their USB sticks, which > is why the manufacturers can get away with this kind of fraud.) > > So this is fundamentally a problem with the quality of the hardware, > and that's not something the file system can really compensate for. > And there's no way to tell whether a particular USB device has has a > high quality flash device, or is a craptastic flash device. It's not > like we can query the device for "I ripped off the purchaser" bit. :-) > > Regards, > > - Ted