On Fri, Aug 30, 2013 at 10:56:17AM +0100, Mark Ballard wrote:> > This is incredible, Mr Sandeen. You mean USB flash manufacturers > (what's their body - the USB Implementer's Forum?) have simply not > provided a means for software to query the underlying hardware in a > USB flash? Have software producers asked them for this?No, they haven't. And yes we have, since there are some things (such as the erase block size) which would be useful for tuning file system performance. And the technical people I've talked to at various Flash manufacturers all agree it's pointless to hide this information, but the product managers tend to be the roadblock. If you are buying several million eMMC flash devices for a mobile handset (i.e., for an Android device), you can find out this information, under NDA. But otherwise, the only way you can find it out is by carrying out timing attacks against the flash device. Even if they did provide this sort of information, there are lots of things which are not quantifiable. Cars don't have interface for telling you whether they have the crash protection of a Ford Pinto, or a Tesla S. Part of this is because product managers aren't going to want to advertise that they have only provided 512MB worth of flash on a device which is labelled as containing 8GB (and which will claim to contain 8GB even if you query it via software; it's just that the 512MB + 4k write will cause some random block to disappear). But the other part is that there isn't a good single metric for "crash safety" (crash safety against head-on collisions? crash safety versus side impact collisions? etc.) Similarly, even for a non-fradulent USB stick or SD card, there is no single way to measure "FTL quality". An FTL which is optimized for use in digital cameras (where writes are usually sequential, and for large jpg images) may be horrible for use a general purpose root file system. There are sites that will do technical analysis and ratings for various flash media (such as www.anandtech.com). For example a quick google search turns up: http://www.anandtech.com/show/4523/usb-30-flash-drive-roundup These are going to be the pricier devices, though. They aren't going to be the five dollar crap that you find at the checkout counter of a computer store. But then again, you get what you pay for.... - Ted
>From the little I have heard about control systems for cars, which wassome years ago, they were blockhead proprietary. The analogy would only work if computing was customarily blackbox technology, which it isn't. I'd be surprised if there were any branded flash drives that contained less than their advertised amount of storage. That leaves the question of what is going on under the hood in what is probably the vast majority of devices where the flash isn't fraudulent. And whether my system handles it correctly. My system leaves me with no idea of either (though my hope holds out for some tools I bookmarked recently). Reference to forums and specialist websites gives genuine cause for doubt. Yet I thought it was usual for system software to have a good angle on how its hardware was constructed and what it was doing. I thought they worked in symbiosis, and that this maintained by mutual necessity. I thought the symbiosis was kept unassailably whole by a common purpose: the user. What you say implies that this symbiosis has been broken by the commercial greed of flash manufacturers. Or that it is by neglect on their part, or lazyness, or some other cause of a fissure in industry relations. Whatever the reason, it raises another question, and that is what must be done so that I can simply format my USB without a concern and get back to my work.> even for a non-fradulent USB stick or SD card, there is no single way to measure "FTL quality". ... there are some things (such as the erase block size) which would be useful for tuning file system performance. And the technical people I've talked to at various Flash manufacturers all agree it's pointless to hide this information, butthe product managers tend to be the roadblock.>This is perhaps telling. One would imagine the USB Industry Forum meeting the Association of (File) System Software Scribes or whatever at routine collegiate meetings in Las Vegas hotels, and so on. Which flash manufacturers have refused to collaborate? Why has the fabled industry forum failed? mb.
On Tue, Sep 03, 2013 at 09:46:39AM +0100, Mark Ballard wrote:> isn't. I'd be surprised if there were any branded flash drives that > contained less than their advertised amount of storage.The vast majority of flash sold, especially the cheap-grade flash (i.e., SD Cards and USB sticks) sold through retail channels, is probably unbranded. Because it's cheaper, and for most users, (a) price is a feature, and (b) they are only using flash as a temporary transport medium (e.g., here let me give you my slide presentation; can I borrow a USB stick?), and (c) they are much more likely to lose said flash device before it is likely to go bad, or even gets 100% filled. It's for the same reason that the quality of experience in airplanes travel has degraded so badly. The market has spoken; and consumers have said, at least by their actions, that price is important than anything else.> Whatever the reason, it raises another question, and that is what must > be done so that I can simply format my USB without a concern and get > back to my work.Buy high quality flash which has been explicitly reviewed by a source you trust. There isn't much else you can really do....> This is perhaps telling. One would imagine the USB Industry Forum > meeting the Association of (File) System Software Scribes or whatever > at routine collegiate meetings in Las Vegas hotels, and so on. > > Which flash manufacturers have refused to collaborate? Why has the > fabled industry forum failed?They are collaborating --- with the mass buyers of their flash. If you are a purchasing flash by the millions, then you can get all of this information (under NDA), and you can dictate the quality of the flash which is appropriate for your use case. This even afflicted Microsoft's Windows Phone, where they had some manufacturers provide an SD Card slot. This meant that end users could replace their carefully tested-and-selected-for-performance SD cards which was shipped with their phone with crap sold at the checkout counter, and since the phone's root file system was stored on the SD card, performance when into the crapper, and guess who the customers blamed? Not the flash manufacturer, and not the handset manufacturer for including a removable SD-card slot instead of using a fixed eMMC flash device, but Microsoft. As a result, many handset manufacutrers these days do *not* have an SD card slot, and if they do, they don't allow the root file system to be stored on the SD card, and the SD card can only be used for auxilary or media storage (for which even really crappy flash is generally good enough). So the market is working; it's just working for the most common use case, and the most common desire of the customers who are doing the buying. And that means there will be high quality stuff that costs $$$, and really cheap stuff where you get what you pay for, and hardware manufacters who buy flash devices by the million unit order will get better deals, and all of the low-level information under NDA. All hail the free market.... as my libertarian friends would say, "Huge success". - Ted