zfs discuss - Jan 2010 - preview of new SSD based on SandForce controller

Interesting article - rumor has it that this is the same controller
that Seagate will use in its upcoming enterprise level SSDs:

http://anandtech.com/storage/showdoc.aspx?i=3702

It reads like  SandForce has implemented a bunch of ZFS like
functionality in firmware.  Hmm, I wonder if they used any ZFS source
code??

Happy new year.

-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

On Fri, 1 Jan 2010, Al Hopper wrote:
> Interesting article - rumor has it that this is the same controller
> that Seagate will use in its upcoming enterprise level SSDs:
>
> http://anandtech.com/storage/showdoc.aspx?i=3702
>
> It reads like  SandForce has implemented a bunch of ZFS like
> functionality in firmware.  Hmm, I wonder if they used any ZFS source
> code??
The article (and product) seem interesting, but (in usual form) the 
article is written as a sort of unsubstantiated guess-work propped up 
by vendor charts and graphs and with links so the gentle reader can 
purchase the product on-line.

It is good to see that Intel is seeing some competition.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Jan 1, 2010, at 11:28 AM, Bob Friesenhahn wrote:
> On Fri, 1 Jan 2010, Al Hopper wrote:
>
>> Interesting article - rumor has it that this is the same controller
>> that Seagate will use in its upcoming enterprise level SSDs:
>>
>> http://anandtech.com/storage/showdoc.aspx?i=3702
>>
>> It reads like  SandForce has implemented a bunch of ZFS like
>> functionality in firmware.  Hmm, I wonder if they used any ZFS source
>> code??
>
> The article (and product) seem interesting, but (in usual form) the  
> article is written as a sort of unsubstantiated guess-work propped  
> up by vendor charts and graphs and with links so the gentle reader  
> can purchase the product on-line.
>
> It is good to see that Intel is seeing some competition.
Yep, it is good to see that people who are being creative are finding
design wins.  IMHO, the rate of change in the SSD world right now is
about 1000x the rate of change in the HDD world.
  -- richard

Bob Friesenhahn wrote:
> On Fri, 1 Jan 2010, Al Hopper wrote:
>
>> Interesting article - rumor has it that this is the same controller
>> that Seagate will use in its upcoming enterprise level SSDs:
>>
>> http://anandtech.com/storage/showdoc.aspx?i=3702
>>
>> It reads like  SandForce has implemented a bunch of ZFS like
>> functionality in firmware.  Hmm, I wonder if they used any ZFS source
>> code??
>
> The article (and product) seem interesting, but (in usual form) the 
> article is written as a sort of unsubstantiated guess-work propped up 
> by vendor charts and graphs and with links so the gentle reader can 
> purchase the product on-line.
>
> It is good to see that Intel is seeing some competition.
>
> Bob
> -- 
Yeah, there were a bunch more "maybe" and "looks like" and
"might be"
than I''m really comfortable with in that article.

The one thing it does bring up is the old problem of Where Intelligence 
Belongs.   You most typically see this in the CPU/coprocessor cycle, 
where the battle between enough performance gain in using a separate 
chip vs the main CPU to perform some task is a never ending cycle.

One of ZFS''s founding ideas is that Intelligence belongs up in the main
system (i.e. running in the OS, on the primary CPU(s)), and that all 
devices are stupid and unreliable.   I''m looking at all the (purported)
features in this SandForce controller, and wondering how they''ll 
interact with a "smart" filesystem like ZFS, rather than a traditional
"stupid" filesystem a la UFS.   I see a lot of overlap, which
I''m not
sure is a good thing.

Maybe it''s approaching time for vendors to just produce really stupid 
SSDs: that is, ones that just do wear-leveling, and expose their true 
page-size info (e.g. for MLC, how many blocks of X size have to be 
written at once) and that''s about it.  Let filesystem makers worry
about
scheduling writes appropriately, doing redundancy, etc.

Oooh!   Oooh!  a whole cluster of USB thumb drives!  Yeah!    <wink>




-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA

On Fri, 1 Jan 2010, Erik Trimble wrote:
>
> Maybe it''s approaching time for vendors to just produce really
stupid SSDs:
> that is, ones that just do wear-leveling, and expose their true page-size 
> info (e.g. for MLC, how many blocks of X size have to be written at once)
and
> that''s about it.  Let filesystem makers worry about scheduling
writes
> appropriately, doing redundancy, etc.
>From the benchmarks, it is clear that the drive interface is already 
often the bottleneck for these new SSDs.  That implies that the 
current development path is in the wrong direction unless we are 
willing to accept legacy-sized devices implementing a complex legacy 
protocol.  If the devices remain the same physical size with more 
storage then we are faced with the same current situation we have with 
rotating media, with huge media density and relatively slow I/O 
performance.  We do need stupider SSDs which fit in a small form 
factor, offer considerable bandwidth (e.g. 300MB/second) per device, 
and use a specialized communication protocol which is not defined by 
legacy disk drives.  This allows more I/O to occur in parallel, for 
much better I/O rates.
> Oooh!   Oooh!  a whole cluster of USB thumb drives!  Yeah!    <wink>
That is not far from what we should have (small chassis-oriented 
modules), but without the crummy USB.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Jan 1, 2010, at 6:33 PM, Bob Friesenhahn wrote:
> On Fri, 1 Jan 2010, Erik Trimble wrote:
>>
>> Maybe it''s approaching time for vendors to just produce really
>> stupid SSDs: that is, ones that just do wear-leveling, and expose  
>> their true page-size info (e.g. for MLC, how many blocks of X size  
>> have to be written at once) and that''s about it.  Let
filesystem
>> makers worry about scheduling writes appropriately, doing  
>> redundancy, etc.
>
> From the benchmarks, it is clear that the drive interface is already  
> often the bottleneck for these new SSDs.  That implies that the  
> current development path is in the wrong direction unless we are  
> willing to accept legacy-sized devices implementing a complex legacy  
> protocol.  If the devices remain the same physical size with more  
> storage then we are faced with the same current situation we have  
> with rotating media, with huge media density and relatively slow I/O  
> performance.  We do need stupider SSDs which fit in a small form  
> factor, offer considerable bandwidth (e.g. 300MB/second) per device,  
> and use a specialized communication protocol which is not defined by  
> legacy disk drives.  This allows more I/O to occur in parallel, for  
> much better I/O rates.
You can already see this affecting the design of high-throughput
storage.  The Sun Storage F1500 Flash Array has 80 SSDs and
uses 64 SAS channels for host connection. Some folks think that
6 Gbps SATA/SAS connections are the Next Great Thing^TM but
that only means you need 32 host connections.  It is quite amazing
to have 1M IOPS and 12.8 GB/s in 1 RU.  Perhaps this is the DAS
of the future?
  -- richard

Bob Friesenhahn wrote:
> On Fri, 1 Jan 2010, Erik Trimble wrote:
>>
>> Maybe it''s approaching time for vendors to just produce really
stupid
>> SSDs: that is, ones that just do wear-leveling, and expose their true 
>> page-size info (e.g. for MLC, how many blocks of X size have to be 
>> written at once) and that''s about it.  Let filesystem makers
worry
>> about scheduling writes appropriately, doing redundancy, etc.
>
> From the benchmarks, it is clear that the drive interface is already 
> often the bottleneck for these new SSDs.  That implies that the 
> current development path is in the wrong direction unless we are 
> willing to accept legacy-sized devices implementing a complex legacy 
> protocol.  If the devices remain the same physical size with more 
> storage then we are faced with the same current situation we have with 
> rotating media, with huge media density and relatively slow I/O 
> performance.  We do need stupider SSDs which fit in a small form 
> factor, offer considerable bandwidth (e.g. 300MB/second) per device, 
> and use a specialized communication protocol which is not defined by 
> legacy disk drives.  This allows more I/O to occur in parallel, for 
> much better I/O rates.
>
>> Oooh!   Oooh!  a whole cluster of USB thumb drives!  Yeah!   
<wink>
>
> That is not far from what we should have (small chassis-oriented 
> modules), but without the crummy USB.
>
> Bob
I tend to like the 2.5" form factor, for a lot of reasons (economies of 
scale, and all).   And, the new SATA III (i.e. 6Gbit/s) interface is 
really sufficient for reasonable I/O, at least until the 12Gbit SAS 
comes along in a year or so.  The 1.8" drive form factor might be useful 
as Flash densities go up (in order to keep down the GB to drive 
interface ratio), but physically, that size is a bit of a pain (it''s 
actually too small for reliability reasons, and makes chassis design 
harder).  I''m actually all for adding a second SATA/SAS I/O connector
on
a 2.5" drive (it''s just possible, physically).

That all said, it certainly would be really nice to get a SSD controller 
which can really push the bandwidth, and the only way I see this 
happening now is to go the "stupid" route, and dumb down the
controller
as much as possible.  I really think we just want the controller to Do 
What I Say, and not try any optimizations or such.  There''s simply much
more benefit to doing the optimization up at the filesystem level than 
down at the device level. For a trivial case, consider the dreaded 
"write-read-write" problem of MLCs:   to write a single bit, a whole 
page has to be read, then the page recomposed with the changed bits, 
before writing again.  If the filesystem was aware that the drive had 
this kind of issue, then in-RAM caching would almost always allow for 
the avoidance of the first "read" cycle, and performance goes back to
a
typical Copy-on-Write style stripe write. 

I can see why having "dumb" controllers might not appear to the 
consumer/desktop market, but certainly, for the Enterprise market, I 
think it''s actually /more/ likely that they start showing up soon.  
Which would be a neat reversal of sorts:   Consumer drives using a 
complex controller with cheap flash (and a large "spare" capacity 
area),  while Enterprise drives use a simple controller, higher-quality 
flash chips, and likely a much smaller spare capacity area.  Which 
means, I expect price parity between the two. 

Whee!

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

On Fri, Jan  1 at 21:21, Erik Trimble wrote:
>That all said, it certainly would be really nice to get a SSD 
>controller which can really push the bandwidth, and the only way I 
>see this happening now is to go the "stupid" route, and dumb down
the
>controller as much as possible.  I really think we just want the 
>controller to Do What I Say, and not try any optimizations or such.  
>There''s simply much more benefit to doing the optimization up at
the
>filesystem level than down at the device level. For a trivial case, 
>consider the dreaded "write-read-write" problem of MLCs:   to
write a
>single bit, a whole page has to be read, then the page recomposed 
>with the changed bits, before writing again.  If the filesystem was 
>aware that the drive had this kind of issue, then in-RAM caching 
>would almost always allow for the avoidance of the first "read" 
>cycle, and performance goes back to a typical Copy-on-Write style 
>stripe write.
I am not convinced that a general purpose CPU, running other software
in parallel, will be able to be timely and responsive enough to
maximize bandwidth in an SSD controller without specialized hardware
support.  This hardware support is, of course, the controller that
exists on modern SSDs.

Drive vendors abstracted these interfaces a long time ago, creating
Integrated Drive Electronics (IDE).  Bringing all of that logic back
up into the CPU would likely not help meaningfully.  Yes, it would
likely be cheaper, but I doubt it would be faster or more reliable.

I also am not convinced that your described RMW semantics are used in
any modern NAND devices.  Those problems were solved years ago.  The
granularity of the implementation has implications on performance in
some workloads, but I believe only those old JMicron-based SSDs did
block-level RMW, and hence wound up doing about ~2-3 IOPS in random
workloads with MLC drives.

SSDs (with good controllers) really strut their stuff when in-RAM
caching isn''t working anyway.  If in-RAM was good enough, then why
bother with SSD?  Just have a spun-down rotating drive at 1/5th-1/15th
the cost.

--eric


-- 
Eric D. Mudama
edmudama at mail.bounceswoosh.org

Eric D. Mudama wrote:
> On Fri, Jan  1 at 21:21, Erik Trimble wrote:
>> That all said, it certainly would be really nice to get a SSD 
>> controller which can really push the bandwidth, and the only way I 
>> see this happening now is to go the "stupid" route, and dumb
down the
>> controller as much as possible.  I really think we just want the 
>> controller to Do What I Say, and not try any optimizations or such.  
>> There''s simply much more benefit to doing the optimization up
at the
>> filesystem level than down at the device level. For a trivial case, 
>> consider the dreaded "write-read-write" problem of MLCs:   to
write a
>> single bit, a whole page has to be read, then the page recomposed 
>> with the changed bits, before writing again.  If the filesystem was 
>> aware that the drive had this kind of issue, then in-RAM caching 
>> would almost always allow for the avoidance of the first
"read"
>> cycle, and performance goes back to a typical Copy-on-Write style 
>> stripe write.
>
> I am not convinced that a general purpose CPU, running other software
> in parallel, will be able to be timely and responsive enough to
> maximize bandwidth in an SSD controller without specialized hardware
> support.  This hardware support is, of course, the controller that
> exists on modern SSDs.
Why not?  My argument is the one that ZFS as a whole is founded on:  
that modern CPUs have so many spare cycles that it''s silly to pay extra
for a smart raid controller when we can just borrow time on the main 
CPU.   It seems to work out just fine for hard drives, so why not for 
SSDs (which, while much faster than HDs, are still many orders of 
magnitude slower than DMA transfers)? 

> Drive vendors abstracted these interfaces a long time ago, creating
> Integrated Drive Electronics (IDE).  Bringing all of that logic back
> up into the CPU would likely not help meaningfully.  Yes, it would
> likely be cheaper, but I doubt it would be faster or more reliable.
>
I''m not advocating a return to something like the old IPI technology
(oh
boy, did I just date myself there...).  That''s silly.  By
"dumb", I''m
referring to things on the level with IDE - a disk controller that 
handles nothing more than internal (to the disk) bad block remapping, 
LBA to actual block mapping, etc.  In the case of a "stupid" SSD 
controller, that would entail sufficient smarts to do wear leveling, 
LBA, bad page/block marking/detection, and very little else.

> I also am not convinced that your described RMW semantics are used in
> any modern NAND devices.  Those problems were solved years ago.  The
> granularity of the implementation has implications on performance in
> some workloads, but I believe only those old JMicron-based SSDs did
> block-level RMW, and hence wound up doing about ~2-3 IOPS in random
> workloads with MLC drives.
ALL modern MLC-based SSDs have exactly the problem I''ve described as
the
example above.  It''s a defining characteristic of the Multi-Level Cell 
design.  A nice modern Intel X25-M can see a loss of 50-80% of it''s 
theoretical maximum write performance once it runs out of unused cells 
to write to.  And that''s with the fancy firmware.
> SSDs (with good controllers) really strut their stuff when in-RAM
> caching isn''t working anyway.  If in-RAM was good enough, then why
> bother with SSD?  Just have a spun-down rotating drive at 1/5th-1/15th
> the cost.
>
> --eric
No, they don''t (strut, that is).  MLC-based SSDs (and, even SLC-based 
ones, to a lesser extent) have a very significant write penalty.  Much 
of the "smarts" that does into current-gen SSDs is an attempt to 
overcome this design limitation.   What Bob and I are saying is that 
locating the "smarts" in the SSD controller is misguided.  Having this
intelligence located in the OS/Filesystem driver is a far better idea, 
as the system has a much more global understanding as to where 
optimizations can occur, and make the appropriate choices.   And, 
frankly, it''s far easier to update a filesystem driver than it is to 
reflash firmware on an SSD, should any changes be necessary.

The example I was giving for R-M-W is that it is /highly/ likely that 
the OS already has a significant bunch of a file to be modified ALREADY 
in the buffer cache (L2ARC, in ZFS''s case).  So, if ZFS is talking to a
stupid SSD, it knows that it cannot just issue a single block write 
should a bit in the file change. Instead, ZFS will know that it should 
issue a TRIM command (or something like it) to have the SSD mark the old 
page (where the bit(s) change) deleted, and then use that page from the 
L2ARC as the template to build a full page with the new bit(s) in it, 
and then issue a full page write to the SSD.  This avoids having the SSD 
itself having to do this.  Worst case scenario is that the SSD will have 
to read the whole page to get it back into the L2ARC, but typical-use 
case is much higher likelihood.    So, typical case is 1 IOPS vs 3 IOPS 
on a "smart" SSD.  My approach uses more interface (SAS/SATA/etc) 
bandwidth to the SSD, but that''s OK, since there''s plenty to
spare.

And, going back to the article where they''re speculating having 
something like RAID and Dedup integrated at the SSD controller-level, 
this is just a /really/ bad idea. 

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

On Fri, Jan 1, 2010 at 8:31 PM, Erik Trimble <Erik.Trimble at sun.com>
wrote:
> Bob Friesenhahn wrote:
>
>> On Fri, 1 Jan 2010, Al Hopper wrote:
>>
>>  Interesting article - rumor has it that this is the same controller
>>> that Seagate will use in its upcoming enterprise level SSDs:
>>>
>>> http://anandtech.com/storage/showdoc.aspx?i=3702
>>>
>>> It reads like  SandForce has implemented a bunch of ZFS like
>>> functionality in firmware.  Hmm, I wonder if they used any ZFS
source
>>> code??
>>>
>>
>> The article (and product) seem interesting, but (in usual form) the
>> article is written as a sort of unsubstantiated guess-work propped up
by
>> vendor charts and graphs and with links so the gentle reader can
purchase
>> the product on-line.
>>
>> It is good to see that Intel is seeing some competition.
>>
>> Bob
>> --
>>
>
> Yeah, there were a bunch more "maybe" and "looks like"
and "might be" than
> I''m really comfortable with in that article.
>
> The one thing it does bring up is the old problem of Where Intelligence
> Belongs.   You most typically see this in the CPU/coprocessor cycle, where
> the battle between enough performance gain in using a separate chip vs the
> main CPU to perform some task is a never ending cycle.
>
> One of ZFS''s founding ideas is that Intelligence belongs up in the
main
> system (i.e. running in the OS, on the primary CPU(s)), and that all
devices
> are stupid and unreliable.   I''m looking at all the (purported)
features in
> this SandForce controller, and wondering how they''ll interact with
a "smart"
> filesystem like ZFS, rather than a traditional "stupid"
filesystem a la UFS.
>   I see a lot of overlap, which I''m not sure is a good thing.
>
> Maybe it''s approaching time for vendors to just produce really
stupid SSDs:
> that is, ones that just do wear-leveling, and expose their true page-size
> info (e.g. for MLC, how many blocks of X size have to be written at once)
> and that''s about it.  Let filesystem makers worry about scheduling
writes
> appropriately, doing redundancy, etc.
>
> Oooh!   Oooh!  a whole cluster of USB thumb drives!  Yeah!    <wink>
>
>
>
While I''m sure to offend someone, it must be stated.  That''s
not going to
happen for the simple fact that there''s all of two vendors that could
utilize it, both niche (in relative terms).  NetApp and Sun.  Why would SSD
MFG''s waste their time building  drives to sell for less money than
their
mainline that a very, very small portion of the market can actually utilize?

Until you convince Hitachi, EMC, HP, IBM, and Microsoft to develop similar
"intelligent" filesystems, the SSD''s you''re
recommending won''t see the light
of day (unless Sun/NetApp decide to make it themselves).

-- 
--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100102/6019c287/attachment.html>

Tim Cook wrote:
> While I''m sure to offend someone, it must be stated. 
That''s not going
> to happen for the simple fact that there''s all of two vendors that
> could utilize it, both niche (in relative terms).  NetApp and Sun.  
> Why would SSD MFG''s waste their time building  drives to sell for
less
> money than their mainline that a very, very small portion of the 
> market can actually utilize?
>
> Until you convince Hitachi, EMC, HP, IBM, and Microsoft to develop 
> similar "intelligent" filesystems, the SSD''s
you''re recommending won''t
> see the light of day (unless Sun/NetApp decide to make it themselves).
>
> -- 
> --Tim
I do think the market is slight larger:  Hitachi and EMC storage 
arrays/big SAN controllers,  plus all Linux boxes once Brtfs actually 
matures enough to be usable.  I don''t see MSFT making any NTFS changes 
to help here, but they are doing some research/development on a next-gen 
filesystem (didn''t make it into Windows 2008, but maybe Win2011), so 
we''ll have to see what that entails.

All that said, it would certainly be limited to Enterprise SSD, which, 
are low-volume.  But, on the up side, they''re High Margin, so maybe we 
can hope...

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

On Jan 2, 2010, at 19:44, Erik Trimble wrote:
> I do think the market is slight larger:  Hitachi and EMC storage  
> arrays/big SAN controllers,  plus all Linux boxes once Brtfs  
> actually matures enough to be usable.  I don''t see MSFT making any
> NTFS changes to help here, but they are doing some research/ 
> development on a next-gen filesystem (didn''t make it into Windows
> 2008, but maybe Win2011), so we''ll have to see what that entails.
Apple is (sadly?) probably developing their own new file system as well.

Of course Microsoft is the 362 kg gorilla, especially when it comes to  
the consumer space.

On Sat, Jan 2, 2010 at 5:40 PM, Tim Cook <tim at cook.ms>
wrote:
>
>
> On Fri, Jan 1, 2010 at 8:31 PM, Erik Trimble <Erik.Trimble at
sun.com> wrote:
>>
>> Bob Friesenhahn wrote:
>>>
>>> On Fri, 1 Jan 2010, Al Hopper wrote:
>>>
>>>> Interesting article - rumor has it that this is the same
controller
>>>> that Seagate will use in its upcoming enterprise level SSDs:
>>>>
>>>> http://anandtech.com/storage/showdoc.aspx?i=3702
>>>>
>>>> It reads like ?SandForce has implemented a bunch of ZFS like
>>>> functionality in firmware. ?Hmm, I wonder if they used any ZFS
source
>>>> code??
>>>
>>> The article (and product) seem interesting, but (in usual form) the
>>> article is written as a sort of unsubstantiated guess-work propped
up by
>>> vendor charts and graphs and with links so the gentle reader can
purchase
>>> the product on-line.
>>>
>>> It is good to see that Intel is seeing some competition.
>>>
>>> Bob
>>> --
>>
>> Yeah, there were a bunch more "maybe" and "looks
like" and "might be" than
>> I''m really comfortable with in that article.
>>
>> The one thing it does bring up is the old problem of Where Intelligence
>> Belongs. ? You most typically see this in the CPU/coprocessor cycle,
where
>> the battle between enough performance gain in using a separate chip vs
the
>> main CPU to perform some task is a never ending cycle.
>>
>> One of ZFS''s founding ideas is that Intelligence belongs up in
the main
>> system (i.e. running in the OS, on the primary CPU(s)), and that all
devices
>> are stupid and unreliable. ? I''m looking at all the
(purported) features in
>> this SandForce controller, and wondering how they''ll interact
with a "smart"
>> filesystem like ZFS, rather than a traditional "stupid"
filesystem a la UFS.
>> ? I see a lot of overlap, which I''m not sure is a good thing.
>>
>> Maybe it''s approaching time for vendors to just produce really
stupid
>> SSDs: that is, ones that just do wear-leveling, and expose their true
>> page-size info (e.g. for MLC, how many blocks of X size have to be
written
>> at once) and that''s about it. ?Let filesystem makers worry
about scheduling
>> writes appropriately, doing redundancy, etc.
>>
>> Oooh! ? Oooh! ?a whole cluster of USB thumb drives! ?Yeah! ?
?<wink>
>>
>
> While I''m sure to offend someone, it must be stated.?
That''s not going to
> happen for the simple fact that there''s all of two vendors that
could
> utilize it, both niche (in relative terms).? NetApp and Sun.? Why would SSD
> MFG''s waste their time building? drives to sell for less money
than their
> mainline that a very, very small portion of the market can actually
utilize?
>
> Until you convince Hitachi, EMC, HP, IBM, and Microsoft to develop similar
> "intelligent" filesystems, the SSD''s you''re
recommending won''t see the light
> of day (unless Sun/NetApp decide to make it themselves).
>
Agreed.  One issue is that there is a limited pool of highly talented
and motivated developers like those that work on Team ZFS.  The 2nd
issue is how to pay for that development work.  Obviously SSD software
IP (intellectual property) providers can recoup their NRE costs
because they can amortize them over large volume commodity hardware
devices.  It appears (to me) that SandForce has thought about their
business model and is targeting both the high-end, high margin, lower
volume enterprise SSD market while also targeting the low-end (desktop
systems) commodity SSD market with a "stripped down" version of the
same software.

Regards,

-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

On Sat, 2 Jan 2010, David Magda wrote:
>
> Apple is (sadly?) probably developing their own new file system as well.
I assume that you are talking about developing a filesystem design 
more suitable for the iNetbook and the iPhone?

Hardly any Apple users are complaining about the advanced filesytem 
they have already.
> Of course Microsoft is the 362 kg gorilla, especially when it comes to the 
> consumer space.
Yes, but Microsoft''s tail does not even wiggle.  It just sort of lays 
there.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Saturday, January 2, 2010, Bob Friesenhahn
<bfriesen at simple.dallas.tx.us> wrote:
> On Sat, 2 Jan 2010, David Magda wrote:
>
>
> Apple is (sadly?) probably developing their own new file system as well.
>
>
> I assume that you are talking about developing a filesystem design more
suitable for the iNetbook and the iPhone?
>
> Hardly any Apple users are complaining about the advanced filesytem they
have already.
That''s a joke right?  Hfs+ is nothing remotely close to an advanced
filesystem. Apple users not complaining is more proof of them having
not only drunk the koolaid but also bathed in it than them knowing any
lImtations of what they have today.  This coming from someone with a
MacBook pro sitting in the other room.

>
>
> Of course Microsoft is the 362 kg gorilla, especially when it comes to the
consumer space.
>
>
> Yes, but Microsoft''s tail does not even wiggle. ?It just sort of
lays there.
>
> Bob
> --
Ahhh, so the truth comes out. Another apple zealot. They to this day
cant touch outlook/exchange or SQL or AD, or anything remotely
resembling enterprise management of wokstations. Win7 is an incredible
release as is 2k8R2. Other than that though, ya, not moving.

-- 
--Tim

On Jan 2, 2010, at 20:51, Tim Cook wrote:
> Apple users not complaining is more proof of them having
> not only drunk the koolaid but also bathed in it than them knowing any
> lImtations of what they have today.  This coming from someone with a
> MacBook pro sitting in the other room.
Apple users not complaining is simply proof that most people don''t  
care about these things because they have more important things to  
worry about (at least to them). There were plenty of users that were  
annoyed when it was announced that Apple and Sun couldn''t agree on  
ZFS--my sister was not one of them.

On Sat, Jan 2, 2010 at 9:45 PM, David Magda <dmagda at ee.ryerson.ca>
wrote:
> On Jan 2, 2010, at 20:51, Tim Cook wrote:
>
>  Apple users not complaining is more proof of them having
>> not only drunk the koolaid but also bathed in it than them knowing any
>> lImtations of what they have today.  This coming from someone with a
>> MacBook pro sitting in the other room.
>>
>
> Apple users not complaining is simply proof that most people don''t
care
> about these things because they have more important things to worry about
> (at least to them). There were plenty of users that were annoyed when it
was
> announced that Apple and Sun couldn''t agree on ZFS--my sister was
not one of
> them.
>
>
Most people didn''t complain when they had dial-up.  Try switching one
of
them back to dial-up from broadband and let me know what kind of reaction
you get.  Just because a user is ignorant as to what they don''t have,
doesn''t mean they don''t want it or wouldn''t use it. 
You''re going to need a
far better argument than that.

-- 
--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100102/e022ad4e/attachment.html>

David Magda <dmagda at ee.ryerson.ca> wrote:
> Apple is (sadly?) probably developing their own new file system as well.
Well, I still don''t understand Apple. Apple likes to get a grant for an
indemnification for something that cannot happen in a country with a proper 
law system. 

The netapps patents contain claims on ideas that I invented for my Diploma 
thesis work between 1989 and 1991, so the netapps patents only describe prior
art. The new ideas introduced with "wofs" include the ideas on how to
use COW
for filesystems and on how to find the most recent "superblock" on a
COW
filesystem. The ieas for the latter method have been developed while 
discussing the "wofs" structure with Casten Bormann at TU-Berlin.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Tim Cook <tim at cook.ms> wrote:
> On Saturday, January 2, 2010, Bob Friesenhahn
> <bfriesen at simple.dallas.tx.us> wrote:
> > On Sat, 2 Jan 2010, David Magda wrote:
> >
> >
> > Apple is (sadly?) probably developing their own new file system as
well.
> >
> >
> > I assume that you are talking about developing a filesystem design
more suitable for the iNetbook and the iPhone?
> >
> > Hardly any Apple users are complaining about the advanced filesytem
they have already.
>
> That''s a joke right?  Hfs+ is nothing remotely close to an
advanced
;-)

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Joerg.Schilling at fokus.fraunhofer.de (Joerg Schilling) writes:
> The netapps patents contain claims on ideas that I invented for my Diploma 
> thesis work between 1989 and 1991, so the netapps patents only describe
prior
> art. The new ideas introduced with "wofs" include the ideas on
how to use COW
> for filesystems and on how to find the most recent "superblock"
on a COW
> filesystem. The ieas for the latter method have been developed while 
> discussing the "wofs" structure with Casten Bormann at TU-Berlin.
Would you perhaps be willing to share the text? Sounds quite
interesting, especially to compare it with ZFS and with Netapp''s
introduction to WAFL that I read a while ago.

(And I know that discussions with Carsten Bormann can
result in remarkable results -- not that I would want to disregard
your own part in these ideas. :-)

Regards, Juergen.

-- 
Viele Informatiker sind Taeter; sie haben beispielweise Sexualdelikte
begangen oder sind verantwortlich fuer Bluttaten in der eigenen
Familie. Meiner Meinung nach ist ihr Gehirn in der Informatiktechnik
ausser Kontrolle geraten.                           -- Karl Notter

Since there''s nothing I love better on a Sunday than a religious OT
discussion:

On January 2, 2010 8:51:25 PM -0500 Tim Cook <tim at cook.ms>
wrote:
> On Saturday, January 2, 2010, Bob Friesenhahn
> <bfriesen at simple.dallas.tx.us> wrote:
>> Hardly any Apple users are complaining about the advanced filesytem
they
>> have already.
>
> That''s a joke right?  Hfs+ is nothing remotely close to an
advanced
> filesystem.
True, but neither are most all existing filesystems, including NTFS.
Yet folks *do* get along just fine.  Let''s not forget, "most"
people
can barely compose an email.  Even on this list you run into the
occasional buffoon (sometimes it''s me).
> Apple users not complaining is more proof of them having
> not only drunk the koolaid but also bathed in it than them knowing any
> lImtations of what they have today.  This coming from someone with a
> MacBook pro sitting in the other room.
I am a lifelong Apple fan and it is amazing to me that there really is
an RDF around Steve Jobs.  Apple is definitely worse than Microsoft
when it comes to propriety and lock-in (or should i say lock-out).
They just don''t have the market share to be a monopoly.
> Ahhh, so the truth comes out. Another apple zealot. They to this day
> cant touch outlook/exchange or SQL or AD, or anything remotely
> resembling enterprise management of wokstations. Win7 is an incredible
> release as is 2k8R2. Other than that though, ya, not moving.
You''re at risk of being an anti-Apple zealot though.  Yup, Apple
products
sure suck balls for the enterprise (and their raid array is laughable),
but at least the OS has unix underpinnings and all the good stuff that
brings with it.  Win7 is great, but only in comparison to XP and Vista.
The security of Windows is a disaster, and well behind what is easily
achievable today.  The apps are great though!

AD is extremely impressive.  It is easier to push a technology when
you control it 100% (as opposed to kerberos on the unix side) but even
taking that into account, it''s quite the state of the art.  But the
underlying platform it runs on is still years behind in many ways.

-frank

Eric D. Mudama wrote:
> I am not convinced that a general purpose CPU, running other software
> in parallel, will be able to be timely and responsive enough to
> maximize bandwidth in an SSD controller without specialized hardware
> support.  
Fusion-io would seem to be a counter-example, since it uses a fairly 
simple controller (I guess the controller still performs ECC and maybe 
XOR) and the driver eats a whole x86 core. The result is very high 
performance.

Wes Felter

On Mon, Jan  4 at 16:43, Wes Felter wrote:
>Eric D. Mudama wrote:
>
>>I am not convinced that a general purpose CPU, running other software
>>in parallel, will be able to be timely and responsive enough to
>>maximize bandwidth in an SSD controller without specialized hardware
>>support.
>
>Fusion-io would seem to be a counter-example, since it uses a fairly 
>simple controller (I guess the controller still performs ECC and 
>maybe XOR) and the driver eats a whole x86 core. The result is very 
>high performance.
>
>Wes Felter
I see what you''re saying, but it isn''t obvious (to me) how
well
they''re using all the hardware at hand.  2GB/s of bandwidth over their
PCI-e link and what looks like a TON of NAND, with a nearly-dedicated
x86 core...  resuting in 600MB/s or something like that?

While the number is very good for NAND flash SSDs, it seems like a TON
of horsepower going to waste, and they still have a large onboard
controller/FPGA.  I guess enough CPU can make the units faster, but
i''m just not sold.

-- 
Eric D. Mudama
edmudama at mail.bounceswoosh.org

600? I''ve heard 1.5GBps reported.

On 1/5/10, Eric D. Mudama <edmudama at bounceswoosh.org>
wrote:
> On Mon, Jan  4 at 16:43, Wes Felter wrote:
>>Eric D. Mudama wrote:
>>
>>>I am not convinced that a general purpose CPU, running other
software
>>>in parallel, will be able to be timely and responsive enough to
>>>maximize bandwidth in an SSD controller without specialized hardware
>>>support.
>>
>>Fusion-io would seem to be a counter-example, since it uses a fairly
>>simple controller (I guess the controller still performs ECC and
>>maybe XOR) and the driver eats a whole x86 core. The result is very
>>high performance.
>>
>>Wes Felter
>
> I see what you''re saying, but it isn''t obvious (to me)
how well
> they''re using all the hardware at hand.  2GB/s of bandwidth over
their
> PCI-e link and what looks like a TON of NAND, with a nearly-dedicated
> x86 core...  resuting in 600MB/s or something like that?
>
> While the number is very good for NAND flash SSDs, it seems like a TON
> of horsepower going to waste, and they still have a large onboard
> controller/FPGA.  I guess enough CPU can make the units faster, but
> i''m just not sold.
>
> --
> Eric D. Mudama
> edmudama at mail.bounceswoosh.org
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

-- 
Regards,
Andrey

Juergen Nickelsen <ni at jnickelsen.de> wrote:
> Joerg.Schilling at fokus.fraunhofer.de (Joerg Schilling) writes:
>
> > The netapps patents contain claims on ideas that I invented for my
Diploma
> > thesis work between 1989 and 1991, so the netapps patents only
describe prior
> > art. The new ideas introduced with "wofs" include the ideas
on how to use COW
> > for filesystems and on how to find the most recent
"superblock" on a COW
> > filesystem. The ieas for the latter method have been developed while 
> > discussing the "wofs" structure with Casten Bormann at
TU-Berlin.
>
> Would you perhaps be willing to share the text? Sounds quite
> interesting, especially to compare it with ZFS and with Netapp''s
> introduction to WAFL that I read a while ago.
If you are interested in the text, this is here:

http://cdrecord.berlios.de/private/wofs.ps.gz	(this is the old original 
						without images as they have
						not been created with troff)
http://cdrecord.berlios.de/private/WoFS.pdf	(this is a reformatted version
						with images included).

If you like to see the program code, I am considering to make it available
at some time.
						
> (And I know that discussions with Carsten Bormann can
> result in remarkable results -- not that I would want to disregard
> your own part in these ideas. :-)
Yes, he is a really helpful discussion partner.

As a note: the basic ideas for implementing COW (such as inverting the tree 
structure in order to avoid to rewrite all directories upwards to the root 
directory in case that a nested file is updated, the idea to use generation 
nodes called "G-nodes" and the idea on how updated super blocks can be
found)
have been invented by me. Carsten helped to develop a method that allows to 
define and locate extension areas for updated super blocks for the case when 
the primary super clock update area has become full. The latter idea is not 
needed on a hard disk based filesystem as hard disks allw to overwrite old 
superblock locations. On a WORM media, this is essential to make sure that 
the medium is usable for writing as long as there are unwritten blockd.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Eric D. Mudama wrote:
> On Mon, Jan  4 at 16:43, Wes Felter wrote:
>> Eric D. Mudama wrote:
>>
>>> I am not convinced that a general purpose CPU, running other
software
>>> in parallel, will be able to be timely and responsive enough to
>>> maximize bandwidth in an SSD controller without specialized
hardware
>>> support.
>>
>> Fusion-io would seem to be a counter-example, since it uses a fairly 
>> simple controller (I guess the controller still performs ECC and maybe 
>> XOR) and the driver eats a whole x86 core. The result is very high 
>> performance.
> 
> I see what you''re saying, but it isn''t obvious (to me)
how well
> they''re using all the hardware at hand.  2GB/s of bandwidth over
their
> PCI-e link and what looks like a TON of NAND, with a nearly-dedicated
> x86 core...  resuting in 600MB/s or something like that?
Actually it''s 600-700MB/s out of a 1+1GB/s slot or 1.5GB/s with two 
cards in a 2+2GB/s slot. I suspect that''s pretty close to the PCIe 
limit. IIRC they have 22 NAND channels at 40MB/s (theoretical peak) 
each, which is 880MB/s. I agree that their CPU efficiency is not great, 
but cores are supposed to be cheap these days.

Wes Felter

Wes Felter wrote:
> Eric D. Mudama wrote:
>> On Mon, Jan  4 at 16:43, Wes Felter wrote:
>>> Eric D. Mudama wrote:
>>>
>>>> I am not convinced that a general purpose CPU, running other
software
>>>> in parallel, will be able to be timely and responsive enough to
>>>> maximize bandwidth in an SSD controller without specialized
hardware
>>>> support.
>>>
>>> Fusion-io would seem to be a counter-example, since it uses a
fairly
>>> simple controller (I guess the controller still performs ECC and 
>>> maybe XOR) and the driver eats a whole x86 core. The result is very
>>> high performance.
>>
>> I see what you''re saying, but it isn''t obvious (to
me) how well
>> they''re using all the hardware at hand.  2GB/s of bandwidth
over their
>> PCI-e link and what looks like a TON of NAND, with a nearly-dedicated
>> x86 core...  resuting in 600MB/s or something like that?
>
> Actually it''s 600-700MB/s out of a 1+1GB/s slot or 1.5GB/s with
two
> cards in a 2+2GB/s slot. I suspect that''s pretty close to the PCIe
> limit. IIRC they have 22 NAND channels at 40MB/s (theoretical peak) 
> each, which is 880MB/s. I agree that their CPU efficiency is not 
> great, but cores are supposed to be cheap these days.
>
> Wes Felter
>
The single Fusion-IO card is a 4x PCI-E 1.1 interface, which means about 
1GB/s max throughput. The Fusion-IO Duo is a 8x PCI-E 2.0 interface, 
which tops out at about 4GB/s.   So, it looks like the single card is at 
least a major fraction of the max throughput of the interface, while the 
Duo card still has plenty of headroom. 

I see the single Fusion-IO card eat about 1/4 the CPU power that a 
8Gbit  Fibre Channel card HBA does, and roughly the same as a 10Gbit 
Ethernet card.  So, it''s not out of line with comparable throughput 
add-in cards.  It does need significantly more CPU than a SAS or SCSI 
controller, though.

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)