zfs discuss - Oct 2009 - SSD over 10gbe not any faster than 10K SAS over GigE

GigE wasn''t giving me the performance I had hoped for so I spring for
some 10Gbe cards.    So what am I doing wrong.

My setup is a Dell 2950 without a raid controller, just a SAS6 card.  The setup
is as such
:
mirror rpool (boot) SAS 10K
raidz SSD  467 GB  on 3 Samsung 256 MLC SSD (220MB/s each)

to create the raidz I did a simple zpool create raidz SSD c1xxxxx c1xxxxxx
c1xxxxx.  I have a single 10GBe card with a single IP on it.

I created a NFS filesystem for vmware by using :  zfs create SSD/vmware .   I
had to set permissoins for Vmware anon=0, but thats it.  Below is what zpool
iostat reads:

File copy 10Gbe to SSD -> 40M max
file copy  1gbe  to SSD ->  5.4M max
File copy  SAS to SSD internal -> 90M
File copy SSD to SAS internal -> 55M

Top shows not matter what I always have 2.5 G free and every other test says the
same thing.  Can anyone tell me why this is seems to be slow?  Does 90M mean
MegaBytes or MegaBits?

Thanks,

Derek
-- 
This message posted from opensolaris.org

On Fri, 9 Oct 2009, Derek Anderson wrote:
> I created a NFS filesystem for vmware by using :  zfs create 
> SSD/vmware .  I had to set permissoins for Vmware anon=0, but thats 
> it.  Below is what zpool iostat reads:
>
> File copy 10Gbe to SSD -> 40M max
My clients here do better than that over gigabit ethernet if the file 
is a large file.  If by ''file'' you mean directories of many
small
files, then there are likely other factors at work.
> Top shows not matter what I always have 2.5 G free and every other 
> test says the same thing.  Can anyone tell me why this is seems to 
> be slow?  Does 90M mean MegaBytes or MegaBits?
The units are megabytes.

It seems that many of these SSD products (e.g. Samsung SSDs) are 
over-specified and the transfer rate specified is the "burst" rate 
rather than the average continuous write rate.  They also often have 
limited buffering and command queing capability.  After the device has 
been used for a while, it needs to start erasing large blocks before 
it can write blocks, which increases write latencies.  Even though you 
are using a SSD, ZFS is going to tell the device to sync its write 
cache due to NFS synchronous writes.

I suggest that you try adjusting this zfs tunable in /etc/system

* Set device I/O maximum concurrency
*
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29
set zfs:zfs_vdev_max_pending = 1

The default is 32.  Since SSDs don''t have heads or rotating media, 
there is likely less benefit to queuing up a bunch of requests.  If 
this helps, try increasing the value until performance drops off.

Also, check the system log file for any messages involving bus resets 
or other problems.

Lastly, you should use a network benchmark tool to verify that you 
really are getting something close to 10Gbits of bandwidth.  It is 
quite possible that you have a broken network.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Thank you so much for the detail.  The 10Gbe is attached to 10Gbe port on a
Vmware ESX server.  I am trying to use NFS for VMware.  When I bought the
SSD''s I was after low seek time not necessarily total bandwidth.  I can
add devices over time to get the bandwidth up.   I am puzzled why even my GigE,
which is through a GigE switch to the same VMware box is only getting 2-5M
throughput.  If I could get Gige to run at 50% that would be 10x what I am
getting now.

Derek
-- 
This message posted from opensolaris.org

On Fri, Oct 9, 2009 at 9:25 PM, Derek Anderson <derek at rockymtndata.net>
wrote:
>
> GigE wasn''t giving me the performance I had hoped for so I spring
for some 10Gbe cards. ? ?So what am I doing wrong.
>
> My setup is a Dell 2950 without a raid controller, just a SAS6 card. ?The
setup is as such
> :
> mirror rpool (boot) SAS 10K
> raidz SSD ?467 GB ?on 3 Samsung 256 MLC SSD (220MB/s each)
>
> to create the raidz I did a simple zpool create raidz SSD c1xxxxx c1xxxxxx
c1xxxxx. ?I have a single 10GBe card with a single IP on it.
>
> I created a NFS filesystem for vmware by using : ?zfs create SSD/vmware . ?
I had to set permissoins for Vmware anon=0, but thats it. ?Below is what zpool
iostat reads:
>
> File copy 10Gbe to SSD -> 40M max
> file copy ?1gbe ?to SSD -> ?5.4M max
> File copy ?SAS to SSD internal -> 90M
> File copy SSD to SAS internal -> 55M
>
> Top shows not matter what I always have 2.5 G free and every other test
says the same thing. ?Can anyone tell me why this is seems to be slow? ?Does 90M
mean MegaBytes or MegaBits?
>
> Thanks,
>
Derek - I think you made a bad choice with the Samsung disks.  I''d
recommend the Intel 160Gb drives if its not too late to return the
Samsungs.  The Intel drives currently offer the best compromise
between different work loads.  There are plenty of SSD reviews and the
Samsungs always come out poorly in comparison testing.

Regards,

--
Al Hopper ?Logical Approach Inc,Plano,TX al at logical-approach.com
? ? ? ? ? ? ? ? ? Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

Thank you for your input folks.  The MTU 9000 idea worked like a charm.  I have
the Intel X25 also, but the capacity was not what I am after for a 6 device
array.   I have looked and looked at review after review and thats why I started
with the Intel path, albeit that firmware upgrade in May was a pain to pull off.
I have seen glowing things about the Samsung''s and Intels both.  What
tipped me over the edge is a youtube video, ( surely paid for by Samsung ). 
Check it out : http://www.youtube.com/watch?v=96dWOEa4Djs

Figuring out how to do jumbo frames on the ixgbe was fun given my newness to
suns platform.

Thanks,

Derek
-- 
This message posted from opensolaris.org

I think after some time we gonna see Derek screaming for f... zfs that toasted
the data on his ssd array :)

Hopefully this setup was non for production.

--
Roman Naumenko

PS
-- 
This message posted from opensolaris.org

Before you all start taking bets, I am having a difficult time understanding why
you would.   If you think I am nuts because SSD''s have a limited
lifespan, I would agree with you, however we all know that SSD''s are
going to get cheaper and cheaper as the days go by.  The Intels I bought in
April are half the price now they were then.  So are the Samsungs.   I suspect
that by next spring, I will replace them all with new ones and they will be half
the cost they are now.   Why would anyone spend 3K on disks and just toss it in
the river?

Simple answer:  Man hour math.  I have 150 virtual machines on these disks for
shared storage.  They hold no actual data so who really cares if they get lost. 
However 150 users of these virtual machines will save 5 minutes or so every day
of work, which translates to $250.   So $3,000 in SSD''s which are
easily replaced one by one with zfs saves the company $250,000 in labor.  So
when I replace these drives in 6 months, for somewhere around $1500 its a
fantastic deal.

The only bad part is I cannot estimate how much of the old disks have life is
left because in a few months, I am going to have a handful of the fastest
SSD''s around and not sure if I would trust them for much of anything.

Am I really that wrong?

Derek
-- 
This message posted from opensolaris.org

On 13 oct. 2009, at 15:24, Derek Anderson wrote:
> Simple answer:  Man hour math.  I have 150 virtual machines on these  
> disks for shared storage.  They hold no actual data so who really  
> cares if they get lost.  However 150 users of these virtual machines  
> will save 5 minutes or so every day of work, which translates to  
> $250.   So $3,000 in SSD''s which are easily replaced one by one
with
> zfs saves the company $250,000 in labor.  So when I replace these  
> drives in 6 months, for somewhere around $1500 its a fantastic deal.
Overall, I think this is a reasonable model for the medium sized  
enterprise to work with. As in most cases the mythical 5 minutes saved  
with be invisible to the overall operations, and difficult to justify  
to management, but if you can squeeze it into an annual operating  
budget rather than a capital expense that requires separate  
justification you should be good.
> The only bad part is I cannot estimate how much of the old disks  
> have life is left because in a few months, I am going to have a  
> handful of the fastest SSD''s around and not sure if I would trust
> them for much of anything.
As for what to do with the SSDs - you can resell them or give them to  
employees (being clear on their usage and provenance) since they  
represent a risk in a high volume enterprise environment, but could   
probably supply several years worth of service in a single-user mode.  
I''d be very happy to get a top of the line SSD at half price for my  
laptop for a year''s projected use...knowing of course that I backup  
daily as a matter of religious observance :-)

Cheers,

Erik

I did bad math, I meant 25,000 in labor dollars saved over 6 months.  There is
one applicatoin called FRx, a reporting engine for their accounting.  Even if
their executives save 10 minutes a day running just that bloated application,
then this plan has payed for itself in just a few weeks.

ZFS is pretty cool.   I have spent just over $6k on a Dell server with 1 TB of
SSD storage and 10Gbe.  It houses 150 Virtual Machines (WinXP) that are
connected to by $35 thin clients we picked up an an auction.  Eat that Netapp!

Derek
-- 
This message posted from opensolaris.org

On Tue, Oct 13, 2009 at 8:24 AM, Derek Anderson <derek at
rockymtndata.net>wrote:
> Before you all start taking bets, I am having a difficult time
> understanding why you would.   If you think I am nuts because
SSD''s have a
> limited lifespan, I would agree with you, however we all know that
SSD''s are
> going to get cheaper and cheaper as the days go by.  The Intels I bought in
> April are half the price now they were then.  So are the Samsungs.   I
> suspect that by next spring, I will replace them all with new ones and they
> will be half the cost they are now.   Why would anyone spend 3K on disks
and
> just toss it in the river?
>
> Simple answer:  Man hour math.  I have 150 virtual machines on these disks
> for shared storage.  They hold no actual data so who really cares if they
> get lost.  However 150 users of these virtual machines will save 5 minutes
> or so every day of work, which translates to $250.   So $3,000 in
SSD''s
> which are easily replaced one by one with zfs saves the company $250,000 in
> labor.  So when I replace these drives in 6 months, for somewhere around
> $1500 its a fantastic deal.
>
> The only bad part is I cannot estimate how much of the old disks have life
> is left because in a few months, I am going to have a handful of the
fastest
> SSD''s around and not sure if I would trust them for much of
anything.
>
> Am I really that wrong?
>
> Derek
>
I''ll take them when you''re done :)

--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091013/d78dcb18/attachment.html>

On Tue, October 13, 2009 08:24, Derek Anderson wrote:
> The only bad part is I cannot estimate how much of the old disks have life
> is left because in a few months, I am going to have a handful of the
> fastest SSD''s around and not sure if I would trust them for much
of
> anything.
In the long run, this information should be exposed in a standard way to
SMART and probably direct query commands.  With that in place, it would
mean you could run them in production longer, since you''d get warning
when
they were reaching their write life limit.  It would also mean you could
precisely characterize the remaining life for resale.

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

Well, your plan on storage usage goes to 1% of those who doesn''t need
reliability and roomy media back-end. So, it can work out well - but
unfortunately this is not a silver bullet.

--
Roman
-- 
This message posted from opensolaris.org