Derek Anderson
2009-Oct-10 02:25 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
GigE wasn''t giving me the performance I had hoped for so I spring for some 10Gbe cards. So what am I doing wrong. My setup is a Dell 2950 without a raid controller, just a SAS6 card. The setup is as such : mirror rpool (boot) SAS 10K raidz SSD 467 GB on 3 Samsung 256 MLC SSD (220MB/s each) to create the raidz I did a simple zpool create raidz SSD c1xxxxx c1xxxxxx c1xxxxx. I have a single 10GBe card with a single IP on it. I created a NFS filesystem for vmware by using : zfs create SSD/vmware . I had to set permissoins for Vmware anon=0, but thats it. Below is what zpool iostat reads: File copy 10Gbe to SSD -> 40M max file copy 1gbe to SSD -> 5.4M max File copy SAS to SSD internal -> 90M File copy SSD to SAS internal -> 55M Top shows not matter what I always have 2.5 G free and every other test says the same thing. Can anyone tell me why this is seems to be slow? Does 90M mean MegaBytes or MegaBits? Thanks, Derek -- This message posted from opensolaris.org
Bob Friesenhahn
2009-Oct-10 15:37 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
On Fri, 9 Oct 2009, Derek Anderson wrote:> I created a NFS filesystem for vmware by using : zfs create > SSD/vmware . I had to set permissoins for Vmware anon=0, but thats > it. Below is what zpool iostat reads: > > File copy 10Gbe to SSD -> 40M maxMy clients here do better than that over gigabit ethernet if the file is a large file. If by ''file'' you mean directories of many small files, then there are likely other factors at work.> Top shows not matter what I always have 2.5 G free and every other > test says the same thing. Can anyone tell me why this is seems to > be slow? Does 90M mean MegaBytes or MegaBits?The units are megabytes. It seems that many of these SSD products (e.g. Samsung SSDs) are over-specified and the transfer rate specified is the "burst" rate rather than the average continuous write rate. They also often have limited buffering and command queing capability. After the device has been used for a while, it needs to start erasing large blocks before it can write blocks, which increases write latencies. Even though you are using a SSD, ZFS is going to tell the device to sync its write cache due to NFS synchronous writes. I suggest that you try adjusting this zfs tunable in /etc/system * Set device I/O maximum concurrency * http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 set zfs:zfs_vdev_max_pending = 1 The default is 32. Since SSDs don''t have heads or rotating media, there is likely less benefit to queuing up a bunch of requests. If this helps, try increasing the value until performance drops off. Also, check the system log file for any messages involving bus resets or other problems. Lastly, you should use a network benchmark tool to verify that you really are getting something close to 10Gbits of bandwidth. It is quite possible that you have a broken network. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Derek Anderson
2009-Oct-10 15:59 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
Thank you so much for the detail. The 10Gbe is attached to 10Gbe port on a Vmware ESX server. I am trying to use NFS for VMware. When I bought the SSD''s I was after low seek time not necessarily total bandwidth. I can add devices over time to get the bandwidth up. I am puzzled why even my GigE, which is through a GigE switch to the same VMware box is only getting 2-5M throughput. If I could get Gige to run at 50% that would be 10x what I am getting now. Derek -- This message posted from opensolaris.org
Al Hopper
2009-Oct-13 06:15 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
On Fri, Oct 9, 2009 at 9:25 PM, Derek Anderson <derek at rockymtndata.net> wrote:> > GigE wasn''t giving me the performance I had hoped for so I spring for some 10Gbe cards. ? ?So what am I doing wrong. > > My setup is a Dell 2950 without a raid controller, just a SAS6 card. ?The setup is as such > : > mirror rpool (boot) SAS 10K > raidz SSD ?467 GB ?on 3 Samsung 256 MLC SSD (220MB/s each) > > to create the raidz I did a simple zpool create raidz SSD c1xxxxx c1xxxxxx c1xxxxx. ?I have a single 10GBe card with a single IP on it. > > I created a NFS filesystem for vmware by using : ?zfs create SSD/vmware . ? I had to set permissoins for Vmware anon=0, but thats it. ?Below is what zpool iostat reads: > > File copy 10Gbe to SSD -> 40M max > file copy ?1gbe ?to SSD -> ?5.4M max > File copy ?SAS to SSD internal -> 90M > File copy SSD to SAS internal -> 55M > > Top shows not matter what I always have 2.5 G free and every other test says the same thing. ?Can anyone tell me why this is seems to be slow? ?Does 90M mean MegaBytes or MegaBits? > > Thanks, >Derek - I think you made a bad choice with the Samsung disks. I''d recommend the Intel 160Gb drives if its not too late to return the Samsungs. The Intel drives currently offer the best compromise between different work loads. There are plenty of SSD reviews and the Samsungs always come out poorly in comparison testing. Regards, -- Al Hopper ?Logical Approach Inc,Plano,TX al at logical-approach.com ? ? ? ? ? ? ? ? ? Voice: 972.379.2133 Timezone: US CDT OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007 http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/
Derek Anderson
2009-Oct-13 06:48 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
Thank you for your input folks. The MTU 9000 idea worked like a charm. I have the Intel X25 also, but the capacity was not what I am after for a 6 device array. I have looked and looked at review after review and thats why I started with the Intel path, albeit that firmware upgrade in May was a pain to pull off. I have seen glowing things about the Samsung''s and Intels both. What tipped me over the edge is a youtube video, ( surely paid for by Samsung ). Check it out : http://www.youtube.com/watch?v=96dWOEa4Djs Figuring out how to do jumbo frames on the ixgbe was fun given my newness to suns platform. Thanks, Derek -- This message posted from opensolaris.org
Roman Naumenko
2009-Oct-13 13:04 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
I think after some time we gonna see Derek screaming for f... zfs that toasted the data on his ssd array :) Hopefully this setup was non for production. -- Roman Naumenko PS -- This message posted from opensolaris.org
Derek Anderson
2009-Oct-13 13:24 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
Before you all start taking bets, I am having a difficult time understanding why you would. If you think I am nuts because SSD''s have a limited lifespan, I would agree with you, however we all know that SSD''s are going to get cheaper and cheaper as the days go by. The Intels I bought in April are half the price now they were then. So are the Samsungs. I suspect that by next spring, I will replace them all with new ones and they will be half the cost they are now. Why would anyone spend 3K on disks and just toss it in the river? Simple answer: Man hour math. I have 150 virtual machines on these disks for shared storage. They hold no actual data so who really cares if they get lost. However 150 users of these virtual machines will save 5 minutes or so every day of work, which translates to $250. So $3,000 in SSD''s which are easily replaced one by one with zfs saves the company $250,000 in labor. So when I replace these drives in 6 months, for somewhere around $1500 its a fantastic deal. The only bad part is I cannot estimate how much of the old disks have life is left because in a few months, I am going to have a handful of the fastest SSD''s around and not sure if I would trust them for much of anything. Am I really that wrong? Derek -- This message posted from opensolaris.org
erik.ableson
2009-Oct-13 13:36 UTC
[zfs-discuss] SSD value [was SSD over 10gbe not any faster than 10K SAS over GigE]
On 13 oct. 2009, at 15:24, Derek Anderson wrote:> Simple answer: Man hour math. I have 150 virtual machines on these > disks for shared storage. They hold no actual data so who really > cares if they get lost. However 150 users of these virtual machines > will save 5 minutes or so every day of work, which translates to > $250. So $3,000 in SSD''s which are easily replaced one by one with > zfs saves the company $250,000 in labor. So when I replace these > drives in 6 months, for somewhere around $1500 its a fantastic deal.Overall, I think this is a reasonable model for the medium sized enterprise to work with. As in most cases the mythical 5 minutes saved with be invisible to the overall operations, and difficult to justify to management, but if you can squeeze it into an annual operating budget rather than a capital expense that requires separate justification you should be good.> The only bad part is I cannot estimate how much of the old disks > have life is left because in a few months, I am going to have a > handful of the fastest SSD''s around and not sure if I would trust > them for much of anything.As for what to do with the SSDs - you can resell them or give them to employees (being clear on their usage and provenance) since they represent a risk in a high volume enterprise environment, but could probably supply several years worth of service in a single-user mode. I''d be very happy to get a top of the line SSD at half price for my laptop for a year''s projected use...knowing of course that I backup daily as a matter of religious observance :-) Cheers, Erik
Derek Anderson
2009-Oct-13 14:09 UTC
[zfs-discuss] SSD value [was SSD over 10gbe not any faster than 10K SAS over GigE]
I did bad math, I meant 25,000 in labor dollars saved over 6 months. There is one applicatoin called FRx, a reporting engine for their accounting. Even if their executives save 10 minutes a day running just that bloated application, then this plan has payed for itself in just a few weeks. ZFS is pretty cool. I have spent just over $6k on a Dell server with 1 TB of SSD storage and 10Gbe. It houses 150 Virtual Machines (WinXP) that are connected to by $35 thin clients we picked up an an auction. Eat that Netapp! Derek -- This message posted from opensolaris.org
Tim Cook
2009-Oct-13 14:44 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
On Tue, Oct 13, 2009 at 8:24 AM, Derek Anderson <derek at rockymtndata.net>wrote:> Before you all start taking bets, I am having a difficult time > understanding why you would. If you think I am nuts because SSD''s have a > limited lifespan, I would agree with you, however we all know that SSD''s are > going to get cheaper and cheaper as the days go by. The Intels I bought in > April are half the price now they were then. So are the Samsungs. I > suspect that by next spring, I will replace them all with new ones and they > will be half the cost they are now. Why would anyone spend 3K on disks and > just toss it in the river? > > Simple answer: Man hour math. I have 150 virtual machines on these disks > for shared storage. They hold no actual data so who really cares if they > get lost. However 150 users of these virtual machines will save 5 minutes > or so every day of work, which translates to $250. So $3,000 in SSD''s > which are easily replaced one by one with zfs saves the company $250,000 in > labor. So when I replace these drives in 6 months, for somewhere around > $1500 its a fantastic deal. > > The only bad part is I cannot estimate how much of the old disks have life > is left because in a few months, I am going to have a handful of the fastest > SSD''s around and not sure if I would trust them for much of anything. > > Am I really that wrong? > > Derek >I''ll take them when you''re done :) --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091013/d78dcb18/attachment.html>
David Dyer-Bennet
2009-Oct-13 16:31 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
On Tue, October 13, 2009 08:24, Derek Anderson wrote:> The only bad part is I cannot estimate how much of the old disks have life > is left because in a few months, I am going to have a handful of the > fastest SSD''s around and not sure if I would trust them for much of > anything.In the long run, this information should be exposed in a standard way to SMART and probably direct query commands. With that in place, it would mean you could run them in production longer, since you''d get warning when they were reaching their write life limit. It would also mean you could precisely characterize the remaining life for resale. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
Roman Naumenko
2009-Oct-13 21:28 UTC
[zfs-discuss] SSD over 10gbe not any faster than 10K SAS over GigE
Well, your plan on storage usage goes to 1% of those who doesn''t need reliability and roomy media back-end. So, it can work out well - but unfortunately this is not a silver bullet. -- Roman -- This message posted from opensolaris.org