Hello, I''m working on building a iSCSI storage server to use as the backend for virtual servers. I am far more familiar with FreeBSD and Linux, but want to use OpenSolaris for this project because of Comstar & ZFS. My plan was to have a 24 2TB Hitachi SATA drives connected via SAS expanders to two OpenSolaris servers working in a fallover setup. These servers would provide iSCSI shares the the virtual hosting servers over a 10Gb infiniband connection. At this point I''m having trouble getting the performance I was expecting out of ZFS. I can get between 100-130MB/s sequential read and write from my drives individually using dd. And I can maintain those speeds up to a point reading and writing to them in parallel to where I can get about 1GB/s raw to the drives until the bus becomes a bottleneck. However, when I create a zpool, either striped, raidz, or raidz2, my sequential write speeds end up around the write speed of a single drive. I wasn''t expecting this, using the same hardware running linux and an md-raid/lvm2 setup I can easily achieve 1100MB/s+ sequential reads and 600MB/s+ sequential writes. Shouldn''t I, in theory, be able to achieve similar results with zfs? I''m at a loss where to go from here to troubleshoot and tune this setup, or if I have unreasonable expectations of the performance I should be able to achieve. If anyone could offer insight or advice, I''d greatly appreciate it. -- Horace Demmink RackSphere Inc.
On Jul 19, 2010, at 2:38 PM, Horace Demmink wrote:> Hello, > > I''m working on building a iSCSI storage server to use as the backend for virtual servers. I am far more familiar with FreeBSD and Linux, but want to use OpenSolaris for this project because of Comstar & ZFS. My plan was to have a 24 2TB Hitachi SATA drives connected via SAS expanders to two OpenSolaris servers working in a fallover setup. These servers would provide iSCSI shares the the virtual hosting servers over a 10Gb infiniband connection. > > At this point I''m having trouble getting the performance I was expecting out of ZFS. I can get between 100-130MB/s sequential read and write from my drives individually using dd. And I can maintain those speeds up to a point reading and writing to them in parallel to where I can get about 1GB/s raw to the drives until the bus becomes a bottleneck. However, when I create a zpool, either striped, raidz, or raidz2, my sequential write speeds end up around the write speed of a single drive. I wasn''t expecting this, using the same hardware running linux and an md-raid/lvm2 setup I can easily achieve 1100MB/s+ sequential reads and 600MB/s+ sequential writes. > > Shouldn''t I, in theory, be able to achieve similar results with zfs?Yes. In the local dd case, you will be testing the memory bandwidth, too.> I''m at a loss where to go from here to troubleshoot and tune this setup, or if I have unreasonable expectations of the performance I should be able to achieve. If anyone could offer insight or advice, I''d greatly appreciate it.There are many, many, many different ways to configure, create load, and measure performance. Perhaps you can start with explaining how you did those things. -- richard -- Richard Elling richard at nexenta.com +1-760-896-4422 Enterprise class storage for everyone www.nexenta.com
I believe, I''m in a very similar situation than yours. Have you figured something out? -- This message posted from opensolaris.org
Horace - I''ve run more tests and come up with basically the exact same numbers you do. On Opensolaris - I get about the same from my drives (140MBps) and hit a 1GBps (almost exactly) top end system bottle neck when pushing data to all drives. However, if I give ZFS more than one drive (mirror, stripe, raidz) it cannot go beyond the performance of a single drive on reads. (However writes seem to perform much better - but that could be due to the ZIL and/or caching. I''ve seen writes jump beyond 900MBps for a pool) I should point out that I tried SVM (solaris volume manager - comparable to mdraid on linux) and SVM was able to push 1GBps during initialization but couldn''t go beyond what ZFS was capable of when doing a dd test. This SVM test was just a quick test before trying Linux since, like Linux, it takes forever to init an SVM device. I''m not very familiar with SVM - so I''m sure tuning could be an issue here - however with the kind of hardware you and I are working with, I would think at a minimum we should expect much better numbers, even without tuning. Unless the Opensolaris code is all tuned for ancient hardware. (Or *gasp* perhaps it''s all tuned for SPARC or AMD) Dunno. I am now installing Linux to test. Would you mind giving me some information on what your Linux distro/configuration is approximately? Our numbers are so similar that I think we may be running into the same issue here - whatever it is. -- This message posted from opensolaris.org
I wonder if this has anything to do with it: http://opensolaris.org/jive/thread.jspa?messageID=33739菋 Anyway, I''ve already blown away my OSOL install to test Linux performance - so I can''t test ZFS at the moment. -- This message posted from opensolaris.org