Hi everyone. I posted this on the OmniOS list as well, but hoping someone here can help. We are running our final tests with our servers before we send them off to the DC. We have tested various ZIL devices, including Zeusram and DDRdrive. I wanted to get some comments on what we should expect when it comes to w/s in iostat for the ZIL. I''ve heard conflicting results. So far, running gnu dd with 512b and oflag=sync, the most we can get is 8k iops on the zil device. I even tried with some SSDs (Crucial M4, Vertex 4) and it is the same result. They all cap at about 8k iops. The only outlier is the DDRDrive, which hovers around 6.9k iops. To rule things out, I tried Nexenta, OmniOS and Solaris 11, all with mostly the same result. In addition to the log device, the pool is made up of 6 mirrored pairs of 2TB Western Digital drives. No cache devices added yet. Is this the expected result? Should I be pushing for more? In IRC I was told that I should be able to get 12k no problem. We are running NFS in a heavily used environment with millions of very small files, so low latency counts. In case it helps, the hardware is a Supermicro chassis (24 disks), X9-DRi-F motherboard, and LSI 9211-8i connected to the SM backplane. Networking is 10gbe (x510-DA2) directly connected via SFP+ twin-axial. Thanks, Chris
On Sat, 11 Aug 2012, Chris Nagele wrote:> > So far, running gnu dd with 512b and oflag=sync, the most we can get is 8k > iops on the zil device. I even tried with some SSDs (Crucial M4,If this is one dd program running, then all you are measuring is "sequential" IOPS. That is, the next I/O will not start until the previous one has returned. What you want to test is threaded IOPS with some number of threads (each one represents a client) running. You can use iozone to effectively test that. This command runs with 16 threads and 8k blocks with a 2GB file: iozone -m -t 16 -T -O -r 8k -o -s 2G If you ''dd'' from /dev/zero then the test is meaningless since zfs is able to compress zeros. If you ''dd'' from /dev/random then the test is meaningless since the random generator is slow.> Is this the expected result? Should I be pushing for more? In IRC I > was told that I should be able to get 12k no problem. We are running > NFS in a heavily used environment with millions of very small files, > so low latency counts.Your test method is not valid. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
> This command runs with 16 threads and 8k blocks with a 2GB file: > > iozone -m -t 16 -T -O -r 8k -o -s 2GThat''s fair, and I realize dd is not an ideal test. I did run multiple instances of dd as well, only getting to about 8.8k iops. The reality is that no benchmark is ideal, which is why our main benchmarks have been using sample sets of production data. My main reason for this post was to find out if I am getting expected (or usual) results compared to others. I say this because: * Chris from DDRDrive was able to get better results with the X1 (well over 9500 iops) on an X8DTH-6F motherboard, using a slightly different setup. * Some people in OmniOS IRC mentioned I should be seeing better results in iostat. When it comes to a zil device I know that both DDRDrive and Zeusram are the better options that exist. I want to make sure we are getting the most out of both of them. I will run the iozone test tomorrow and post the results. -Chris