Hi all,
among many other things I recently restarted benchmarking ZFS over NFS3
performance between X4500 (host) and Linux clients. I''ve just iozone
quite a while ago and am still a bit at a loss understanding the
results. The automatic mode is pretty ok (and generates nice 3D plots
for the people higher up the ladder), but someone gave a hint to use
multiple threads for testing the ops/s and here I''m a bit at a loss how
to understand the results and if the values are reasonable or not.
Here is the current example - can anyone with deeper knowledge tell me
if these are reasonable values to start with?
Thanks a lot
Carsten
Iozone: Performance Test of File I/O
Version $Revision: 3.315 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby
Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin
Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.
Run began: Wed Jan 7 09:31:49 2009
Multi_buffer. Work area 16777216 bytes
OPS Mode. Output is in operations per second.
Record Size 8 KB
SYNC Mode.
File size set to 4194304 KB
Command line used: ../iozone3_315/src/current/iozone -m -t 8 -T
-O -r 8k -o -s 4G iozone
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 8 threads
Each thread writes a 4194304 Kbyte file in 8 Kbyte records
Children see throughput for 8 initial writers = 4925.20 ops/sec
Parent sees throughput for 8 initial writers = 4924.65 ops/sec
Min throughput per thread = 615.61 ops/sec
Max throughput per thread = 615.69 ops/sec
Avg throughput per thread = 615.65 ops/sec
Min xfer = 524219.00 ops
Children see throughput for 8 rewriters = 4208.45 ops/sec
Parent sees throughput for 8 rewriters = 4208.42 ops/sec
Min throughput per thread = 525.88 ops/sec
Max throughput per thread = 526.22 ops/sec
Avg throughput per thread = 526.06 ops/sec
Min xfer = 523944.00 ops
Children see throughput for 8 readers = 11986.99 ops/sec
Parent sees throughput for 8 readers = 11986.46 ops/sec
Min throughput per thread = 1481.13 ops/sec
Max throughput per thread = 1512.71 ops/sec
Avg throughput per thread = 1498.37 ops/sec
Min xfer = 513361.00 ops
Children see throughput for 8 re-readers = 12017.70 ops/sec
Parent sees throughput for 8 re-readers = 12017.22 ops/sec
Min throughput per thread = 1486.72 ops/sec
Max throughput per thread = 1520.35 ops/sec
Avg throughput per thread = 1502.21 ops/sec
Min xfer = 512761.00 ops
Children see throughput for 8 reverse readers = 25741.62 ops/sec
Parent sees throughput for 8 reverse readers = 25735.91 ops/sec
Min throughput per thread = 3141.50 ops/sec
Max throughput per thread = 3282.11 ops/sec
Avg throughput per thread = 3217.70 ops/sec
Min xfer = 501956.00 ops
Children see throughput for 8 stride readers = 1434.73 ops/sec
Parent sees throughput for 8 stride readers = 1434.71 ops/sec
Min throughput per thread = 122.51 ops/sec
Max throughput per thread = 297.87 ops/sec
Avg throughput per thread = 179.34 ops/sec
Min xfer = 215638.00 ops
Children see throughput for 8 random readers = 529.83 ops/sec
Parent sees throughput for 8 random readers = 529.83 ops/sec
Min throughput per thread = 55.63 ops/sec
Max throughput per thread = 101.03 ops/sec
Avg throughput per thread = 66.23 ops/sec
Min xfer = 288717.00 ops
Children see throughput for 8 mixed workload = 352.39 ops/sec
Parent sees throughput for 8 mixed workload = 147.72 ops/sec
Min throughput per thread = 19.24 ops/sec
Max throughput per thread = 72.45 ops/sec
Avg throughput per thread = 44.05 ops/sec
Min xfer = 139211.00 ops
Children see throughput for 8 random writers = 153.29 ops/sec
Parent sees throughput for 8 random writers = 152.62 ops/sec
Min throughput per thread = 19.08 ops/sec
Max throughput per thread = 19.24 ops/sec
Avg throughput per thread = 19.16 ops/sec
Min xfer = 519969.00 ops
iozone test complete.
On Thu, 8 Jan 2009, Carsten Aulbert wrote:> for the people higher up the ladder), but someone gave a hint to use > multiple threads for testing the ops/s and here I''m a bit at a loss how > to understand the results and if the values are reasonable or not.I will admit that some research is required to understand what is meant by "Parent" and "Children". It seems that "Parent" takes an extra hit by communicating with the "Children".> Here is the current example - can anyone with deeper knowledge tell me > if these are reasonable values to start with?Everything depends on what you are planning do with your NFS access. For example, the default blocksize for zfs is 128K. My example tests performance when doing I/O with small 8K blocks (like a database), which will severely penalize zfs configured for 128K blocks. While NFS writes are synchronous, most NFS I/O is sequential reads and writes of bulk data without much random access. This means that typical NFS I/O will produce larger reads and writes which work ok with ZFS''s default configuration. The main penalty for NFS will be for when doing small operations like creating/deleting files, or changing file attributes. My experience with iozone is that it refuses to run on an NFS client of a Solaris server using ZFS since it performs a test and then refuses to work since it says that the filesystem is not implemented correctly. Commenting a line of code in iozone will get over this hurdle. This seems to be a religious issue with the iozone maintainer. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Hi Bob. Bob Friesenhahn wrote:>> Here is the current example - can anyone with deeper knowledge tell me >> if these are reasonable values to start with? > > Everything depends on what you are planning do with your NFS access. For > example, the default blocksize for zfs is 128K. My example tests > performance when doing I/O with small 8K blocks (like a database), which > will severely penalize zfs configured for 128K blocks. > [...]My plans don''t count in here, I need to optimize what the users want and they don''t have a clue what they will do in 6 months from now, so I guess all detailed planning will fail anyway and I''m just searching for the one size fits almost all...> > My experience with iozone is that it refuses to run on an NFS client of > a Solaris server using ZFS since it performs a test and then refuses to > work since it says that the filesystem is not implemented correctly. > Commenting a line of code in iozone will get over this hurdle. This > seems to be a religious issue with the iozone maintainer.Interesting, I''ve been running this on a Linux client accessing a ZFS file system from one of our Thumpers without any source modifications and problems. Cheers Carsten
On Thu, 8 Jan 2009, Carsten Aulbert wrote:>> >> My experience with iozone is that it refuses to run on an NFS client of >> a Solaris server using ZFS since it performs a test and then refuses to >> work since it says that the filesystem is not implemented correctly. >> Commenting a line of code in iozone will get over this hurdle. This >> seems to be a religious issue with the iozone maintainer. > > Interesting, I''ve been running this on a Linux client accessing a ZFS > file system from one of our Thumpers without any source modifications > and problems.I think that the problem only occurs when the client is also Solaris. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/