freebsd stable - May 2009 - current zfs tuning in RELENG_7 (AMD64) suggestions ?

I gave the AMD64 version of 7.2 RC2 a spin and all installed as 
expected off the dvd

INTEL S3200SHV MB, Core2Duo, 4G of RAM

In the past it had been suggested that for zfs tuning, something like

vm.kmem_size_max="1073741824"
vm.kmem_size="1073741824"
vfs.zfs.prefetch_disable=1

However doing a simple test with bonnie and dd, there does not seem 
to be very much difference in 4 configs.  Am I better off just with 
the defaults ?  The machine is acting as anoffline storage site, so a 
steady stream of data via rsync/ssh

The writes are all within the normal variance of the tests except for 
b).  Is there anything else that should be tuned ? Not that I am 
looking for any "magic bullets" but I just want to run this backup 
server as best as possible

               -------Sequential Output-------- ---Sequential Input-- 
--Random--
               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- 
--Seeks---
           MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
a        5000  98772 54.7 153111 31.5 100015 21.1 178730 85.3 368782 
32.5 161.6  0.6
b        5000 101271 57.9 154765 31.5  61325 13.9 176741 84.6 372477 
32.8 149.3  0.6
c        5000 102331 57.1 159559 29.5 105767 17.4 144410 63.8 299317 
19.9 167.9  0.6
d        5000 107308 58.6 175004 32.4 117926 18.8 143657 63.4 305126 
20.0 167.6  0.6


a) defaults
b) vm.kmem_size_max="1073741824" vm.kmem_size="1073741824"
c) vfs.zfs.prefetch_disable=1 plus b)
d) vfs.zfs.zil_disable="1" plus c) plus b)



Results tend to fluctuate a bit.

offsitetmp# dd if=/dev/zero of=/tank1/test bs=2048k count=1000
1000+0 records in
1000+0 records out
2097152000 bytes transferred in 10.016818 secs (209363092 bytes/sec)
offsitetmp#
offsitetmp# dd if=/dev/zero of=/tank1/test bs=2048k count=1000
1000+0 records in
1000+0 records out
2097152000 bytes transferred in 10.733547 secs (195382943 bytes/sec)
offsitetmp#

Drives are raidz

ad1: 1430799MB <Seagate ST31500341AS CC1H> at ata3-master SATA300
ad2: 1430799MB <Seagate ST31500341AS CC1H> at ata4-master SATA300
ad3: 1430799MB <Seagate ST31500341AS CC1H> at ata5-master SATA300
ad4: 1430799MB <Seagate ST31500341AS CC1H> at ata6-master SATA300

on ich9

         ---Mike



--------------------------------------------------------------------
Mike Tancsa,                                      tel +1 519 651 3400
Sentex Communications,                            mike@sentex.net
Providing Internet since 1994                    www.sentex.net
Cambridge, Ontario Canada                         www.sentex.net/mike

> In the past it had been suggested that for zfs tuning, something like
>
> vm.kmem_size_max="1073741824"
> vm.kmem_size="1073741824"
> vfs.zfs.prefetch_disable=1
>
> However doing a simple test with bonnie and dd, there does not seem 
> to be very much difference in 4 configs.  Am I better off just with 
The tuning isn't there to improve performance, it's there to prevent
the box going titus due to a panic when the ARC gets too big, and
you are missing the mian one, which is to limit the size of the ARC.
On recent versions of BSD (and you are running 7.2, so thats fine) then
the defaults for kmem size are fine, but you still need something like this:

vfs.zfs.arc_max="256M"

In there to stop the ARC growing. thats the only tuning I have on
my 4 gig machine, which takes a steady stream of data and is used
for taking backup snapshots. ZFS is excellent, and for me is perfectly
stable, to the point where I am starting to roll it out to production
machines, with the above tuning.

-pete.

On Fri, May 01, 2009 at 04:42:09PM -0400, Mike Tancsa wrote:

  I gave the AMD64 version of 7.2 RC2 a spin and all installed as 
  expected off the dvd
  
  INTEL S3200SHV MB, Core2Duo, 4G of RAM
  
<snip>

  The writes are all within the normal variance of the tests except for 
  b).  Is there anything else that should be tuned ? Not that I am 
  looking for any "magic bullets" but I just want to run this backup 
  server as best as possible
  
                 -------Sequential Output-------- ---Sequential Input-- 
  --Random--
                 -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- 
  --Seeks---
             MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec
%CPU
  a        5000  98772 54.7 153111 31.5 100015 21.1 178730 85.3 368782 
  32.5 161.6  0.6
  b        5000 101271 57.9 154765 31.5  61325 13.9 176741 84.6 372477 
  32.8 149.3  0.6
  c        5000 102331 57.1 159559 29.5 105767 17.4 144410 63.8 299317 
  19.9 167.9  0.6
  d        5000 107308 58.6 175004 32.4 117926 18.8 143657 63.4 305126 
  20.0 167.6  0.6


<begin brain dump>
  
You might want to try running gstat -I 100000 during the test to see   
how fast each drive flushes the cache from ram and if there are any
disks slower than the others.  I've found some cards or slots cause 
drives to perform slower than other drives in the system, dragging down
performance of the raid to the slowest drive(s).  Individual performance
testing of the drives outside of the raid might reveal something too,
even just to find out what the maximum sequential speed of one drive is
so you know that 4x(speed) is the best to hope for in raid tests.

ZFS tends to cache heavily at the start of each write and you will 
probably see it bounce between no IO and furious writes, until the ram 
cache fills up more and it has no choice but to write almost constantly.  
This can affect the results between runs.  I would recommend a larger 
count= that results in a test run of 30-60 seconds at least.

Additionally, try other zfs raid types such as mirror and stripe to see 
if raidz is acting as an unexpectedly large bottleneck, I've found its 
serial write speed usually leaves something to be desired.  Even if the 
other raid levels won't work realistically in the long run, its useful 
to raise the bar to find out what extra performance your IO setup can 
push. It could be useful to compare with gstripe and graid3 for further 
hardware performance evaluation.  On the other hand, if you can 
read/write data faster than your network connection can push, you're 
probably at a workable level.

Also, I believe that zfs uses a cluster size up to 128k (queueing 
multiple writes if it can, depending on the disk subsystem) so I think 
the computer has to do extra work if you are giving it bs=2048k since 
zfs will have to cut that into 16 pieces, sending one piece to each 
drive.  You might try bs=512k or bs=128k for example to see if this has 
a positive effect.  In a traditional raid5 setup, I've found I get head 
over heals the best performance when my bs= is the same size as the raid 
stripe size multiplied by the number of drives, and this gets weird when 
you have an odd number of drives because your optimum write size might 
be something like 768k which probably no application is going to produce 
:)  Also it makes it hard to optimize UFS for a larger stripe size when
the cluster sizes are generally limited to 16k such as in Solaris.

  Results tend to fluctuate a bit.
  
  offsitetmp# dd if=/dev/zero of=/tank1/test bs=2048k count=1000
  1000+0 records in
  1000+0 records out
  2097152000 bytes transferred in 10.016818 secs (209363092 bytes/sec)
  offsitetmp#
  offsitetmp# dd if=/dev/zero of=/tank1/test bs=2048k count=1000
  1000+0 records in
  1000+0 records out
  2097152000 bytes transferred in 10.733547 secs (195382943 bytes/sec)
  offsitetmp#
  
  Drives are raidz
  
  ad1: 1430799MB <Seagate ST31500341AS CC1H> at ata3-master SATA300
  ad2: 1430799MB <Seagate ST31500341AS CC1H> at ata4-master SATA300
  ad3: 1430799MB <Seagate ST31500341AS CC1H> at ata5-master SATA300
  ad4: 1430799MB <Seagate ST31500341AS CC1H> at ata6-master SATA300
  
  on ich9
  
           ---Mike
  
  
  
  --------------------------------------------------------------------
  Mike Tancsa,                                      tel +1 519 651 3400
  Sentex Communications,                            mike@sentex.net
  Providing Internet since 1994                    www.sentex.net
  Cambridge, Ontario Canada                         www.sentex.net/mike
  
  _______________________________________________
  freebsd-stable@freebsd.org mailing list
  http://lists.freebsd.org/mailman/listinfo/freebsd-stable
  To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"