freebsd stable - Jan 2010 - ZFS performance degradation over time

Hello all,

I posted a thread about ZFS performance issues in the General section of the
forums and there were other people having the same issue, so I thought that
it might be helpful to send the details to the mailing list as well. The
thread can be found at:
http://forums.freebsd.org/showthread.php?p=59680&posted=1#post59680

My original post:

---
I'm having problems with ZFS performance. When my system comes up,
read/write speeds are excellent (testing with dd if=/dev/zero
of=/tank/bigfile and dd if=/tank/bigfile of=/dev/null); I get at least
100MB/s on both reads and writes, and I'm happy with that.

The longer the system is up, the worse my performance gets. Currently my
system has been up for 4 days, and read/write performance is down to about
10MB/s at best.

The system is only accessed by 3 clients: myself, my roommate, and our HTPC.
Usually, only one client will be doing anything at a time, so it is not
under heavy load or anything.

*Software:*
 Code:

FreeBSD leviathan 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21
15:02:08 UTC 2009

root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64

The following apps are running and touching data on the zpool:

   - rTorrent - read and write, usually active, not doing much for reads
   (not doing any major seeding)
   - SABnzbd+ - write only, not always active
   - Lighttpd - running ruTorrent (web interface for rTorrent); nothing else
   - samba - all of our clients are running Windows, so we use samba to
   network-mount the zpool


*Hardware:*

   - AMD Athlon II X2 250 Dual Core Processor Socket AM3 3.0GHZ
   - Gigabyte MA790GP-UD4H AMD790GX ATX AM2+/AM3 Sideport 2PCI-E Sound GBLAN
   HDMI CrossFireX Motherboard
   - Corsair XMS2 TWIN2X4096-6400C5 4GB DDR2 2X2GB
   - Supermicro AOC-USASLP-L8I LSI 1068E 8-PORT RAID 0/1/10 Uio SATA/SAS
   Controller W/ 16MB Low Profile
   - *8x* Western Digital WD15EADS Caviar Green 1.5TB SATA 32MB Cache 3.5IN


*ZFS setup:*
I have the 1.5TB drives in one RAIDZ pool. All 8 drives are connected to the
Supermicro L8I controller. The controller is set to 'disabled', so it
isn't
doing anything with the drives except presenting them to the system
untouched. (So I'm really only using it as an expansion card, for the extra
ports).
 Code:

[root@leviathan ~]# zpool status
  pool: tank
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0

            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0

            da7     ONLINE       0     0     0

errors: No known data errors

Any suggestions as to what might be causing the performance to degrade with
system uptime? If I missed anything or more information is needed, please
let me know. Thanks in advance.
---

Also just to note, as suggested by someone in the thread, it's probably not
directly related to system uptime, but instead related to usage - the more
usage, the worse the performance.

Garrett Moore <garrettmoore@gmail.com> writes:
> I'm having problems with ZFS performance. When my system comes up,
> read/write speeds are excellent (testing with dd if=/dev/zero
> of=/tank/bigfile and dd if=/tank/bigfile of=/dev/null); I get at least
> 100MB/s on both reads and writes, and I'm happy with that.
>
> The longer the system is up, the worse my performance gets. Currently my
> system has been up for 4 days, and read/write performance is down to about
> 10MB/s at best.
>
> The system is only accessed by 3 clients: myself, my roommate, and our
HTPC.
> Usually, only one client will be doing anything at a time, so it is not
> under heavy load or anything.
This could be due to the amount of memory available for ZFS caching declining
as time goes on (for reasons that are not entirely clear, though I suspect
it may be due to increasing fragmentation in the kernel memory).  You might
try doing "sysctl kstat.zfs.misc.arcstats.size" repeatedly to monitor
the
amount of memory the ZFS cache is taking up as your system uptime increases.

On 3.1.2010 17:42, Garrett Moore wrote:
> I'm having problems with ZFS performance. When my system comes up,
> read/write speeds are excellent (testing with dd if=/dev/zero
> of=/tank/bigfile and dd if=/tank/bigfile of=/dev/null); I get at least
> 100MB/s on both reads and writes, and I'm happy with that.
> 
> The longer the system is up, the worse my performance gets. Currently my
> system has been up for 4 days, and read/write performance is down to about
> 10MB/s at best.
Are you sure you have isolated the cause to be only the uptime of the
machine? Is there no other change between the runs? E.g. did you stop
all other services and applications on the machine before doing the test
for the second time? Can you create a big file (2x memory size) when the
machine boots, measure the time to read it, then read it again after a
few days when you notice performance problems?