Hello,
I'm running ZFS V28 with 8.2-RELEASE. Its a stock system patched with
the following
http://people.freebsd.org/~mm/patches/zfs/v28/releng-8.2-zfsv28-20110616.patch.xz
https://www.illumos.org/attachments/292/txg.c.patch
The zfsv28 patch did have a failed hunk in
/usr/src/sys/cddl/compat/opensolaris/sys/sysmacros.h but it appeared
that was because the file already existed identically in my src tree.
I have not upgraded the zpool, it remains at v15 in case I need to
drop back to the old kernel. I've removed all zfs related
loader/sysctl options I had in from v15.
What I'm seeing is strange write behavior. When I first set it up I'd
see writes in small bursts of 100 megs or so every 5 seconds. Now I'm
seeing small writes and reads are stalling.
This is what it looks like when IO is horrible:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
san 3.97T 8.66T 269 228 6.54M 21.9M
san 3.97T 8.66T 2 142 131K 1.18M
san 3.97T 8.66T 952 0 52.7M 0
san 3.97T 8.66T 527 149 24.7M 1.22M
san 3.97T 8.66T 2 181 192K 1.35M
san 3.97T 8.66T 1.17K 0 47.9M 0
san 3.97T 8.66T 452 247 23.6M 3.40M
san 3.97T 8.66T 702 268 23.8M 3.38M
san 3.97T 8.66T 0 0 0 0
san 3.97T 8.66T 686 255 30.7M 3.39M
san 3.97T 8.66T 379 229 19.3M 2.82M
san 3.97T 8.66T 94 0 5.59M 0
san 3.97T 8.66T 1004 243 52.4M 3.32M
san 3.97T 8.66T 3 270 195K 3.48M
san 3.97T 8.66T 250 0 14.6M 0
san 3.97T 8.66T 439 250 15.5M 3.42M
san 3.97T 8.66T 1 257 128K 3.65M
It gets better for short periods and looks like this
san 3.95T 8.68T 1004 55 53.1M 360K
san 3.95T 8.68T 341 533 10.6M 42.4M
san 3.95T 8.68T 783 0 43.5M 0
san 3.95T 8.68T 409 374 17.1M 35.5M
san 3.95T 8.68T 423 117 18.4M 3.83M
san 3.95T 8.68T 1.01K 136 43.2M 2.26M
san 3.95T 8.68T 447 454 16.8M 35.4M
san 3.95T 8.68T 991 0 46.3M 0
san 3.95T 8.68T 420 394 21.8M 32.9M
san 3.95T 8.68T 1.16K 241 54.4M 6.00M
san 3.95T 8.68T 575 0 20.4M 0
san 3.95T 8.68T 284 328 9.77M 30.3M
san 3.95T 8.68T 663 203 33.6M 5.15M
san 3.95T 8.68T 319 0 10.0M 0
san 3.95T 8.68T 318 453 10.0M 24.0M
san 3.95T 8.68T 1.06K 299 30.9M 25.0M
I also see very high CPU usage on a few ZFS related threads
38 root -8 - 0K 7536K zio->i 3 64:23 25.20%
{txg_thread_enter}
0 root -16 0 0K 2704K sm->sm 0 66:48 21.39%
{zio_write_issue_}
0 root -16 0 0K 2704K sm->sm 1 66:36 18.16%
{zio_write_issue_}
0 root -16 0 0K 2704K sm->sm 1 67:50 13.77%
{zio_write_issue_}
0 root -16 0 0K 2704K CPU3 3 67:28 12.16% {zio_write_issue_}
These would be a 1-2% each prior. Any help tracking this down would be
greatly appreciated.
Thanks,
--
Dave Cundiff
System Administrator
A2Hosting, Inc
http://www.a2hosting.com