Greetings! We are setting up a new server using 8.2-PRERELEASE (as of
12/12/10) and we are encountering performance issues with ZFS. The
system boots from a ZFS volume consisting of two Western Digital 2TB
RE4 drives in a ZFS mirror. The rest of the system consists of an
Intel-based Tyan motherboard, 2 x Xeon 5520, and 12GB ECC ram. I have
attached the dmesg output to this email as a text file for more
hardware details.
The performance degradation occurs when copying large amounts of data
to or from the ZFS volume. It happens both when copying from the
network onto the volume, and when copying from the volume back onto
itself. When copying a large (22GB) file, the disk throughput
according to `zpool iostat -v` is only about 30-35 MB/s, but yet the
system is extremely slow interactively. SSH-ing into the machine takes
a good 5-10 seconds just to get a password prompt, and running simple
commands like `ls` and `top` takes several seconds to respond.
Compression is enabled for some volumes in the pool, but not for any of
the volumes that have been copied to or from. CPU load stays very low
(<5%) during all of this.
We have tried tuning some of the ZFS paramters. The default for
vfs.zfs.arc_max was about 11GB, but we have tried setting it as low as
400MB with no change. vfs.numvnodes has typically been below 10000
during all of this, but even such, we have toyed with the
kern.maxvnodes parameter (ranging from 10000 to 500000) with no effect.
The only thing that had any real effect was setting
vfs.zfs.txg.write_limit_override to 500000000 or so, which helped
slightly, but the system was still very slow to respond.
When not under IO load the system is very fast, running a buildworld
in 13 minutes. Unfortunately, I am not sure whether the issue existed
under 8.1-RELEASE because we did not copy any large files before it was
updated. I've attached the outputs of `zpool status` and `zpool iostat
-v` below:
pool: zroot
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror ONLINE 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
errors: No known data errors
pool used avail read write read write
------------- ----- ----- ----- ----- ----- -----
zroot 240G 1.58T 0 1 6.40K 3.18K
mirror 240G 1.58T 0 1 6.40K 3.18K
gpt/disk0 - - 0 0 5.64K 3.21K
gpt/disk1 - - 0 0 4.75K 3.21K
------------- ----- ----- ----- ----- ----- -----
I greatly appreciate any input!
-------------- next part --------------
cam: using minimum scsi_delay (100ms)
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-PRERELEASE #0: Sun Dec 12 16:06:48 EST 2010
root@blah:/usr/obj/usr/src/sys/XEON amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2266.76-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a
Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 12884901888 (12288 MB)
avail memory = 12370169856 (11797 MB)
ACPI APIC Table: <031710 APIC1617>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
cpu8 (AP): APIC ID: 16
cpu9 (AP): APIC ID: 17
cpu10 (AP): APIC ID: 18
cpu11 (AP): APIC ID: 19
cpu12 (AP): APIC ID: 20
cpu13 (AP): APIC ID: 21
cpu14 (AP): APIC ID: 22
cpu15 (AP): APIC ID: 23
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: <031710 XSDT1617> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, bff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0xAD, should be 0xAA
(20101013/tbutils-354)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
cpu12: <ACPI CPU> on acpi0
cpu13: <ACPI CPU> on acpi0
cpu14: <ACPI CPU> on acpi0
cpu15: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci10: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci9: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci8: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 8.0 on pci0
pci7: <PCI bus> on pcib4
pcib5: <PCI-PCI bridge> at device 9.0 on pci0
pci6: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 10.0 on pci0
pci5: <PCI bus> on pcib6
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no driver
attached)
pci0: <base peripheral> at device 22.0 (no driver attached)
pci0: <base peripheral> at device 22.1 (no driver attached)
pci0: <base peripheral> at device 22.2 (no driver attached)
pci0: <base peripheral> at device 22.3 (no driver attached)
pci0: <base peripheral> at device 22.4 (no driver attached)
pci0: <base peripheral> at device 22.5 (no driver attached)
pci0: <base peripheral> at device 22.6 (no driver attached)
pci0: <base peripheral> at device 22.7 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0x9400-0x941f irq
16 at device 26.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801JI (ICH10) USB controller USB-D> on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0x9480-0x949f irq
21 at device 26.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801JI (ICH10) USB controller USB-E> on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0x9800-0x981f irq
19 at device 26.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus2: <Intel 82801JI (ICH10) USB controller USB-F> on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem
0xfbcf6000-0xfbcf63ff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> on ehci0
pcib7: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci4: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib8
em0: <Intel(R) PRO/1000 Network Connection 7.1.8> port 0xec00-0xec1f mem
0xfbee0000-0xfbefffff,0xfbedc000-0xfbedffff irq 16 at device 0.0 on pci3
em0: Using MSIX interrupts with 3 vectors
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
em0: Ethernet address: 00:e0:81:ba:bf:e7
pcib9: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci2: <ACPI PCI bus> on pcib9
em1: <Intel(R) PRO/1000 Network Connection 7.1.8> port 0xdc00-0xdc1f mem
0xfbde0000-0xfbdfffff,0xfbddc000-0xfbddffff irq 17 at device 0.0 on pci2
em1: Using MSIX interrupts with 3 vectors
em1: [ITHREAD]
em1: [ITHREAD]
em1: [ITHREAD]
em1: Ethernet address: 00:e0:81:ba:bf:e8
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0x9880-0x989f irq
23 at device 29.0 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x2f00
usbus4: <Intel 82801JI (ICH10) USB controller USB-A> on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0x9c00-0x9c1f irq
19 at device 29.1 on pci0
uhci4: [ITHREAD]
uhci4: LegSup = 0x2f00
usbus5: <Intel 82801JI (ICH10) USB controller USB-B> on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xa000-0xa01f irq
18 at device 29.2 on pci0
uhci5: [ITHREAD]
uhci5: LegSup = 0x2f00
usbus6: <Intel 82801JI (ICH10) USB controller USB-C> on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem
0xfbcf8000-0xfbcf83ff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus7: EHCI version 1.0
usbus7: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> on ehci1
pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib10
vgapci0: <VGA-compatible display> port 0xcc00-0xcc7f mem
0xfb000000-0xfb7fffff,0xfafe0000-0xfaffffff irq 16 at device 0.0 on pci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH10 SATA300 controller> port
0xac00-0xac07,0xa880-0xa883,0xa800-0xa807,0xa480-0xa483,0xa400-0xa40f,0xa080-0xa08f
irq 19 at device 31.2 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci1: <Intel ICH10 SATA300 controller> port
0xbc00-0xbc07,0xb880-0xb883,0xb800-0xb807,0xb480-0xb483,0xb400-0xb40f,0xb080-0xb08f
irq 19 at device 31.5 on pci0
atapci1: [ITHREAD]
ata4: <ATA channel 0> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci1
ata5: [ITHREAD]
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
qpi0: <QPI system bus> on motherboard
pcib11: <QPI Host-PCI bridge> pcibus 255 on qpi0
pci255: <PCI bus> on pcib11
pcib12: <QPI Host-PCI bridge> pcibus 254 on qpi0
pci254: <PCI bus> on pcib12
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
est8: <Enhanced SpeedStep Frequency Control> on cpu8
p4tcc8: <CPU Frequency Thermal Control> on cpu8
est9: <Enhanced SpeedStep Frequency Control> on cpu9
p4tcc9: <CPU Frequency Thermal Control> on cpu9
est10: <Enhanced SpeedStep Frequency Control> on cpu10
p4tcc10: <CPU Frequency Thermal Control> on cpu10
est11: <Enhanced SpeedStep Frequency Control> on cpu11
p4tcc11: <CPU Frequency Thermal Control> on cpu11
est12: <Enhanced SpeedStep Frequency Control> on cpu12
p4tcc12: <CPU Frequency Thermal Control> on cpu12
est13: <Enhanced SpeedStep Frequency Control> on cpu13
p4tcc13: <CPU Frequency Thermal Control> on cpu13
est14: <Enhanced SpeedStep Frequency Control> on cpu14
p4tcc14: <CPU Frequency Thermal Control> on cpu14
est15: <Enhanced SpeedStep Frequency Control> on cpu15
p4tcc15: <CPU Frequency Thermal Control> on cpu15
ZFS filesystem version 4
ZFS storage pool version 15
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x5d4 offMax=0x878
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ad4: 1907729MB <WDC WD2003FYYS-02W0B0 01.01D01> at ata2-master UDMA100
SATA 3Gb/s
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
ad6: 1907729MB <WDC WD2003FYYS-02W0B0 01.01D01> at ata3-master UDMA100
SATA 3Gb/s
SMP: AP CPU #1 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #15 Launched!
Root mount waiting for: usbus7 usbus6 usbus5 usbus4 usbus3 usbus2 usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
Root mount waiting for: usbus7 usbus3
Root mount waiting for: usbus7 usbus3
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
ugen7.2: <American Megatrends Inc.> at usbus7
Trying to mount root from zfs:zroot
vboxnet0: Ethernet address: 0a:00:27:00:00:00
em1: link state changed to UP