Can anyone suggest why I am getting poor write performance from my nfs setup.
I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards,
4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with
onboard Gb network cards connected to an idle network. The results below show
that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It
improves if I use async but a smbfs mount still beats it. I am using the same
file, source and destinations for all tests. I have tried alternate Network
cards with no resulting benefit.
oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2
1950511+1 records in
1950511+1 records out
998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec)
1950477+74 records in
1950511+1 records out
998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s)
oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt
oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
7619+1 records in
7619+1 records out
998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s)
oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt
oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
7619+1 records in
7619+1 records out
998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s)
oguido# mount -t smbfs //gemini/ufs /mnt
oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
7619+1 records in
7619+1 records out
998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s)
Looking at a systat -v on the destination I see that the nfs test does not
exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t.
For the record I get reads of 22Mb/s without and 77Mb/s with async turned on
for the nfs mount.
A copy of dmesg:
===============
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011
root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping
= 11
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 4294967296 (4096 MB)
avail memory = 3141234688 (2995 MB)
ACPI APIC Table: <MSTEST OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <MSTEST TESTONLY> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, bff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1
(20101013/tbutils-354)
cpu1: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mpt0: <LSILogic SAS/SATA Adapter> port 0x7800-0x78ff mem
0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.18.0
mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt0: 0 Active Volumes (2 Max)
mpt0: 0 Hidden Drive Members (14 Max)
uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0xdc00-0xdc1f irq
16 at device 26.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801H (ICH8) USB controller USB-D> on uhci0
uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0xe000-0xe01f irq
17 at device 26.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801H (ICH8) USB controller USB-E> on uhci1
ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem
0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus2: EHCI version 1.0
usbus2: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> on ehci0
pci0: <multimedia, HDA> at device 27.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci5: <ACPI PCI bus> on pcib2
atapci0: <SiI 3132 SATA300 controller> port 0xac00-0xac7f mem
0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci4: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x9c00-0x9c1f mem
0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4
em0: Using an MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:1b:21:04:ac:11
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci3: <ACPI PCI bus> on pcib4
age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem
0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3
age0: 1280 Tx FIFO, 2364 Rx FIFO
age0: Using 1 MSI messages.
miibus0: <MII bus> on age0
atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0
atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX,
1000baseT-FDX-master, auto
age0: Ethernet address: 00:1a:92:d2:de:cc
age0: [FILTER]
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci2: <ACPI PCI bus> on pcib5
atapci1: <JMicron JMB363 UDMA133 controller> port
0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem
0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2
atapci1: [ITHREAD]
atapci2: <AHCI SATA controller> on atapci1
atapci2: [ITHREAD]
atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported
ata4: <ATA channel 0> on atapci2
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci2
ata5: [ITHREAD]
ata6: <ATA channel 0> on atapci1
ata6: [ITHREAD]
uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0xd480-0xd49f irq
23 at device 29.0 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus3: <Intel 82801H (ICH8) USB controller USB-A> on uhci2
uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0xd800-0xd81f irq
19 at device 29.1 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x2f00
usbus4: <Intel 82801H (ICH8) USB controller USB-B> on uhci3
uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0xd880-0xd89f irq
18 at device 29.2 on pci0
uhci4: [ITHREAD]
uhci4: LegSup = 0x2f00
usbus5: <Intel 82801H (ICH8) USB controller USB-C> on uhci4
ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem
0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus6: EHCI version 1.0
usbus6: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> on ehci1
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> mem
0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at
device 1.0 on pci6
fwohci0: <VIA Fire II (VT6306)> port 0xbc00-0xbc7f mem
0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:11:d8:00:01:39:52:d3
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x1494000
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3
fwe0: Ethernet address: 02:11:d8:39:52:d3
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec
2048
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci3: <Intel ICH8 SATA300 controller> port
0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f
irq 19 at device 31.2 on pci0
atapci3: [ITHREAD]
ata7: <ATA channel 0> on atapci3
ata7: [ITHREAD]
ata8: <ATA channel 1> on atapci3
ata8: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci4: <Intel ICH8 SATA300 controller> port
0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f
irq 19 at device 31.5 on pci0
atapci4: [ITHREAD]
ata9: <ATA channel 0> on atapci4
ata9: [ITHREAD]
ata10: <ATA channel 1> on atapci4
ata10: [ITHREAD]
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
on acpi0
fdc0: [FILTER]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata0: [ITHREAD]
ata1 at port 0x170-0x177,0x376 irq 15 on isa0
ata1: [ITHREAD]
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
acd0: DVDR <HL-DT-ST DVDRAM GH22NS50/TN02> at ata7-slave UDMA100 SATA
1.5Gb/s
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub2: 4 ports with 4 removable, self powered
uhub6: 6 ports with 6 removable, self powered
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
da2 at mpt0 bus 0 scbus0 target 2 lun 0
da2: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <FUJITSU MBC2073RC 0104> Fixed Direct Access SCSI-5 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C)
SMP: AP CPU #1 Launched!
ugen1.2: <NOVATEK> at usbus1
ugen3.2: <Microsoft> at usbus3
ukbd1: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1
ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev
1.10/1.99, addr 2> on usbus3
ums0: 3 buttons and [XYZ] coordinates ID=0
kbd2 at ukbd1
uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1
Trying to mount root from ufs:/dev/da0s1a
--
Daryl Sayers Direct: +612 95525510
Corinthian Engineering Office: +612 95525500
Suite 54, Jones Bay Wharf Fax: +612 95525549
26-32 Pirrama Rd email: daryl@ci.com.au
Pyrmont NSW 2009 Australia www: http://www.ci.com.au
did you try this ? sysctl -w vfs.nfsrv.async=1 On 11/18/11 04:10, Daryl Sayers wrote:> Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) > > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) > > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount. > > > A copy of dmesg: > ===============> > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 > root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> > AMD Features=0x20100000<NX,LM> > AMD Features2=0x1<LAHF> > TSC: P-state invariant > real memory = 4294967296 (4096 MB) > avail memory = 3141234688 (2995 MB) > ACPI APIC Table: <MSTEST OEMAPIC > > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 1 package(s) x 2 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0 <Version 2.0> irqs 0-23 on motherboard > kbd1 at kbdmux0 > cryptosoft0: <software crypto> on motherboard > acpi0: <MSTEST TESTONLY> on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, bff00000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > cpu0: <ACPI CPU> on acpi0 > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354) > cpu1: <ACPI CPU> on acpi0 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 > pci1: <ACPI PCI bus> on pcib1 > mpt0: <LSILogic SAS/SATA Adapter> port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1 > mpt0: [ITHREAD] > mpt0: MPI Version=1.5.18.0 > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > mpt0: 0 Active Volumes (2 Max) > mpt0: 0 Hidden Drive Members (14 Max) > uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0 > uhci0: [ITHREAD] > uhci0: LegSup = 0x2f00 > usbus0: <Intel 82801H (ICH8) USB controller USB-D> on uhci0 > uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0xe000-0xe01f irq 17 at device 26.1 on pci0 > uhci1: [ITHREAD] > uhci1: LegSup = 0x2f00 > usbus1: <Intel 82801H (ICH8) USB controller USB-E> on uhci1 > ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 > ehci0: [ITHREAD] > usbus2: EHCI version 1.0 > usbus2: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> on ehci0 > pci0: <multimedia, HDA> at device 27.0 (no driver attached) > pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 > pci5: <ACPI PCI bus> on pcib2 > atapci0: <SiI 3132 SATA300 controller> port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5 > atapci0: [ITHREAD] > ata2: <ATA channel 0> on atapci0 > ata2: [ITHREAD] > ata3: <ATA channel 1> on atapci0 > ata3: [ITHREAD] > pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0 > pci4: <ACPI PCI bus> on pcib3 > em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4 > em0: Using an MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:1b:21:04:ac:11 > pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 > pci3: <ACPI PCI bus> on pcib4 > age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 > age0: 1280 Tx FIFO, 2364 Rx FIFO > age0: Using 1 MSI messages. > miibus0: <MII bus> on age0 > atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0 > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto > age0: Ethernet address: 00:1a:92:d2:de:cc > age0: [FILTER] > pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 > pci2: <ACPI PCI bus> on pcib5 > atapci1: <JMicron JMB363 UDMA133 controller> port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 > atapci1: [ITHREAD] > atapci2: <AHCI SATA controller> on atapci1 > atapci2: [ITHREAD] > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported > ata4: <ATA channel 0> on atapci2 > ata4: [ITHREAD] > ata5: <ATA channel 1> on atapci2 > ata5: [ITHREAD] > ata6: <ATA channel 0> on atapci1 > ata6: [ITHREAD] > uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0xd480-0xd49f irq 23 at device 29.0 on pci0 > uhci2: [ITHREAD] > uhci2: LegSup = 0x2f00 > usbus3: <Intel 82801H (ICH8) USB controller USB-A> on uhci2 > uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0xd800-0xd81f irq 19 at device 29.1 on pci0 > uhci3: [ITHREAD] > uhci3: LegSup = 0x2f00 > usbus4: <Intel 82801H (ICH8) USB controller USB-B> on uhci3 > uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0xd880-0xd89f irq 18 at device 29.2 on pci0 > uhci4: [ITHREAD] > uhci4: LegSup = 0x2f00 > usbus5: <Intel 82801H (ICH8) USB controller USB-C> on uhci4 > ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 > ehci1: [ITHREAD] > usbus6: EHCI version 1.0 > usbus6: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> on ehci1 > pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0 > pci6: <ACPI PCI bus> on pcib6 > vgapci0: <VGA-compatible display> mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6 > fwohci0: <VIA Fire II (VT6306)> port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 > fwohci0: [ITHREAD] > fwohci0: OHCI version 1.10 (ROM=1) > fwohci0: No. of Isochronous channels is 4. > fwohci0: EUI64 00:11:d8:00:01:39:52:d3 > fwohci0: Phy 1394a available S400, 2 ports. > fwohci0: Link S400, max_rec 2048 bytes. > firewire0: <IEEE1394(FireWire) bus> on fwohci0 > dcons_crom0: <dcons configuration ROM> on firewire0 > dcons_crom0: bus_addr 0x1494000 > fwe0: <Ethernet over FireWire> on firewire0 > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 > fwe0: Ethernet address: 02:11:d8:39:52:d3 > fwip0: <IP over FireWire> on firewire0 > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048 > fwohci0: Initiate bus reset > fwohci0: fwohci_intr_core: BUS reset > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode > isab0: <PCI-ISA bridge> at device 31.0 on pci0 > isa0: <ISA bus> on isab0 > atapci3: <Intel ICH8 SATA300 controller> port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0 > atapci3: [ITHREAD] > ata7: <ATA channel 0> on atapci3 > ata7: [ITHREAD] > ata8: <ATA channel 1> on atapci3 > ata8: [ITHREAD] > pci0: <serial bus, SMBus> at device 31.3 (no driver attached) > atapci4: <Intel ICH8 SATA300 controller> port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0 > atapci4: [ITHREAD] > ata9: <ATA channel 0> on atapci4 > ata9: [ITHREAD] > ata10: <ATA channel 1> on atapci4 > ata10: [ITHREAD] > acpi_button0: <Power Button> on acpi0 > atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart0: [FILTER] > fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > fdc0: [FILTER] > acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppc0: [ITHREAD] > ppbus0: <Parallel port bus> on ppc0 > plip0: <PLIP network interface> on ppbus0 > plip0: [ITHREAD] > lpt0: <Printer> on ppbus0 > lpt0: [ITHREAD] > lpt0: Interrupt-driven port > ppi0: <Parallel I/O> on ppbus0 > pmtimer0 on isa0 > orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 > sc0: <System console> at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 > ata0: [ITHREAD] > ata1 at port 0x170-0x177,0x376 irq 15 on isa0 > ata1: [ITHREAD] > atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 > atkbd0: <AT Keyboard> irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > est0: <Enhanced SpeedStep Frequency Control> on cpu0 > p4tcc0: <CPU Frequency Thermal Control> on cpu0 > est1: <Enhanced SpeedStep Frequency Control> on cpu1 > p4tcc1: <CPU Frequency Thermal Control> on cpu1 > Timecounters tick every 1.000 msec > IPsec: Initialized Security Association Processing. > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) > firewire0: bus manager 0 > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 480Mbps High Speed USB v2.0 > usbus3: 12Mbps Full Speed USB v1.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 12Mbps Full Speed USB v1.0 > usbus6: 480Mbps High Speed USB v2.0 > ugen0.1: <Intel> at usbus0 > uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 > ugen1.1: <Intel> at usbus1 > uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 > ugen2.1: <Intel> at usbus2 > uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 > ugen3.1: <Intel> at usbus3 > uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 > ugen4.1: <Intel> at usbus4 > uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 > ugen5.1: <Intel> at usbus5 > uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 > ugen6.1: <Intel> at usbus6 > uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6 > acd0: DVDR <HL-DT-ST DVDRAM GH22NS50/TN02> at ata7-slave UDMA100 SATA 1.5Gb/s > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub3: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > uhub5: 2 ports with 2 removable, self powered > uhub2: 4 ports with 4 removable, self powered > uhub6: 6 ports with 6 removable, self powered > da1 at mpt0 bus 0 scbus0 target 1 lun 0 > da1: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device > da1: 300.000MB/s transfers > da1: Command Queueing enabled > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da2 at mpt0 bus 0 scbus0 target 2 lun 0 > da2: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device > da2: 300.000MB/s transfers > da2: Command Queueing enabled > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da0 at mpt0 bus 0 scbus0 target 0 lun 0 > da0: <FUJITSU MBC2073RC 0104> Fixed Direct Access SCSI-5 device > da0: 300.000MB/s transfers > da0: Command Queueing enabled > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) > SMP: AP CPU #1 Launched! > ugen1.2: <NOVATEK> at usbus1 > ugen3.2: <Microsoft> at usbus3 > ukbd1: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1 > ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev 1.10/1.99, addr 2> on usbus3 > ums0: 3 buttons and [XYZ] coordinates ID=0 > kbd2 at ukbd1 > uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1 > Trying to mount root from ufs:/dev/da0s1a > >
and if you use zfs also try this zfs set sync=disabled On 11/18/11 04:10, Daryl Sayers wrote:> Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) > > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) > > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount. > > > A copy of dmesg: > ===============> > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 > root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> > AMD Features=0x20100000<NX,LM> > AMD Features2=0x1<LAHF> > TSC: P-state invariant > real memory = 4294967296 (4096 MB) > avail memory = 3141234688 (2995 MB) > ACPI APIC Table: <MSTEST OEMAPIC > > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 1 package(s) x 2 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0 <Version 2.0> irqs 0-23 on motherboard > kbd1 at kbdmux0 > cryptosoft0: <software crypto> on motherboard > acpi0: <MSTEST TESTONLY> on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, bff00000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > cpu0: <ACPI CPU> on acpi0 > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354) > cpu1: <ACPI CPU> on acpi0 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 > pci1: <ACPI PCI bus> on pcib1 > mpt0: <LSILogic SAS/SATA Adapter> port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1 > mpt0: [ITHREAD] > mpt0: MPI Version=1.5.18.0 > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > mpt0: 0 Active Volumes (2 Max) > mpt0: 0 Hidden Drive Members (14 Max) > uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0 > uhci0: [ITHREAD] > uhci0: LegSup = 0x2f00 > usbus0: <Intel 82801H (ICH8) USB controller USB-D> on uhci0 > uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0xe000-0xe01f irq 17 at device 26.1 on pci0 > uhci1: [ITHREAD] > uhci1: LegSup = 0x2f00 > usbus1: <Intel 82801H (ICH8) USB controller USB-E> on uhci1 > ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 > ehci0: [ITHREAD] > usbus2: EHCI version 1.0 > usbus2: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> on ehci0 > pci0: <multimedia, HDA> at device 27.0 (no driver attached) > pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 > pci5: <ACPI PCI bus> on pcib2 > atapci0: <SiI 3132 SATA300 controller> port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5 > atapci0: [ITHREAD] > ata2: <ATA channel 0> on atapci0 > ata2: [ITHREAD] > ata3: <ATA channel 1> on atapci0 > ata3: [ITHREAD] > pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0 > pci4: <ACPI PCI bus> on pcib3 > em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4 > em0: Using an MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:1b:21:04:ac:11 > pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 > pci3: <ACPI PCI bus> on pcib4 > age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 > age0: 1280 Tx FIFO, 2364 Rx FIFO > age0: Using 1 MSI messages. > miibus0: <MII bus> on age0 > atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0 > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto > age0: Ethernet address: 00:1a:92:d2:de:cc > age0: [FILTER] > pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 > pci2: <ACPI PCI bus> on pcib5 > atapci1: <JMicron JMB363 UDMA133 controller> port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 > atapci1: [ITHREAD] > atapci2: <AHCI SATA controller> on atapci1 > atapci2: [ITHREAD] > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported > ata4: <ATA channel 0> on atapci2 > ata4: [ITHREAD] > ata5: <ATA channel 1> on atapci2 > ata5: [ITHREAD] > ata6: <ATA channel 0> on atapci1 > ata6: [ITHREAD] > uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0xd480-0xd49f irq 23 at device 29.0 on pci0 > uhci2: [ITHREAD] > uhci2: LegSup = 0x2f00 > usbus3: <Intel 82801H (ICH8) USB controller USB-A> on uhci2 > uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0xd800-0xd81f irq 19 at device 29.1 on pci0 > uhci3: [ITHREAD] > uhci3: LegSup = 0x2f00 > usbus4: <Intel 82801H (ICH8) USB controller USB-B> on uhci3 > uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0xd880-0xd89f irq 18 at device 29.2 on pci0 > uhci4: [ITHREAD] > uhci4: LegSup = 0x2f00 > usbus5: <Intel 82801H (ICH8) USB controller USB-C> on uhci4 > ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 > ehci1: [ITHREAD] > usbus6: EHCI version 1.0 > usbus6: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> on ehci1 > pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0 > pci6: <ACPI PCI bus> on pcib6 > vgapci0: <VGA-compatible display> mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6 > fwohci0: <VIA Fire II (VT6306)> port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 > fwohci0: [ITHREAD] > fwohci0: OHCI version 1.10 (ROM=1) > fwohci0: No. of Isochronous channels is 4. > fwohci0: EUI64 00:11:d8:00:01:39:52:d3 > fwohci0: Phy 1394a available S400, 2 ports. > fwohci0: Link S400, max_rec 2048 bytes. > firewire0: <IEEE1394(FireWire) bus> on fwohci0 > dcons_crom0: <dcons configuration ROM> on firewire0 > dcons_crom0: bus_addr 0x1494000 > fwe0: <Ethernet over FireWire> on firewire0 > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 > fwe0: Ethernet address: 02:11:d8:39:52:d3 > fwip0: <IP over FireWire> on firewire0 > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048 > fwohci0: Initiate bus reset > fwohci0: fwohci_intr_core: BUS reset > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode > isab0: <PCI-ISA bridge> at device 31.0 on pci0 > isa0: <ISA bus> on isab0 > atapci3: <Intel ICH8 SATA300 controller> port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0 > atapci3: [ITHREAD] > ata7: <ATA channel 0> on atapci3 > ata7: [ITHREAD] > ata8: <ATA channel 1> on atapci3 > ata8: [ITHREAD] > pci0: <serial bus, SMBus> at device 31.3 (no driver attached) > atapci4: <Intel ICH8 SATA300 controller> port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0 > atapci4: [ITHREAD] > ata9: <ATA channel 0> on atapci4 > ata9: [ITHREAD] > ata10: <ATA channel 1> on atapci4 > ata10: [ITHREAD] > acpi_button0: <Power Button> on acpi0 > atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart0: [FILTER] > fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > fdc0: [FILTER] > acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppc0: [ITHREAD] > ppbus0: <Parallel port bus> on ppc0 > plip0: <PLIP network interface> on ppbus0 > plip0: [ITHREAD] > lpt0: <Printer> on ppbus0 > lpt0: [ITHREAD] > lpt0: Interrupt-driven port > ppi0: <Parallel I/O> on ppbus0 > pmtimer0 on isa0 > orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 > sc0: <System console> at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 > ata0: [ITHREAD] > ata1 at port 0x170-0x177,0x376 irq 15 on isa0 > ata1: [ITHREAD] > atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 > atkbd0: <AT Keyboard> irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > est0: <Enhanced SpeedStep Frequency Control> on cpu0 > p4tcc0: <CPU Frequency Thermal Control> on cpu0 > est1: <Enhanced SpeedStep Frequency Control> on cpu1 > p4tcc1: <CPU Frequency Thermal Control> on cpu1 > Timecounters tick every 1.000 msec > IPsec: Initialized Security Association Processing. > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) > firewire0: bus manager 0 > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 480Mbps High Speed USB v2.0 > usbus3: 12Mbps Full Speed USB v1.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 12Mbps Full Speed USB v1.0 > usbus6: 480Mbps High Speed USB v2.0 > ugen0.1: <Intel> at usbus0 > uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 > ugen1.1: <Intel> at usbus1 > uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 > ugen2.1: <Intel> at usbus2 > uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 > ugen3.1: <Intel> at usbus3 > uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 > ugen4.1: <Intel> at usbus4 > uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 > ugen5.1: <Intel> at usbus5 > uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 > ugen6.1: <Intel> at usbus6 > uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6 > acd0: DVDR <HL-DT-ST DVDRAM GH22NS50/TN02> at ata7-slave UDMA100 SATA 1.5Gb/s > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub3: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > uhub5: 2 ports with 2 removable, self powered > uhub2: 4 ports with 4 removable, self powered > uhub6: 6 ports with 6 removable, self powered > da1 at mpt0 bus 0 scbus0 target 1 lun 0 > da1: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device > da1: 300.000MB/s transfers > da1: Command Queueing enabled > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da2 at mpt0 bus 0 scbus0 target 2 lun 0 > da2: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device > da2: 300.000MB/s transfers > da2: Command Queueing enabled > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da0 at mpt0 bus 0 scbus0 target 0 lun 0 > da0: <FUJITSU MBC2073RC 0104> Fixed Direct Access SCSI-5 device > da0: 300.000MB/s transfers > da0: Command Queueing enabled > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) > SMP: AP CPU #1 Launched! > ugen1.2: <NOVATEK> at usbus1 > ugen3.2: <Microsoft> at usbus3 > ukbd1: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1 > ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev 1.10/1.99, addr 2> on usbus3 > ums0: 3 buttons and [XYZ] coordinates ID=0 > kbd2 at ukbd1 > uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1 > Trying to mount root from ufs:/dev/da0s1a > >
On Thursday, November 17, 2011 10:10:27 pm Daryl Sayers wrote:> > Can anyone suggest why I am getting poor write performance from my nfssetup.> I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus motherboards,> 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results belowshow> that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using thesame> file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec)(98Mb/s)> > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec)(15Mb/s)> > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs/mnt> oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec)(19Mb/s)> > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec)(33Mb/s)> > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount.I don't know if it will help with your performance, but I have some patches to allow the NFS server to cluster writes. You can try www.freebsd.org/~jhb/patches/nfs_server_cluster.patch. I've tested it on 8, but it should probably apply fine to 9. -- John Baldwin
Daryl Sayers <daryl@ci.com.au> writes:> Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit.[...]> Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount.On an UFS filesystem you get NFS writes with the same size as the filesystem blocksize. So an easy way to improve performance is to create a filesystem with larger blocks. I accidentally found this out when I had two NFS exported filesystems from the same box with 16K and 64K blocksizes respectively. (Larger blocksize also tremendously improves the performance of UFS snapshots!) Bengt