Hello,
I've got a new Supermicro X58 system with an Intel Core i7 930 with 6
GB ram that is not performing nearly as fast as it should in many ways
(compiling, network transfers). ?To give an example, it has been
building the gcc44 port for about 10 hours now and at the same time
rsync'ing from a Linux box on the same Gigabit network is only getting
throughput of between 10-25 MB/sec. ?When I did a buildkernel for
8-STABLE, it took 17 hours! My investigations have shown inhibited
performance on compute, network and storage activities.
In the BIOS, I have played with a few settings and some actually made
it worse. ?What I have done now is disabled Hyperthreading and Turbo
Boost.
Thanks in advance for any ideas. Here's some system info and stats:
bryce@tahiti[~]>uname -a
FreeBSD tahiti.bryce.net 8.0-STABLE FreeBSD 8.0-STABLE #0: Wed Apr 28
10:53:37 CDT 2010
root@tahiti.bryce.net:/usr/obj/usr/src/sys/GENERIC ?amd64
bryce@tahiti[~]>cat /boot/loader.conf
ahci_load="YES"
ichsmb_load="YES"
smb_load="YES"
coretemp_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:system"
hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1
bryce@tahiti[~]>cat /etc/sysctl.conf
kern.timecounter.hardware=HPET
bryce@tahiti[~]>vmstat 1
?procs ? ? ?memory ? ? ?page ? ? ? ? ? ? ? ? ? ?disks ? ? faults ? ? ? ? cpu
?r b w ? ? avm ? ?fre ? flt ?re ?pi ?po ? ?fr ?sr ad0 ad1 ? in ? sy
cs us sy id
?5 0 0 ? 1068M ?3478M ? 572 ? 1 ? 1 ? 0 ? 862 ? 0 ? 0 ? 0 9370 16514
16157 71 22 ?7
?5 0 0 ? 1068M ?3478M ? ? 2 ? 0 ? 0 ? 0 ? ? 0 ? 0 ? 0 ? 0 8008 14504
11716 81 17 ?2
?5 0 0 ? 1068M ?3478M ? ? 0 ? 0 ? 0 ? 0 ? ? 0 ? 0 ? 0 ? 0 12429 22323
18125 77 23 ?0
?5 0 0 ? 1068M ?3478M ? ? 0 ? 0 ? 0 ? 0 ? ? 0 ? 0 ? 0 ? 0 12348 22125
17988 73 27 ?0
bryce@tahiti[~]>vmstat -i
interrupt ? ? ? ? ? ? ? ? ? ? ? ? ?total ? ? ? rate
irq1: atkbd0 ? ? ? ? ? ? ? ? ? ? ? ?9291 ? ? ? ? ?0
irq17: fwohci0 ? ? ? ? ? ? ? ? ? ? ? ? 1 ? ? ? ? ?0
cpu0: timer ? ? ? ? ? ? ? ? ? ? 75416246 ? ? ? 2000
irq256: em0 ? ? ? ? ? ? ? ? ? ?137590284 ? ? ? 3649
irq257: em0 ? ? ? ? ? ? ? ? ? ?206367605 ? ? ? 5473
irq260: em0 ? ? ? ? ? ? ? ? ? ? ? ? ? ?1 ? ? ? ? ?0
irq266: ahci0 ? ? ? ? ? ? ? ? ? ?9892384 ? ? ? ?262
cpu2: timer ? ? ? ? ? ? ? ? ? ? 75415653 ? ? ? 2000
cpu3: timer ? ? ? ? ? ? ? ? ? ? 75415702 ? ? ? 2000
cpu1: timer ? ? ? ? ? ? ? ? ? ? 75415561 ? ? ? 2000
Total ? ? ? ? ? ? ? ? ? ? ? ? ?655522728 ? ? ?17385
bryce@tahiti[~]>netstat -I em0 -h 1
? ? ? ? ? ?input ? ? ? ? ?(em0) ? ? ? ? ? output
? packets ?errs idrops ? ? ?bytes ? ?packets ?errs ? ? ?bytes colls
? ? ?7.7K ? ? 0 ? ? 0 ? ? ? ?11M ? ? ? 7.2K ? ? 0 ? ? ? 475K ? ? 0
? ? ?8.1K ? ? 0 ? ? 0 ? ? ? ?12M ? ? ? 7.4K ? ? 0 ? ? ? 491K ? ? 0
? ? ?7.8K ? ? 0 ? ? 0 ? ? ? ?11M ? ? ? 7.2K ? ? 0 ? ? ? 476K ? ? 0
bryce@tahiti[/usr/adm]>iostat 1
? ? ? tty ? ? ? ? ? ?ada0 ? ? ? ? ? ? ada1 ? ? ? ? ? ? ada2 ? ? ? ? ? ? cpu
?tin ?tout ?KB/t tps ?MB/s ? KB/t tps ?MB/s ? KB/t tps ?MB/s ?us ni sy in id
? 0 ? 108 22.35 ? 3 ?0.07 ?20.61 ? 3 ?0.07 ?58.60 ? 0 ?0.00 ?71 ?0 ?4 17 ?7
? 0 ? 222 64.00 ? 1 ?0.06 ?128.00 ? 1 ?0.12 ? 0.00 ? 0 ?0.00 ?87 ?0 ?2 11 ?0
Dmesg output:
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-STABLE #0: Wed Apr 28 10:53:37 CDT 2010
root@tahiti.bryce.net:/usr/obj/usr/src/sys/GENERIC amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM) i7 CPU 930 @ 2.80GHz (2786.02-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a
Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x98e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 6446645248 (6148 MB)
avail memory = 6169243648 (5883 MB)
ACPI APIC Table: <021210 APIC1519>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 6
ioapic0: Changing APIC ID to 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <SMCI > on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, cbf00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0x7B, should be
0x74 (20100331/tbutils-354)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci3: <ACPI PCI bus> on pcib3
vgapci0: <VGA-compatible display> port 0xcc00-0xcc7f mem
0xfa000000-0xfaffffff,0xd0000000-0xdfffffff,0xce000000-0xcfffffff irq
16 at device 0.0 on pci3
pci3: <multimedia, HDA> at device 0.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.0 (no
driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no
driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no
driver attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no
driver attached)
pci0: <base peripheral> at device 22.0 (no driver attached)
pci0: <base peripheral> at device 22.1 (no driver attached)
pci0: <base peripheral> at device 22.2 (no driver attached)
pci0: <base peripheral> at device 22.3 (no driver attached)
pci0: <base peripheral> at device 22.4 (no driver attached)
pci0: <base peripheral> at device 22.5 (no driver attached)
pci0: <base peripheral> at device 22.6 (no driver attached)
pci0: <base peripheral> at device 22.7 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0xbc00-0xbc1f
irq 16 at device 26.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801JI (ICH10) USB controller USB-D> on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0xb880-0xb89f
irq 21 at device 26.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801JI (ICH10) USB controller USB-E> on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0xb800-0xb81f
irq 19 at device 26.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus2: <Intel 82801JI (ICH10) USB controller USB-F> on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem
0xf9fde000-0xf9fde3ff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> on ehci0
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0xdc00-0xdc1f
mem 0xfbce0000-0xfbcfffff,0xfbcdc000-0xfbcdffff irq 16 at device 0.0
on pci5
em0: Using MSIX interrupts with 5 vectors
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
em0: Ethernet address: 00:30:48:fb:8a:06
pcib6: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
em1: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0xec00-0xec1f
mem 0xfbde0000-0xfbdfffff,0xfbddc000-0xfbddffff irq 17 at device 0.0
on pci6
em1: Using MSIX interrupts with 5 vectors
em1: [ITHREAD]
em1: [ITHREAD]
em1: [ITHREAD]
em1: [ITHREAD]
em1: [ITHREAD]
em1: Ethernet address: 00:30:48:fb:8a:07
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0xb480-0xb49f
irq 23 at device 29.0 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x2f00
usbus4: <Intel 82801JI (ICH10) USB controller USB-A> on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0xb400-0xb41f
irq 19 at device 29.1 on pci0
uhci4: [ITHREAD]
uhci4: LegSup = 0x2f00
usbus5: <Intel 82801JI (ICH10) USB controller USB-B> on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xb080-0xb09f
irq 18 at device 29.2 on pci0
uhci5: [ITHREAD]
uhci5: LegSup = 0x2f00
usbus6: <Intel 82801JI (ICH10) USB controller USB-C> on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem
0xf9fdc000-0xf9fdc3ff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus7: EHCI version 1.0
usbus7: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> on ehci1
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
fwohci0: <Texas Instruments TSB43AB22/A> mem
0xfbefb800-0xfbefbfff,0xfbefc000-0xfbefffff irq 17 at device 8.0 on
pci7
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:30:48:00:00:01:7a:2d
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x2718000
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:30:48:01:7a:2d
fwe0: Ethernet address: 02:30:48:01:7a:2d
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:30:48:00:00:01:7a:2d @ 0xfffe00000000,
S400, maxrec 2048
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH10 AHCI SATA controller> port
0xa480-0xa487,0xb000-0xb003,0xac00-0xac07,0xa880-0xa883,0xa800-0xa81f
mem 0xf9fda000-0xf9fda7ff irq 19 at device 31.2 on pci0
ahci0: [ITHREAD]
ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich5: [ITHREAD]
ichsmb0: <Intel 82801JI (ICH10) SMBus controller> port 0x400-0x41f mem
0xf9fd8000-0xf9fd80ff irq 18 at device 31.3 on pci0
ichsmb0: [ITHREAD]
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
ZFS filesystem version 3
ZFS storage pool version 14
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD1600BEKT-00F3T0 11.01A11> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD1600BEKT-00F3T0 11.01A11> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <SAMSUNG HD203WI 1AN10002> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <SAMSUNG HD203WI 1AN10002> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
ada5: <Hitachi HDS722020ALA330 JKAOA28A> ATA-8 SATA 2.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
Root mount waiting for: usbus7 usbus3
Root mount waiting for: usbus7 usbus3
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
Trying to mount root from zfs:system