Alexey Popov
2007-Nov-19 05:32 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
Hi.
I have a large pool of web backends (Apache + mod_php5) with
2 x Xeon 3.2GHz processors and 2 x Xeon 5120 dual-core processors. The
workload is mostly CPU-bound. I'm using 6-STABLE-amd64 and also tried
7-STABLE.
Now I'm trying to use new hardware with 2 x Xeon 5320 (quad-core), but
it can not work under the same load as dual-core. It shows up to 80%
system CPU load in top:
last pid: 3850; load averages: 22.51, 19.75, 12.18
up
0+02:41:41 15:54:40
71 processes: 25 running, 46 sleeping
CPU states: 9.5% user, 0.0% nice, 79.9% system, 1.2% interrupt, 9.5%
idle
Mem: 464M Active, 26M Inact, 75M Wired, 108K Cache, 51M Buf, 3323M Free
Swap: 2048M Total, 2048M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
3729 www 1 118 0 99400K 50840K CPU4 1 1:55 23.24% httpd
3751 www 1 -4 0 96312K 46380K RUN 0 1:53 22.85% httpd
3732 www 1 118 0 96028K 47752K RUN 5 1:57 22.36% httpd
3744 www 1 -4 0 96036K 47756K RUN 7 1:54 22.27% httpd
3747 www 1 -4 0 96600K 48536K CPU5 1 1:53 22.07% httpd
3734 www 1 -4 0 96360K 46388K RUN 7 1:57 22.02% httpd
3758 www 1 -4 0 96004K 48208K CPU7 2 1:52 21.97% httpd
3763 www 1 -4 0 94824K 46800K RUN 1 1:55 21.88% httpd
3745 www 1 -4 0 97304K 47540K RUN 1 1:53 21.78% httpd
3750 www 1 -4 0 98672K 49920K RUN 0 1:54 21.73% httpd
3606 www 1 -4 0 96428K 48012K RUN 5 2:31 21.68% httpd
3762 www 1 -4 0 96008K 47332K CPU1 1 1:54 21.63% httpd
3757 www 1 -4 0 96368K 48020K RUN 0 1:53 21.48% httpd
3730 www 1 -4 0 96648K 48096K CPU7 5 1:57 21.44% httpd
3746 www 1 -4 0 96440K 48384K RUN 0 1:55 21.44% httpd
3756 www 1 -4 0 95972K 46984K RUN 1 1:55 21.44% httpd
3748 www 1 118 0 99756K 52056K select 6 1:55 21.39% httpd
3743 www 1 117 0 98432K 49984K select 6 1:52 21.34% httpd
3754 www 1 -4 0 98756K 50464K RUN 2 1:52 21.24% httpd
731 www 1 117 0 99232K 51028K select 4 2:37 21.14% httpd
3761 www 1 117 0 97384K 49136K RUN 0 1:53 21.14% httpd
Here's the output from 2xdual-core backend running under the same load
and with the same software:
last pid: 52086; load averages: 3.84, 4.15, 4.11 up 42+01:14:01
16:06:10
45 processes: 8 running, 37 sleeping
CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0%
idle
Mem: 2264M Active, 704M Inact, 277M Wired, 181M Cache, 214M Buf, 384M Free
Swap: 2048M Total, 1412K Used, 2046M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
52067 www 1 111 0 96476K 46868K CPU2 0 0:19 40.45% httpd
52069 www 1 110 0 99M 51464K CPU3 0 0:13 37.62% httpd
52064 www 1 4 0 97168K 47636K sbwait 2 0:18 35.27% httpd
52078 www 1 112 0 96348K 45752K RUN 1 0:04 34.13% httpd
52083 www 1 112 0 96892K 42384K RUN 0 0:02 32.37% httpd
52058 www 1 108 0 96156K 47228K CPU0 2 0:21 32.32% httpd
52071 www 1 108 0 99M 50528K RUN 0 0:09 31.80% httpd
52074 www 1 111 0 99M 51180K select 1 0:07 29.35% httpd
787 nobody 1 4 0 2079M 2077M kqread 1 20.7H 3.61%
memcached
660 root 1 96 0 18120K 2108K select 0 25:17 0.00% snmpd
795 root 6 20 0 14652K 2992K kserel 0 13:28 0.00%
bacula-fd
I tried Linux and it works much better than old (2 x dual-core)
backends. It handles 2 times more requests than FreeBSD on the old
backends. So there's a real scalability problem in FreeBSD. The more
processors it have the more CPU time it consumes.
Also I faced the same problem moving heavily loaded MySQL-server to new
hardware. That time I thought that the problem is in the mysql-server
itself and I had to install Linux.
See in attach: mutex statistics for quad-core system and dmesg and
vmstat for dual- and quad-core systems.
What can I do to make FreeBSD run faster on many-CPU systems???
With best regards,
Alexey Popov
-------------- next part --------------
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.3-PRERELEASE #0: Mon Nov 19 12:50:10 MSK 2007
llp@hydrogen.loadup.ru:/usr/obj/usr/src/sys/xxxSMP-amd64-HWPMC
module_register: module accf_http already exists!
Module accf_http failed to register: 17
module_register: module mfi_linux already exists!
Module mfi_linux failed to register: 17
ACPI APIC Table: <INTEL S5000PAL>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5320 @ 1.86GHz (1873.10-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
real memory = 5905580032 (5632 MB)
avail memory = 4096229376 (3906 MB)
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
lapic0: Forcing LINT1 to edge trigger
kbd1 at kbdmux0
module_register_init: MOD_LOAD (accf_http, 0xffffffff8033a0e0,
0xffffffff807a9720) error 17
acpi0: <INTEL S5000PAL> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
cpu4: <ACPI CPU> on acpi0
acpi_throttle4: <ACPI CPU Throttling> on cpu4
acpi_throttle4: failed to attach P_CNT
device_attach: acpi_throttle4 attach returned 6
cpu5: <ACPI CPU> on acpi0
acpi_throttle5: <ACPI CPU Throttling> on cpu5
acpi_throttle5: failed to attach P_CNT
device_attach: acpi_throttle5 attach returned 6
cpu6: <ACPI CPU> on acpi0
acpi_throttle6: <ACPI CPU Throttling> on cpu6
acpi_throttle6: failed to attach P_CNT
device_attach: acpi_throttle6 attach returned 6
cpu7: <ACPI CPU> on acpi0
acpi_throttle7: <ACPI CPU Throttling> on cpu7
acpi_throttle7: failed to attach P_CNT
device_attach: acpi_throttle7 attach returned 6
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xca2,0xca3,0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci2
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 2.0 on pci2
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.2> port
0x2020-0x203f mem 0xb8820000-0xb883ffff,0xb8400000-0xb87fffff irq 18 at device
0.0 on pci5
em0: Ethernet address: 00:04:23:e1:30:ea
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.2> port
0x2000-0x201f mem 0xb8800000-0xb881ffff,0xb8000000-0xb83fffff irq 19 at device
0.1 on pci5
em1: Ethernet address: 00:04:23:e1:30:eb
pcib6: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci6: <ACPI PCI bus> on pcib6
pcib7: <PCI-PCI bridge> at device 3.0 on pci0
pci7: <PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> at device 0.0 on pci10
pci11: <PCI bus> on pcib11
mfi0: <LSI MegaSAS 1064R> mem 0xb8a00000-0xb8a0ffff,0xb8c00000-0xb8c1ffff
irq 18 at device 14.0 on pci11
mfi0: Megaraid SAS driver Ver 2.00
mfi0: 280 (248793038s/0x0020/0) - Shutdown command received from host
mfi0: 281 (4278190080s/0x0020/0) - PCI 0x041000 0x04411 0x041000 0x041004:
Firmware initialization started (PCI ID 0411/1000/1004/1000)
mfi0: 282 (4278190080s/0x0020/0) - Type 18: Firmware version 1.03.60-0255
mfi0: 283 (4278190118s/0x0008/0) - Battery temperature is normal
mfi0: 284 (4278190118s/0x0008/0) - Battery Present
mfi0: 285 (4278190156s/0x0002/0) - PD 08(e0/s8) event: Inserted: PD 08(e0/s8)
mfi0: 286 (4278190156s/0x0002/0) - Type 29: Inserted: PD 08(e0/s8) Info:
enclPd=ffff, scsiType=0, portMap=01, sasAddr=500000e016917972,0000000000000000
mfi0: 287 (4278190156s/0x0002/0) - PD 09(e0/s9) event: Inserted: PD 09(e0/s9)
mfi0: 288 (4278190156s/0x0002/0) - Type 29: Inserted: PD 09(e0/s9) Info:
enclPd=ffff, scsiType=0, portMap=02, sasAddr=500000e016916272,0000000000000000
mfi0: 289 (248793151s/0x0020/0) - Adapter ticks 248793151 elapsed 77s: Time
established as 11/19/07 13:12:31; (77 seconds since power on)
pcib12: <PCI-PCI bridge> at device 0.2 on pci10
pci12: <PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci13: <ACPI PCI bus> on pcib13
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0x3080-0x309f irq 23 at device
29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x3060-0x307f irq 22 at device
29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x3040-0x305f irq 23 at device
29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x3020-0x303f irq 22 at device
29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xb8d00400-0xb8d007ff irq
23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <EHCI (generic) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
pci14: <display, VGA> at device 12.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30b0-0x30bf irq 20 at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
atapci1: <Intel 63XXESB2 SATA300 controller> port
0x30c8-0x30cf,0x30e4-0x30e7,0x30c0-0x30c7,0x30e0-0x30e3,0x30a0-0x30af mem
0xb8d00000-0xb8d003ff irq 20 at device 31.2 on pci0
ata2: <ATA channel 0> on atapci1
ata3: <ATA channel 1> on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A, console
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xcdfff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
IP Filter: v4.1.28 initialized. Default = pass all, Logging = enabled
ipfw2 initialized, divert enabled, rule-based forwarding disabled, default to
accept, logging unlimited
acd0: DVDR <Optiarc DVD RW AD-7543A/1-00> at ata0-slave UDMA33
mfid0: <MFI Logical Disk> on mfi0
mfid0: 139236MB (285155328 sectors) RAID volume '' is optimal
lapic2: Forcing LINT1 to edge trigger
SMP: AP CPU #2 Launched!
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
lapic3: Forcing LINT1 to edge trigger
SMP: AP CPU #3 Launched!
lapic4: Forcing LINT1 to edge trigger
SMP: AP CPU #4 Launched!
lapic7: Forcing LINT1 to edge trigger
SMP: AP CPU #7 Launched!
lapic5: Forcing LINT1 to edge trigger
SMP: AP CPU #5 Launched!
lapic6: Forcing LINT1 to edge trigger
SMP: AP CPU #6 Launched!
Trying to mount root from ufs:/dev/mfid0s1a
em0: Hardware Initialization Failed
em0: Unable to initialize the hardware
em0: link state changed to UP
1 users Load 24.97 24.71 22.97 Nov 19 16:25
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 487264 19648 621692 33304 3400752 count
All 535596 20936 5003244 37628 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt cow 16839 total
27 1 39 137k 3390 33k 2490 313 2519 2519 zfod sio0 irq4
2519 ozfod ata0 irq14
78.0%Sys 1.1%Intr 13.1%User 0.0%Nice 7.8%Idle 100%ozfod 832 em0 mfi0 1
| | | | | | | | | | | daefr uhci0 uhci
=======================================+>>>>>>
prcfr 1999 cpu0: time
37 dtbuf 2979 totfr 2000 cpu2: time
Namei Name-cache Dir-cache 100000 desvn react 2000 cpu1: time
Calls hits % hits % 5491 numvn pdwak 2000 cpu3: time
51775 51775 100 3686 frevn pdpgs 2002 cpu4: time
intrn 2002 cpu7: time
Disks mfid0 80080 wire 2002 cpu5: time
KB/t 0.00 472368 act 2002 cpu6: time
tps 0 26720 inact
MB/s 0.00 108 cache
%busy 0 3400644 free
59296 buf
debug.mutex.prof.stats:
max total count avg cnt_hold cnt_lock name
14830 12696030 1840283 6 622811 981441
/usr/src/sys/kern/kern_lock.c:168 (lockbuilder mtxpool)
1152 67642 25424 2 0 0
/usr/src/sys/kern/kern_descrip.c:1962 (filedesc structure)
2343 1526816 557240 2 13043 481687
/usr/src/sys/kern/vfs_subr.c:1997 (vnode interlock)
1285 71570 25363 2 1 1
/usr/src/sys/kern/kern_descrip.c:1983 (sleep mtxpool)
328 7238962 2175947 3 6165708 7064072
/usr/src/sys/kern/kern_synch.c:220 (lockbuilder mtxpool)
1209 71686 25363 2 0 0
/usr/src/sys/kern/kern_descrip.c:1984 (filedesc structure)
1553 2069549 806581 2 555036 175103
/usr/src/sys/kern/kern_lock.c:532 (lockbuilder mtxpool)
14836 10286955 769265 13 36020 28870
/usr/src/sys/kern/vfs_subr.c:2119 (vnode interlock)
272 51743 23159 2 19 62
/usr/src/sys/kern/kern_conf.c:61 (cdev)
4189 8785941 350672 25 36714 13130
/usr/src/sys/kern/vfs_subr.c:2236 (vnode interlock)
11 100 28 3 0 2
/usr/src/sys/kern/tty.c:2093 (process lock)
3630 270809 103261 2 688 2045
/usr/src/sys/ufs/ufs/ufs_vnops.c:133 (vnode interlock)
243 22329 2023 11 48 9491
/usr/src/sys/kern/sys_generic.c:1147 (sellck)
2185 278665 102786 2 1002 450
/usr/src/sys/ufs/ufs/ufs_vnops.c:408 (vnode interlock)
200 2425 38 63 4 2750
/usr/src/sys/kern/kern_conf.c:329 (Giant)
353 326270 104708 3 3037 24148
/usr/src/sys/kern/vfs_mount.c:438 (struct mount mtx)
2529 228837 81898 2 0 3
/usr/src/sys/kern/kern_descrip.c:2112 (sleep mtxpool)
2307 1583215 257982 6 13461 23127
/usr/src/sys/kern/vfs_vnops.c:806 (vnode interlock)
24007 185216 346 535 161 23
/usr/src/sys/kern/kern_sysctl.c:1313 (Giant)
851 67462 24917 2 0 0
/usr/src/sys/kern/kern_descrip.c:1376 (filedesc structure)
1316 69326 24925 2 2 169
/usr/src/sys/kern/kern_descrip.c:1268 (process lock)
1420 77550 24917 3 0 0
/usr/src/sys/kern/kern_descrip.c:1391 (filedesc structure)
516 66280 24783 2 0 0
/usr/src/sys/kern/vfs_syscalls.c:1007 (filedesc structure)
11467 81199 24783 3 0 0
/usr/src/sys/kern/vfs_syscalls.c:1015 (filedesc structure)
5259 308813 107752 2 0 0
/usr/src/sys/kern/vfs_lookup.c:188 (filedesc structure)
599 2618267 455837 5 40555 177047
/usr/src/sys/kern/vfs_cache.c:356 (Name Cache)
1474 4246486 452087 9 9871 85861
/usr/src/sys/kern/vfs_cache.c:457 (vnode interlock)
2544 847434 317780 2 27777 152916
/usr/src/sys/kern/vfs_subr.c:2020 (vnode interlock)
1888 305492 107752 2 0 0
/usr/src/sys/kern/vfs_lookup.c:195 (filedesc structure)
437 277687 105799 2 11707 18106
/usr/src/sys/kern/vfs_subr.c:2060 (vnode interlock)
63 1325 150 8 14 3
/usr/src/sys/kern/kern_condvar.c:189 (sellck)
13 8322 2322 3 0 0
/usr/src/sys/kern/sys_generic.c:849 (filedesc structure)
451 9400 2670 3 26 39
/usr/src/sys/kern/sys_generic.c:1099 (sellck)
1146 41574 5704 7 62 169
/usr/src/sys/kern/uipc_socket.c:2049 (so_rcv)
1155 89277 5704 15 34 19
/usr/src/sys/kern/uipc_socket.c:2048 (so_snd)
5 27 8 3 0 0
/usr/src/sys/kern/kern_exit.c:139 (process lock)
11 31 8 3 0 0
/usr/src/sys/kern/kern_exit.c:212 (p_peers)
357 4300 493 8 0 46
/usr/src/sys/kern/sys_pipe.c:1345 (pipe mutex)
376 567533 104712 5 3856 3057
/usr/src/sys/kern/vfs_subr.c:343 (struct mount mtx)
8 103 16 6 0 0
/usr/src/sys/kern/subr_eventhandler.c:212 (eventhandler)
8 55 9 6 0 0
/usr/src/sys/kern/subr_eventhandler.c:215 (process_exit)
46 1333 121 11 5 9
/usr/src/sys/kern/kern_conf.c:341 (Giant)
4 23 8 2 0 0
/usr/src/sys/kern/sysv_sem.c:1268 (sem)
4 46 16 2 0 0
/usr/src/sys/kern/kern_exit.c:230 (process_exit)
216 8050 2322 3 0 0
/usr/src/sys/kern/sys_generic.c:872 (filedesc structure)
8 49 8 6 0 0
/usr/src/sys/fs/pseudofs/pseudofs_vncache.c:286 (pfs_vncache)
116 24710 2322 10 807 784
/usr/src/sys/kern/sys_generic.c:771 (sellck)
17 110 8 13 0 0
/usr/src/sys/fs/pseudofs/pseudofs_vncache.c:285 (Giant)
11973 107437 33819 3 0 0
/usr/src/sys/kern/kern_descrip.c:994 (filedesc structure)
2774 117520 37693 3 0 13
/usr/src/sys/kern/kern_sig.c:641 (process lock)
1819 3983950 104533 38 7818 38206
/usr/src/sys/kern/vfs_hash.c:71 (vfs hash)
2753 64202 22718 2 0 0
/usr/src/sys/kern/kern_descrip.c:1021 (filedesc structure)
5 25 8 3 0 0
/usr/src/sys/kern/kern_exit.c:238 (process lock)
5 24 8 3 0 0
/usr/src/sys/kern/kern_descrip.c:1639 (filedesc structure)
1035 4256701 104533 40 78201 60906
/usr/src/sys/kern/vfs_hash.c:79 (vnode interlock)
365 63008 22698 2 953 1293
/usr/src/sys/kern/vfs_mount.c:429 (struct mount mtx)
475 62166 22651 2 1384 496
/usr/src/sys/kern/vfs_vnops.c:902 (struct mount mtx)
5 22 8 2 0 0
/usr/src/sys/kern/kern_descrip.c:1653 (filedesc structure)
10 448 127 3 0 0
/usr/src/sys/kern/kern_event.c:1643 (tty)
833 60258 22565 2 1 8
/usr/src/sys/ufs/ufs/ufs_vnops.c:282 (vnode interlock)
912 1536 44 34 1 2
/usr/src/sys/kern/kern_conf.c:317 (Giant)
5 23 8 2 0 0
/usr/src/sys/kern/kern_descrip.c:1656 (fdesc)
5 23 8 2 0 0
/usr/src/sys/kern/kern_descrip.c:1673 (filedesc structure)
5 24 8 3 0 0
/usr/src/sys/kern/kern_descrip.c:1454 (fdesc)
371 58195 22651 2 875 1016
/usr/src/sys/kern/vfs_vnops.c:1045 (struct mount mtx)
192 7020 2283 3 0 0
/usr/src/sys/kern/sys_generic.c:698 (filedesc structure)
5 23 8 2 0 0
/usr/src/sys/kern/kern_exit.c:285 (p_peers)
502 20605 2283 9 1172 923
/usr/src/sys/kern/sys_generic.c:762 (sellck)
101 98352 10938 8 0 19
/usr/src/sys/dev/em/if_em.c:1464 (em0)
26 26048 7350 3 370 726
/usr/src/sys/net/if.c:2329 (ip_inq)
480 2053 9 228 0 0
/usr/src/sys/amd64/amd64/pmap.c:2748 (pmap)
94 44497 10927 4 148 658
/usr/src/sys/dev/em/if_em.c:1492 (em0)
488 2124 9 236 4 107126
/usr/src/sys/amd64/amd64/pmap.c:2747 (vm page queue mutex)
47 85817 7350 11 2 1
/usr/src/sys/dev/em/if_em.c:4546 (em0)
99 10733 828 12 0 5
/usr/src/sys/vm/vm_map.c:2297 (vm object)
25 41183 12051 3 282 384
/usr/src/sys/net/netisr.c:233 (ip_inq)
1138 57777 18145 3 0 0
/usr/src/sys/vm/vm_object.c:446 (vm object)
360 44497 13232 3 59 196
/usr/src/sys/net/pfil.c:63 (pfil_head_mtx)
297 42979 13232 3 33 154
/usr/src/sys/netinet/ip_fw2.c:156 (IPFW static rules)
1137 45016 13232 3 48 35
/usr/src/sys/netinet/ip_fw2.c:164 (IPFW static rules)
5 268 92 2 0 6
/usr/src/sys/vm/vm_object.c:572 (vm object)
122 730 135 5 2 3
/usr/src/sys/vm/vm_object.c:646 (vm page queue mutex)
183 42700 13228 3 0 0
/usr/src/sys/contrib/ipfilter/netinet/fil.c:2255 (ipf cache rwlock)
6 413 135 3 0 391
/usr/src/sys/vm/vm_object.c:669 (vm object_list)
173 23927 7835 3 0 0
/usr/src/sys/contrib/ipfilter/netinet/fil.c:2282 (ipf cache rwlock)
975 201344 13228 15 0 0
/usr/src/sys/contrib/ipfilter/netinet/fil.c:2562 (ipf filter rwlock)
982 307062 13228 23 639 954
/usr/src/sys/contrib/ipfilter/netinet/fil.c:2432 (ipf filter load/unload mutex)
62 7689 2507 3 0 37
/usr/src/sys/kern/uipc_socket.c:668 (so_snd)
36 42097 13232 3 43 60
/usr/src/sys/net/pfil.c:71 (pfil_head_mtx)
26 86698 27094 3 0 0
/usr/src/sys/netinet/ip_dummynet.c:743 (dummynet)
11 9540 3060 3 0 0
/usr/src/sys/kern/uipc_socket2.c:770 (so_snd)
94 6937 320 21 8 14
/usr/src/sys/vm/vm_object.c:1809 (vm page queue mutex)
760 41669 3060 13 38 1025
/usr/src/sys/netinet/tcp_usrreq.c:630 (tcp)
463 46791 13301 3 15 33
/usr/src/sys/netinet/tcp_output.c:253 (so_snd)
25 10126 2874 3 2 4
/usr/src/sys/netinet/tcp_input.c:2145 (so_snd)
1198 52186 7374 7 7 54
/usr/src/sys/net/route.c:147 (radix node head)
458 52801 7374 7 30 158
/usr/src/sys/net/route.c:197 (rtentry)
252 35852 2510 14 3 14
/usr/src/sys/netinet/tcp_input.c:2375 (so_rcv)
282 220788 6888 32 190 100
/usr/src/sys/netinet/tcp_input.c:625 (tcp)
545 453792 6888 65 701 672
/usr/src/sys/netinet/tcp_input.c:752 (inp)
96 39388 6315 6 35 30
/usr/src/sys/net/route.c:1265 (rtentry)
2461 49157 6315 7 1 4
/usr/src/sys/net/route.c:1281 (rtentry)
72 19734 6315 3 3 36
/usr/src/sys/net/if_ethersubr.c:409 (em0)
5 10 2 5 0 0
/usr/src/sys/kern/kern_proc.c:551 (process lock)
48 18573 6315 2 0 10
/usr/src/sys/dev/em/if_em.c:943 (em0)
19 46 4 11 0 0
/usr/src/sys/kern/kern_proc.c:473 (process group)
1896 73387 6618 11 586 454
/usr/src/sys/dev/em/if_em.c:974 (em0)
32 88 8 11 0 1
/usr/src/sys/kern/kern_exit.c:297 (Giant)
524 20223 6315 3 27 51
/usr/src/sys/netinet/ip_output.c:835 (rtentry)
2661 396213 3060 129 0 578
/usr/src/sys/netinet/tcp_usrreq.c:647 (inp)
5 23 8 2 0 0
/usr/src/sys/kern/kern_exit.c:365 (ktrace)
130 7625 2502 3 0 0
/usr/src/sys/kern/uipc_socket.c:864 (so_snd)
23 84 8 10 0 0
/usr/src/sys/kern/kern_exit.c:364 (process lock)
5 205 78 2 0 0
/usr/src/sys/kern/vfs_subr.c:2090 (vnode interlock)
5 177 60 2 1 0
/usr/src/sys/kern/vfs_vnops.c:1008 (struct mount mtx)
216 2641 374 7 0 3
/usr/src/sys/kern/vfs_bio.c:2387 (vnode interlock)
24 2411 373 6 4 1
/usr/src/sys/sys/buf.h:292 (lockbuilder mtxpool)
7 218 63 3 0 225
/usr/src/sys/ufs/ffs/ffs_softdep.c:4887 (Softdep Lock)
19 388 57 6 2 2
/usr/src/sys/kern/vfs_subr.c:1818 (vnode interlock)
6 497 162 3 0 3
/usr/src/sys/kern/vfs_bio.c:421 (buffer daemon lock)
10 583 183 3 2 6
/usr/src/sys/kern/vfs_bio.c:2259 (vm object)
8 535 145 3 2 3
/usr/src/sys/kern/vfs_bio.c:3403 (vm page queue mutex)
26 1546 145 10 0 0
/usr/src/sys/kern/vfs_bio.c:3402 (vm object)
476 1981 494 4 4 9
/usr/src/sys/kern/kern_lock.c:556 (lockbuilder mtxpool)
352 14144 3973 3 0 0
/usr/src/sys/kern/subr_sleepqueue.c:391 (sigacts)
50 3218 411 7 0 114
/usr/src/sys/kern/vfs_bio.c:1421 (buf queue lock)
450 68483 3973 17 5 10
/usr/src/sys/kern/subr_sleepqueue.c:389 (process lock)
5 182 60 3 0 0
/usr/src/sys/kern/vfs_vnops.c:1067 (struct mount mtx)
12 36 8 4 0 0
/usr/src/sys/kern/kern_exit.c:394 (process lock)
6 25 8 3 0 0
/usr/src/sys/kern/kern_resource.c:932 (sleep mtxpool)
5 30 8 3 0 0
/usr/src/sys/kern/kern_exit.c:468 (sigacts)
5 134 40 3 0 0
/usr/src/sys/kern/kern_sig.c:1719 (sigacts)
41 230 8 28 0 11
/usr/src/sys/kern/kern_exit.c:467 (process lock)
57 314 8 39 0 0
/usr/src/sys/kern/kern_exit.c:444 (process lock)
85 11508 2415 4 254 300
/usr/src/sys/netinet/tcp_input.c:1303 (so_rcv)
274 55363 22570 2 0 401
/usr/src/sys/vm/vnode_pager.c:131 (vm object)
211 60598 24777 2 0 0
/usr/src/sys/kern/vfs_syscalls.c:1017 (filedesc structure)
282 59815 24777 2 0 0
/usr/src/sys/kern/vfs_syscalls.c:1019 (filedesc structure)
250 55202 22576 2 1 23
/usr/src/sys/kern/vfs_syscalls.c:1062 (sleep mtxpool)
1323 189186 22576 8 0 0
/usr/src/sys/kern/vfs_syscalls.c:1061 (filedesc structure)
1451 61760 22836 2 0 10
/usr/src/sys/kern/kern_prot.c:92 (process lock)
1143 65390 23912 2 0 0
/usr/src/sys/kern/vfs_cache.c:720 (filedesc structure)
233 72388 23912 3 2150 3714
/usr/src/sys/kern/vfs_cache.c:801 (Name Cache)
1140 69010 23912 2 0 0
/usr/src/sys/kern/vfs_cache.c:723 (filedesc structure)
4278 609427 23912 25 755 595
/usr/src/sys/kern/vfs_cache.c:719 (Giant)
1016 34871 3676 9 1409 773
/usr/src/sys/kern/kern_condvar.c:301 (sellck)
705 18904 5013 3 0 0
/usr/src/sys/kern/sys_generic.c:1019 (filedesc structure)
1970 22990 5013 4 0 0
/usr/src/sys/kern/sys_generic.c:1043 (filedesc structure)
3820 68563 5013 13 67 93
/usr/src/sys/kern/sys_generic.c:951 (sellck)
17 255 31 8 0 0
/usr/src/sys/kern/subr_trap.c:268 (sigacts)
74 572 31 18 0 0
/usr/src/sys/kern/subr_trap.c:267 (process lock)
487 22678 6407 3 2 1
/usr/src/sys/kern/uipc_syscalls.c:128 (sleep mtxpool)
11 116 31 3 0 0
/usr/src/sys/amd64/amd64/machdep.c:335 (sigacts)
537 73109 6409 11 0 0
/usr/src/sys/kern/uipc_syscalls.c:120 (filedesc structure)
19 344 31 11 0 0
/usr/src/sys/amd64/amd64/machdep.c:334 (process lock)
309 16717 4650 3 107 77
/usr/src/sys/kern/uipc_socket.c:1018 (so_rcv)
119 6216 894 6 4 35
/usr/src/sys/kern/kern_exit.c:598 (process lock)
1318 26635 6040 4 77 139
/usr/src/sys/kern/uipc_socket.c:1265 (so_rcv)
2446 249683 8954 27 780 853
/usr/src/sys/netinet/tcp_usrreq.c:599 (tcp)
37 14088 4481 3 49 1
/usr/src/sys/kern/uipc_socket.c:1423 (so_rcv)
232 11614 3381 3 1 14
/usr/src/sys/kern/sys_generic.c:911 (process lock)
1469 31754 3381 9 19 26
/usr/src/sys/kern/sys_generic.c:942 (sellck)
11 46 8 5 0 0
/usr/src/sys/kern/kern_proc.c:423 (process lock)
17 110 8 13 0 0
/usr/src/sys/kern/kern_proc.c:422 (process group)
29 172 8 21 0 1
/usr/src/sys/kern/kern_proc.c:421 (Giant)
4 12 4 3 0 0
/usr/src/sys/kern/kern_proc.c:571 (session)
18 51 4 12 0 0
/usr/src/sys/kern/kern_proc.c:454 (process group)
29 88 4 22 0 0
/usr/src/sys/kern/kern_proc.c:453 (Giant)
5 25 8 3 0 0
/usr/src/sys/kern/kern_exit.c:680 (process lock)
5 25 8 3 0 0
/usr/src/sys/kern/kern_exit.c:683 (process lock)
5 32 14 2 0 41
/usr/src/sys/kern/kern_resource.c:1149 (sleep mtxpool)
12 64 17 3 0 0
/usr/src/sys/kern/kern_proc.c:1152 (struct pargs.ref)
12 121 31 3 0 0
/usr/src/sys/amd64/amd64/machdep.c:426 (process lock)
7 41 10 4 0 0
/usr/src/sys/kern/tty.c:850 (process lock)
5 15 4 3 0 0
/usr/src/sys/kern/kern_proc.c:260 (process group)
1173 1510 38 39 1 0
/usr/src/sys/kern/kern_conf.c:305 (Giant)
20 1260 388 3 0 0
/usr/src/sys/vm/uma_core.c:2349 (mbuf)
22 8389 2712 3 0 0
/usr/src/sys/nfsserver/nfs_srvsock.c:813 (nfsd_mtx)
24 8009 2712 2 13 41
/usr/src/sys/netinet/tcp_subr.c:1483 (tcp)
6 57 14 4 0 0
/usr/src/sys/kern/kern_time.c:581 (process lock)
64 1914 612 3 0 18
/usr/src/sys/vm/vm_unix.c:81 (process lock)
10 1028 343 2 0 6
/usr/src/sys/vm/vm_object.c:1874 (vm object)
11 549 199 2 0 0
/usr/src/sys/vm/vm_mmap.c:274 (process lock)
165 843 227 3 0 0
/usr/src/sys/vm/vm_mmap.c:1261 (process lock)
109 50814 16766 3 0 6
/usr/src/sys/amd64/amd64/trap.c:552 (process lock)
10 350 111 3 0 0
/usr/src/sys/vm/vm_object.c:229 (vm object_list)
140 50738 17036 2 269 295
/usr/src/sys/vm/vm_fault.c:854 (vm page queue mutex)
3163 202962 17181 11 0 3605
/usr/src/sys/vm/vm_fault.c:297 (vm object)
950 110673 17181 6 372 372
/usr/src/sys/amd64/amd64/pmap.c:2054 (vm page queue mutex)
957 122361 17181 7 0 0
/usr/src/sys/amd64/amd64/pmap.c:2055 (pmap)
1112 120548 39135 3 0 0
/usr/src/sys/amd64/amd64/pmap.c:2856 (pmap)
2069 90833 17789 5 0 0
/usr/src/sys/vm/vm_fault.c:989 (vm object)
793 53945 17181 3 215 211
/usr/src/sys/vm/vm_fault.c:912 (vm page queue mutex)
849 171876 17181 10 0 0
/usr/src/sys/vm/vm_fault.c:911 (vm object)
1439 55321 17181 3 1 2
/usr/src/sys/vm/vm_fault.c:934 (process lock)
1334 54611 16766 3 0 1
/usr/src/sys/amd64/amd64/trap.c:561 (process lock)
15 4925 830 5 2 14
/usr/src/sys/netinet/tcp_timer.c:139 (tcp)
256 103420 830 124 0 23
/usr/src/sys/netinet/tcp_timer.c:145 (inp)
91 11925 504 23 16 6
/usr/src/sys/amd64/amd64/pmap.c:1755 (vm page queue mutex)
92 12410 504 24 0 0
/usr/src/sys/amd64/amd64/pmap.c:1756 (pmap)
14 1353 370 3 0 4
/usr/src/sys/vm/uma_core.c:1888 (mbuf)
55 854 60 14 2 3
/usr/src/sys/netinet/tcp_input.c:373 (so_rcv)
105 13067 1649 7 41 43
/usr/src/sys/kern/kern_timeout.c:258 (Giant)
295 11605 3929 2 9 23
/usr/src/sys/kern/kern_resource.c:1075 (sleep mtxpool)
22 2541 873 2 0 17
/usr/src/sys/kern/vfs_subr.c:2941 (vnode_free_list)
91 2578 872 2 0 0
/usr/src/sys/kern/vfs_subr.c:2913 (vnode_free_list)
53 10898 3931 2 7 7
/usr/src/sys/kern/kern_resource.c:1101 (sleep mtxpool)
17 492 181 2 0 0
/usr/src/sys/kern/vfs_syscalls.c:787 (filedesc structure)
448 4343 894 4 0 0
/usr/src/sys/kern/kern_sig.c:297 (sigacts)
453 10221 894 11 0 0
/usr/src/sys/kern/kern_sig.c:295 (process lock)
245 7085 1081 6 29 2370
/usr/src/sys/geom/geom_io.c:67 (bio queue)
49 2184 56 39 0 0
/usr/src/sys/dev/random/yarrow.c:193 (random reseed)
13 1307 429 3 0 0
/usr/src/sys/vm/vm_object.c:357 (vm object)
8 202 60 3 0 0
/usr/src/sys/ufs/ffs/ffs_vnops.c:675 (process lock)
10 204 60 3 0 2
/usr/src/sys/vm/vnode_pager.c:382 (vm object)
62 912 120 7 0 0
/usr/src/sys/kern/kern_resource.c:855 (process lock)
5 171 63 2 0 0
/usr/src/sys/kern/uipc_socket2.c:324 (so_snd)
229 12029 128 93 2 1
/usr/src/sys/netinet/tcp_usrreq.c:580 (inp)
14 302 100 3 0 1
/usr/src/sys/kern/uipc_socket2.c:192 (so_rcv)
33 430 100 4 0 0
/usr/src/sys/kern/uipc_socket2.c:197 (so_snd)
5 119 41 2 0 0
/usr/src/sys/netinet/in_pcb.c:708 (so_rcv)
15 376 41 9 0 2
/usr/src/sys/netinet/in_pcb.c:707 (accept)
16 54 5 10 0 0
/usr/src/sys/kern/sys_socket.c:121 (process lock)
13 931 271 3 0 0
/usr/src/sys/geom/geom_event.c:185 (GEOM orphanage)
12 914 271 3 0 0
/usr/src/sys/geom/geom_event.c:199 (GEOM orphanage)
6 80 28 2 0 0
/usr/src/sys/net/if.c:1214 (ifnet)
11 154 55 2 0 0
/usr/src/sys/contrib/ipfilter/netinet/fil.c:6587 (ipf token rwlock)
12 153 55 2 0 0
/usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:786 (ipf fragment rwlock)
5 158 55 2 0 0
/usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:799 (ipf IP NAT-Frag rwlock)
6 245 55 4 0 0
/usr/src/sys/contrib/ipfilter/netinet/ip_state.c:3057 (ipf IP state rwlock)
6 217 55 3 0 0
/usr/src/sys/contrib/ipfilter/netinet/ip_nat.c:4464 (ipf IP NAT rwlock)
5 212 55 3 0 0
/usr/src/sys/contrib/ipfilter/netinet/ip_auth.c:563 (ipf IP User-Auth rwlock)
78 2626 55 47 0 1
/usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:855 (ipf filter load/unload
mutex)
61 471 142 3 0 0
/usr/src/sys/kern/kern_descrip.c:748 (sigio lock)
5 319 113 2 0 0
/usr/src/sys/kern/uipc_socket.c:488 (so_rcv)
61 1087 113 9 0 0
/usr/src/sys/kern/uipc_socket.c:487 (accept)
5 316 113 2 0 0
/usr/src/sys/kern/uipc_socket.c:400 (so_snd)
13 1656 610 2 2 0
/usr/src/sys/kern/kern_resource.c:1174 (sleep mtxpool)
89 1174 113 10 0 0
/usr/src/sys/kern/uipc_socket.c:409 (so_snd)
688 1042 123 8 0 0
/usr/src/sys/kern/uipc_socket.c:1477 (so_rcv)
13 369 123 3 0 0
/usr/src/sys/kern/uipc_socket.c:1486 (so_rcv)
21 1143 123 9 0 0
/usr/src/sys/kern/uipc_socket2.c:581 (so_rcv)
5 315 113 2 0 2
/usr/src/sys/kern/uipc_socket.c:253 (so_glabel)
10 324 113 2 0 0
/usr/src/sys/kern/uipc_socket.c:274 (so_glabel)
7 237 65 3 0 1
/usr/src/sys/kern/sys_pipe.c:591 (pipe mutex)
10 172 60 2 0 0
/usr/src/sys/kern/kern_descrip.c:2214 (sleep mtxpool)
20 1490 120 12 0 6
/usr/src/sys/kern/kern_descrip.c:2193 (Giant)
5 172 61 2 0 0
/usr/src/sys/kern/kern_descrip.c:2086 (so_rcv)
5 176 61 2 0 0
/usr/src/sys/kern/kern_descrip.c:2090 (filedesc structure)
11 179 61 2 0 0
/usr/src/sys/kern/uipc_syscalls.c:367 (so_rcv)
19 599 61 9 0 0
/usr/src/sys/kern/uipc_syscalls.c:334 (accept)
7 154 61 2 0 0
/usr/src/sys/kern/kern_descrip.c:951 (sigio lock)
5 160 61 2 0 0
/usr/src/sys/kern/uipc_syscalls.c:389 (sleep mtxpool)
11 762 277 2 0 0
/usr/src/sys/kern/sys_socket.c:144 (so_rcv)
11 777 279 2 0 0
/usr/src/sys/kern/sys_socket.c:170 (so_rcv)
6 777 279 2 0 0
/usr/src/sys/kern/sys_socket.c:173 (so_rcv)
46 841 279 3 0 0
/usr/src/sys/kern/sys_socket.c:176 (so_snd)
5 173 61 2 0 0
/usr/src/sys/kern/uipc_socket.c:524 (so_rcv)
13 350 61 5 1 2
/usr/src/sys/netinet/tcp_usrreq.c:466 (tcp)
15 384 61 6 0 0
/usr/src/sys/netinet/tcp_usrreq.c:472 (inp)
5 153 61 2 0 0
/usr/src/sys/kern/kern_descrip.c:2104 (so_rcv)
15 529 61 8 0 0
/usr/src/sys/kern/kern_descrip.c:2103 (accept)
12 162 60 2 0 0
/usr/src/sys/kern/kern_descrip.c:2200 (sleep mtxpool)
6 169 55 3 0 0
/usr/src/sys/netinet/ip_input.c:1215 (ipqlock)
106 2119 55 38 1 0
/usr/src/sys/netinet/tcp_timer.c:115 (tcp)
5 143 55 2 0 0
/usr/src/sys/netinet/igmp.c:443 (igmp_mtx)
16 171 28 6 0 1
/usr/src/sys/kern/vfs_subr.c:1641 (Syncer mtx)
11 82 28 2 0 0
/usr/src/sys/kern/vfs_subr.c:1717 (Syncer mtx)
28832 48031 94 510 31 2
/usr/src/sys/kern/kern_synch.c:218 (Giant)
6 372 131 2 0 0
/usr/src/sys/kern/uipc_socket2.c:346 (so_rcv)
24 640 187 3 0 0
/usr/src/sys/netinet/tcp_hostcache.c:291 (tcp_hc_entry)
5 184 66 2 0 0
/usr/src/sys/netinet/tcp_subr.c:1792 (so_rcv)
20 616 66 9 0 0
/usr/src/sys/netinet/tcp_subr.c:1791 (accept)
20 2261 368 6 4 360
/usr/src/sys/vm/vnode_pager.c:1199 (vm object)
286 4186 1274 3 21 25
/usr/src/sys/vm/vm_fault.c:346 (vm page queue mutex)
11 811 246 3 1 8
/usr/src/sys/vm/vm_fault.c:136 (vm page queue mutex)
308 7178 314 22 2 3
/usr/src/sys/vm/vm_fault.c:688 (vm object)
32 2582 246 10 0 0
/usr/src/sys/vm/vm_fault.c:792 (vm object)
32 1000 120 8 0 0
/usr/src/sys/vm/vm_fault.c:996 (vm object)
5 20 6 3 0 0
/usr/src/sys/vm/vm_pageout.c:1511 (vm page queue mutex)
24 238 81 2 0 0
/usr/src/sys/kern/kern_kthread.c:181 (process lock)
9 170 27 6 0 0
/usr/src/sys/kern/vfs_bio.c:2040 (buffer daemon lock)
15 195 27 7 0 0
/usr/src/sys/kern/vfs_subr.c:719 (vnode_free_list)
14 1138 264 4 0 0
/usr/src/sys/kern/vfs_subr.c:1977 (vnode interlock)
10 493 185 2 0 0
/usr/src/sys/vm/vnode_pager.c:229 (vm object)
10 498 195 2 0 0
/usr/src/sys/vm/vm_map.c:986 (vm object)
21 1236 192 6 0 0
/usr/src/sys/amd64/amd64/pmap.c:2233 (pmap)
34 2708 192 14 3 4
/usr/src/sys/vm/vm_map.c:1528 (vm page queue mutex)
41 3977 195 20 0 0
/usr/src/sys/vm/vm_map.c:1472 (vm object)
5 427 152 2 0 0
/usr/src/sys/netinet/tcp_subr.c:1649 (rtentry)
22 175 60 2 0 0
/usr/src/sys/kern/uipc_socket2.c:222 (accept)
7 280 108 2 0 0
/usr/src/sys/kern/uipc_socket.c:167 (so_glabel)
5 585 216 2 0 0
/usr/src/sys/kern/uipc_socket2.c:529 (process lock)
51 2961 108 27 0 0
/usr/src/sys/kern/uipc_socket2.c:468 (so_rcv)
60 3576 108 33 0 0
/usr/src/sys/kern/uipc_socket2.c:467 (so_snd)
31 505 102 4 0 0
/usr/src/sys/netinet/in_pcb.c:217 (inp)
5 155 60 2 0 0
/usr/src/sys/kern/uipc_socket2.c:262 (accept)
16 664 76 8 0 0
/usr/src/sys/netinet/tcp_input.c:2975 (so_snd)
17 647 76 8 0 0
/usr/src/sys/netinet/tcp_input.c:2992 (so_rcv)
112 3597 60 59 0 0
/usr/src/sys/netinet/tcp_syncache.c:564 (inp)
548 1815 161 11 0 0
/usr/src/sys/kern/uipc_socket2.c:127 (accept)
13 682 161 4 0 0
/usr/src/sys/kern/uipc_socket2.c:128 (so_rcv)
53 2072 60 34 0 0
/usr/src/sys/netinet/tcp_input.c:867 (inp)
4 149 59 2 0 0
/usr/src/sys/kern/uipc_socket2.c:142 (so_rcv)
653 6607 2202 3 0 0
/usr/src/sys/kern/kern_descrip.c:1761 (filedesc structure)
247 6505 2202 2 0 0
/usr/src/sys/kern/kern_descrip.c:1765 (filedesc structure)
5 95 27 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:735 (Softdep Lock)
16 182 27 6 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:754 (mountlist)
10 446 162 2 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:847 (Softdep Lock)
42 2863 162 17 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:767 (mountlist)
5 75 27 2 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:774 (Softdep Lock)
3 24 8 3 10 5
/usr/src/sys/kern/kern_kse.c:965 (process lock)
5 31 9 3 7 4
/usr/src/sys/kern/kern_kse.c:422 (process lock)
513 7851 2242 3 0 0
/usr/src/sys/amd64/amd64/pmap.c:2255 (pmap)
2063 25059 2242 11 32 40
/usr/src/sys/vm/vm_fault.c:1011 (vm page queue mutex)
5 9 2 4 0 0
/usr/src/sys/vm/uma_core.c:988 (vm object)
5 81 30 2 0 0
/usr/src/sys/vm/uma_core.c:891 (PV ENTRY)
168 898 27 33 0 2
/usr/src/sys/kern/kern_time.c:627 (process lock)
5 147 52 2 0 0
/usr/src/sys/kern/uipc_socket2.c:172 (so_rcv)
5 148 52 2 0 0
/usr/src/sys/kern/uipc_socket2.c:177 (so_snd)
5 146 52 2 0 0
/usr/src/sys/kern/uipc_socket2.c:1081 (so_rcv)
215 3941 52 75 0 0
/usr/src/sys/netinet/tcp_usrreq.c:441 (inp)
45 742 26 28 0 0
/usr/src/sys/netinet/tcp_usrreq.c:165 (inp)
59 930 26 35 0 0
/usr/src/sys/netinet/tcp_usrreq.c:159 (tcp)
5 14 4 3 0 0
/usr/src/sys/kern/uipc_socket2.c:1153 (so_snd)
4 11 4 2 0 0
/usr/src/sys/netinet/tcp_input.c:1261 (so_snd)
11 456 153 2 0 0
/usr/src/sys/libkern/arc4random.c:137 (arc4_mtx)
68 222 6 37 0 0
/usr/src/sys/netinet/ip_fw2.c:4315 (IPFW dynamic rules)
62 224 46 4 1 0
/usr/src/sys/sys/buf.h:272 (lockbuilder mtxpool)
70 121 5 24 0 0
/usr/src/sys/kern/vfs_default.c:401 (vnode interlock)
9 151 35 4 3 3
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1671 (vnode interlock)
11 162 38 4 0 16
/usr/src/sys/kern/vfs_bio.c:286 (needsbuffer lock)
5 163 43 3 0 3
/usr/src/sys/kern/vfs_bio.c:3745 (vnode interlock)
7 165 38 4 2 3
/usr/src/sys/kern/vfs_bio.c:3334 (vm page queue mutex)
27 504 38 13 3 1
/usr/src/sys/kern/vfs_bio.c:3332 (vm object)
16 339 31 10 0 1
/usr/src/sys/kern/vfs_default.c:433 (vnode interlock)
15 307 42 7 4 22
/usr/src/sys/dev/mfi/mfi_disk.c:258 (MFI I/O lock)
6 687 168 4 8 267
/usr/src/sys/kern/vfs_bio.c:2923 (bdone lock)
38 1053 42 25 0 0
/usr/src/sys/geom/geom_disk.c:198 (g_disk_done)
99 1933 40 48 10 4
/usr/src/sys/dev/mfi/mfi.c:912 (MFI I/O lock)
7 165 41 4 0 0
/usr/src/sys/kern/vfs_bio.c:332 (runningbufspace lock)
6 162 39 4 3 2
/usr/src/sys/kern/vfs_bio.c:3139 (vm page queue mutex)
27 521 39 13 2 4
/usr/src/sys/kern/vfs_bio.c:3120 (vm object)
11 785 265 2 4 3
/usr/src/sys/kern/vfs_bio.c:1437 (vnode interlock)
53 864 267 3 1 0
/usr/src/sys/kern/vfs_bio.c:355 (needsbuffer lock)
6 837 268 3 4 1
/usr/src/sys/kern/vfs_bio.c:313 (needsbuffer lock)
5 155 44 3 5 1
/usr/src/sys/kern/vfs_bio.c:3755 (vnode interlock)
12 45 10 4 0 0
/usr/src/sys/kern/vfs_subr.c:1846 (Syncer mtx)
13 96 16 6 0 0
/usr/src/sys/kern/vfs_subr.c:1596 (vnode interlock)
16 111 16 6 0 0
/usr/src/sys/kern/vfs_subr.c:1607 (Syncer mtx)
14 200 54 3 0 0
/usr/src/sys/netinet/ip_fw2.c:1327 (IPFW dynamic rules)
110 640 25 25 0 1
/usr/src/sys/netinet/udp_usrreq.c:510 (so_rcv)
116 830 25 33 0 0
/usr/src/sys/netinet/udp_usrreq.c:425 (inp)
123 996 25 39 0 6
/usr/src/sys/netinet/udp_usrreq.c:285 (udp)
5 689 248 2 0 0
/usr/src/sys/kern/vfs_vnops.c:499 (sleep mtxpool)
79 740 224 3 0 0
/usr/src/sys/ufs/ffs/ffs_vnops.c:597 (vnode interlock)
11 709 248 2 0 0
/usr/src/sys/kern/vfs_vnops.c:520 (sleep mtxpool)
5 701 414 1 0 0
/usr/src/sys/amd64/amd64/pmap.c:881 (pmap)
5 21 6 3 0 7
/usr/src/sys/vm/vm_glue.c:150 (system map)
905 2104 3 701 0 0
/usr/src/sys/vm/vm_meter.c:125 (vm object_list)
7 27136 17853 1 0 9629
/usr/src/sys/vm/vm_meter.c:198 (vm object)
132 337 3 112 0 0
/usr/src/sys/vm/vm_meter.c:212 (vm object_list)
5 36 8 4 0 0
/usr/src/sys/vm/swap_pager.c:2243 (swapdev)
5 112 41 2 0 0
/usr/src/sys/netinet/in_pcb.c:618 (rtentry)
44 115 3 38 0 0
/usr/src/sys/netinet/udp_usrreq.c:801 (udp)
8 97 23 4 0 0
/usr/src/sys/netinet/ip_fw2.c:1490 (IPFW dynamic rules)
269 3673 25 146 0 0
/usr/src/sys/netinet/udp_usrreq.c:805 (inp)
7 39 7 5 0 0
/usr/src/sys/netinet/tcp_timer.c:415 (tcp)
129 837 7 119 0 0
/usr/src/sys/netinet/tcp_timer.c:422 (inp)
33 33 1 33 0 0
/usr/src/sys/kern/uipc_socket2.c:292 (so_rcv)
67 337 43 7 0 0
/usr/src/sys/kern/uipc_socket2.c:159 (so_rcv)
97 257 43 5 0 0
/usr/src/sys/kern/uipc_socket2.c:160 (so_snd)
11 724 280 2 0 0
/usr/src/sys/kern/kern_descrip.c:383 (filedesc structure)
10 69 25 2 0 0
/usr/src/sys/kern/kern_descrip.c:423 (sleep mtxpool)
12 72 25 2 0 0
/usr/src/sys/kern/kern_descrip.c:426 (filedesc structure)
384 965 218 4 0 0
/usr/src/sys/kern/kern_descrip.c:431 (sleep mtxpool)
10 602 218 2 0 0
/usr/src/sys/kern/kern_descrip.c:436 (filedesc structure)
424 12200 218 55 11 6
/usr/src/sys/kern/kern_descrip.c:376 (Giant)
3 3 1 3 0 0
/usr/src/sys/kern/sys_pipe.c:385 (sleep mtxpool)
2 2 1 2 0 0
/usr/src/sys/kern/sys_pipe.c:400 (sleep mtxpool)
3 15 6 2 0 32
/usr/src/sys/kern/uipc_usrreq.c:920 (unp)
63 112 2 56 0 0
/usr/src/sys/kern/uipc_usrreq.c:399 (unp)
3 4 2 2 0 0
/usr/src/sys/kern/uipc_syscalls.c:628 (sleep mtxpool)
3 4 2 2 0 0
/usr/src/sys/kern/uipc_syscalls.c:633 (sleep mtxpool)
3 11 4 2 0 0
/usr/src/sys/kern/kern_fork.c:400 (process lock)
14 39 4 9 0 27
/usr/src/sys/kern/kern_fork.c:399 (process lock)
3 10 4 2 0 0
/usr/src/sys/kern/kern_descrip.c:1411 (filedesc structure)
5 12 4 3 0 0
/usr/src/sys/kern/kern_descrip.c:1421 (filedesc structure)
12 91 26 3 0 0
/usr/src/sys/kern/kern_descrip.c:1527 (sleep mtxpool)
76 208 4 52 0 0
/usr/src/sys/kern/kern_descrip.c:1511 (filedesc structure)
5 13 4 3 0 0
/usr/src/sys/kern/kern_descrip.c:1535 (filedesc structure)
5 13 4 3 0 0
/usr/src/sys/kern/kern_descrip.c:1539 (filedesc structure)
7 17 4 4 0 0
/usr/src/sys/kern/kern_descrip.c:1540 (filedesc structure)
7 28 7 4 0 0
/usr/src/sys/kern/kern_proc.c:1141 (struct pargs.ref)
5 16 4 4 0 0
/usr/src/sys/kern/kern_sig.c:2798 (sigacts)
5 10 4 2 0 0
/usr/src/sys/kern/kern_resource.c:921 (sleep mtxpool)
41 113 4 28 0 0
/usr/src/sys/kern/kern_fork.c:470 (process lock)
53 144 4 36 0 0
/usr/src/sys/kern/kern_fork.c:469 (process lock)
5 16 4 4 0 0
/usr/src/sys/kern/kern_fork.c:582 (session)
28 76 4 19 0 0
/usr/src/sys/kern/kern_fork.c:572 (process group)
5 15 4 3 0 0
/usr/src/sys/kern/kern_fork.c:600 (ktrace)
39 107 4 26 0 0
/usr/src/sys/kern/kern_fork.c:574 (process lock)
51 139 4 34 0 0
/usr/src/sys/kern/kern_fork.c:573 (process lock)
19 1419 144 9 0 0
/usr/src/sys/vm/vm_map.c:2508 (vm object)
111 2272 154 14 0 0
/usr/src/sys/amd64/amd64/pmap.c:2528 (vm page queue mutex)
95 958 76 12 0 0
/usr/src/sys/amd64/amd64/pmap.c:2531 (pmap)
104 1562 76 20 0 0
/usr/src/sys/amd64/amd64/pmap.c:2530 (pmap)
176 1262 82 15 0 0
/usr/src/sys/vm/vm_object.c:1627 (vm object)
15 609 95 6 0 0
/usr/src/sys/amd64/amd64/pmap.c:1933 (vm page queue mutex)
55 732 95 7 0 0
/usr/src/sys/amd64/amd64/pmap.c:1934 (pmap)
5 15 4 3 0 0
/usr/src/sys/kern/kern_fork.c:700 (process lock)
5 13 4 3 0 0
/usr/src/sys/kern/kern_fork.c:715 (process lock)
6 278 90 3 1 1
/usr/src/sys/vm/vm_object.c:1218 (vm object)
7 278 82 3 0 1
/usr/src/sys/vm/vm_object.c:1251 (vm object)
5 10 4 2 0 0
/usr/src/sys/kern/kern_fork.c:794 (process lock)
262 4189 329 12 0 240
/usr/src/sys/vm/vnode_pager.c:1211 (vnode interlock)
31 2903 329 8 0 0
/usr/src/sys/vm/vnode_pager.c:1229 (vm object)
60 88 4 22 0 0
/usr/src/sys/kern/uipc_usrreq.c:656 (so_rcv)
20 55 4 13 0 0
/usr/src/sys/kern/uipc_usrreq.c:680 (so_snd)
86 201 4 50 0 0
/usr/src/sys/kern/uipc_usrreq.c:564 (unp)
13 149 22 6 0 0
/usr/src/sys/fs/devfs/devfs_vnops.c:177 (devfs interlock)
31 288 22 13 0 0
/usr/src/sys/fs/devfs/devfs_vnops.c:180 (vnode interlock)
21 31061 11101 2 0 0
/usr/src/sys/kern/kern_descrip.c:998 (filedesc structure)
51 93 2 46 0 0
/usr/src/sys/kern/uipc_usrreq.c:433 (unp)
29 105 6 17 0 0
/usr/src/sys/kern/uipc_usrreq.c:417 (unp)
11 33 4 8 0 0
/usr/src/sys/dev/random/yarrow.c:280 (random reseed)
15 82 8 10 0 0
/usr/src/sys/fs/devfs/devfs_vnops.c:337 (vnode interlock)
3 6 4 1 0 0
/usr/src/sys/kern/vfs_subr.c:2192 (vnode interlock)
25 276 26 10 0 0
/usr/src/sys/vm/vm_fault.c:670 (vm object)
30 6812 2445 2 1 2
/usr/src/sys/kern/kern_proc.c:239 (process lock)
4 8 3 2 0 0
/usr/src/sys/kern/kern_proc.c:1209 (process lock)
1 1 1 1 0 0
/usr/src/sys/kern/kern_proc.c:302 (process lock)
1 1 1 1 0 0
/usr/src/sys/kern/kern_proc.c:305 (process group)
14 14 1 14 0 0
/usr/src/sys/kern/kern_proc.c:301 (Giant)
4 11 3 3 0 0
/usr/src/sys/kern/kern_proc.c:398 (process lock)
16 40 3 13 0 0
/usr/src/sys/kern/kern_proc.c:397 (process group)
25 64 3 21 0 0
/usr/src/sys/kern/kern_proc.c:396 (process group)
36 90 3 30 0 0
/usr/src/sys/kern/kern_proc.c:395 (Giant)
5 37 14 2 0 0
/usr/src/sys/kern/kern_descrip.c:625 (process lock)
5 44 14 3 0 0
/usr/src/sys/kern/kern_descrip.c:631 (filedesc structure)
11 48 14 3 0 0
/usr/src/sys/kern/kern_descrip.c:642 (sleep mtxpool)
10 21 5 4 0 0
/usr/src/sys/kern/kern_descrip.c:720 (filedesc structure)
5 25 9 2 0 0
/usr/src/sys/kern/kern_descrip.c:733 (filedesc structure)
5 24 7 3 0 0
/usr/src/sys/vm/vm_map.c:3105 (system map)
4 9 3 3 0 0
/usr/src/sys/kern/kern_exec.c:331 (process lock)
11 41 10 4 0 0
/usr/src/sys/vm/vm_page.c:1481 (vm page queue mutex)
5 15 5 3 0 0
/usr/src/sys/kern/kern_exec.c:876 (vm page queue mutex)
30 104 5 20 0 0
/usr/src/sys/kern/kern_exec.c:837 (vm object)
6 90 24 3 2 0
/usr/src/sys/vm/vm_object.c:1551 (vm page queue mutex)
5 334 106 3 12 6
/usr/src/sys/vm/vm_object.c:1570 (vm page queue mutex)
12 58 16 3 0 0
/usr/src/sys/vm/vm_object.c:1714 (vm object_list)
5 11 3 3 0 0
/usr/src/sys/vm/vm_map.c:2722 (process lock)
5 15 5 3 0 0
/usr/src/sys/vm/vm_glue.c:273 (vm page queue mutex)
26 93 5 18 0 0
/usr/src/sys/vm/vm_glue.c:256 (vm object)
3 11 5 2 0 0
/usr/src/sys/vm/vm_glue.c:309 (vm page queue mutex)
5 10 3 3 0 0
/usr/src/sys/kern/imgact_elf.c:762 (process lock)
5 13 5 2 0 0
/usr/src/sys/kern/kern_exec.c:898 (vm page queue mutex)
5 12 3 4 0 0
/usr/src/sys/kern/kern_descrip.c:1483 (filedesc structure)
15 108 13 8 0 2
/usr/src/sys/kern/kern_resource.c:1041 (uidinfo hash)
4 10 3 3 0 0
/usr/src/sys/kern/kern_descrip.c:1786 (filedesc structure)
1 5 3 1 0 0
/usr/src/sys/kern/kern_descrip.c:1811 (filedesc structure)
5 10 3 3 0 0
/usr/src/sys/kern/kern_sig.c:2808 (sigacts)
5 9 3 3 0 0
/usr/src/sys/kern/kern_sig.c:595 (sigacts)
26 63 3 21 0 0
/usr/src/sys/kern/kern_exec.c:518 (process lock)
5 27 10 2 0 0
/usr/src/sys/kern/kern_prot.c:1257 (process lock)
18 90 7 12 0 0
/usr/src/sys/kern/kern_prot.c:834 (process lock)
5 56 19 2 0 0
/usr/src/sys/kern/kern_prot.c:1988 (process lock)
2 2 1 2 0 0
/usr/src/sys/kern/kern_resource.c:689 (process lock)
3 9 4 2 0 0
/usr/src/sys/fs/devfs/devfs_vnops.c:611 (dev_clone)
4 11 4 2 0 0
/usr/src/sys/kern/uipc_usrreq.c:515 (so_rcv)
24 63 4 15 0 0
/usr/src/sys/kern/uipc_usrreq.c:499 (unp)
65 1180 26 45 0 0
/usr/src/sys/netinet/udp_usrreq.c:985 (udp)
5 118 44 2 0 0
/usr/src/sys/kern/uipc_syscalls.c:179 (filedesc structure)
26 441 26 16 0 0
/usr/src/sys/netinet/udp_usrreq.c:1070 (inp)
36 629 26 24 0 0
/usr/src/sys/netinet/udp_usrreq.c:1064 (udp)
17 85 14 6 0 0
/usr/src/sys/ufs/ffs/ffs_vnops.c:208 (vnode interlock)
3 5 2 2 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:6021 (Softdep Lock)
29 54 3 18 0 0
/usr/src/sys/kern/vfs_bio.c:1600 (vnode interlock)
12 12 1 12 0 0
/usr/src/sys/kern/vfs_cluster.c:753 (vnode interlock)
1 1 1 1 0 0
/usr/src/sys/vm/vm_pager.c:324 (pbuf mutex)
5 9 2 4 0 0
/usr/src/sys/kern/vfs_cluster.c:895 (vm object)
6 6 1 6 0 0
/usr/src/sys/kern/vfs_cluster.c:835 (vnode interlock)
5 46 13 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:3740 (process lock)
5 54 13 4 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:3742 (Softdep Lock)
5 55 13 4 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:3813 (process lock)
9 28 6 4 0 0
/usr/src/sys/ufs/ffs/ffs_vnops.c:283 (vnode interlock)
10 67 20 3 0 0
/usr/src/sys/kern/vfs_subr.c:1527 (Syncer mtx)
16 84 7 12 0 0
/usr/src/sys/kern/vfs_subr.c:3097 (vnode interlock)
13 50 7 7 0 0
/usr/src/sys/kern/vfs_subr.c:3105 (mountlist)
39 135 13 10 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:4237 (Softdep Lock)
4 17 7 2 0 0
/usr/src/sys/kern/vfs_subr.c:3114 (struct mount mtx)
5 18 7 2 1 0
/usr/src/sys/kern/vfs_subr.c:2872 (struct mount mtx)
11 7193 4957 1 1 109
/usr/src/sys/kern/vfs_subr.c:2874 (vnode interlock)
14209 34629 12 2885 4 1
/usr/src/sys/kern/vfs_mount.c:1895 (struct mount mtx)
5 31 12 2 2 2
/usr/src/sys/kern/vfs_mount.c:1933 (struct mount mtx)
5 16 5 3 0 1
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1139 (struct mount mtx)
4 17 5 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:6242 (Softdep Lock)
3 13 5 2 0 0
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1148 (struct mount mtx)
13 7658 4930 1 1 0
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1157 (vnode interlock)
3 3 1 3 0 0
/usr/src/sys/vm/vm_pager.c:463 (vnode interlock)
5 5 1 5 0 0
/usr/src/sys/vm/vm_pager.c:401 (pbuf mutex)
5 13 5 2 0 67
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1200 (vnode interlock)
5 22 7 3 0 0
/usr/src/sys/kern/vfs_subr.c:3120 (struct mount mtx)
5 97 37 2 0 0
/usr/src/sys/kern/kern_descrip.c:418 (filedesc structure)
4 4 1 4 0 0
/usr/src/sys/kern/kern_descrip.c:256 (process lock)
4 22 11 2 0 0
/usr/src/sys/kern/uipc_socket.c:1616 (so_rcv)
14 196 38 5 0 0
/usr/src/sys/netinet/tcp_usrreq.c:1023 (tcp)
15 239 38 6 0 0
/usr/src/sys/netinet/tcp_usrreq.c:1029 (inp)
1 6 4 1 0 0
/usr/src/sys/netinet/in_pcb.c:788 (inp)
10 27 4 6 0 0
/usr/src/sys/netinet/in_pcb.c:782 (tcp)
9 22 7 3 0 0
/usr/src/sys/netinet/in_pcb.c:761 (inp)
18 66 7 9 0 0
/usr/src/sys/netinet/in_pcb.c:755 (tcp)
112 1090 78 13 0 0
/usr/src/sys/amd64/amd64/pmap.c:2533 (pmap)
112 1142 78 14 0 0
/usr/src/sys/amd64/amd64/pmap.c:2534 (pmap)
7 12 2 6 0 0
/usr/src/sys/vm/vm_object.c:1333 (vm page queue mutex)
14 23 2 11 0 0
/usr/src/sys/vm/vm_object.c:1309 (vm object)
3 5 2 2 0 0
/usr/src/sys/vm/vm_object.c:1370 (vm page queue mutex)
23 44 2 22 0 5
/usr/src/sys/vm/vm_object.c:1308 (vm object)
2 3 2 1 0 0
/usr/src/sys/vm/vm_object.c:1378 (vm object)
3 5 2 2 0 0
/usr/src/sys/vm/vm_map.c:2644 (vm object)
3 3 1 3 0 0
/usr/src/sys/kern/vfs_syscalls.c:924 (filedesc structure)
4 4 1 4 0 0
/usr/src/sys/kern/vfs_syscalls.c:940 (filedesc structure)
15 15 1 15 0 0
/usr/src/sys/kern/kern_prot.c:1130 (process lock)
9 9 1 9 0 0
/usr/src/sys/kern/kern_resource.c:1046 (uidinfo hash)
52 52 1 52 0 0
/usr/src/sys/kern/kern_prot.c:1049 (process lock)
95 114 3 38 0 0
/usr/src/sys/kern/kern_prot.c:674 (process lock)
32 43 3 14 0 0
/usr/src/sys/kern/kern_prot.c:775 (process lock)
31 44 3 14 0 0
/usr/src/sys/kern/kern_prot.c:504 (process lock)
35 81 5 16 0 0
/usr/src/sys/kern/kern_prot.c:618 (process lock)
4 17 6 2 0 0
/usr/src/sys/kern/uipc_usrreq.c:382 (unp)
110 566 6 94 1 1
/usr/src/sys/kern/uipc_usrreq.c:1037 (Giant)
5 24 6 4 0 0
/usr/src/sys/kern/uipc_usrreq.c:1134 (unp)
1 3 2 1 0 0
/usr/src/sys/vm/vm_object.c:1689 (vm object)
26 50 7 7 0 1
/usr/src/sys/kern/sys_pipe.c:1479 (pipe mutex)
4 21 7 3 0 0
/usr/src/sys/kern/sys_pipe.c:1519 (pipe mutex)
3 6 2 3 0 0
/usr/src/sys/kern/kern_exit.c:350 (session)
2 2 1 2 0 0
/usr/src/sys/kern/kern_exit.c:426 (process lock)
9 9 1 9 0 0
/usr/src/sys/kern/kern_resource.c:1111 (uidinfo hash)
10 10 1 10 0 0
/usr/src/sys/kern/kern_resource.c:1112 (sleep mtxpool)
8 13 3 4 0 2
/usr/src/sys/kern/kern_exit.c:766 (process lock)
5 22 5 4 0 0
/usr/src/sys/kern/sys_pipe.c:993 (pipe mutex)
34 70 5 14 0 0
/usr/src/sys/kern/sys_pipe.c:1145 (pipe mutex)
4 14 5 2 0 0
/usr/src/sys/kern/sys_pipe.c:628 (pipe mutex)
5 5 1 5 0 0
/usr/src/sys/kern/kern_prot.c:135 (process lock)
5 7 2 3 0 0
/usr/src/sys/kern/kern_proc.c:319 (session)
6 9 2 4 0 0
/usr/src/sys/kern/kern_proc.c:322 (process group)
27 45 2 22 0 1
/usr/src/sys/kern/kern_proc.c:317 (Giant)
5 26 10 2 0 0
/usr/src/sys/kern/kern_descrip.c:1806 (filedesc structure)
4 31 10 3 0 0
/usr/src/sys/kern/kern_descrip.c:1808 (filedesc structure)
5 6 2 3 0 0
/usr/src/sys/kern/kern_sig.c:1164 (process lock)
11 6608 2464 2 0 0
/usr/src/sys/kern/kern_proc.c:638 (ktrace)
11 6671 2464 2 0 0
/usr/src/sys/kern/kern_proc.c:666 (sigacts)
10 6572 2464 2 0 0
/usr/src/sys/kern/kern_proc.c:717 (session)
361 62735 2464 25 3 24
/usr/src/sys/kern/kern_proc.c:1017 (process lock)
5 47 12 3 0 0
/usr/src/sys/vm/vm_glue.c:204 (process lock)
4 676 408 1 0 0
/usr/src/sys/amd64/amd64/pmap.c:2491 (pmap)
4 682 408 1 11 5
/usr/src/sys/vm/vm_fault.c:1093 (vm page queue mutex)
5 48 17 2 0 0
/usr/src/sys/ufs/ffs/ffs_alloc.c:1880 (FFS)
5 9 3 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:2765 (FFS)
5 11 3 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:2791 (Softdep Lock)
5 29 8 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:986 (Softdep Lock)
11 19 3 6 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:3669 (Softdep Lock)
4 11 3 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:3675 (FFS)
5 9 3 3 0 0
/usr/src/sys/ufs/ffs/ffs_alloc.c:2072 (FFS)
4 10 3 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:3681 (Softdep Lock)
3 11 5 2 0 0
/usr/src/sys/kern/subr_devstat.c:381 (devstat)
4 12 5 2 0 0
/usr/src/sys/kern/subr_devstat.c:394 (devstat)
5 60 22 2 0 0
/usr/src/sys/kern/kern_event.c:524 (filedesc structure)
5 62 22 2 0 0
/usr/src/sys/kern/kern_event.c:528 (sleep mtxpool)
596 1894 22 86 0 0
/usr/src/sys/netinet/udp_usrreq.c:1042 (inp)
602 2032 22 92 0 0
/usr/src/sys/netinet/udp_usrreq.c:1036 (udp)
5 69 22 3 0 0
/usr/src/sys/kern/uipc_syscalls.c:543 (so_rcv)
5 119 44 2 0 0
/usr/src/sys/kern/kern_event.c:980 (kqueue)
17 395 44 8 0 0
/usr/src/sys/kern/kern_event.c:972 (sleep mtxpool)
5 63 22 2 0 0
/usr/src/sys/kern/kern_event.c:723 (protect sysfilt_ops)
5 71 22 3 0 0
/usr/src/sys/kern/kern_event.c:779 (filedesc structure)
5 75 22 3 0 0
/usr/src/sys/kern/kern_event.c:788 (sleep mtxpool)
5 86 22 3 0 0
/usr/src/sys/kern/kern_event.c:1053 (kqueue)
11 76 22 3 0 0
/usr/src/sys/kern/kern_event.c:819 (filedesc structure)
15 86 22 3 0 0
/usr/src/sys/kern/kern_event.c:820 (kqueue)
5 71 22 3 0 0
/usr/src/sys/kern/kern_event.c:1577 (kqueue)
645 869 22 39 1 0
/usr/src/sys/kern/uipc_socket.c:2108 (so_rcv)
12 171 22 7 0 0
/usr/src/sys/kern/kern_event.c:926 (kqueue)
5 74 22 3 0 0
/usr/src/sys/kern/kern_event.c:1161 (kqueue)
96 243 21 11 0 0
/usr/src/sys/kern/kern_event.c:1547 (kqueue)
5 54 22 2 0 0
/usr/src/sys/kern/kern_event.c:1599 (kqueue)
21 211 22 9 0 0
/usr/src/sys/kern/uipc_socket.c:2120 (so_rcv)
12 80 22 3 0 0
/usr/src/sys/kern/kern_event.c:1847 (kqueue)
11 72 22 3 0 0
/usr/src/sys/kern/kern_event.c:739 (protect sysfilt_ops)
4 49 22 2 0 0
/usr/src/sys/kern/kern_event.c:1236 (kqueue)
4 50 22 2 0 0
/usr/src/sys/kern/kern_event.c:1000 (kqueue)
5 119 44 2 0 0
/usr/src/sys/kern/kern_event.c:1781 (kqueue)
5 74 22 3 0 0
/usr/src/sys/netinet/udp_usrreq.c:1087 (inp)
14 218 22 9 0 0
/usr/src/sys/netinet/udp_usrreq.c:1081 (udp)
5 74 22 3 0 0
/usr/src/sys/kern/kern_event.c:1421 (kqueue)
11 79 22 3 0 0
/usr/src/sys/kern/kern_event.c:1474 (filedesc structure)
84 799 16 49 0 0
/usr/src/sys/netinet/tcp_usrreq.c:121 (tcp)
4 41 16 2 0 0
/usr/src/sys/kern/uipc_socket2.c:115 (so_rcv)
1232 8691 32 271 0 0
/usr/src/sys/netinet/tcp_usrreq.c:368 (inp)
7 15 2 7 0 0
/usr/src/sys/kern/vfs_bio.c:1689 (buf queue lock)
7 7 1 7 0 0
/usr/src/sys/kern/vfs_bio.c:3545 (vm object)
3 3 1 3 0 0
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1715 (vnode interlock)
2 2 1 2 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:1003 (Softdep Lock)
5 5 1 5 0 0
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1598 (vnode interlock)
8 20 4 5 0 0
/usr/src/sys/kern/vfs_bio.c:3596 (vm page queue mutex)
70 70 1 70 0 0
/usr/src/sys/kern/vfs_bio.c:3584 (vm object)
2 2 1 2 0 0
/usr/src/sys/kern/vfs_subr.c:1500 (vnode interlock)
3 5 2 2 0 0
/usr/src/sys/kern/vfs_bio.c:1321 (buf queue lock)
4 4 1 4 0 0
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1620 (vnode interlock)
1102 2187 10 218 0 0
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1181 (struct mount mtx)
5 6 2 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:1826 (Softdep Lock)
10 15 2 7 0 0
/usr/src/sys/net/rtsock.c:1100 (ifnet)
5 17 4 4 0 0
/usr/src/sys/net/if.c:1235 (ifnet)
1579 3160 3 1053 0 0
/usr/src/sys/dev/em/if_em.c:1652 (em0)
23 797 246 3 0 217
/usr/src/sys/netinet/tcp_timer.c:252 (inp)
3 37 17 2 0 0
/usr/src/sys/vm/uma_core.c:2209 (UMA Slabs)
4 7 2 3 0 0
/usr/src/sys/ufs/ffs/ffs_balloc.c:654 (FFS)
2 3 2 1 0 0
/usr/src/sys/ufs/ffs/ffs_alloc.c:307 (FFS)
1 3 2 1 0 0
/usr/src/sys/ufs/ffs/ffs_alloc.c:1367 (FFS)
2 4 2 2 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:1556 (Softdep Lock)
2 4 2 2 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:1368 (Softdep Lock)
3 10 4 2 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:653 (Softdep Lock)
3 3 1 3 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:1591 (Softdep Lock)
26 33 3 11 0 0
/usr/src/sys/kern/vfs_bio.c:2793 (vm object)
5 10 2 5 0 0
/usr/src/sys/kern/vfs_bio.c:3483 (vm object)
6 9 2 4 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:1659 (Softdep Lock)
2 2 1 2 0 0
/usr/src/sys/ufs/ffs/ffs_softdep.c:1291 (Softdep Lock)
4 22 8 2 0 0
/usr/src/sys/vm/uma_core.c:2408 (2048)
14 53 10 5 0 0
/usr/src/sys/netinet/ip_output.c:1240 (tcp)
15 80 10 8 0 0
/usr/src/sys/netinet/ip_output.c:1246 (inp)
4 5 2 2 0 0
/usr/src/sys/fs/fifofs/fifo_vnops.c:224 (fifo mutex)
1874 1922 2 961 2 0
/usr/src/sys/fs/fifofs/fifo_vnops.c:733 (Giant)
4 6 2 3 0 0
/usr/src/sys/ufs/ufs/ufs_vnops.c:2014 (vnode interlock)
9 18 2 9 0 0
/usr/src/sys/kern/uipc_usrreq.c:522 (so_snd)
58 109 2 54 0 0
/usr/src/sys/fs/fifofs/fifo_vnops.c:711 (Giant)
5 22 6 3 0 0
/usr/src/sys/kern/vfs_syscalls.c:3883 (filedesc structure)
5 23 6 3 0 0
/usr/src/sys/kern/vfs_syscalls.c:3891 (sleep mtxpool)
4 19 6 3 0 0
/usr/src/sys/kern/vfs_syscalls.c:3894 (filedesc structure)
5 10 2 5 0 0
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1080 (FFS)
70 130 2 65 0 0
/usr/src/sys/kern/vfs_syscalls.c:334 (Giant)
5 5 1 5 0 0
/usr/src/sys/kern/sys_pipe.c:1270 (pipe mutex)
5 224 73 3 0 0
/usr/src/sys/vm/uma_core.c:416 (FFS2 dinode)
506 506 1 506 0 0
/usr/src/sys/vm/uma_core.c:1494 (UMA lock)
1 1 1 1 0 5
/usr/src/sys/vm/vnode_pager.c:157 (vm object)
1 1 1 1 0 0
/usr/src/sys/kern/vfs_bio.c:2535 (vnode interlock)
1 1 1 1 0 0
/usr/src/sys/kern/vfs_subr.c:2170 (vnode interlock)
1 1 1 1 0 0
/usr/src/sys/kern/vfs_bio.c:3704 (bdone lock)
3 3 1 3 0 0
/usr/src/sys/kern/vfs_bio.c:3694 (bdone lock)
2 2 1 2 0 0
/usr/src/sys/kern/vfs_bio.c:1248 (vm object)
2 2 1 2 0 0
/usr/src/sys/kern/vfs_cluster.c:133 (vnode interlock)
3 3 1 3 0 0
/usr/src/sys/dev/mfi/mfi.c:2497 (MFI I/O lock)
-------------- next part --------------
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-STABLE #1: Wed Oct 3 16:49:23 MSD 2007
llp@cover.xxx.ru:/usr/obj/usr/src/sys/xxxSMP-amd64
module_register: module accf_http already exists!
Module accf_http failed to register: 17
module_register: module mfi_linux already exists!
Module mfi_linux failed to register: 17
ACPI APIC Table: <INTEL S5000PAL>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5120 @ 1.86GHz (1871.23-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
real memory = 5905580032 (5632 MB)
avail memory = 4096319488 (3906 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
lapic0: Forcing LINT1 to edge trigger
kbd1 at kbdmux0
module_register_init: MOD_LOAD (accf_http, 0xffffffff80340ea0,
0xffffffff807b4720) error 17
acpi0: <INTEL S5000PAL> on motherboard
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 2000
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xca2,0xca3,0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci2
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 2.0 on pci2
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x2020-0x203f mem 0xb8820000-0xb883ffff,0xb8400000-0xb87fffff irq 18 at device
0.0 on pci5
em0: Ethernet address: 00:04:23:dd:5a:26
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x2000-0x201f mem 0xb8800000-0xb881ffff,0xb8000000-0xb83fffff irq 19 at device
0.1 on pci5
em1: Ethernet address: 00:04:23:dd:5a:27
pcib6: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci6: <ACPI PCI bus> on pcib6
pcib7: <PCI-PCI bridge> at device 3.0 on pci0
pci7: <PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> at device 0.0 on pci10
pci11: <PCI bus> on pcib11
mfi0: <LSI MegaSAS 1064R> mem 0xb8a00000-0xb8a0ffff,0xb8c00000-0xb8c1ffff
irq 18 at device 14.0 on pci11
mfi0: Megaraid SAS driver Ver 2.00
mfi0: 593 (245173707s/0x0020/0) - Shutdown command received from host
mfi0: 594 (4278190080s/0x0020/0) - PCI 0x041000 0x04411 0x041000 0x041004:
Firmware initialization started (PCI ID 0411/1000/1004/1000)
mfi0: 595 (4278190080s/0x0020/0) - Type 18: Firmware version 1.03.60-0255
mfi0: 596 (4278190080s/0x0020/0) - PCI 0x041000 0x04411 0x041000 0x041004:
Firmware initialization started (PCI ID 0411/1000/1004/1000)
mfi0: 597 (4278190080s/0x0020/0) - Type 18: Firmware version 1.03.60-0255
mfi0: 598 (4278190080s/0x0020/0) - PCI 0x041000 0x04411 0x041000 0x041004:
Firmware initialization started (PCI ID 0411/1000/1004/1000)
mfi0: 599 (4278190080s/0x0020/0) - Type 18: Firmware version 1.03.60-0255
mfi0: 600 (4278190080s/0x0020/0) - PCI 0x041000 0x04411 0x041000 0x041004:
Firmware initialization started (PCI ID 0411/1000/1004/1000)
mfi0: 601 (4278190080s/0x0020/0) - Type 18: Firmware version 1.03.60-0255
mfi0: 602 (4278190126s/0x0008/0) - Battery temperature is normal
mfi0: 603 (4278190126s/0x0008/0) - Battery Present
mfi0: 604 (4278190161s/0x0002/0) - PD 08(e0/s8) event: Inserted: PD 08(e0/s8)
mfi0: 605 (4278190161s/0x0002/0) - Type 29: Inserted: PD 08(e0/s8) Info:
enclPd=ffff, scsiType=0, portMap=01, sasAddr=500000e0158537c2,0000000000000000
mfi0: 606 (4278190161s/0x0002/0) - PD 09(e0/s9) event: Inserted: PD 09(e0/s9)
mfi0: 607 (4278190161s/0x0002/0) - Type 29: Inserted: PD 09(e0/s9) Info:
enclPd=ffff, scsiType=0, portMap=02, sasAddr=500000e0158539f2,0000000000000000
mfi0: 608 (245173902s/0x0020/0) - Adapter ticks 245173902 elapsed 82s: Time
established as 10/08/07 15:51:42; (82 seconds since power on)
pcib12: <PCI-PCI bridge> at device 0.2 on pci10
pci12: <PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci13: <ACPI PCI bus> on pcib13
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0x3080-0x309f irq 23 at device
29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x3060-0x307f irq 22 at device
29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x3040-0x305f irq 23 at device
29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x3020-0x303f irq 22 at device
29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xb8d00400-0xb8d007ff irq
23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <EHCI (generic) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
pci14: <display, VGA> at device 12.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30b0-0x30bf irq 20 at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
atapci1: <Intel 63XXESB2 SATA300 controller> port
0x30c8-0x30cf,0x30e4-0x30e7,0x30c0-0x30c7,0x30e0-0x30e3,0x30a0-0x30af mem
0xb8d00000-0xb8d003ff irq 20 at device 31.2 on pci0
ata2: <ATA channel 0> on atapci1
ata3: <ATA channel 1> on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A, console
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xcdfff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
IP Filter: v4.1.13 initialized. Default = pass all, Logging = enabled
ipfw2 initialized, divert enabled, rule-based forwarding disabled, default to
accept, logging unlimited
acd0: DVDR <Slimtype DVDRW SSM-8515S/GS06> at ata0-slave UDMA33
mfid0: <MFI Logical Disk> on mfi0
mfid0: 139236MB (285155328 sectors) RAID volume '' is optimal
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
lapic7: Forcing LINT1 to edge trigger
SMP: AP CPU #3 Launched!
lapic6: Forcing LINT1 to edge trigger
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/mfid0s1a
em0: link state changed to UP
1 users Load 5.27 7.25 6.15 Nov 19 16:22
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 2309932 13812 2385372 32628 880612 count
All 2387968 14912 6766444 36300 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt 1 cow 13149 total
38 4 30k 6327 117k 15k 233 3596 3525 zfod sio0 irq4
3523 ozfod ata0 irq14
17.9%Sys 3.5%Intr 41.5%User 0.0%Nice 37.1%Idle 99%ozfod 5148 em0 mfi0 1
| | | | | | | | | | | daefr uhci0 uhci
=========++>>>>>>>>>>>>>>>>>>>>
prcfr 1993 cpu0: time
36 dtbuf 3933 totfr 1993 cpu1: time
Namei Name-cache Dir-cache 100000 desvn react 2007 cpu3: time
Calls hits % hits % 27886 numvn pdwak 2008 cpu2: time
169660 169660 100 24717 frevn pdpgs
intrn
Disks mfid0 278668 wire
KB/t 0.00 2318668 act
tps 0 426152 inact
MB/s 0.00 190808 cache
%busy 0 689804 free
Ivan Voras
2007-Nov-19 05:46 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
Alexey Popov wrote:> Hi. > > I have a large pool of web backends (Apache + mod_php5) with > 2 x Xeon 3.2GHz processors and 2 x Xeon 5120 dual-core processors. The > workload is mostly CPU-bound. I'm using 6-STABLE-amd64 and also tried > 7-STABLE.If you haven't tried mod_fcgid, give it a try - it can dramatically benefit PHP applications. And with mod_fcgid, you can use apache with a multi-threaded MPM (i.e. worker-mpm).> Now I'm trying to use new hardware with 2 x Xeon 5320 (quad-core), but > it can not work under the same load as dual-core. It shows up to 80% > system CPU load in top:On what version of FreeBSD is this? If it's 6-STABLE, this might be expected.> CPU states: 9.5% user, 0.0% nice, 79.9% system, 1.2% interrupt, 9.5% > idleCan you try hitting "S" to see if a kernel process is gobbling up CPU time?> Here's the output from 2xdual-core backend running under the same load > and with the same software:> CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% > idleThis line is bogus - where is the load?> What can I do to make FreeBSD run faster on many-CPU systems???Except for trying 7-STABLE, there's not much you can do.
Robert Watson
2007-Nov-19 06:02 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
On Mon, 19 Nov 2007, Alexey Popov wrote:> I tried Linux and it works much better than old (2 x dual-core) backends. It > handles 2 times more requests than FreeBSD on the old backends. So there's a > real scalability problem in FreeBSD. The more processors it have the more > CPU time it consumes. > > Also I faced the same problem moving heavily loaded MySQL-server to new > hardware. That time I thought that the problem is in the mysql-server itself > and I had to install Linux. > > See in attach: mutex statistics for quad-core system and dmesg and vmstat > for dual- and quad-core systems. > > What can I do to make FreeBSD run faster on many-CPU systems???Have you configured libmap.conf to force MySQL to use libthr instead of libpthread? libpthread is known to have serious performance bottlenecks for MySQL as compared to libthr. FreeBSD 7 contains significant optimization for increased numbers of cores, and is where a lot of the work optimizing MySQL has ended up. I see you're trying out a 6.3 beta, any chance you could try out a 7.0 beta instead? Also, consider switching to "options SCHED_ULE" in the 7.0 kernel rather than "options SCHED_4BSD". Robert N M Watson Computer Laboratory University of Cambridge
Krassimir Slavchev
2007-Nov-19 08:27 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi All, What version of apache do you use and what are: StartServers MinSpareServers MaxSpareServers MaxClients KeepAliveTimeout settings in both configurations? Best Regards Alexey Popov wrote:> Hi. > > I have a large pool of web backends (Apache + mod_php5) with > 2 x Xeon 3.2GHz processors and 2 x Xeon 5120 dual-core processors. The > workload is mostly CPU-bound. I'm using 6-STABLE-amd64 and also tried > 7-STABLE. > > Now I'm trying to use new hardware with 2 x Xeon 5320 (quad-core), but > it can not work under the same load as dual-core. It shows up to 80% > system CPU load in top: >-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHQblfxJBWvpalMpkRAgTQAJ4uy8qhmpCVWevAI0LSYXPrXiIUSQCeNE8y +dkavLoDzqrILkqVGZNZZDM=xI6R -----END PGP SIGNATURE-----
Claus Guttesen
2007-Nov-19 10:06 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
On Nov 19, 2007 2:32 PM, Alexey Popov <lol@chistydom.ru> wrote:> Hi. > > I have a large pool of web backends (Apache + mod_php5) with > 2 x Xeon 3.2GHz processors and 2 x Xeon 5120 dual-core processors. The > workload is mostly CPU-bound. I'm using 6-STABLE-amd64 and also tried > 7-STABLE. > > Now I'm trying to use new hardware with 2 x Xeon 5320 (quad-core), but > it can not work under the same load as dual-core. It shows up to 80% > system CPU load in top: > > last pid: 3850; load averages: 22.51, 19.75, 12.18Very high load. Could it be the raid-controller? I had a db-server with horibble performance due to a cheap raid-controller. Moving to a ciss-controller (DL380 G5) solved all my issues. My load decreased 100 fold. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
Stefan Lambrev
2007-Nov-20 04:02 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
Hi Alexey, Can you please send and dmesg from FreeBSD 7 on this server? As I'm little puzzled what you mean by 7-stable :) Alexey Popov wrote:> Hi. > > I have a large pool of web backends (Apache + mod_php5) with > 2 x Xeon 3.2GHz processors and 2 x Xeon 5120 dual-core processors. The > workload is mostly CPU-bound. I'm using 6-STABLE-amd64 and also tried > 7-STABLE. > > Now I'm trying to use new hardware with 2 x Xeon 5320 (quad-core), but > it can not work under the same load as dual-core. It shows up to 80% > system CPU load in top: > > ------------------------------------------------------------------------ > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"-- Best Wishes, Stefan Lambrev ICQ# 24134177
Karl M. Joch
2007-Nov-20 04:48 UTC
Software for distribution of configuration files and changes
Hello, i have searched alot for a software to: - distribut configuration files from one master to different systems - maintain configuration files on one machine for all systemes and then send it out - push the files, not download them like cvsup - maintaining files for all systems and files only affecting one system any ideas and hints would be greatly appreziatet. Many thanks, best regards, Karl
Richard Arends
2007-Nov-20 04:58 UTC
Software for distribution of configuration files and changes
On Tue, Nov 20, 2007 at 01:15:56PM +0100, Karl M. Joch wrote: Karl,> i have searched alot for a software to: > > - distribut configuration files from one master to different systems > - maintain configuration files on one machine for all systemes and then send > it out > - push the files, not download them like cvsup > - maintaining files for all systems and files only affecting one system > > any ideas and hints would be greatly appreziatet.http://en.wikipedia.org/wiki/Comparison_of_open_source_configuration_management_software -- Regards, Richard. /* Homo Sapiens non urinat in ventum */
Alexey Popov
2007-Nov-20 09:45 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
Hi. Tom Evans wrote:>> After that I rebuilt with SMP GENERIC kernel and put on that server 2 >> times more requests that UP could handle. For the first time it worked >> good. Then I increased load to 2.5 times more than UP. Immediately >> Apache child count increased to MaxClients (24), most of them in RUN >> state, and %sys became greater than %user (see attach). I think after >> some threshold of load FreeBSD is paying more CPU time to the management >> of running processes than to run them. > MaxClients of 24 seems very low for a 8 cpu box, running prefork MPM. On > our quad CPU boxes, running custom apache modules, we use > MaxClients 70 > MinSpareServers 5 > MaxSpareServers 15 > StartServers 20 > Perhaps you are seeing high system load because the system is having to > maintain a lot of queued connections. Certainly, our load remains > in-between comfortable margins, except when heavily stressed.I believe 8-core FreeBSD server is able to maintain 1024 waiting TCP connections without measurable CPU load. As of this problem: increasing MaxClients leads to growing %sys part of CPU load. Generally large MaxClients value is useful when most Apache children are waiting for I/O or something else but CPU. With best regards, Alexey Popov
Aryeh M. Friedman
2007-Nov-20 10:21 UTC
Software for distribution of configuration files and changes
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Karl M. Joch wrote:> Hello, > > i have searched alot for a software to: > > - distribut configuration files from one master to different > systems - maintain configuration files on one machine for all > systemes and then send it out - push the files, not download them > like cvsup - maintaining files for all systems and files only > affecting one system > > any ideas and hints would be greatly appreziatet. >Have you looked at aegis (aegis.sf.net)? - -- Aryeh M. Friedman Developer, not business, friendly http://www.flosoft-systems.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHQyWgJ9+1V27SttsRAvd0AJ0XkDvAUPZhvoONYp+yEbSUxxWpxQCfdrnk fz7gOeOmsHPkDxtf6bQo480=0z9l -----END PGP SIGNATURE-----
Joseph Koshy
2007-Nov-22 11:54 UTC
Software for distribution of configuration files and changes
> i have searched alot for a software to: > > - distribut configuration files from one master to different systems > - maintain configuration files on one machine for all systemes and then send > it out > - push the files, not download them like cvsup > - maintaining files for all systems and files only affecting one system > > any ideas and hints would be greatly appreziatet.You could take a look at ISCONF: http://trac.t7a.org/isconf/ http://www.infrastructures.org/bootstrap/isconf.shtml -- FreeBSD Volunteer, http://people.freebsd.org/~jkoshy
Brandon S. Allbery KF8NH
2007-Nov-22 17:49 UTC
Software for distribution of configuration files and changes
On Nov 21, 2007, at 21:51 , Joseph Koshy wrote:>> i have searched alot for a software to: >> >> - distribut configuration files from one master to different systems >> - maintain configuration files on one machine for all systemes and >> then send >> it out >> - push the files, not download them like cvsup >> - maintaining files for all systems and files only affecting one >> system >> >> any ideas and hints would be greatly appreziatet. > > You could take a look at ISCONF: > > http://trac.t7a.org/isconf/ > http://www.infrastructures.org/bootstrap/isconf.shtmlisconf, cfengine, puppet, lcfg, bcfg2, radmind... http:// www.infrastructures.org is in general a good resource for such things. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH
Alexey Vlasov
2007-Dec-01 13:58 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
Hi,
It seems that I'm not the only one who faced the problem that FreeBSD is
non productive on multiprocessors platforms.
I use OS Linux on my hosting for web-servers, base for all servers is
the same m/b S5000PAL ( SR1500), 2 quad kernel cpu Xeon E5320 or E5345,
8Gb RAM. I decided to install FreeBSD 6.2 i386 on one of the servers,
and the result was totally non productive.
Used software, Apache 2.2.6 (worker) as frontend Proxy, backend Apache
2.2.3 (prefork)
By the time we understood,that something was wrong with FreeBSD, there
had already been placed about 10 sites with high-capacity and about
hundred of usual ones. And this was the limit for FreeBSD. It came along
with a great amount of Context Switches, about hundred thousands.
I attached the log what was then happening with FreeBSD.
After playing with ab (ApacheBenchmark) options, it turned out that even
with the following options you can totally down the server:
./ab -n 100 -c 20 -t 30 http://somesite-freebsd.com
I copied at the same time somesite.com (php scripts) to Linux server,
launched ab with the same options, and saw that it has no influence on
work of the server. (And by the way there work about 1.5 virtual hosts
on that server)
All options for Apache on Linux and FreeBSD are the same:
FreeBSD:
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking somesite-freebsd.com (be patient)
Finished 29 requests
Server Software: Apache/2.2.3
Server Hostname: somesite-freebsd.com
Server Port: 80
Document Path: /ab/
Document Length: 41450 bytes
Concurrency Level: 20
Time taken for tests: 30.44765 seconds
Complete requests: 29
Failed requests: 22
(Connect: 0, Length: 22, Exceptions: 0)
Write errors: 0
Total transferred: 1529557 bytes
HTML transferred: 1513497 bytes
Requests per second: 0.97 [#/sec] (mean)
Time per request: 20720.527 [ms] (mean)
Time per request: 1036.026 [ms] (mean, across all concurrent
requests)
Transfer rate: 49.69 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1760 1503.4 3002 3002
Processing: 866 13328 9460.5 13853 26246
Waiting: 139 2286 2319.0 1129 6764
Total: 871 15089 10642.6 16855 29248
Percentage of the requests served within a certain time (ms)
50% 16705
66% 22670
75% 25439
80% 26342
90% 29160
95% 29188
98% 29248
99% 29248
100% 29248 (longest request)
Linux: (the same site)
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking linux.1gb.ru (be patient)
Finished 814 requests
Server Software: Apache/2.2.3
Server Hostname: somesite-linux.com
Server Port: 80
Document Path: /ab/
Document Length: 41451 bytes
Concurrency Level: 20
Time taken for tests: 30.3216 seconds
Complete requests: 814
Failed requests: 759
(Connect: 0, Length: 759, Exceptions: 0)
Write errors: 0
Non-2xx responses: 1
Total transferred: 34430291 bytes
HTML transferred: 34126461 bytes
Requests per second: 27.13 [#/sec] (mean)
Time per request: 737.180 [ms] (mean)
Time per request: 36.859 [ms] (mean, across all concurrent
requests)
Transfer rate: 1120.65 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 214 725 575.2 557 5001
Waiting: 41 265 376.0 131 3280
Total: 214 725 575.2 557 5001
Percentage of the requests served within a certain time (ms)
50% 557
66% 716
75% 863
80% 967
90% 1398
95% 1749
98% 2529
99% 3064
100% 5001 (longest request)
# cat /etc/sysctl.conf
security.bsd.see_other_uids=0
kern.maxfiles=204800
kern.maxfilesperproc=202400
kernel:
machine i386
cpu I686_CPU
ident F1RNT1
options PAE
options SMP
options SCHED_4BSD
options PREEMPTION
options INET
options FFS
options SOFTUPDATES
options UFS_ACL
options UFS_DIRHASH
options NULLFS
options MD_ROOT
options CD9660
options PROCFS
options PSEUDOFS
options GEOM_GPT
options GEOM_LABEL
options GEOM_MIRROR
options COMPAT_43
options COMPAT_FREEBSD4
options COMPAT_FREEBSD5
options SCSI_DELAY=5000
options KTRACE
options SYSVSHM
options SYSVMSG
options SYSVSEM
options _KPOSIX_PRIORITY_SCHEDULING
options KBD_INSTALL_CDEV
options ADAPTIVE_GIANT
options LIBICONV
options NETSMB
options NETSMBCRYPTO
options LIBMCHAIN
options SMBFS
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=1000
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFIREWALL_FORWARD
options IPDIVERT
options IPSTEALTH
options TCPDEBUG
options DUMMYNET
options IPSEC
options IPSEC_DEBUG
options ACCEPT_FILTER_DATA
options ACCEPT_FILTER_HTTP
# cat /etc/make.conf
CPUTYPE?=nocona
CFLAGS=-O2 -pipe
CXXFLAGS+=-fconserve-space
COPTFLAGS=-O2 -pipe
NO_CPU_CFLAGS=false
NO_CPU_COPTFLAGS=false
NO_MODULES
--
BRGDS. Alexey Vlasov.
-------------- next part --------------
last pid: 31039; load averages: 51.33, 26.31, 18.09
up 15+22:58:54 16:47:30
692 processes: 80 running, 608 sleeping, 2 stopped, 1 zombie, 1 lock
CPU states: 2.4% user, 0.0% nice, 89.1% system, 1.6% interrupt, 6.9%
idle
Mem: 1587M Active, 5614M Inact, 416M Wired, 369M Cache, 112M Buf, 14M
Free
Swap: 16G Total, 46M Used, 16G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
31027 w_istok2_d424 1 -4 0 76240K 36532K ufs 4 0:00 9.74% httpd
30826 w_nova-i_7638 1 -4 0 79340K 40116K ufs 4 0:02 7.78% httpd
31025 w_shop2y_b290 1 -4 0 76212K 36512K RUN 7 0:00 7.70% httpd
28349 w_urtest_7dc3 1 -4 0 79616K 39496K RUN 4 1:27 7.32% httpd
30801 w_new-vw_5e17 1 -4 0 76092K 37184K ufs 5 0:02 7.25% httpd
30914 w_avtolo_3761 1 -4 0 76316K 37304K RUN 7 0:01 7.18% httpd
29952 w_svekon_9de8 1 -4 0 77976K 42916K RUN 6 0:11 7.04% httpd
29875 w_lancer_12cf 1 -4 0 79584K 41408K RUN 6 0:12 6.94% httpd
29818 w_iprit-_be41 1 -4 0 77340K 39880K RUN 6 0:13 6.92% httpd
30811 w_dfase__508a 1 -4 0 76456K 37928K RUN 4 0:02 6.91% httpd
30200 w_iprit-_be41 1 -4 0 76292K 37260K ufs 1 0:07 6.86% httpd
30445 w_new-vw_5e17 1 119 0 77048K 40196K CPU3 0 0:04 6.74% httpd
30025 w_forum-_89b3 1 -4 0 76824K 41240K RUN 4 0:10 6.71% httpd
30687 w_new-vw_5e17 1 -4 0 76232K 38616K RUN 1 0:02 6.69% httpd
30561 w_avtolo_3761 1 105 0 76512K 39500K select 2 0:03 6.67% httpd
30519 w_new-vw_5e17 1 -4 0 76784K 39396K ufs 4 0:04 6.62% httpd
31018 w_tucson_bd87 1 -4 0 76000K 36464K RUN 6 0:00 6.62% httpd
30867 w_as-vid_ac7e 1 -4 0 76356K 37452K RUN 6 0:01 6.60% httpd
30997 w_rfscla_06fc 1 -4 0 76820K 38652K RUN 2 0:01 6.60% httpd
31015 w_tucson_bd87 1 -4 0 76024K 36908K RUN 1 0:00 6.59% httpd
30532 w_shop2y_b290 1 -4 0 76904K 38232K ufs 4 0:04 6.55% httpd
30936 w_tucson_bd87 1 -4 0 77836K 39740K RUN 6 0:01 6.53% httpd
30376 w_vannay_4606 1 -4 0 78252K 41596K RUN 1 0:05 6.53% httpd
30452 w_new-vw_5e17 1 -4 0 77356K 40124K RUN 6 0:04 6.50% httpd
30201 w_berega_3091 1 -4 0 81636K 42992K RUN 4 0:07 6.46% httpd
30840 w_forum-_89b3 1 -4 0 76412K 38752K RUN 4 0:02 6.44% httpd
30809 w_rm-for_636d 1 -4 0 76148K 38028K RUN 6 0:02 6.43% httpd
30700 w_ott-gu_2def 1 -4 0 76888K 39272K RUN 4 0:02 6.40% httpd
29971 w_mobi-m_2e46 1 -4 0 76872K 41484K RUN 4 0:05 6.40% httpd
31016 w_wplane_ebed 1 -4 0 76320K 37056K RUN 7 0:00 6.39% httpd
30839 w_lyubar_f1e2 1 -4 0 77332K 39788K RUN 4 0:02 6.36% httpd
31004 w_digitr_77d4 1 -4 0 76016K 36656K CPU5 5 0:01 6.36% httpd
# vmstat
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id
110 9 0 10884836 384132 1167 2 2 1 1340 273 0 0 19 809 1282 13 9
78
110 10 0 10950600 382964 4530 0 0 0 3968 0 1 13 3925 35129 162137 9
87 4
131 9 0 10855252 387356 3552 0 2 0 4782 0 1 9 2505 27529 166233 5
92 3
129 11 0 10924792 379056 6723 0 4 0 4658 0 1 26 4015 29908 162047 3
94 3
122 10 1 10936140 367864 5289 0 2 0 2145 0 0 17 3788 27790 162641 3
95 3
66 55 0 10807504 378600 4236 0 4 0 6818 0 1 6 3164 26302 167760 2
95 2
128 11 0 10815488 374308 4092 0 3 0 2910 0 1 13 2962 32777 167559 2
93 5
80 42 1 10784792 377988 4592 0 3 0 5821 0 5 13 3296 28117 163950 3
94 2
64 62 0 10785528 377076 5640 0 3 0 5238 0 1 6 2890 31425 163924 3
94 3
52 77 0 10656100 386120 2994 0 1 0 4768 0 1 8 2427 23684 159985 2
95 3
70 52 0 10591528 393632 3381 0 0 0 5335 0 1 4 3010 27593 158225 3
94 3
105 9 0 10549508 402488 5426 0 0 0 7436 0 2 7 2318 31753 168856 3
94 4
77 40 0 10554208 397512 5848 0 0 0 4448 0 1 9 2275 31423 156746 3
92 5
114 9 4 10483152 403624 3369 0 0 0 4974 0 0 5 2243 27000 160225 2
93 5
109 9 0 10550332 394796 4972 0 1 0 2708 0 1 11 2515 26056 158628 3
92 5
125 9 0 10518288 423240 4256 0 3 0 11099 0 1 9 2779 28773 158967 3
93 4
118 11 0 10529144 415176 5887 4 3 0 3670 0 1 11 2595 30282 159381 2
95 3
116 9 0 10497144 418012 3407 0 1 0 3929 0 1 11 2815 24890 159056 2
93 5
119 9 0 10463764 419260 3391 0 0 0 3797 0 1 9 2558 28508 159086 2
94 3
5 116 0 10436304 420156 4263 0 0 0 4368 0 24 8 3348 27563 159025 3
94 4
84 43 0 10563104 412824 5012 0 0 0 3630 0 2 32 2658 32205 158341 1
94 5
82 59 0 10620056 415188 6158 4 2 0 4858 0 11 11 2743 30929 159186 3
94 4
87 39 0 10453904 431328 3867 0 0 0 8894 0 1 5 2560 28903 168316 2
95 3
102 37 2 10642596 417724 6146 0 1 0 3285 0 5 11 2396 35490 158978 2
95 4
145 10 0 10470216 461180 5202 4 0 0 15462 0 77 10 2522 30525 163784 4
93 3
122 12 0 10507604 452820 4561 0 0 50 2341 0 138 6 2368 31650 158428 3
94 3
108 29 0 10669764 439152 4808 0 0 0 1294 0 4 8 3456 27089 166675 2
95 3
124 10 0 10631948 444132 6171 0 0 0 7239 0 6 6 2875 31822 167193 4
94 3
7 users Load 54.35 28.49 23.64 Nov 27 17:04
Mem:KB REAL VIRTUAL VN PAGER SWAP
PAGER
Tot Share Tot Share Free in out in out
Act 927804 34556 8233412 53740 489892 count
All 8071840 267008983194432 3143860 pages
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt 1189 cow 16584
total
89 9454 153k 533325086 2022 309 5066 425580 wire 1:
atkb
1499680 act
14: ata
90.0%Sys 0.6%Intr 2.9%User 0.0%Nice 6.4%Idl 5776020 inact 546
18: em0
| | | | | | | | | | 425236 cache 31
20: ata
=============================================>> 64968 free 2001
cpu0: time
daefr 2002
cpu1: time
Namei Name-cache Dir-cache 2333 prcfr 2001
cpu2: time
Calls hits % hits % react 2001
cpu3: time
207699 207669 100 5 0 pdwak 2000
cpu7: time
3514 zfod pdpgs 2001
cpu4: time
Disks ad4 ad6 ad8 3463 ozfod intrn 2000
cpu6: time
KB/t 16.00 4.20 6.90 98 %slo-z 114784 buf 2001
cpu5: time
tps 13 1 1 3077 tfree 229 dirtybuf
MB/s 0.21 0.00 0.01 100000 desiredvnodes
% busy 0 1 1 89091 numvnodes
24971 freevnodes
Mark Linimon
2007-Dec-01 14:34 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
On Sun, Dec 02, 2007 at 12:37:32AM +0300, Alexey Vlasov wrote:> I decided to install FreeBSD 6.2 i386 on one of the servers, and the > result was totally non productive.The 6.x series was intended to get us back to the stability that we had had pre-SMP integration. I believe we mostly succeeded. One of the major thrusts for 7.0 development was to fix the performance regressions that had been introduced. From the results that I have seen (I am not one of the participants), there has been major progress over the past 2 years in removing yet one bottleneck after another. Recent tests show us to be on a par with Linux on a number of benchmarks; of course, we need more people testing 7.0 in real-world environments to confirm this. You may want to try the 7.0 release candidate on a testbed to see if your results have improved as much as we think that they will have. mcl
Daniel Gerzo
2007-Dec-01 14:39 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
Hello Alexey, Saturday, December 1, 2007, 10:37:32 PM, you wrote:> I use OS Linux on my hosting for web-servers, base for all servers is > the same m/b S5000PAL ( SR1500), 2 quad kernel cpu Xeon E5320 or E5345, > 8Gb RAM. I decided to install FreeBSD 6.2 i386 on one of the servers, > and the result was totally non productive.Please try with RELENG_7 (aka. FreeBSD 7.0-BETA3) and ULE scheduler. -- Best regards, Daniel mailto:danger@FreeBSD.org
Alexey Popov
2007-Dec-03 01:19 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
Hi Mark Linimon wrote:>> I used 7.0-BETA3 and it is much worse. > Ouch. A lot of systems see improvement. Thanks for trying it > out. I hope that one of the people that has been doing the actual > work can now comment (I am just an onlooker), and that you can be > patient in the meantime. > Unfortunately, Kris, who often looks at these kind of issues, is > traveling for all of December and thus off the net.Is there any other FreeBSD developer who can take care of performance problems on many-cores systems? Seems like upcoming 7-RELEASE and 6.3-RELEASE would be completely unusable for us on that kind of systems i.e. mostly on all modern hardware. Now we also have terribly performing PostgreSQL on 8-core server. We noticed the slowdown after moving PostgreSQL from 2xXeon 3.0 Apache+PostgreSQL server to dedicated PostgreSQL server. I collected some stats (see attach) before moving to Linux. With best regards, Alexey Popov -------------- next part -------------- last pid: 58755; load averages: 26.42, 20.88, 14.00 up 25+22:12:42 11:51:11 84 processes: 29 running, 55 sleeping CPU states: % user, % nice, % system, % interrupt, % idle Mem: 1149M Active, 1971M Inact, 464M Wired, 120M Cache, 214M Buf, 161M Free Swap: 2048M Total, 72K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 58541 pgsql 1 -4 0 1068M 655M semwai 5 0:17 27.08% postgres 58664 pgsql 1 4 0 1068M 458M sblock 0 0:05 25.49% postgres 58677 pgsql 1 129 0 1067M 291M RUN 2 0:04 24.55% postgres 58713 pgsql 1 130 0 1067M 210M RUN 5 0:03 23.99% postgres 58705 pgsql 1 130 0 1069M 214M CPU7 4 0:03 23.03% postgres 58679 pgsql 1 129 0 1068M 306M RUN 1 0:04 22.45% postgres 58724 pgsql 1 130 0 1068M 179M RUN 4 0:02 22.19% postgres 58698 pgsql 1 129 0 1068M 238M RUN 0 0:03 22.19% postgres 58715 pgsql 1 130 0 1068M 188M RUN 0 0:02 21.68% postgres 58727 pgsql 1 131 0 1069M 119M RUN 1 0:01 20.15% postgres 58658 pgsql 1 125 0 1069M 304M CPU0 0 0:03 19.99% postgres 58728 pgsql 1 131 0 1068M 104M RUN 3 0:01 19.57% postgres 58726 pgsql 1 -4 0 1067M 140M semwai 6 0:01 18.83% postgres 58730 pgsql 1 131 0 1067M 96504K RUN 2 0:01 17.42% postgres 58695 pgsql 1 128 0 1069M 194M RUN 0 0:02 16.37% postgres 58731 pgsql 1 131 0 1068M 57016K CPU2 4 0:01 14.77% postgres 58737 pgsql 1 131 0 1067M 53680K RUN 3 0:01 13.45% postgres 58738 pgsql 1 131 0 1067M 50508K RUN 4 0:00 13.45% postgres 58743 pgsql 1 131 0 1067M 29588K CPU4 2 0:00 9.74% postgres 58712 pgsql 1 -4 0 1069M 60488K semwai 6 0:01 9.57% postgres 58733 pgsql 1 131 0 1068M 42968K RUN 6 0:00 8.61% postgres 58742 pgsql 1 131 0 1067M 27284K RUN 1 0:00 6.65% postgres 58740 pgsql 1 131 0 1067M 20096K RUN 7 0:00 5.60% postgres 58736 pgsql 1 -4 0 1067M 26164K semwai 6 0:00 5.38% postgres 58734 pgsql 1 130 0 1068M 33496K RUN 7 0:00 4.04% postgres 58741 pgsql 1 4 0 1067M 23308K sbwait 7 0:00 3.85% postgres 58735 pgsql 1 -4 0 1067M 26152K semwai 5 0:00 3.50% postgres 47990 pgsql 1 132 0 1066M 4300K select 6 163:53 1.51% postgres 58750 pgsql 1 131 0 1067M 6816K RUN 5 0:00 1.00% postgres 58751 pgsql 1 131 0 1067M 6368K RUN 6 0:00 1.00% postgres 58748 pgsql 1 131 0 1067M 6456K CPU6 6 0:00 1.00% postgres 58732 pgsql 1 4 0 1067M 6772K sbwait 4 0:00 0.88% postgres 58744 pgsql 1 -4 0 1067M 10956K semwai 6 0:00 0.51% postgres 58745 pgsql 1 4 0 1067M 6804K sbwait 1 0:00 0.51% postgres 2 users Load 27.56 21.69 14.53 Dec 3 11:51 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 1272944 6956 1373904 10528 156944 count All 1516224 8576 5907668 18076 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 222 cow 16147 total 38 46 50 9210 29k 2349 1313 223 28k 3475 zfod atkbd0 1 3449 ozfod ata0 irq14 75.9%Sys 0.4%Intr 21.3%User 0.0%Nice 2.4%Idle 99%ozfod 161 em0 mfi0 1 | | | | | | | | | | | daefr 1999 cpu0: time ======================================>>>>>>>>>>> 1893 prcfr 1999 cpu1: time 282 dtbuf 4131 totfr 1998 cpu2: time Namei Name-cache Dir-cache 100000 desvn react 1998 cpu3: time Calls hits % hits % 6850 numvn pdwak 1995 cpu4: time 447 439 98 5474 frevn pdpgs 2000 cpu7: time 20 intrn 1996 cpu5: time Disks mfid0 502808 wire 2001 cpu6: time KB/t 113 1274232 act tps 26 2017344 inact MB/s 2.85 122864 cache %busy 2 39272 free 219632 buf %vmstat 1 procs memory page disk faults cpu r b w avm fre flt re pi po fr sr mf0 in sy cs us sy id 1 2 0 1141400 370652 1762 0 0 0 670 0 0 1184 602 1007 6 11 84 3 5 0 1147876 365412 30021 0 0 0 3867 0 8 1407 2393 5490 4 6 91 1 0 0 1141628 370788 28359 0 0 0 10891 0 4 1303 1771 5241 14 6 81 2 0 0 1152000 361456 37957 0 0 0 2057 0 2 1204 1325 4791 11 12 78 1 0 0 1149292 363388 25155 0 0 0 4254 0 1 1182 1259 4712 10 5 85 0 0 0 1138188 373212 17983 0 0 0 6909 0 5 1341 2058 5265 4 4 93 3 4 0 1147944 365664 28170 0 0 0 2215 0 5 1235 1636 4875 8 6 86 1 0 0 1144412 367376 38756 0 0 0 4171 0 12 1372 2020 5604 10 16 74 2 0 0 1147844 364192 45418 0 0 0 6197 0 4 1336 2084 5256 9 10 80 2 1 0 1149456 362972 35356 0 0 0 2852 0 4 1234 1432 4942 16 9 75 0 0 0 1141436 369512 30652 0 0 0 7443 0 8 1289 1720 5047 6 8 86 1 1 0 1141628 370820 22387 0 0 0 2351 0 0 1161 1076 4660 10 4 86 ^C
Zoran Kolic
2007-Dec-04 07:36 UTC
2 x quad-core system is slower that 2 x dual core on FreeBSD
> > Now we also have terribly performing PostgreSQL on 8-core server. We > > noticed the slowdown after moving PostgreSQL from 2xXeon 3.0 > > Apache+PostgreSQL server to dedicated PostgreSQL server. I collected > > some stats (see attach) before moving to Linux.I'm sure that some code optimization could help to have multicore enhancement. Rapidmind example comes as proof that there is room to make things better. I saw some text on Sun compiler making code more able to use cores and get the speed. Intel also tried to do the same. Maybe the same extent of work should be put on app optimization, aside to os changes. Despite I like bsd better, must to say that linux does not sleep and wait. Zoran