Greg Rivers
2005-Nov-21 15:55 UTC
Recurring problem: processes block accessing UFS file system
I've recently put up three busy email relay hosts running 6.0-STABLE.
Performance is excellent except for a nagging critical issue that keeps
cropping up.
/var/spool is its own file system mounted on a geom stripe of four BSD
partitions (details below). Once every two or three days all the
processes accessing /var/spool block forever in disk wait. All three
machines suffer this problem. No diagnostic messages are generated and
the machines continue running fine otherwise, but a reboot is required to
clear the condition. This problem occurs during normal operation, but is
particularly likely to occur during a backup when dump makes a snapshot.
There doesn't appear to be a problem with gstripe, as gstripe status is
"UP" and I can read the raw device just fine while processes continue
to
block on the file system. I tried running a kernel with WITNESS and
DIAGNOSTIC, but these options shed no light.
If I catch the problem early enough I can break successfully into kdb;
otherwise, if too many processes stack up, the machine hangs going into
kdb and must be power-cycled.
I'd appreciate any insight anyone may have into this problem or advise on
turning this report into a coherent PR.
--
Greg Rivers
$ uname -a
FreeBSD mx17.sac.fedex.com 6.0-STABLE FreeBSD 6.0-STABLE #0: Fri Nov 18 23:34:59
CST 2005 root@mx17.sac.fedex.com:/usr/obj/usr/src/sys/MX i386
(kernel built from sources current at time of build)
$ df -h /var/spool
Filesystem Size Used Avail Capacity Mounted on
/dev/stripe/gs0 260G 2.2G 237G 1% /var/spool
$ mount
/dev/da0a on / (ufs, local, soft-updates)
devfs on /dev (devfs, local)
/dev/md0 on /tmp (ufs, asynchronous, local, noatime)
/dev/mirror/gm0 on /var (ufs, local, soft-updates)
/dev/mirror/gm1d on /var/log (ufs, local, soft-updates)
/dev/stripe/gs0 on /var/spool (ufs, local, noatime, soft-updates)
/dev/md1 on /var/spool/mqueue/.hoststat (ufs, asynchronous, local, noatime)
/dev/mirror/gm1a on /usr (ufs, local, soft-updates)
/dev/mirror/gm2 on /home (ufs, local, soft-updates)
devfs on /var/named/dev (devfs, local)
$ gstripe list
Geom name: gs0
State: UP
Status: Total=4, Online=4
Type: AUTOMATIC
Stripesize: 131072
ID: 3224140609
Providers:
1. Name: stripe/gs0
Mediasize: 283460501504 (264G)
Sectorsize: 512
Mode: r1w1e1
Consumers:
1. Name: da0d
Mediasize: 70865144320 (66G)
Sectorsize: 512
Mode: r1w1e2
Number: 0
2. Name: da1d
Mediasize: 70865144320 (66G)
Sectorsize: 512
Mode: r1w1e2
Number: 2
3. Name: da2d
Mediasize: 70865144320 (66G)
Sectorsize: 512
Mode: r1w1e2
Number: 1
4. Name: da3d
Mediasize: 70865144320 (66G)
Sectorsize: 512
Mode: r1w1e2
Number: 3
(Host is a HP/Compaq DL380 G4)
$ dmesg
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.0-STABLE #0: Fri Nov 18 23:34:59 CST 2005
root@mx17.sac.fedex.com:/usr/obj/usr/src/sys/MX
ACPI APIC Table: <HP 00000083>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.40GHz (3400.14-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf41 Stepping = 1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x649d<SSE3,RSVD2,MON,DS_CPL,EST,CNTX-ID,CX16,<b14>>
AMD Features=0x20000000<LM>
Hyperthreading: 2 logical CPUs
real memory = 3758043136 (3583 MB)
avail memory = 3683098624 (3512 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
ioapic3 <Version 2.0> irqs 72-95 on motherboard
ioapic4 <Version 2.0> irqs 96-119 on motherboard
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <HP P51> on motherboard
acpi0: Power Button (fixed)
pci_link0: <ACPI PCI Link LNKA> irq 7 on acpi0
pci_link1: <ACPI PCI Link LNKB> irq 5 on acpi0
pci_link2: <ACPI PCI Link LNKC> irq 5 on acpi0
pci_link3: <ACPI PCI Link LNKD> irq 5 on acpi0
pci_link4: <ACPI PCI Link LNKE> irq 0 on acpi0
pci_link5: <ACPI PCI Link LNKF> irq 5 on acpi0
pci_link6: <ACPI PCI Link LNKG> irq 5 on acpi0
pci_link7: <ACPI PCI Link LNKH> irq 7 on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib2
bge0: <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2100> mem
0xfddf0000-0xfddfffff irq 25 at device 1.0 on pci3
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
bge0: Ethernet address: 00:13:21:09:89:79
bge1: <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2100> mem
0xfdde0000-0xfddeffff irq 26 at device 1.1 on pci3
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
bge1: Ethernet address: 00:13:21:09:89:78
pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci2
pci4: <ACPI PCI bus> on pcib3
ciss0: <HP Smart Array 6i> port 0x4000-0x40ff mem
0xfdef0000-0xfdef1fff,0xfde80000-0xfdebffff irq 51 at device 3.0 on pci4
ciss0: [GIANT-LOCKED]
pcib4: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci5: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 0.2 on pci5
pci10: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection, Version - 2.1.7> port
0x5000-0x503f mem 0xfdfe0000-0xfdffffff,0xfdf80000-0xfdfbffff irq 97 at device
1.0 on pci10
em0: Ethernet address: 00:11:0a:56:07:2e
em1: <Intel(R) PRO/1000 Network Connection, Version - 2.1.7> port
0x5040-0x507f mem 0xfdf60000-0xfdf7ffff irq 98 at device 1.1 on pci10
em1: Ethernet address: 00:11:0a:56:07:2f
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib7
pci1: <display, VGA> at device 3.0 (no driver attached)
pci1: <base peripheral> at device 4.0 (no driver attached)
pci1: <base peripheral> at device 4.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x500-0x50f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <Standard PC COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xcbfff,0xcc000-0xcd7ff,0xcd800-0xcefff,0xee000-0xeffff
on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
Timecounters tick every 1.000 msec
acd0: DVDROM <DV-28E-C/B.4F> at ata0-master PIO4
da0 at ciss0 bus 0 target 0 lun 0
da0: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-0 device
da0: 135.168MB/s transfers
da0: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C)
da1 at ciss0 bus 0 target 1 lun 0
da1: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-0 device
da1: 135.168MB/s transfers
da1: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C)
da2 at ciss0 bus 0 target 2 lun 0
da2: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-0 device
da2: 135.168MB/s transfers
da2: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C)
da3 at ciss0 bus 0 target 3 lun 0
da3: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-0 device
da3: 135.168MB/s transfers
da3: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C)
da4 at ciss0 bus 0 target 4 lun 0
da4: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-0 device
da4: 135.168MB/s transfers
da4: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C)
da5 at ciss0 bus 0 target 5 lun 0
da5: <COMPAQ RAID 0 VOLUME OK> Fixed Direct Access SCSI-0 device
da5: 135.168MB/s transfers
da5: 69459MB (142253280 512 byte sectors: 255H 32S/T 17433C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
GEOM_STRIPE: Device gs0 created (id=3224140609).
GEOM_STRIPE: Disk da0d attached to gs0.
GEOM_MIRROR: Device gm2 created (id=3517057742).
GEOM_MIRROR: Device gm2: provider da1a detected.
GEOM_STRIPE: Disk da1d attached to gs0.
GEOM_STRIPE: Disk da2d attached to gs0.
GEOM_MIRROR: Device gm2: provider da3a detected.
GEOM_MIRROR: Device gm2: provider da3a activated.
GEOM_MIRROR: Device gm2: provider da1a activated.
GEOM_MIRROR: Device gm2: provider mirror/gm2 launched.
GEOM_STRIPE: Disk da3d attached to gs0.
GEOM_STRIPE: Device gs0 activated.
GEOM_MIRROR: Device gm0 created (id=3319228670).
GEOM_MIRROR: Device gm0: provider da4a detected.
GEOM_MIRROR: Device gm1 created (id=283379314).
GEOM_MIRROR: Device gm1: provider da4d detected.
GEOM_MIRROR: Device gm0: provider da5a detected.
GEOM_MIRROR: Device gm0: provider da5a activated.
GEOM_MIRROR: Device gm0: provider da4a activated.
GEOM_MIRROR: Device gm0: provider mirror/gm0 launched.
GEOM_MIRROR: Device gm1: provider da5d detected.
GEOM_MIRROR: Device gm1: provider da5d activated.
GEOM_MIRROR: Device gm1: provider da4d activated.
GEOM_MIRROR: Device gm1: provider mirror/gm1 launched.
Trying to mount root from ufs:/dev/da0a
em1: link state changed to UP
I obtained the following process listing and traces from kdb. I traced
mksnap_ffs which was blocked in "ufs", and two random sendmail
processes
that were blocked in "ufs" and "suspfs" respectively.
...
Expensive timeout(9) function: 0xc04c9082(0xc4c97d80) 0.002908205 s
...
# set -o vi
# sync;sync
# KDB: enter: Line break on console
timeout stopping cpus
[thread pid 11 tid 100005 ]
Stopped at kdb_enter+0x30: leave
db> ps
pid proc uid ppid pgrp flag stat wmesg wchan cmd
13273 c9301a3c 11521 13271 13270 0004000 [SLPQ piperd 0xc56897f8][SLP] awk
13272 c9301c48 11521 13271 13270 0004100 [SLPQ ufs 0xc502ce28][SLP] sendmail
13271 c9266a3c 11521 13270 13270 0004000 [SLPQ piperd 0xc4ff3cc0][SLP] ksh
13270 c53d8418 11521 13269 13270 0004000 [SLPQ wait 0xc53d8418][SLP] sh
13269 c92cca3c 0 689 689 0000000 [SLPQ piperd 0xc530bcc0][SLP] cron
13267 c9302000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13266 c92ccc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13265 c92ce000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13264 c92ce20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13263 c92ce418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13262 c92ce624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13261 c930220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13260 c92ce830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13259 c9302418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13258 c926a000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13257 c9302624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13256 c926a20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13255 c9302830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13254 c926a418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13253 c926a624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13252 c926a830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13251 c9302a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13250 c8feec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13249 c60fdc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13248 c926aa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13247 c92cea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13246 c92cec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13245 c926ac48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13244 c926d000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13243 c92cf000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13242 c926d20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13241 c92cf20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13240 c91a8000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13239 c60f6418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13238 c9119418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13237 c926d418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13235 c926d624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13234 c9190a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13233 c926d830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13232 c90a4830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13231 c9190624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13230 c926da3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13229 c586ca3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13228 c918a000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13227 c90a4c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13226 c926dc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13225 c918a20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13224 c918a418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13223 c918a624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13222 c4fa2a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13221 c9193830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13220 c926e000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13219 c918a830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13218 c9193a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13217 c9193c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13216 c918aa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13214 c926e418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13213 c90a4418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13212 c926e624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13211 c91a6000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13210 c926e830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13209 c9190830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13208 c501da3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13207 c91a620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13206 c91aa000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13205 c90a4624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13204 c91a6418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13203 c918ac48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13202 c91a6624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13201 c9190000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13199 c91a6830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13198 c919020c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13197 c9190418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13196 c91a6a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13195 c90a4a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13194 c91a6c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13186 c501d000 11521 13181 13178 0004000 [SLPQ piperd 0xc4ff0000][SLP] awk
13185 c587420c 11521 13181 13178 0004100 [SLPQ ufs 0xc502ce28][SLP] sendmail
13181 c9191000 11521 13178 13178 0004000 [SLPQ piperd 0xc8b44990][SLP] ksh
13178 c9191418 11521 13175 13178 0004000 [SLPQ wait 0xc9191418][SLP] sh
13175 c91a820c 0 689 689 0000000 [SLPQ piperd 0xc4fef990][SLP] cron
13174 c91a8418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13173 c9191624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13172 c9119624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13171 c91a8624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13170 c91a8830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13169 c91a8a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13168 c91a8c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13167 c91a9000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13166 c91a920c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13165 c91a9418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13164 c9119830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13163 c91a9624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13162 c9191830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13161 c9119a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13160 c9119c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13159 c91a9830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13158 c9191a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13157 c9122000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13156 c91a9a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13155 c91a9c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13153 c91aa20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13152 c9191c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13151 c91aa418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13150 c91aa624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13149 c91aa830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13148 c9192000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13147 c912220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13146 c9122418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13145 c919220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13144 c9122624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13143 c9192418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13142 c9122830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13141 c9192624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13140 c91aaa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13139 c9122a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13138 c91aac48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13137 c9192830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13136 c9192a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13135 c9192c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13134 c91af000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13133 c9050c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13132 c9122c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13131 c9193000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13130 c909c000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13129 c919320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13128 c9193418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13127 c909c20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13126 c9193624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13125 c8ff6418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13124 c8ff6624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13123 c8ff6830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13122 c9123000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13121 c909c418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13120 c8ff6a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13119 c912320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13118 c909c624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13117 c8ff6c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13116 c909c830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13115 c909ca3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13114 c9123418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13113 c9123624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13112 c9123830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13111 c909cc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13110 c9123a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13109 c9123c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13108 c909e000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13107 c9124000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13106 c9045000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13105 c909e20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13104 c909e418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13103 c912420c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13102 c904520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13101 c909e624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13100 c909e830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13099 c9124418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13098 c909ea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13097 c909ec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13096 c90a1000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13095 c9124624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13094 c90a120c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13093 c90a1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13092 c90a1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13091 c9045418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13090 c9045624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13089 c9045830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13088 c9045a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13087 c9124830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13086 c9124a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13085 c9124c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13084 c90a1830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13083 c90a1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13082 c8fa8a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13081 c90a1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13080 c90a2000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13079 c90a220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13078 c8fa8c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13077 c90a2418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13076 c8fee000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13075 c9045c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13074 c90a2624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13073 c8fee20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13072 c8fee418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13071 c904c000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13070 c90a2830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13069 c8fee624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13068 c90a2a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13067 c90a2c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13066 c90a4000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13065 c90a420c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13064 c8fee830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13063 c904c20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13062 c904c418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13061 c904c624 11521 13059 13057 0004000 [SLPQ piperd 0xc5689990][SLP] awk
13060 c8feea3c 11521 13059 13057 0004100 [SLPQ ufs 0xc502ce28][SLP] sendmail
13059 c904c830 11521 13057 13057 0004000 [SLPQ piperd 0xc8b44198][SLP] ksh
13057 c904ca3c 11521 13055 13057 0004000 [SLPQ wait 0xc904ca3c][SLP] sh
13055 c8ff2000 0 689 689 0000000 [SLPQ piperd 0xc4ff3b28][SLP] cron
13053 c904cc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13052 c8ff220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13051 c904d000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13050 c904d20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13049 c533520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13048 c8ff2418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13047 c8ef6418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13046 c904d418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13045 c53d1830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13044 c8ef6624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13043 c904d624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13042 c8ff2624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13041 c8ff2830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13040 c904d830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13039 c904da3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13038 c904dc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13037 c8ef6830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13036 c8ef6a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13035 c8ff2a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13034 c8ff2c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13033 c9050000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13032 c8ef6c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13031 c8fa1000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13030 c8fa120c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13029 c8fa1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13028 c8ff3000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13027 c905020c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13026 c9050418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13025 c8fa1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13024 c8ff320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13023 c8fa1830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13022 c8ff3418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13021 c8ff3624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13020 c8fa1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13019 c60fc830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13018 c9050624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13017 c8ff3830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13016 c8fa1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13015 c8fa4000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13014 c8ff3a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13013 c9050830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13012 c8ff3c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13011 c8ff5000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13010 c502320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13009 c8ff520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13008 c9050a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13007 c8fa420c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13006 c8fa4418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13005 c8ff5418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13004 c8fa4624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13003 c5023830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13002 c8ff5624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13001 c8ff5830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
13000 c8fa4830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12999 c8fa4a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12998 c4fed830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12997 c8ff5a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12996 c8fa4c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12995 c8ff5c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12994 c8ff6000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12993 c586f000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12992 c8fa5000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12991 c8fa520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12990 c586c000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12989 c8fa5418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12988 c4fa1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12987 c8fa5624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12986 c8ff620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12985 c8fa5830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12984 c8eee000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12983 c8eee20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12982 c8eee418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12981 c5550624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12980 c8fa5a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12979 c60fea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12978 c8fa5c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12977 c8fa8000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12976 c8fa820c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12975 c5588418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12974 c8eee624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12973 c8fa8418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12972 c8fa8624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12971 c5335c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12970 c8fa8830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12969 c60fec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12968 c8eee830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12967 c8eeea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12966 c8eac000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12965 c8eeec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12964 c4d7ac48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12963 c4fa1000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12962 c8ef3000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12961 c8ef320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12960 c8ef3418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12959 c8eac20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12958 c8ef3624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12957 c501d20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12956 c8ef3830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12955 c8eac418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12954 c8ef3a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12953 c8eac624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12952 c5024000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12951 c8ef3c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12950 c586c830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12949 c8ef4000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12948 c8ef420c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12947 c8eac830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12946 c8eaca3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12945 c8eacc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12944 c8eaf000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12943 c8eaf20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12942 c8eaf418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12941 c8ef4418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12940 c8eaf624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12939 c8ef4624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12938 c8ef4830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12937 c4f9d624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12936 c8eaf830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12935 c8eafa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12934 c8eafc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12933 c8eb1000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12932 c4fa220c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12931 c8eb120c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12930 c559020c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12929 c8eb1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12928 c8eb1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12927 c8ef4a3c 11521 12925 12924 0004000 [SLPQ piperd 0xc4ff4cc0][SLP] awk
12926 c8ef4c48 11521 12925 12924 0004100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12925 c554b000 11521 12924 12924 0004000 [SLPQ piperd 0xc4ff3990][SLP] ksh
12924 c586cc48 11521 12923 12924 0004000 [SLPQ wait 0xc586cc48][SLP] sh
12923 c8ef5000 0 689 689 0000000 [SLPQ piperd 0xc501ccc0][SLP] cron
12922 c532e830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12921 c4fa1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12920 c53d9830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12919 c53d1418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12918 c8ef520c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12917 c8ef5418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12916 c8ef5624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12915 c53d9418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12914 c8ef5830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12913 c8ef5a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12912 c8ef5c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12911 c8eb1830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12910 c8ef6000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12909 c8eb1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12908 c8ef620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12907 c586fc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12906 c60f6000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12905 c8eb1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12904 c4f9d000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12903 c5873c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12902 c5023a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12901 c4f9da3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12900 c5024c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12899 c8eb3000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12898 c4f9d20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12897 c5586418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12896 c8eb320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12895 c8eb3418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12894 c8eb3624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12893 c8eb3830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12892 c8eb3a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12891 c8eb3c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12890 c4d7a418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12889 c5873418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12888 c5551830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12887 c501d418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12886 c53d820c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12885 c586fa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12884 c5023c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12883 c532e000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12882 c5335830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12881 c53d8c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12880 c4fec20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12879 c60fd20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12878 c586c418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12877 c558620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12876 c5550418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12875 c555020c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12874 c4d76000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12873 c532ea3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12872 c5874418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12871 c532ec48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12870 c5339c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12869 c4fec624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12868 c5023418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12867 c60fe830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12866 c4d7aa3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12865 c60fd830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12864 c586f20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12863 c5551418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12862 c53d1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12861 c60fe000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12860 c53d1a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12859 c60fe624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12858 c532e418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12857 c5873000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12856 c5024418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12855 c554b20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12854 c5550c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12853 c5874000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12852 c4fed418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12851 c554bc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12850 c4fa1c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12849 c5335000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12847 c5874830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12846 c4d7a624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12845 c53d1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12844 c5586624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12843 c60f6a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12842 c4fec830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12841 c5588c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12840 c4fa1624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12839 c5550830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12838 c53d8624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12837 c60fc000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12836 c60fc20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12835 c5590c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12834 c5588624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12833 c60fe20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12832 c5335418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12831 c5590418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12830 c5335624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12829 c60f620c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12828 c5586000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12827 c586c20c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12826 c4fec000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12825 c5873624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12824 c60fcc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12823 c5588830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12822 c4fa2624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12821 c60fd624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12820 c5586c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12819 c53d8830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12818 c587320c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12817 c53d120c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12816 c60fda3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12815 c60fc624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12814 c5551624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12813 c4f9dc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12812 c53d9624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12811 c4fa2418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12810 c60f6624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12809 c60f6c48 100 673 673 0000100 [SLPQ ufs 0xc53db5a8][SLP] sendmail
12808 c53d8a3c 100 673 673 0000100 [SLPQ ufs 0xc53e6388][SLP] sendmail
12807 c554b624 100 673 673 0000100 [SLPQ ufs 0xc53db388][SLP] sendmail
12806 c5588000 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12805 c533920c 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12804 c5339000 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12803 c5339624 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12801 c554ba3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12800 c60fd418 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12799 c4fa2c48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12798 c558820c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12797 c5551a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12788 c5024624 11521 12784 12781 0004000 [SLPQ piperd 0xc501c4c8][SLP] awk
12787 c5590830 11521 12784 12781 0004100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12784 c4fe8c48 11521 12781 12781 0004000 [SLPQ piperd 0xc69167f8][SLP] ksh
12781 c5551000 11521 12778 12781 0004000 [SLPQ wait 0xc5551000][SLP] sh
12778 c532e624 0 689 689 0000000 [SLPQ piperd 0xc56894c8][SLP] cron
12777 c5550a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12776 c60fe418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12775 c5874c48 100 1 12775 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12773 c5023624 100 673 673 0000100 [SLPQ ufs 0xc53f1e28][SLP] sendmail
12772 c5590624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12771 c5586830 100 1 12771 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12769 c4d7a830 100 1 12769 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12765 c53d9a3c 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12762 c4f9d418 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12761 c5586a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12760 c554b830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12759 c4fe8a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12758 c53d1000 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12752 c4fe8418 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12750 c53d9c48 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12732 c5874a3c 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12656 c5339830 100 1 12656 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12559 c5588a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12542 c554b418 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12500 c586f830 100 673 673 0000100 [SLPQ suspfs 0xc4d6946c][SLP] sendmail
12463 c4fed20c 100 1 12463 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12454 c586f418 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail
12384 c555120c 100 1 12384 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
12079 c4fa1830 100 1 12079 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
11910 c586c624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
11707 c4fec418 11008 11706 11151 0004102 [SLPQ ufs 0xc502ce28][SLP] mksnap_ffs
11706 c5024a3c 11008 11151 11151 0004002 [SLPQ wait 0xc5024a3c][SLP] sh
11654 c60fd000 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail
11646 c5873a3c 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail
11161 c60fc418 11008 11107 11161 0004002 [SLPQ ttyin 0xc8bf1410][SLP] ksh
11151 c501dc48 11008 11110 11151 0004002 [SLPQ wait 0xc501dc48][SLP] dump
11110 c60f6830 11008 11107 11110 0004002 [SLPQ pause 0xc60f6864][SLP] ksh
11107 c4d7620c 11008 11106 11107 0000100 [SLPQ select 0xc067dac4][SLP] screen
11106 c501d830 11008 5725 11106 0004102 [SLPQ pause 0xc501d864][SLP] screen
10540 c4fed624 100 1 10540 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
9815 c4fa2830 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail
9623 c4f9d830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
8980 c4fecc48 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
8719 c5873830 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
7925 c53d9000 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail
7149 c4fedc48 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail
6972 c586f624 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
5725 c5024830 11008 5724 5725 0004002 [SLPQ pause 0xc5024864][SLP] ksh
5724 c60fca3c 11008 5719 5719 0000100 [SLPQ select 0xc067dac4][SLP] sshd
5719 c4fa120c 0 668 5719 0004100 [SLPQ sbwait 0xc588d900][SLP] sshd
5077 c5335a3c 100 673 673 0000100 [SLPQ select 0xc067dac4][SLP] sendmail
2513 c502420c 0 2512 2513 0004002 [SLPQ ttyin 0xc4da1810][SLP] ksh
2512 c4fe8000 0 1 2512 0004102 [SLPQ wait 0xc4fe8000][SLP] login
2511 c5550000 0 1 2511 0004002 [SLPQ ttyin 0xc4d97010][SLP] getty
2510 c5590000 0 1 2510 0004002 [SLPQ ttyin 0xc4da1010][SLP] getty
2509 c4feda3c 0 1 2509 0004002 [SLPQ ttyin 0xc4da3810][SLP] getty
2508 c53d920c 0 1 2508 0004002 [SLPQ ttyin 0xc4da1410][SLP] getty
2497 c5590a3c 0 1 2497 0000000 [SLPQ select 0xc067dac4][SLP] inetd
689 c4fed000 0 1 689 0000000 [SLPQ nanslp 0xc06307ac][SLP] cron
677 c53d8000 25 1 677 0000100 [SLPQ pause 0xc53d8034][SLP] sendmail
676 c5339a3c 100 673 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
673 c5339418 0 1 673 0000100 [SLPQ ufs 0xc50088d8][SLP] sendmail
668 c532e20c 0 1 668 0000100 [SLPQ select 0xc067dac4][SLP] sshd
650 c4fe8830 0 1 650 0000000 [SLPQ select 0xc067dac4][SLP] ntpd
558 c5023000 53 1 558 0000100 [SLPQ select 0xc067dac4][SLP] named
497 c4fe820c 0 1 497 0000000 [SLPQ select 0xc067dac4][SLP] syslogd
460 c4feca3c 0 1 460 0000000 [SLPQ select 0xc067dac4][SLP] devd
223 c4fe8624 0 0 0 0000204 [SLPQ mdwait 0xc5017000][SLP] md1
214 c501d624 0 0 0 0000204 [SLPQ mdwait 0xc4fc5000][SLP] md0
155 c4fa2000 0 0 0 0000204 [SLPQ m:w1 0xc4f43400][SLP] g_mirror
gm1
154 c4d76418 0 0 0 0000204 [SLPQ m:w2 0xc4f9ce00][SLP] g_mirror
gm0
153 c4d76624 0 0 0 0000204 [SLPQ m:w2 0xc4dc1600][SLP] g_mirror
gm2
152 c4d76830 0 0 0 0000204 [SLPQ - 0xe8ccecfc][SLP] schedcpu
151 c4d76a3c 0 0 0 0000204 [SLPQ vlruwt 0xc4d76a3c][SLP] vnlru
150 c4d76c48 0 0 0 0000204 [SLPQ syncer 0xc063051c][SLP] syncer
149 c4d79000 0 0 0 0000204 [SLPQ psleep 0xc067e00c][SLP] bufdaemon
148 c4d7920c 0 0 0 0000204 [SLPQ pollid 0xc062fb60][SLP] idlepoll
147 c4d79418 0 0 0 000020c [SLPQ pgzero 0xc06817a4][SLP] pagezero
146 c4d79624 0 0 0 0000204 [SLPQ psleep 0xc06812f4][SLP] vmdaemon
145 c4d79830 0 0 0 0000204 [SLPQ psleep 0xc06812b0][SLP]
pagedaemon
144 c4d79a3c 0 0 0 0000204 [IWAIT] swi0: sio
143 c4d79c48 0 0 0 0000204 [SLPQ cooling 0xc4d828d4][SLP]
acpi_cooling0
142 c4d7a000 0 0 0 0000204 [SLPQ tzpoll 0xc07b14d4][SLP]
acpi_thermal
141 c4d7a20c 0 0 0 0000204 [SLPQ idle 0xc4d37c74][SLP]
ciss_notify0
9 c4c9aa3c 0 0 0 0000204 [SLPQ - 0xc4cba500][SLP] thread taskq
140 c4c9ac48 0 0 0 0000204 [IWAIT] swi6:+
8 c4cad000 0 0 0 0000204 [SLPQ - 0xc4cba680][SLP] acpi_task2
7 c4cad20c 0 0 0 0000204 [SLPQ - 0xc4cba680][SLP] acpi_task1
6 c4cad418 0 0 0 0000204 [SLPQ - 0xc4cba680][SLP] acpi_task0
139 c4cad624 0 0 0 0000204 [IWAIT] swi6: task queue
138 c4cad830 0 0 0 0000204 [IWAIT] swi2: cambio
5 c4cada3c 0 0 0 0000204 [SLPQ - 0xc4cba900][SLP] kqueue taskq
137 c4cadc48 0 0 0 0000204 [IWAIT] swi5:+
136 c4cb1000 0 0 0 0000204 [SLPQ - 0xc062cee0][SLP] yarrow
4 c4cb120c 0 0 0 0000204 [SLPQ - 0xc062d848][SLP] g_down
3 c4cb1418 0 0 0 0000204 [SLPQ - 0xc062d844][SLP] g_up
2 c4cb1624 0 0 0 0000204 [SLPQ - 0xc062d83c][SLP] g_event
135 c4cb1830 0 0 0 0000204 [IWAIT] swi3: vm
134 c4cb1a3c 0 0 0 000020c [IWAIT] swi4: clock sio
133 c4cb1c48 0 0 0 0000204 [IWAIT] swi1: net
132 c4c83624 0 0 0 0000204 [IWAIT] irq119:
131 c4c83830 0 0 0 0000204 [IWAIT] irq118:
130 c4c83a3c 0 0 0 0000204 [IWAIT] irq117:
129 c4c83c48 0 0 0 0000204 [IWAIT] irq116:
128 c4c96000 0 0 0 0000204 [IWAIT] irq115:
127 c4c9620c 0 0 0 0000204 [IWAIT] irq114:
126 c4c96418 0 0 0 0000204 [IWAIT] irq113:
125 c4c96624 0 0 0 0000204 [IWAIT] irq112:
124 c4c96830 0 0 0 0000204 [IWAIT] irq111:
123 c4c96a3c 0 0 0 0000204 [IWAIT] irq110:
122 c4c96c48 0 0 0 0000204 [IWAIT] irq109:
121 c4c9a000 0 0 0 0000204 [IWAIT] irq108:
120 c4c9a20c 0 0 0 0000204 [IWAIT] irq107:
119 c4c9a418 0 0 0 0000204 [IWAIT] irq106:
118 c4c9a624 0 0 0 0000204 [IWAIT] irq105:
117 c4c9a830 0 0 0 0000204 [IWAIT] irq104:
116 c4c6f20c 0 0 0 0000204 [IWAIT] irq103:
115 c4c6f418 0 0 0 0000204 [IWAIT] irq102:
114 c4c6f624 0 0 0 0000204 [IWAIT] irq101:
113 c4c6f830 0 0 0 0000204 [IWAIT] irq100:
112 c4c6fa3c 0 0 0 0000204 [IWAIT] irq99:
111 c4c6fc48 0 0 0 0000204 [IWAIT] irq98: em1
110 c4c82000 0 0 0 0000204 [IWAIT] irq97: em0
109 c4c8220c 0 0 0 0000204 [IWAIT] irq96:
108 c4c82418 0 0 0 0000204 [IWAIT] irq95:
107 c4c82624 0 0 0 0000204 [IWAIT] irq94:
106 c4c82830 0 0 0 0000204 [IWAIT] irq93:
105 c4c82a3c 0 0 0 0000204 [IWAIT] irq92:
104 c4c82c48 0 0 0 0000204 [IWAIT] irq91:
103 c4c83000 0 0 0 0000204 [IWAIT] irq90:
102 c4c8320c 0 0 0 0000204 [IWAIT] irq89:
101 c4c83418 0 0 0 0000204 [IWAIT] irq88:
100 c4c6b000 0 0 0 0000204 [IWAIT] irq87:
99 c4c6b20c 0 0 0 0000204 [IWAIT] irq86:
98 c4c6b418 0 0 0 0000204 [IWAIT] irq85:
97 c4c6b624 0 0 0 0000204 [IWAIT] irq84:
96 c4c6b830 0 0 0 0000204 [IWAIT] irq83:
95 c4c6ba3c 0 0 0 0000204 [IWAIT] irq82:
94 c4c6bc48 0 0 0 0000204 [IWAIT] irq81:
93 c4c6e000 0 0 0 0000204 [IWAIT] irq80:
92 c4c6e20c 0 0 0 0000204 [IWAIT] irq79:
91 c4c6e418 0 0 0 0000204 [IWAIT] irq78:
90 c4c6e624 0 0 0 0000204 [IWAIT] irq77:
89 c4c6e830 0 0 0 0000204 [IWAIT] irq76:
88 c4c6ea3c 0 0 0 0000204 [IWAIT] irq75:
87 c4c6ec48 0 0 0 0000204 [IWAIT] irq74:
86 c4c6f000 0 0 0 0000204 [IWAIT] irq73:
85 c4c57000 0 0 0 0000204 [IWAIT] irq72:
84 c4c5720c 0 0 0 0000204 [IWAIT] irq71:
83 c4c57418 0 0 0 0000204 [IWAIT] irq70:
82 c4c57624 0 0 0 0000204 [IWAIT] irq69:
81 c4c57830 0 0 0 0000204 [IWAIT] irq68:
80 c4c57a3c 0 0 0 0000204 [IWAIT] irq67:
79 c4c57c48 0 0 0 0000204 [IWAIT] irq66:
78 c4c5a000 0 0 0 0000204 [IWAIT] irq65:
77 c4c5a20c 0 0 0 0000204 [IWAIT] irq64:
76 c4c5a418 0 0 0 0000204 [IWAIT] irq63:
75 c4c5a624 0 0 0 0000204 [IWAIT] irq62:
74 c4c5a830 0 0 0 0000204 [IWAIT] irq61:
73 c4c5aa3c 0 0 0 0000204 [IWAIT] irq60:
72 c4c5ac48 0 0 0 0000204 [IWAIT] irq59:
71 c4c2c20c 0 0 0 0000204 [IWAIT] irq58:
70 c4c2c418 0 0 0 0000204 [IWAIT] irq57:
69 c4c2c624 0 0 0 0000204 [IWAIT] irq56:
68 c4c2c830 0 0 0 0000204 [IWAIT] irq55:
67 c4c2ca3c 0 0 0 0000204 [IWAIT] irq54:
66 c4c2cc48 0 0 0 0000204 [IWAIT] irq53:
65 c4c48000 0 0 0 0000204 [IWAIT] irq52:
64 c4c4820c 0 0 0 0000204 [IWAIT] irq51: ciss0
63 c4c48418 0 0 0 0000204 [IWAIT] irq50:
62 c4c48624 0 0 0 0000204 [IWAIT] irq49:
61 c4c48830 0 0 0 0000204 [IWAIT] irq48:
60 c4c48a3c 0 0 0 0000204 [IWAIT] irq47:
59 c4c48c48 0 0 0 0000204 [IWAIT] irq46:
58 c4c1d624 0 0 0 0000204 [IWAIT] irq45:
57 c4c1d830 0 0 0 0000204 [IWAIT] irq44:
56 c4c1da3c 0 0 0 0000204 [IWAIT] irq43:
55 c4c1dc48 0 0 0 0000204 [IWAIT] irq42:
54 c4c2b000 0 0 0 0000204 [IWAIT] irq41:
53 c4c2b20c 0 0 0 0000204 [IWAIT] irq40:
52 c4c2b418 0 0 0 0000204 [IWAIT] irq39:
51 c4c2b624 0 0 0 0000204 [IWAIT] irq38:
50 c4c2b830 0 0 0 0000204 [IWAIT] irq37:
49 c4c2ba3c 0 0 0 0000204 [IWAIT] irq36:
48 c4c2bc48 0 0 0 0000204 [IWAIT] irq35:
47 c4c2c000 0 0 0 0000204 [IWAIT] irq34:
46 c4c10c48 0 0 0 0000204 [IWAIT] irq33:
45 c4c1c000 0 0 0 0000204 [IWAIT] irq32:
44 c4c1c20c 0 0 0 0000204 [IWAIT] irq31:
43 c4c1c418 0 0 0 0000204 [IWAIT] irq30:
42 c4c1c624 0 0 0 0000204 [IWAIT] irq29:
41 c4c1c830 0 0 0 0000204 [IWAIT] irq28:
40 c4c1ca3c 0 0 0 0000204 [IWAIT] irq27:
39 c4c1cc48 0 0 0 0000204 [IWAIT] irq26: bge1
38 c4c1d000 0 0 0 0000204 [IWAIT] irq25: bge0
37 c4c1d20c 0 0 0 0000204 [IWAIT] irq24:
36 c4c1d418 0 0 0 0000204 [IWAIT] irq23:
35 c4c02624 0 0 0 0000204 [IWAIT] irq22:
34 c4c02830 0 0 0 0000204 [IWAIT] irq21:
33 c4c02a3c 0 0 0 0000204 [IWAIT] irq20:
32 c4c02c48 0 0 0 0000204 [IWAIT] irq19:
31 c4c10000 0 0 0 0000204 [IWAIT] irq18:
30 c4c1020c 0 0 0 0000204 [IWAIT] irq17:
29 c4c10418 0 0 0 0000204 [IWAIT] irq16:
28 c4c10624 0 0 0 0000204 [IWAIT] irq15: ata1
27 c4c10830 0 0 0 0000204 [IWAIT] irq14: ata0
26 c4c10a3c 0 0 0 0000204 [IWAIT] irq13:
25 c4bb020c 0 0 0 0000204 [IWAIT] irq12:
24 c4bb0418 0 0 0 0000204 [IWAIT] irq11:
23 c4bb0624 0 0 0 0000204 [IWAIT] irq10:
22 c4bb0830 0 0 0 0000204 [IWAIT] irq9: acpi0
21 c4bb0a3c 0 0 0 0000204 [IWAIT] irq8:
20 c4bb0c48 0 0 0 0000204 [IWAIT] irq7:
19 c4c02000 0 0 0 0000204 [IWAIT] irq6:
18 c4c0220c 0 0 0 0000204 [IWAIT] irq5:
17 c4c02418 0 0 0 0000204 [IWAIT] irq4: sio0
16 c4bab000 0 0 0 0000204 [IWAIT] irq3: sio1
15 c4bab20c 0 0 0 0000204 [IWAIT] irq0:
14 c4bab418 0 0 0 0000204 [IWAIT] irq1: atkbd0
13 c4bab624 0 0 0 000020c [CPU 0] idle: cpu0
12 c4bab830 0 0 0 000020c [CPU 1] idle: cpu1
11 c4baba3c 0 0 0 000020c [CPU 2] idle: cpu2
10 c4babc48 0 0 0 000020c [CPU 3] idle: cpu3
1 c4bb0000 0 0 1 0004200 [SLPQ wait 0xc4bb0000][SLP] init
0 c062d940 0 0 0 0000200 [IWAIT] swapper
db> trace 11707
Tracing pid 11707 tid 100191 td 0xc4fe9900
sched_switch(c4fe9900,0,1,c067d8a0,e3d78fa4) at sched_switch+0x158
mi_switch(1,0,c05ea3ba,1a7) at mi_switch+0x1d1
sleepq_switch(c502ce28,c05e85df,da,2,eb6f8548) at sleepq_switch+0xe4
sleepq_wait(c502ce28,0,c05e85df,da,0) at sleepq_wait+0x11
msleep(c502ce28,c062f98c,50,c05ed658,0) at msleep+0x32e
acquire(eb6f85a4,40,60000,b6,c4fe9900) at acquire+0x88
lockmgr(c502ce28,2002,c502ce4c,c4fe9900,eb6f85cc) at lockmgr+0x426
vop_stdlock(eb6f861c,c05eaa4f,c061dda0,eb6f861c,eb6f85dc) at vop_stdlock+0x2f
VOP_LOCK_APV(c061e300,eb6f861c,eb6f85f4,c05cc673,eb6f861c) at VOP_LOCK_APV+0x44
ffs_lock(eb6f861c,246,2002,c502cdd0,eb6f8638) at ffs_lock+0x19
VOP_LOCK_APV(c061dda0,eb6f861c,c060f004,c4d69444,1b2) at VOP_LOCK_APV+0x44
vn_lock(c502cdd0,2002,c4fe9900,1b2,c4ba9300) at vn_lock+0xd0
ffs_snapshot(c4d69400,c4f36900,eb6f89b0,6c,0) at ffs_snapshot+0x14a9
ffs_mount(c4d69400,c4fe9900,c05ee869,29b,eb6f8ab0) at ffs_mount+0x9a9
vfs_domount(c4fe9900,c4d9e4d0,c4d30b40,11211000,c575c7c0) at vfs_domount+0x63e
vfs_donmount(c4fe9900,11211000,eb6f8bf4,c5481080,e) at vfs_donmount+0x108
kernel_mount(c59ba6a0,11211000,eb6f8c38,6c,bfbfe8c8) at kernel_mount+0x46
ffs_cmount(c59ba6a0,bfbfe110,11211000,c4fe9900,c061dac0) at ffs_cmount+0x85
mount(c4fe9900,eb6f8d04,c05fc65c,3c6,4) at mount+0x1d7
syscall(3b,3b,3b,8814819c,bfbfe0b0) at syscall+0x2a6
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (21, FreeBSD ELF32, mount), eip = 0x880b8b03, esp = 0xbfbfe07c, ebp
= 0xbfbfe778 ---
db> trace 13267
Tracing pid 13267 tid 100735 td 0xc92fc900
sched_switch(c92fc900,0,1,c067d600,c5bb72dc) at sched_switch+0x158
mi_switch(1,0,c05ea3ba,1a7) at mi_switch+0x1d1
sleepq_switch(c50088d8,c05e85df,da,1,ebfcc6b4) at sleepq_switch+0xe4
sleepq_wait(c50088d8,0,c05e85df,da,0) at sleepq_wait+0x11
msleep(c50088d8,c062e7d4,50,c05ed658,0) at msleep+0x32e
acquire(ebfcc710,40,60000,b6,c92fc900) at acquire+0x88
lockmgr(c50088d8,3002,c50088fc,c92fc900,ebfcc738) at lockmgr+0x426
vop_stdlock(ebfcc788,c50088fc,c061dda0,ebfcc788,ebfcc748) at vop_stdlock+0x2f
VOP_LOCK_APV(c061e300,ebfcc788,ebfcc760,c05cc673,ebfcc788) at VOP_LOCK_APV+0x44
ffs_lock(ebfcc788,1002,1002,c5008880,ebfcc7a4) at ffs_lock+0x19
VOP_LOCK_APV(c061dda0,ebfcc788,c05ef151,31f,c92fc900) at VOP_LOCK_APV+0x44
vn_lock(c5008880,1002,c92fc900,c92fc900,0) at vn_lock+0xd0
lookup(ebfcc898,0,c05ee77c,b6,38) at lookup+0xc3
namei(ebfcc898,0,c05ef041,f9,8140af0) at namei+0x37a
kern_statfs(c92fc900,8140af0,0,ebfccaec,c92fc900) at kern_statfs+0x6d
statfs(c92fc900,ebfccd04,8,c05eb70b,2) at statfs+0x35
syscall(3b,3b,bfbf003b,0,80ef6a0) at syscall+0x2a6
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (396, FreeBSD ELF32, statfs), eip = 0x882966e3, esp = 0xbfbfae4c,
ebp = 0xbfbfb048 ---
db> trace 12800
Tracing pid 12800 tid 100352 td 0xc60f7c00
sched_switch(c60f7c00,0,1,c067d328,77790a88) at sched_switch+0x158
mi_switch(1,0,c05ea3ba,1a7) at mi_switch+0x1d1
sleepq_switch(c4d6946c,c05e85df,da,0,eb9c3944) at sleepq_switch+0xe4
sleepq_wait(c4d6946c,0,c05e85df,da,0) at sleepq_wait+0x11
msleep(c4d6946c,c4d69444,29f,c05ef16f,0,c062a820,c53f0880,eb9c3980,c062f8b4) at
msleep+0x32e
vn_write_suspend_wait(c53f0880,c4d69400,1,887,c05eec46) at
vn_write_suspend_wait+0x11b
ufs_inactive(eb9c39cc,eb9c39e4,c0503c6f,c061dda0,eb9c39cc) at ufs_inactive+0x1b4
VOP_INACTIVE_APV(c061dda0,eb9c39cc,c05eec46,887,c062a6a0) at
VOP_INACTIVE_APV+0x3a
vinactive(c53f0880,c60f7c00,c05eec46,819,c4e49000) at vinactive+0x62
vput(c53f0880,ffffffdf,2,c60f7c00,0) at vput+0x159
vn_open_cred(eb9c3bc4,eb9c3cc4,180,c8e22280,5) at vn_open_cred+0xe7
vn_open(eb9c3bc4,eb9c3cc4,180,5,eb9c3b50) at vn_open+0x33
kern_open(c60f7c00,8141480,0,a03,180) at kern_open+0xca
open(c60f7c00,eb9c3d04,c,c05eb482,3) at open+0x36
syscall(3b,bfbf003b,bfbf003b,0,a02) at syscall+0x2a6
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbfa5ac, ebp =
0xbfbfa638 ---
db> continue
spin lock sio held by 0xc4bac780 for > 5 seconds
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 07
fault virtual address = 0x144
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc04ce557
stack pointer = 0x28:0xe7134bc4
frame pointer = 0x28:0xe7134bcc
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 10 (idle: cpu3)
trap number = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
kdb_backtrace(c05fb775,3,c05defdf,e7134b18,c4bac600) at kdb_backtrace+0x2f
panic(c05defdf,c05fc499,c4babdd0,1,1) at panic+0x129
trap_fatal(e7134b84,144,c4bac600,c4c2da80,144) at trap_fatal+0x31e
trap(e7130008,c0490028,c05f0028,c4bac600,c06836c0) at trap+0x119
calltrap() at calltrap+0x5
--- trap 0xc, eip = 0xc04ce557, esp = 0xe7134bc4, ebp = 0xe7134bcc ---
witness_display_spinlock(c06836c0,0,c4bac780,c4bac600,c06836c0) at
witness_display_spinlock+0x9
_mtx_lock_spin(c06836c0,c4bac600,0,c05f8952,56e) at _mtx_lock_spin+0xb2
_mtx_lock_spin_flags(c06836c0,0,c05f8952,56e,e7134c54) at
_mtx_lock_spin_flags+0x82
siointr(c4da1c00,c062f9c0,2,4,c4bac600) at siointr+0x2a
intr_execute_handlers(c4ba4490,e7134c6c,e7134cb0,c05af203,34) at
intr_execute_handlers+0x8c
lapic_handle_intr(34) at lapic_handle_intr+0x3b
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc07a3272, esp = 0xe7134cb0, ebp = 0xe7134cb0 ---
acpi_cpu_c1(c4bac600,e7134cc8,246,c062f9c0,e7134ce4) at acpi_cpu_c1+0x5
acpi_cpu_idle(e7134d04,c0493346,c062f9c0,0,c05e5dac) at acpi_cpu_idle+0x18a
cpu_idle(c062f9c0,0,c05e5dac,7d,0) at cpu_idle+0x29
idle_proc(0,e7134d38,c05e5cb0,30d,0) at idle_proc+0x6e
fork_exit(c04932d8,0,e7134d38) at fork_exit+0x78
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe7134d6c, ebp = 0 ---
Uptime: 26m8s
Kris Kennaway
2005-Nov-21 18:13 UTC
Recurring problem: processes block accessing UFS file system
On Mon, Nov 21, 2005 at 05:54:09PM -0600, Greg Rivers wrote:> I've recently put up three busy email relay hosts running 6.0-STABLE. > Performance is excellent except for a nagging critical issue that keeps > cropping up. > > /var/spool is its own file system mounted on a geom stripe of four BSD > partitions (details below). Once every two or three days all the > processes accessing /var/spool block forever in disk wait. All three > machines suffer this problem. No diagnostic messages are generated and > the machines continue running fine otherwise, but a reboot is required to > clear the condition. This problem occurs during normal operation, but is > particularly likely to occur during a backup when dump makes a snapshot. > > There doesn't appear to be a problem with gstripe, as gstripe status is > "UP" and I can read the raw device just fine while processes continue to > block on the file system. I tried running a kernel with WITNESS and > DIAGNOSTIC, but these options shed no light. > > If I catch the problem early enough I can break successfully into kdb; > otherwise, if too many processes stack up, the machine hangs going into > kdb and must be power-cycled.Make sure you have KDB_STOP_NMI in your kernel.> I obtained the following process listing and traces from kdb. I traced > mksnap_ffs which was blocked in "ufs", and two random sendmail processes > that were blocked in "ufs" and "suspfs" respectively.Looks like a UFS snapshot deadlock. Are you running something like dump -L on this filesystem, or making other use of snapshots? fsck -B also uses them, but shouldn't be running except at boot time. You should take this up with Kirk McKusick <freebsd@McKusick.COM> - in the meantime you can work around it by not making use of UFS snapshots. Kris -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20051121/9ed4981f/attachment.bin
Ronald Klop
2005-Nov-21 19:02 UTC
Recurring problem: processes block accessing UFS file system
On Tue, 22 Nov 2005 00:54:09 +0100, Greg Rivers <gcr+freebsd-stable@tharned.org> wrote:> I've recently put up three busy email relay hosts running 6.0-STABLE. > Performance is excellent except for a nagging critical issue that keeps > cropping up. > > /var/spool is its own file system mounted on a geom stripe of four BSD > partitions (details below). Once every two or three days all the > processes accessing /var/spool block forever in disk wait. All three > machines suffer this problem. No diagnostic messages are generated and > the machines continue running fine otherwise, but a reboot is required > to clear the condition. This problem occurs during normal operation, > but is particularly likely to occur during a backup when dump makes a > snapshot. > > There doesn't appear to be a problem with gstripe, as gstripe status is > "UP" and I can read the raw device just fine while processes continue to > block on the file system. I tried running a kernel with WITNESS and > DIAGNOSTIC, but these options shed no light. > > If I catch the problem early enough I can break successfully into kdb; > otherwise, if too many processes stack up, the machine hangs going into > kdb and must be power-cycled. > > I'd appreciate any insight anyone may have into this problem or advise > on turning this report into a coherent PR.I have a machine with 5.4-STABLE with the same problem. It hangs every couple of days if I make regular snapshots. It is a remote machine which I don't have easy access to. I disabled the snapshots and since than it didn't hang a single time. I hoped it would be fixed in 6.0, but this sounds the same. Ronald. -- Ronald Klop Amsterdam, The Netherlands