thr3ads.net - freebsd stable - system hangup

If this information is useful, please help other people find it:
Share via:

Oliver Lehmann

2008-Sep-29 20:41 UTC

system hangup - I'm lost

Hi,

My fileserver has sporadical hangups running 6.3:

FreeBSD 6.3-STABLE #0: Thu Jun 19 00:21:00 CEST 2008
   
olivleh1@nudel.salatschuessel.net:/usr/obj/i386-pentium3-6.3/usr/src/sys/NUDEL

The exact release doesn't matter since it happened before. It always
happens afer some time of having some load on the system (I'm building
ports with tinderbox and during the build process it just hangs up).

The system does nothing write out on the console, neither the CRT, nor
the serial console.

The system itself is:

CPU: Intel Pentium III (845.64-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x683  Stepping = 3
 
Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory  = 805240832 (767 MB)
avail memory = 778481664 (742 MB)
ACPI APIC Table: <Intel  N440BX  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  1
 cpu1 (AP): APIC ID:  0
ioapic0 <Version 1.1> irqs 0-23 on motherboard

while the diskspace is provided by an 3ware RAID:

twa0: <3ware 9000 series Storage Controller> port 0x2400-0x24ff mem
0xf4101000-0xf41010ff,0xf4800000-0xf4ffffff irq 18 at device 11.0 on pci0
twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue: 
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9500S-4LP, 4 ports,
Firmware FE9X 2.08.00.009, BIOS BE9X 2.03.01.052

da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9500S-4LP  DISK 2.08> Fixed Direct Access SCSI-3 device 
da0: 100.000MB/s transfers
da0: 715224MB (1464778752 512 byte sectors: 255H 63S/T 91178C)

I had - in the past - sometimes messages left which where indicating,
that the system was not able to allocate swap space fast enough if I
recall it correctly (_not_ out of swap space!) but the RAID is kinda
fast imho.

  Any idea what I could do to shed some more light on this behaviour?
  Why it is happening and what really is causing it?
  Would enabling the kernel debugger really help here? I mean the system
  is really hanging up - except ping response it is not responding to
  anything except the reset switch ;)

   Greetings, Oliver


-- 
 Oliver Lehmann
  http://www.pofo.de/
  http://wishlist.ans-netz.de/

Gavin Atkinson

2008-Sep-30 10:32 UTC

head link

system hangup - I'm lost

On Mon, 2008-09-29 at 22:14 +0200, Oliver Lehmann wrote:
>   Any idea what I could do to shed some more light on this behaviour?
>   Why it is happening and what really is causing it?
>   Would enabling the kernel debugger really help here? I mean the system
>   is really hanging up - except ping response it is not responding to
>   anything except the reset switch ;)
If it's responding to ping, you should be able to get into the debugger.
Compile it in, along with "options WITNESS" and "options
WITNESS_SKIPSPIN", and press ctrl-alt-escape when the machine next
hangs.
>From there, it should hopefully be possible to get more info.  It's beena long time since I've used the debugger under 6.x so some of the more
useful commands may not exist, but the output of at least "sh locks",
"sh alllocks" and "bt" on any processes that seem to be
holding locks.
Also "sh pcpu" and "ps" will help to determine exactly what
was running
at the time.

Gavin

Bartosz Stec

2008-Sep-30 10:39 UTC

head link

system hangup - I'm lost

Oliver Lehmann wrote:> Hi,
>
> My fileserver has sporadical hangups running 6.3:
>
> FreeBSD 6.3-STABLE #0: Thu Jun 19 00:21:00 CEST 2008
>    
olivleh1@nudel.salatschuessel.net:/usr/obj/i386-pentium3-6.3/usr/src/sys/NUDEL
>
> The exact release doesn't matter since it happened before. It always
> happens afer some time of having some load on the system (I'm building
> ports with tinderbox and during the build process it just hangs up).
>
> The system does nothing write out on the console, neither the CRT, nor
> the serial console.
>
> The system itself is:
>
> CPU: Intel Pentium III (845.64-MHz 686-class CPU)
>   Origin = "GenuineIntel"  Id = 0x683  Stepping = 3
>  
Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
> real memory  = 805240832 (767 MB)
> avail memory = 778481664 (742 MB)
> ACPI APIC Table: <Intel  N440BX  >
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
>  cpu0 (BSP): APIC ID:  1
>  cpu1 (AP): APIC ID:  0
> ioapic0 <Version 1.1> irqs 0-23 on motherboard
>
> while the diskspace is provided by an 3ware RAID:
>
> twa0: <3ware 9000 series Storage Controller> port 0x2400-0x24ff mem
0xf4101000-0xf41010ff,0xf4800000-0xf4ffffff irq 18 at device 11.0 on pci0
> twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue: 
> twa0: INFO: (0x15: 0x1300): Controller details:: Model 9500S-4LP, 4 ports,
Firmware FE9X 2.08.00.009, BIOS BE9X 2.03.01.052
>
> da0 at twa0 bus 0 target 0 lun 0
> da0: <AMCC 9500S-4LP  DISK 2.08> Fixed Direct Access SCSI-3 device 
> da0: 100.000MB/s transfers
> da0: 715224MB (1464778752 512 byte sectors: 255H 63S/T 91178C)
>
> I had - in the past - sometimes messages left which where indicating,
> that the system was not able to allocate swap space fast enough if I
> recall it correctly (_not_ out of swap space!) but the RAID is kinda
> fast imho.
>
>   Any idea what I could do to shed some more light on this behaviour?
>   Why it is happening and what really is causing it?
>   Would enabling the kernel debugger really help here? I mean the system
>   is really hanging up - except ping response it is not responding to
>   anything except the reset switch ;)
>
>    Greetings, Oliver
>
>
>   Personally I'd rather bet on some hardware problem (overheating?) Try to 
install mbmon from ports. I had also similiar problems with old 
motherboards with swelled capacitors.

-- 
Bartosz Stec

John Baldwin

2008-Sep-30 14:53 UTC

head link

system hangup - I'm lost

On Monday 29 September 2008 04:14:08 pm Oliver Lehmann
wrote:> Hi,
> 
> My fileserver has sporadical hangups running 6.3:
> 
> FreeBSD 6.3-STABLE #0: Thu Jun 19 00:21:00 CEST 2008
>     
olivleh1@nudel.salatschuessel.net:/usr/obj/i386-pentium3-6.3/usr/src/sys/NUDEL> 
> The exact release doesn't matter since it happened before. It always
> happens afer some time of having some load on the system (I'm building
> ports with tinderbox and during the build process it just hangs up).
> 
> The system does nothing write out on the console, neither the CRT, nor
> the serial console.
1) Setup support for crashdumps.
2) Add 'DDB' and 'KDB' to your kernel.  When it hangs, break
into the debugger
(CTRL+ALT+ESC) and run 'panic' to generate a crash dump.
3) ps -axl -M /var/crash/vmcore.X -N /boot/kernel/kernel

(where vmcore.X is the core file generated, probably vmcore.0).  That's the 
first place to start.

-- 
John Baldwin

Oliver Lehmann

2008-Oct-01 15:30 UTC

head link

system hangup - I'm lost

Hi,

today I'd a crash again - I was not able to get a crash dump (thought a
"panic" at the end of the kdb would do it but didn't - should have
called
dumpon before ;)) - so here now the information I was able to retrieve:

Ok, what I've got so far is wrinting stuff out to the console when the
system hangs up:

swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2, size: 4096
...

and now the debugger stuff:

KDB: enter: manual escape to debugger
[thread pid 40 tid 100048 ]
Stopped at      kdb_enter+0x30: leave   
db> sh locks
exclusive sleep mutex Giant r = 0 (0xc07c73c0) locked
@ /usr/src/sys/kern/kern_intr.c:681
db> sh alllocks
Process 40 (irq1: atkbd0) thread 0xc4503a80 (100048)
exclusive sleep mutex Giant r = 0 (0xc07c73c0) locked
@ /usr/src/sys/kern/kern_intr.c:681
db> 

so there are no locks except the one I caused but anyhow:

db> bt 100048
Tracing pid 40 tid 100048 td 0xc4503a80
kdb_enter(c077aee6,4,1,0,1,...) at kdb_enter+0x30
scgetc(c0842b60,2,de391c88,c05ad0b7,c4609340,...) at scgetc+0x575
sckbdevent(c0823740,0,c0842b60,c07c73c0,8,...) at sckbdevent+0x210
atkbd_intr(c0823740,0,de391cd8,c05695b8,c0823740,...) at atkbd_intr+0xa1
atkbdintr(c0823740,0,c076448a,2a9,8,...) at atkbdintr+0x21
ithread_execute_handlers(c460cc90,c4449680,c076448a,30e,c4503a80,...) at
ithread_execute_handlers+0x108 ithread_loop
(c45f66c0,de391d38,c07642ea,30c,0,...) at ithread_loop+0x64 fork_exit
(c05696b0,c45f66c0,de391d38) at fork_exit+0x78 fork_trampoline() at
fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xde391d6c, ebp = 0 ---

db> sh pcpu
cpuid        = 0
curthread    = 0xc4503a80: pid 40 "irq1: atkbd0"
curpcb       = 0xde391d90
fpcurthread  = none
idlethread   = 0xc444c780: pid 11 "idle: cpu0"
APIC ID      = 1
currentldt   = 0x50
spin locks held:

and now the output of ps (beware, it is long, no idea why there are so
many cron - maybe the crond still schedules but they don't get processed?)

show lockedvnods follows afterwards

db> ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
57919 57918   692     0  SV      ufs      0xc47857c8 cron
57918   692   692     0  S       ppwait   0xc6e63a78 cron
57917 57916   692     0  SV      ufs      0xc47857c8 cron
57916   692   692     0  S       ppwait   0xc6e63c90 cron
57915 57914   692     0  SV      ufs      0xc47857c8 cron
57914   692   692     0  S       ppwait   0xc6eb3000 cron
57913 57912   692     0  SV      ufs      0xc47857c8 cron
57912   692   692     0  S       ppwait   0xc70a9430 cron
57911 57908   692     0  SV      ufs      0xc47857c8 cron
57910 57907   692     0  SV      ufs      0xc47857c8 cron
57909 57906   692     0  SV      ufs      0xc47857c8 cron
57908   692   692     0  S       ppwait   0xc6eb3648 cron
57907   692   692     0  S       ppwait   0xc6eb3860 cron
57906   692   692     0  S       ppwait   0xc6eb3a78 cron
57905   686   686    25  S       ufs      0xc4953388 sendmail
57904 57902   692     0  SV      ufs      0xc47857c8 cron
57903 57901   692     0  SV      ufs      0xc47857c8 cron
57902   692   692     0  S       ppwait   0xc49a4430 cron
57901   692   692     0  S       ppwait   0xc49a4648 cron
57900 57899   692     0  SV      ufs      0xc47857c8 cron
57899   692   692     0  S       ppwait   0xc49a4860 cron
57898 57897   692     0  SV      ufs      0xc47857c8 cron
57897   692   692     0  S       ppwait   0xc49a4a78 cron
57896 57895   692     0  SV      ufs      0xc47857c8 cron
57895   692   692     0  S       ppwait   0xc49a4c90 cron
57894 57893   692     0  SV      ufs      0xc47857c8 cron
57893   692   692     0  S       ppwait   0xc6b7c648 cron
57892 57891   692     0  SV      ufs      0xc47857c8 cron
57891   692   692     0  S       ppwait   0xc66bc430 cron
57890 57889   692     0  SV      ufs      0xc47857c8 cron
57889   692   692     0  S       ppwait   0xc6b7c860 cron
57888 57887   692     0  SV      ufs      0xc47857c8 cron
57887   692   692     0  S       ppwait   0xc66bc860 cron
57886   686   686    25  S       ufs      0xc4953388 sendmail
57885 57884   692     0  SV      ufs      0xc47857c8 cron
57884   692   692     0  S       ppwait   0xc66bca78 cron
57883 57882   692     0  SV      ufs      0xc47857c8 cron
57882   692   692     0  S       ppwait   0xc66bcc90 cron
57881 57880   692     0  SV      ufs      0xc47857c8 cron
57880   692   692     0  S       ppwait   0xc6a65000 cron
57879 57878   692     0  SV      ufs      0xc47857c8 cron
57878   692   692     0  S       ppwait   0xc6a65218 cron
57877 57876   692     0  SV      ufs      0xc47857c8 cron
57876   692   692     0  S       ppwait   0xc6a65430 cron
57875 57874   692     0  SV      ufs      0xc47857c8 cron
57874   692   692     0  S       ppwait   0xc6a65648 cron
57873 57872   692     0  SV      ufs      0xc47857c8 cron
57872   692   692     0  S       ppwait   0xc48ed000 cron
57871 57867   692     0  SV      ufs      0xc47857c8 cron
57870 57868   692     0  SV      ufs      0xc47857c8 cron
57869 57866   692     0  SV      ufs      0xc47857c8 cron
57868   692   692     0  S       ppwait   0xc48ed430 cron
57867   692   692     0  S       ppwait   0xc48ed648 cron
57866   692   692     0  S       ppwait   0xc48ed860 cron
57865   686   686    25  S       ufs      0xc4953388 sendmail
57864 57862   692     0  SV      ufs      0xc47857c8 cron
57862   692   692     0  S       ppwait   0xc4910000 cron
57861   692   692     0  S       ppwait   0xc4910218 cron
57860 57859   692     0  SV      ufs      0xc47857c8 cron
57859   692   692     0  S       ppwait   0xc4910430 cron
57858 57857   692     0  SV      ufs      0xc47857c8 cron
57857   692   692     0  S       ppwait   0xc4910648 cron
57856 57855   692     0  SV      ufs      0xc47857c8 cron
57855   692   692     0  S       ppwait   0xc4910860 cron
57854 57853   692     0  SV      ufs      0xc47857c8 cron
57853   692   692     0  S       ppwait   0xc4910a78 cron
57852 57851   692     0  SV      ufs      0xc47857c8 cron
57851   692   692     0  S       ppwait   0xc4910c90 cron
57850 57849   692     0  SV      ufs      0xc47857c8 cron
57849   692   692     0  S       ppwait   0xc49a4000 cron
57848 57847   692     0  SV      ufs      0xc47857c8 cron
57847   692   692     0  S       ppwait   0xc49a4218 cron
57846   686   686    25  S       ufs      0xc4953388 sendmail
57845 57844   692     0  SV      ufs      0xc47857c8 cron
57844   692   692     0  S       ppwait   0xc4e47860 cron
57843 57842   692     0  SV      ufs      0xc47857c8 cron
57842   692   692     0  S       ppwait   0xc66cd430 cron
57841 57840   692     0  SV      ufs      0xc47857c8 cron
57840   692   692     0  S       ppwait   0xc66cd648 cron
57839 57838   692     0  SV      ufs      0xc47857c8 cron
57838   692   692     0  S       ppwait   0xc4e59218 cron
57837 57836   692     0  SV      ufs      0xc47857c8 cron
57836   692   692     0  S       ppwait   0xc5cd6860 cron
57835 57834   692     0  SV      ufs      0xc47857c8 cron
57834   692   692     0  S       ppwait   0xc5cd5430 cron
57833 57832   692     0  SV      ufs      0xc47857c8 cron
57832   692   692     0  S       ppwait   0xc66c8c90 cron
57831 57828   692     0  SV      ufs      0xc47857c8 cron
57830 57827   692     0  SV      ufs      0xc47857c8 cron
57829 57826   692     0  SV      ufs      0xc47857c8 cron
57828   692   692     0  S       ppwait   0xc691e218 cron
57827   692   692     0  S       ppwait   0xc4bd3218 cron
57826   692   692     0  S       ppwait   0xc66c8430 cron
57825   686   686    25  S       ufs      0xc4953388 sendmail
57824 57822   692     0  SV      ufs      0xc47857c8 cron
57823 57821   692     0  SV      ufs      0xc47857c8 cron
57822   692   692     0  S       ppwait   0xc66cd860 cron
57821   692   692     0  S       ppwait   0xc474ac90 cron
57820 57819   692     0  SV      ufs      0xc47857c8 cron
57819   692   692     0  S       ppwait   0xc4cdf860 cron
57818 57817   692     0  SV      ufs      0xc47857c8 cron
57817   692   692     0  S       ppwait   0xc4e5f430 cron
57816 57815   692     0  SV      ufs      0xc47857c8 cron
57815   692   692     0  S       ppwait   0xc5cd5000 cron
57814 57813   692     0  SV      ufs      0xc47857c8 cron
57813   692   692     0  S       ppwait   0xc4e47430 cron
57812 57811   692     0  SV      ufs      0xc47857c8 cron
57811   692   692     0  S       ppwait   0xc4bcd648 cron
57810 57809   692     0  SV      ufs      0xc47857c8 cron
57809   692   692     0  S       ppwait   0xc4cdfa78 cron
57808 57807   692     0  SV      ufs      0xc47857c8 cron
57807   692   692     0  S       ppwait   0xc4a34a78 cron
57806   686   686    25  S       ufs      0xc4953388 sendmail
57805 57804   692     0  SV      ufs      0xc47857c8 cron
57804   692   692     0  S       ppwait   0xc6923648 cron
57803 57802   692     0  SV      ufs      0xc47857c8 cron
57802   692   692     0  S       ppwait   0xc4cdf430 cron
57801 57800   692     0  SV      ufs      0xc47857c8 cron
57800   692   692     0  S       ppwait   0xc691e430 cron
57799 57798   692     0  SV      ufs      0xc47857c8 cron
57798   692   692     0  S       ppwait   0xc4e59860 cron
57797 57796   692     0  SV      ufs      0xc47857c8 cron
57796   692   692     0  S       ppwait   0xc4e5fa78 cron
57795 57794   692     0  SV      ufs      0xc47857c8 cron
57794   692   692     0  S       ppwait   0xc4e4cc90 cron
57793  1035  1035     0  SLJ     vmpfw    0xc1639568 httpd
57792  1035  1035     0  SLJ     vmpfw    0xc1639568 httpd
57791 57790   692     0  SV      ufs      0xc47857c8 cron
57790   692   692     0  S       ppwait   0xc4e59000 cron
57789  1035  1035     0  SLJ     vmpfw    0xc1639568 httpd
57788 57785 57788     0  SVs     ufs      0xc47857c8 cron
57787 57784 57787     2  SVs     ufs      0xc47857c8 cron
57786 57783 57786     0  SVs     ufs      0xc47d9168 cron
57785   692   692     0  S       ppwait   0xc66cda78 cron
57784   692   692     0  S       ppwait   0xc66c8000 cron
57783   692   692     0  S       ppwait   0xc66cd000 cron
57782   686   686    25  S       biord    0xd3e374b8 sendmail
57780 57778 57778     2  S       ufs      0xc6ebd9e8 sh
57778 57776 57778     2  Ss      wait     0xc4a33000 sh
57776   692   692     0  S       piperd   0xc719e660 cron
57772  1035  1035     0  SLJ     vmpfw    0xc1639568 httpd
57771  1035  1035     0  SLJ     swread   0xc1639568 httpd
57770 57769 57770     0  SVsJ    ufs      0xc5502058 cron
57769  1048  1048     0  SJ      ppwait   0xc4e5c000 cron
57765 57764 57765     0  SVsJ    getblk   0xd3eba538 cron
57764  1048  1048     0  SJ      ppwait   0xc5cd6000 cron
57762 57761 57761     2  S       biord    0xd3e875f8 sh
57761 57760 57761     2  Ss      wait     0xc4bd3a78 sh
57760   692   692     0  S       piperd   0xc68137f8 cron
57759   936   936     0  SJ      biowr    0xd3eba4d8 rateup
57732 51526 46607     0  S+      biowr    0xd3e5b170 sh
51526 51268 46607     0  S+      wait     0xc4cdf000 sh
51268 51267 46607     0  S+      wait     0xc66cd218 sh
51267 51264 46607     0  S+      wait     0xc5cd5c90 sh
51264 49494 46607     0  S+      select   0xc0815024 make
49494 46607 46607     0  S+      wait     0xc4bcd000 sh
78290  1035  1035    80  SJ      ufs      0xc47668d8 httpd
78289  1035  1035    80  SJ      ufs      0xc4ad87c8 httpd
78282  1035  1035    80  SJ      ufs      0xc47d1168 httpd
75486  1035  1035    80  SJ      ufs      0xc47d1168 httpd
72716  1035  1035    80  SJ      ufs      0xc47d1168 httpd
72715  1035  1035    80  SJ      ufs      0xc4ac5168 httpd
72603  1035  1035    80  SJ      ufs      0xc4ac5388 httpd
70352  1035  1035    80  SJ      ufs      0xc47d1168 httpd
53179 53008 53179     0  S+      ufs      0xc47857c8 csh
53008 49451 53008  1000  S+      wait     0xc4bcda78 su
49451 49449 49451  1000  Ss+     pause    0xc4e5c67c tcsh
49449 49270 49270  1000  S       select   0xc0815024 sshd
49270   678 49270     0  Ss      sbwait   0xc508179c sshd
46607  1208 46607     0  S+      wait     0xc66c8860 sh
 1208  1206  1208     0  Ss+     pause    0xc4bd3cc4 csh
 1206     1  1206     0  Ss      select   0xc0815024 screen
 1151  1117  1151     0  S+      ttyin    0xc4616010 csh
 1132   928   928    89  SJ      select   0xc0815024 perl5.8.8
 1131   928   928    89  SJ      select   0xc0815024 perl5.8.8
 1118     1     1     0  S       ttydcd   0xc4614000 getty
 1117     1  1117     0  Ss+     wait     0xc4bcdc90 login
 1116     1  1116     0  Ss+     ttyin    0xc4613010 getty
 1115     1  1115     0  Ss+     ttyin    0xc461f410 getty
 1114     1  1114     0  Ss+     ttyin    0xc45fc810 getty
 1113     1  1113     0  Ss+     ttyin    0xc4615010 getty
 1112     1  1112     0  Ss+     ttyin    0xc461ec10 getty
 1111     1  1111     0  Ss+     ttyin    0xc461e810 getty
 1110     1  1110     0  Ss+     ttyin    0xc461d810 getty
 1109     1  1109     0  Ss+     ttyin    0xc461d410 getty
 1093     1  1093     0  Ss      select   0xc0815024 inetd
 1092     1  1092  1000  SsJ     ufs      0xc47d1168 fetchmail
 1048     1  1048     0  SsJ     ufs      0xc498a9e8 cron
 1043     1  1043     0  SsJ     select   0xc0815024 sshd
 1035     1  1035     0  SsJ     nanslp   0xc07c7cac httpd
 1019     1  1019     0  SsJ     (threaded)          bacula-fd
100137                   S       kserel   0xc475fc94 bacula-fd
100158                   S       kserel   0xc475fc94 bacula-fd
100164                   S       select   0xc0815024 bacula-fd
100162                   S       ksesigwa 0xc49b1350 bacula-fd
 1001  1000  1001     0  S+J     select   0xc0815024 couriertcpd
 1000     1  1000     0  S+J     piperd   0xc48e5660 courierlogger
  988   987   988     0  S+J     select   0xc0815024 couriertcpd
  987     1   987     0  S+J     piperd   0xc4819cc0 courierlogger
  976   975   976     0  S+J     select   0xc0815024 couriertcpd
  975     1   975     0  S+J     piperd   0xc4a72198 courierlogger
  971   946   945     0  S+J     select   0xc0815024 authdaemond
  970   946   945     0  S+J     select   0xc0815024 authdaemond
  969   946   945     0  S+J     select   0xc0815024 authdaemond
  968   946   945     0  S+J     select   0xc0815024 authdaemond
  967   946   945     0  S+J     select   0xc0815024 authdaemond
  946   945   945     0  S+J     select   0xc0815024 authdaemond
  945     1   945     0  S+J     piperd   0xc48197f8 courierlogger
  936     1   936     0  SsJ     piperd   0xc6813cc0 perl
  928     1   928     0  SsJ     select   0xc0815024 perl5.8.8
  926     1   907     0  S+J     select   0xc0815024 sqwebmaild
  924     1   907     0  S+J     select   0xc0815024 sqwebmaild
  922     1   907     0  S+J     select   0xc0815024 sqwebmaild
  920     1   907     0  S+J     select   0xc0815024 sqwebmaild
  918     1   907     0  S+J     select   0xc0815024 sqwebmaild
  908   907   907     0  S+J     select   0xc0815024 sqwebmaild
  907     1   907     0  S+J     piperd   0xc48e5000 courierlogger
  904   897   877    85  SJ      piperd   0xc476fcc0 qmail-clean
  903   897   877    86  SJ      select   0xc0815024 qmail-rspawn
  902   897   877     0  SJ      select   0xc0815024 qmail-lspawn
  901   897   877    83  SJ      piperd   0xc4763330 multilog
  897   887   877    87  SJ      biord    0xd3e299a0 qmail-send
  896   886   877    83  SJ      piperd   0xc48e5198 multilog
  895   885   877    89  SJ      accept   0xc4a22302 tcpserver
  894   883   877     0  SJ      accept   0xc4a2772e tcpserver
  893   888   877    83  SJ      piperd   0xc47627f8 multilog
  892   884   877    83  SJ      piperd   0xc48e5330 multilog
  888   879   877     0  SJ      select   0xc0815024 supervise
  887   879   877     0  SJ      select   0xc0815024 supervise
  886   879   877     0  SJ      select   0xc0815024 supervise
  885   879   877     0  SJ      select   0xc0815024 supervise
  884   879   877     0  SJ      select   0xc0815024 supervise
  883   879   877     0  SJ      select   0xc0815024 supervise
  880     1   877     0  SJ      piperd   0xc476f330 readproctitle
  879     1   877     0  SJ      ufs      0xc47d1168 svscan
  837     1   837     0  SsJ     select   0xc0815024 syslogd
  692     1   692     0  Ss      nanslp   0xc07c7cac cron
  686     1   686    25  Ss      pause    0xc47f267c sendmail
  683     1   683     0  Ss      select   0xc0815024 sendmail
  678     1   678     0  Ss      select   0xc0815024 sshd
  663     1   663   910  Ss      (threaded)          bacula-dir
100118                   S       kserel   0xc45d4754 bacula-dir
100121                   S       kserel   0xc45d4754 bacula-dir
100160                   S       select   0xc0815024 bacula-dir
100127                   S       ksesigwa 0xc47f4bb0 bacula-dir
  657     1   657     0  Ss      (threaded)          bacula-fd
100123                   S       kserel   0xc45d4b14 bacula-fd
100114                   S       kserel   0xc45d4b14 bacula-fd
100163                   S       select   0xc0815024 bacula-fd
100115                   S       ksesigwa 0xc47f2568 bacula-fd
  650     1   650   910  Ss      (threaded)          bacula-sd
100143                   S       kserel   0xc4450874 bacula-sd
100281                   S       kserel   0xc4450874 bacula-sd
100116                   S       select   0xc0815024 bacula-sd
100112                   S       ksesigwa 0xc474abb0 bacula-sd
  649   623   620    88  S+      (threaded)          mysqld
100119                   S       kserel   0xc45d4814 mysqld
100165                   S       select   0xc0815024 mysqld
100089                   S       kserel   0xc45d4814 mysqld
100125                   S       kserel   0xc44506f4 mysqld
100124                   S       sigwait  0xe4239c0c mysqld
100122                   S       ksesigwa 0xc47f4780 mysqld
  623     1   620    88  S+      wait     0xc4760218 sh
  610     1   610     0  Ss      select   0xc0815024 usbd
  588     1   588     0  Ss      biord    0xd3d73060 ntpd
  578     1   578     0  Ss      select   0xc0815024 lpd
  570     1    51     0  S+      select   0xc0815024 3dm2
  569   555   555     0  S       pause    0xc474a894 smbd
  555     1   555     0  Ss      select   0xc0815024 smbd
  551     1   551     0  Ss      select   0xc0815024 nmbd
  532     1   531     0  S       select   0xc0815024 snmpd
  519   513   513     0  S       rpcsvc   0xc4722a6c rpc.lockd
  518   513   513     0  S       nfslockd 0xc081d3c8 rpc.lockd
  513     1   513     0  Ss      select   0xc0815024 rpc.lockd
  506     1   506     0  Ss      select   0xc0815024 rpc.statd
  500   496   496     0  S       -        0xc4560c00 nfsd
  499   496   496     0  S       -        0xc46e6200 nfsd
  498   496   496     0  S       -        0xc46cec00 nfsd
  497   496   496     0  S       -        0xc4551a00 nfsd
  496     1   496     0  Ss      accept   0xc4802b5a nfsd
  488     1   488     0  Ss      ufs      0xc47857c8 mountd
  452     1   452     0  Ss      select   0xc0815024 rpcbind
  435     1   435     0  Ss      biord    0xd3ea6488 syslogd
  377     1   377     0  Ss      select   0xc0815024 devd
  139     1   139     0  Ss      pause    0xc474a034 adjkerntz
   50     0     0     0  SL      -        0xe2618cf8 [schedcpu]
   49     0     0     0  SL      sdflush  0xc0822714 [softdepflush]
   48     0     0     0  SL      vlruwt   0xc45d2a78 [vnlru]
   47     0     0     0  SL      getblk   0xd3e47410 [syncer]
   46     0     0     0  SL      psleep   0xc08155a0 [bufdaemon]
   45     0     0     0  SL      pgzero   0xc0823724 [pagezero]
   44     0     0     0  SL      psleep   0xc082322c [vmdaemon]
   43     0     0     0  SL      psleep   0xc08231e8 [pagedaemon]
   42     0     0     0  WL                          [swi0: sio]
   41     0     0     0  SL      -        0xc45f283c [fdc0]
   40     0     0     0  RL      CPU 0               [irq1: atkbd0]
   39     0     0     0  SL      usbevt   0xc45e6210 [usb4]
   38     0     0     0  WL                          [irq15: ata1]
   37     0     0     0  WL                          [irq14: ata0]
   36     0     0     0  SL      usbevt   0xc44f1210 [usb3]
   35     0     0     0  WL                          [irq16: ehci0]
   34     0     0     0  SL      usbevt   0xc45da210 [usb2]
   33     0     0     0  WL                          [irq23: ohci2]
   32     0     0     0  SL      usbevt   0xc45d8210 [usb1]
   31     0     0     0  WL                          [irq22: ohci1]
   30     0     0     0  SL      usbtsk   0xc07c4364 [usbtask]
   29     0     0     0  SL      usbevt   0xc4577210 [usb0]
   28     0     0     0  WL                          [irq21: fxp0 ohci0+]
   27     0     0     0  SL      idle     0xc4560600 [aic_recovery1]
   26     0     0     0  SL      idle     0xc4560600 [aic_recovery1]
   25     0     0     0  SL      idle     0xc4560400 [aic_recovery0]
   24     0     0     0  WL                          [irq19: ahc0 ahc1]
   23     0     0     0  SL      idle     0xc4560400 [aic_recovery0]
   22     0     0     0  WL                          [irq18: twa0]
   21     0     0     0  WL                          [irq9: acpi0]
   20     0     0     0  WL                          [swi6: Giant taskq]
   19     0     0     0  WL                          [swi6: task queue]
   18     0     0     0  WL                          [swi2: cambio]
   17     0     0     0  SL      ccb_scan 0xc07c0d04 [xpt_thrd]
    9     0     0     0  SL      -        0xc44a5700 [kqueue taskq]
    8     0     0     0  SL      -        0xc44a5800 [acpi_task_2]
    7     0     0     0  SL      -        0xc44a5800 [acpi_task_1]
    6     0     0     0  SL      -        0xc44a5800 [acpi_task_0]
   16     0     0     0  WL                          [swi5: +]
    5     0     0     0  SL      -        0xc44a5980 [thread taskq]
   15     0     0     0  SL      -        0xc07c3c40 [yarrow]
    4     0     0     0  SL      -        0xc07c4c68 [g_down]
    3     0     0     0  SL      -        0xc07c4c64 [g_up]
    2     0     0     0  SL      -        0xc07c4c5c [g_event]
   14     0     0     0  WL                          [swi3: vm]
   13     0     0     0  WL                          [swi4: clock sio]
   12     0     0     0  WL                          [swi1: net]
   11     0     0     0  RL                          [idle: cpu0]
   10     0     0     0  RL      CPU 1               [idle: cpu1]
    1     0     1     0  SLs     wait     0xc4451000 [init]
    0     0     0     0  WLs                         [swapper]
db> show lockedvnods
Locked vnodes

0xc4785770: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 67 mountedhere 0
    flags (VV_ROOT)
    v_object 0xc6401000 ref 0 pages 0
     lock type ufs: EXCL (count 1) by thread 0xc4bd0180 (pid 57786) with
65 pending ino 2, on dev da0s1d

0xc47da660: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 2 mountedhere 0
    flags ()
    v_object 0xc5f8239c ref 0 pages 0
     lock type ufs: EXCL (count 1) by thread 0xc4e48c00 (pid 57780)
        ino 32899, on dev da0s1d

0xc47d9110: tag ufs, type VDIR
    usecount 5, writecount 0, refcount 9 mountedhere 0
    flags ()
    v_object 0xc6f788c4 ref 0 pages 1
     lock type ufs: EXCL (count 1) by thread 0xc47f6900 (pid 588) with 1
pending ino 49347, on dev da0s1d

0xc4827110: tag ufs, type VREG
    usecount 1, writecount 1, refcount 3 mountedhere 0
    flags ()
    v_object 0xc481d948 ref 0 pages 3
     lock type ufs: EXCL (count 1) by thread 0xc4611600 (pid 435)
        ino 49480, on dev da0s1d

0xc4953330: tag ufs, type VDIR
    usecount 15, writecount 0, refcount 18 mountedhere 0
    flags ()
    v_object 0xc4960318 ref 0 pages 1
     lock type ufs: EXCL (count 1) by thread 0xc5cdac00 (pid 57782) with
6 pending ino 49359, on dev da0s1d

0xc6ebd990: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 3 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc4e4a480 (pid 57762) with
1 pending ino 32900, on dev da0s1d

0xc47b6aa0: tag syncer, type VNON
    usecount 1, writecount 0, refcount 2 mountedhere 0
    flags ()
     lock type syncer: EXCL (count 1) by thread 0xc45d3600 (pid 47)

0xc5540330: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 4 mountedhere 0
    flags ()
    v_object 0xc6cca840 ref 0 pages 1
     lock type ufs: EXCL (count 1) by thread 0xc4f19000 (pid 57732)
        ino 1013632, on dev da0s1f

0xc78e6dd0: tag ufs, type VREG
    usecount 1, writecount 0, refcount 1 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc4f19000 (pid 57732)
        ino 1016580, on dev da0s1f

0xc47d1110: tag ufs, type VDIR
    usecount 165, writecount 0, refcount 168 mountedhere 0
    flags ()
    v_object 0xc4a2fdec ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc4951480 (pid 1048) with 6
pending ino 9212984, on dev da0s1g

0xc47d2000: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 4 mountedhere 0
    flags ()
    v_object 0xc51f5d68 ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc5cd8d80 (pid 72603)
        ino 9213777, on dev da0s1g

0xc498a990: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 4 mountedhere 0
    flags ()
    v_object 0xc5c4839c ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc4e5e000 (pid 57770) with
1 pending ino 9213027, on dev da0s1g

0xc4ac5330: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 5 mountedhere 0
    flags ()
    v_object 0xc51f5dec ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc4e4a000 (pid 72715) with
1 pending ino 9213787, on dev da0s1g

0xc4ac5110: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 5 mountedhere 0
    flags ()
    v_object 0xc51f6000 ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc4a35300 (pid 78289) with
1 pending ino 9214698, on dev da0s1g

0xc4ad8770: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 5 mountedhere 0
    flags ()
    v_object 0xc51f6084 ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc4a35600 (pid 78290) with
1 pending ino 9214827, on dev da0s1g

0xc4766880: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 5 mountedhere 0
    flags ()
    v_object 0xc5976108 ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc4a36a80 (pid 57759) with
1 pending ino 9214859, on dev da0s1g

0xc5502000: tag ufs, type VREG
    usecount 0, writecount 0, refcount 4 mountedhere 0
    flags ()
    v_object 0xc6979528 ref 0 pages 10
     lock type ufs: EXCL (count 1) by thread 0xc6cd9900 (pid 57765) with
1 pending ino 9213198, on dev da0s1g

0xc7327cc0: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 3 mountedhere 0
    flags ()
    v_object 0xc73d439c ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc47f7780 (pid 897)
        ino 19661150, on dev da0s1g

0xc6447770: tag ufs, type VREG
    usecount 1, writecount 0, refcount 1 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc4a36a80 (pid 57759)
        ino 9213267, on dev da0s1g

0xc47d1dd0: tag null, type VDIR
    usecount 2, writecount 0, refcount 2 mountedhere 0
    flags (VV_ROOT)
    v_object 0xc51f5d68 ref 0 pages 2
     lock type ufs: EXCL (count 1) by thread 0xc5cd8d80 (pid 72603)
        vp=0xc47d1dd0, lowervp=0xc47d2000

0xc594f110: tag null, type VDIR
    usecount 2, writecount 0, refcount 2 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc4a36a80 (pid 57759) with
1 pending vp=0xc594f110, lowervp=0xc4766880

0xc7096660: tag null, type VDIR
    usecount 1, writecount 0, refcount 1 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc4a35600 (pid 78290) with
1 pending vp=0xc7096660, lowervp=0xc4ad8770

0xc6183660: tag null, type VDIR
    usecount 1, writecount 0, refcount 1 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc4a35300 (pid 78289) with
1 pending vp=0xc6183660, lowervp=0xc4ac5110

0xc6f2e220: tag null, type VDIR
    usecount 1, writecount 0, refcount 1 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc4e4a000 (pid 72715) with
1 pending vp=0xc6f2e220, lowervp=0xc4ac5330


-- 
 Oliver Lehmann
  http://www.pofo.de/
  http://wishlist.ans-netz.de/

freebsd stable - Sep 2008 - system hangup - I'm lost

system hangup - I'm lost

system hangup - I'm lost

system hangup - I'm lost

system hangup - I'm lost

system hangup - I'm lost