thr3ads.net - freebsd stable - Processes stuck in *unp

If this information is useful, please help other people find it:
Share via:

Sergey Aleynikov

2009-Oct-21 10:59 UTC

Processes stuck in *unp_mtx state

I'm running FreeBSD 7.1-RELEASE and have application that spawnes some
children and (heavily) communicates between them using pipes. But
under load parent and one child both get stuck in unp_mtx state, they
can't be killed (killing them puts whole system on hang). This is
reproducable (i've written client emulation that causes this situation
after ~5 min run). Everything else keeps running, but trying to access
these stuck porcesses puts whole system on hang, so i have to perform
reboot. Panic is not invoked, so i've set up ddb panic autocollecting
script and had manually put system into panic.

Is this a known issue in 7.1, so upgrade to 7.2 would help? I list
below some info about system+ddb output:

begom% uname -a
FreeBSD begom.com 7.1-RELEASE FreeBSD 7.1-RELEASE #1: Mon Sep 14
07:37:56 MSD 2009
inferno@begom.com:/usr/obj/usr/src/release-7.1/sys/BEGOM  i386

db:0:kdb.enter.panic>  show pcpu
cpuid        = 1
curthread    = 0x84d00d20: pid 1 "init"
curpcb       = 0xa940bd90
fpcurthread  = none
idlethread   = 0x84d00af0: pid 11 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
db:0:kdb.enter.panic>  show allpcpu
Current CPU: 1

cpuid        = 0
curthread    = 0x850e08c0: pid 1078 "mysqld"
curpcb       = 0xae5d4d90
fpcurthread  = none
idlethread   = 0x84d008c0: pid 12 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50

cpuid        = 1
curthread    = 0x84d00d20: pid 1 "init"
curpcb       = 0xa940bd90
fpcurthread  = none
idlethread   = 0x84d00af0: pid 11 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50

db:0:kdb.enter.panic>  ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
...
89782 89764 89764  1013  L      *unp_mtx  0x850b9b40 perl5.8.8
89764     1 89764  1013  Ls     *unp_mtx  0x856a70f0 perl5.8.8


db:0:kdb.enter.panic>  alltrace
...
Tracing command perl5.8.8 pid 89764 tid 100434 td 0x85672690
sched_switch(85672690,0,1,23f76982,201f8b,...) at sched_switch+0x43b
mi_switch(1,0,80831849,2d9,85bf2000,...) at mi_switch+0x146
turnstile_wait(856a70f0,85bf2000,0,8675d690,87087690,...) at
turnstile_wait+0x2ec
_mtx_lock_sleep(8675d720,85672690,0,0,0,...) at _mtx_lock_sleep+0x10e
uipc_peeraddr(8bb431a0,ae6cac70,85672690,ae6cac80,86b74b94,...) at
uipc_peeraddr+0xc5
kern_getpeername(85672690,11,ae6cac70,ae6cac6c,100,...) at kern_getpeername+0x60
getpeername(85672690,ae6cacfc,c,16,ae6cad2c,...) at getpeername+0x52
syscall(ae6cad38) at syscall+0x335
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (31, FreeBSD ELF32, getpeername), eip = 0x282b88db, esp 0x7fbfeb5c,
ebp = 0x7fbfeba8 ---

Tracing command perl5.8.8 pid 89782 tid 100248 td 0x85bf2000
sched_switch(85bf2000,0,1,23f73c9c,201f8b,...) at sched_switch+0x43b
mi_switch(1,0,80831849,2d9,805d7233,...) at mi_switch+0x146
turnstile_wait(850b9b40,85672690,0,87087690,8675d690,...) at
turnstile_wait+0x2ec
_mtx_lock_sleep(87087720,85bf2000,0,0,0,...) at _mtx_lock_sleep+0x10e
uipc_peeraddr(8f1a8d00,ae3f4c70,ae3f4c44,4aded50c,84ff5da8,...) at
uipc_peeraddr+0xc5
kern_getpeername(85bf2000,6,ae3f4c70,ae3f4c6c,100,...) at kern_getpeername+0x60
getpeername(85bf2000,ae3f4cfc,c,16,ae3f4d2c,...) at getpeername+0x52
syscall(ae3f4d38) at syscall+0x335
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (31, FreeBSD ELF32, getpeername), eip = 0x282b88db, esp 0x7fbfeb5c,
ebp = 0x7fbfeba8 ---

db:0:kdb.enter.panic> show locks
No such command
db:0:kdb.enter.panic>  show alllocks
No such command
db:0:kdb.enter.panic>  show lockedvnods
Locked vnodes



options CONFIG_AUTOGENERATED
ident   BEGOM
machine i386
cpu     I686_CPU
makeoptions     DEBUG=-g
options ATA_STATIC_ID
options SMP
options AUDIT
options STOP_NMI
options ADAPTIVE_GIANT
options KBD_INSTALL_CDEV
options _KPOSIX_PRIORITY_SCHEDULING
options SYSVSEM
options SYSVMSG
options SYSVSHM
options KTRACE
options SCSI_DELAY=5000
options COMPAT_FREEBSD6
options COMPAT_FREEBSD5
options COMPAT_FREEBSD4
options COMPAT_43TTY
options GEOM_LABEL
options GEOM_PART_GPT
options PSEUDOFS
options PROCFS
options CD9660
options MD_ROOT
options UFS_GJOURNAL
options UFS_DIRHASH
options UFS_ACL
options SOFTUPDATES
options FFS
options INET
options PREEMPTION
options SCHED_ULE
options KVA_PAGES=512
options DEVICE_POLLING
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFIREWALL
options INCLUDE_CONFIG_FILE
options ALT_BREAK_TO_DEBUGGER
options KDB_UNATTENDED
options DDB
options KDB
options KSE
options GEOM_MBR
options GEOM_BSD
options ISAPNP

Loaded kernel modules are: linux, acpi, aio (but i've seen this both
with and without aio).


(part from fstat)
game     perl5.8.8  89764 root /             2 drwxr-xr-x     512  r
game     perl5.8.8  89764   wd /home    15191040 drwxr-x--x    3072  r
game     perl5.8.8  89764 text /usr     615835 -rwxr-xr-x   29935  r
game     perl5.8.8  89764    0 /dev        117 crw--w----   ttyp1 rw
game     perl5.8.8  89764    1 /dev        117 crw--w----   ttyp1 rw
game     perl5.8.8  89764    2 /dev        117 crw--w----   ttyp1 rw
game     perl5.8.8  89764    3* local dgram 86a8e3f0 <-> 852dc0a8
game     perl5.8.8  89764    4 /var     329768 -rw-r--r--       6 rw
game     perl5.8.8  89764    5 /usr     733593 -r--r--r--    6611  r
game     perl5.8.8  89764    7* local stream 8678cc78 <-> 8579bc78
game     perl5.8.8  89764    8* local stream 8579b000 <-> 8579bbd0
game     perl5.8.8  89764    9* local stream 8678c000 <-> 86e07150
game     perl5.8.8  89764   10* local stream 86e07888 <-> 86e07d20
game     perl5.8.8  89764   11* local stream 865ca0a8 <-> 875d5a80
game     perl5.8.8  89764   12* local stream 8678c0a8 <-> 865ca738
game     perl5.8.8  89764   13* local stream 8736b540 <-> 8678c690
game     perl5.8.8  89764   14* local stream 8579bdc8 <-> 8736bbd0
game     perl5.8.8  89764   15* local stream 85a5a498 <-> 875d5498
game     perl5.8.8  89764   16* local stream 85c2a3f0 <-> 85c2a348
game     perl5.8.8  89764   17* local stream 87087690 <-> 8675d690
game     perl5.8.8  89764   18* local stream 85a5a888 <-> 865ca2a0
game     perl5.8.8  89764   19* local stream 852c79d8 <-> 8528e3f0
game     perl5.8.8  89764   20* internet stream tcp 8d7d9570
game     perl5.8.8  89764   21* internet stream tcp 85e021d0
game     perl5.8.8  89764   22* internet stream tcp 899f71d0
game     perl5.8.8  89764   23* internet stream tcp 8e8bd1d0
game     perl5.8.8  89764   24* internet stream tcp 8c137000
game     perl5.8.8  89764   25* internet stream tcp 8d39b3a0
game     perl5.8.8  89764   26* internet stream tcp 85bc0000
game     perl5.8.8  89764   27* internet stream tcp 8c78e570
game     perl5.8.8  89764   28* internet stream tcp 8a7fa910
game     perl5.8.8  89764   29* internet stream tcp 8dd6bcb0
game     perl5.8.8  89764   30* internet stream tcp 8e801570
game     perl5.8.8  89764   31* internet stream tcp 86635cb0
game     perl5.8.8  89764   32* internet stream tcp 8cd50740
game     perl5.8.8  89764   33* internet stream tcp 86656000

Kostik Belousov

2009-Oct-21 13:34 UTC

head link

Processes stuck in *unp_mtx state

On Wed, Oct 21, 2009 at 07:36:50PM +0900, Sergey Aleynikov
wrote:> I'm running FreeBSD 7.1-RELEASE and have application that spawnes some
> children and (heavily) communicates between them using pipes. But
> under load parent and one child both get stuck in unp_mtx state, they
> can't be killed (killing them puts whole system on hang). This is
> reproducable (i've written client emulation that causes this situation
> after ~5 min run). Everything else keeps running, but trying to access
> these stuck porcesses puts whole system on hang, so i have to perform
> reboot. Panic is not invoked, so i've set up ddb panic autocollecting
> script and had manually put system into panic.
> 
> Is this a known issue in 7.1, so upgrade to 7.2 would help? I list
> below some info about system+ddb output:
I believe this is fixed by r194460 in HEAD and merged to RELENG_7 by
194967at the end of June, that was after 7.2.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20091021/5280f8e5/attachment.pgp

Sergey Aleynikov

2009-Oct-22 03:53 UTC

head link

Processes stuck in *unp_mtx state

2009/10/21 Kostik Belousov <kostikbel@gmail.com>:> I believe this is fixed by r194460 in HEAD and merged to RELENG_7 by
> 194967at the end of June, that was after 7.2.
After half an our of stress testing problem seams to disappear. Thanks
for your advice.

freebsd stable - Oct 2009 - Processes stuck in *unp_mtx state

Processes stuck in *unp_mtx state

Processes stuck in *unp_mtx state

Processes stuck in *unp_mtx state