Garrett Wollman
2013-Jan-29 03:21 UTC
ZFS deadlock on rrl->rr_ -- look familiar to anyone?
I just had a big fileserver deadlock in an odd way. I was
investigating a user's problem, and decided for various reasons to
restart mountd. It had been complaining like this:
Jan 28 21:06:43 nfs-prod-1 mountd[1108]: can't delete exports for
/usr/local/.zfs/snapshot/monthly-2013-01: Invalid argument
for a while, which is odd because /usr/local was never exported. When
I restarted mountd, it hung waiting on rrl->rr_, but the system may
already have been deadlocked at that point. procstat reported:
87678 104365 mountd - mi_switch sleepq_wait _cv_wait
rrw_enter zfs_root lookup namei vfs_donmount sys_nmount amd64_syscall
Xfast_syscall
I was able to run shutdown, and the rc scripts eventually hung in
sync(1) and timed out. The kernel then hung trying to do the same
thing, but I was able to break into the debugger. The debugger
interrupted an idle thread, which was not particularly helpful, but I
was able to quickly gather the following information before I had to
reset the machine to restore normal service.
Locked vnodes
0xfffffe00536383c0: 0xfffffe00536383c0: tag syncer, type VNON
tag syncer, type VNON
usecount 1, writecount 0, refcount 2 mountedhere 0
usecount 1, writecount 0, refcount 2 mountedhere 0
flags (VI(0x200))
flags (VI(0x200))
lock type syncer: EXCL by thread 0xfffffe00348cc470 (pid 22)
lock type syncer: EXCL by thread 0xfffffe00348cc470 (pid 22)
db> ps
pid ppid pgrp uid state wmesg wchan cmd
87996 1 87994 65534 D rrl->rr_ 0xfffffe0048ff8108 df
87976 1 87726 0 D+ rrl->rr_ 0xfffffe0048ff8108 sync
87707 1 87705 65534 D rrl->rr_ 0xfffffe0048ff8108 df
87700 1 87698 65534 D rrl->rr_ 0xfffffe0048ff8108 df
87678 1 87657 0 D+ rrl->rr_ 0xfffffe0048ff8108 mountd
87531 1 87529 65534 D rrl->rr_ 0xfffffe0048ff8108 df
87387 1 87385 65534 D rrl->rr_ 0xfffffe0048ff8108 df
87380 1 87378 65534 D rrl->rr_ 0xfffffe0048ff8108 df
87103 1 87101 65534 D rrl->rr_ 0xfffffe0048ff8108 df
87096 1 87094 65534 D rrl->rr_ 0xfffffe0048ff8108 df
85193 1 85192 0 D zio->io_ 0xfffffe10d3e75320 zfs
24 0 0 0 DL sdflush 0xffffffff80e50878 [softdepflush]
23 0 0 0 DL vlruwt 0xfffffe0048c0a940 [vnlru]
22 0 0 0 DL rrl->rr_ 0xfffffe0048ff8108 [syncer]
21 0 0 0 DL psleep 0xffffffff80e3c048 [bufdaemon]
20 0 0 0 DL pgzero 0xffffffff80e5a81c [pagezero]
19 0 0 0 DL psleep 0xffffffff80e599e8 [vmdaemon]
18 0 0 0 DL psleep 0xffffffff80e599ac [pagedaemon]
17 0 0 0 DL gkt:wait 0xffffffff80de6c0c [g_mp_kt]
16 0 0 0 DL ipmireq 0xfffffe00347400b8 [ipmi0: kcs]
9 0 0 0 DL ccb_scan 0xffffffff80dc1360 [xpt_thrd]
8 0 0 0 DL waiting_ 0xffffffff80e41e80 [sctp_iterator]
7 0 0 0 DL (threaded) [zfskern]
101355 D tx->tx_s 0xfffffe0050342e10
[txg_thread_enter]
101354 D tx->tx_q 0xfffffe0050342e30
[txg_thread_enter]
100989 D tx->tx_s 0xfffffe004fd27a10
[txg_thread_enter]
100988 D tx->tx_q 0xfffffe004fd27a30
[txg_thread_enter]
100593 D tx->tx_s 0xfffffe004a8c0a10
[txg_thread_enter]
100592 D tx->tx_q 0xfffffe004a8c0a30
[txg_thread_enter]
100216 D l2arc_fe 0xffffffff81228bc0 [l2arc_feed_thread]
100215 D arc_recl 0xffffffff81218d20
[arc_reclaim_thread]
15 0 0 0 DL (threaded) [usb]
[32 uninteresting and identical threads deleted]
6 0 0 0 DL mps_scan 0xfffffe00276816a8 [mps_scan2]
5 0 0 0 DL mps_scan 0xfffffe0027612ca8 [mps_scan1]
4 0 0 0 DL mps_scan 0xfffffe00274ef4a8 [mps_scan0]
14 0 0 0 DL - 0xffffffff80ded764 [yarrow]
3 0 0 0 DL crypto_r 0xffffffff80e4e0a0 [crypto returns]
2 0 0 0 DL crypto_w 0xffffffff80e4e060 [crypto]
13 0 0 0 DL (threaded) [geom]
100055 D - 0xffffffff80de6b90 [g_down]
100054 D - 0xffffffff80de6b88 [g_up]
100053 D - 0xffffffff80de6b78 [g_event]
12 0 0 0 WL (threaded) [intr]
100189 I [irq1: atkbd0]
100188 I [swi0: uart uart]
100187 I [irq19: atapci1]
100186 I [irq18: atapci0+]
100169 I [irq294: igb1:link]
100167 I [irq293: igb1:que
7]
100165 I [irq292: igb1:que
6]
100163 I [irq291: igb1:que
5]
100161 I [irq290: igb1:que
4]
100159 I [irq289: igb1:que
3]
100157 I [irq288: igb1:que
2]
100155 I [irq287: igb1:que
1]
100153 I [irq286: igb1:que
0]
100152 I [irq285: igb0:link]
100150 I [irq284: igb0:que
7]
100148 I [irq283: igb0:que
6]
100146 I [irq282: igb0:que
5]
100144 I [irq281: igb0:que
4]
100142 I [irq280: igb0:que
3]
100140 I [irq279: igb0:que
2]
100138 I [irq278: igb0:que
1]
100136 I [irq277: igb0:que
0]
100131 I [irq20: hpet0
ehci0]
100126 I [irq21: uhci2
uhci5]
100121 I [irq22: uhci1
uhci4]
100116 I [irq23: uhci0
uhci3+]
100115 I [irq276: mps2]
100112 I [irq275: mps1]
100108 I [irq274: ix1:link]
100106 I [irq273: ix1:que 7]
100104 I [irq272: ix1:que 6]
100102 I [irq271: ix1:que 5]
100100 I [irq270: ix1:que 4]
100098 I [irq269: ix1:que 3]
100096 I [irq268: ix1:que 2]
100094 I [irq267: ix1:que 1]
100092 I [irq266: ix1:que 0]
100090 I [irq265: ix0:link]
100088 I [irq264: ix0:que 7]
100086 I [irq263: ix0:que 6]
100084 I [irq262: ix0:que 5]
100082 I [irq261: ix0:que 4]
100080 I [irq260: ix0:que 3]
100078 I [irq259: ix0:que 2]
100076 I [irq258: ix0:que 1]
100074 I [irq257: ix0:que 0]
100073 I [irq256: mps0]
100065 I [swi2: cambio]
100064 I [swi6: task queue]
100063 I [swi6: Giant taskq]
100060 I [swi5: +]
[24 identical [swi4: clock] threads deleted]
100028 I [swi1: netisr 0]
100027 I [swi3: vm]
11 0 0 0 RL (threaded) [idle]
[24 identical idle threads deleted]
1 0 1 0 DLs rrl->rr_ 0xfffffe0048ff8108 [init]
10 0 0 0 DL audit_wo 0xffffffff80e4f7f0 [audit]
0 0 0 0 DLs (threaded) [kernel]
420220 D - 0xfffffe07bf578380 [zil_clean]
[66 similar zil_clean threads deleted]
101353 D - 0xfffffe004a481c80 [zfs_vn_rele_taskq]
101352 D - 0xfffffe005324fa80 [zio_ioctl_intr]
101351 D - 0xfffffe005324fb00 [zio_ioctl_issue]
101350 D - 0xfffffe005324fb80 [zio_claim_intr]
101349 D - 0xfffffe005324fc00 [zio_claim_issue]
101348 D - 0xfffffe005324fc80 [zio_free_intr]
101347 D - 0xfffffe005324fd00 [zio_free_issue_99]
[99 similar zio_free_issue_* threads deleted]
101247 D - 0xfffffe005324fd80
[zio_write_intr_high]
101246 D - 0xfffffe005324fd80
[zio_write_intr_high]
101245 D - 0xfffffe005324fd80
[zio_write_intr_high]
101244 D - 0xfffffe005324fd80
[zio_write_intr_high]
101243 D - 0xfffffe005324fd80
[zio_write_intr_high]
101242 D - 0xfffffe005324fe00 [zio_write_intr_7]
101241 D - 0xfffffe005324fe00 [zio_write_intr_6]
101240 D - 0xfffffe005324fe00 [zio_write_intr_5]
101239 D - 0xfffffe005324fe00 [zio_write_intr_4]
101238 D - 0xfffffe005324fe00 [zio_write_intr_3]
101237 D - 0xfffffe005324fe00 [zio_write_intr_2]
101236 D - 0xfffffe005324fe00 [zio_write_intr_1]
101235 D - 0xfffffe005324fe00 [zio_write_intr_0]
101234 D - 0xfffffe0053250000
[zio_write_issue_hig]
101233 D - 0xfffffe0053250000
[zio_write_issue_hig]
101232 D - 0xfffffe0053250000
[zio_write_issue_hig]
101231 D - 0xfffffe0053250000
[zio_write_issue_hig]
101230 D - 0xfffffe0053250000
[zio_write_issue_hig]
101229 D - 0xfffffe0053250080
[zio_write_issue_23]
101228 D - 0xfffffe0053250080
[zio_write_issue_22]
101227 D - 0xfffffe0053250080
[zio_write_issue_21]
101226 D - 0xfffffe0053250080
[zio_write_issue_20]
101225 D - 0xfffffe0053250080
[zio_write_issue_19]
101224 D - 0xfffffe0053250080
[zio_write_issue_18]
101223 D - 0xfffffe0053250080
[zio_write_issue_17]
101222 D - 0xfffffe0053250080
[zio_write_issue_16]
101221 D - 0xfffffe0053250080
[zio_write_issue_15]
101220 D - 0xfffffe0053250080
[zio_write_issue_14]
101219 D - 0xfffffe0053250080
[zio_write_issue_13]
101218 D - 0xfffffe0053250080
[zio_write_issue_12]
101217 D - 0xfffffe0053250080
[zio_write_issue_11]
101216 D - 0xfffffe0053250080
[zio_write_issue_10]
101215 D - 0xfffffe0053250080 [zio_write_issue_9]
101214 D - 0xfffffe0053250080 [zio_write_issue_8]
101213 D - 0xfffffe0053250080 [zio_write_issue_7]
101212 D - 0xfffffe0053250080 [zio_write_issue_6]
101211 D - 0xfffffe0053250080 [zio_write_issue_5]
101210 D - 0xfffffe0053250080 [zio_write_issue_4]
101209 D - 0xfffffe0053250080 [zio_write_issue_3]
101208 D - 0xfffffe0053250080 [zio_write_issue_2]
101207 D - 0xfffffe0053250080 [zio_write_issue_1]
101206 D - 0xfffffe0053250080 [zio_write_issue_0]
101205 D - 0xfffffe0053250100 [zio_read_intr_23]
101204 D - 0xfffffe0053250100 [zio_read_intr_22]
101203 D - 0xfffffe0053250100 [zio_read_intr_21]
101202 D - 0xfffffe0053250100 [zio_read_intr_20]
101201 D - 0xfffffe0053250100 [zio_read_intr_19]
101200 D - 0xfffffe0053250100 [zio_read_intr_18]
101199 D - 0xfffffe0053250100 [zio_read_intr_17]
101198 D - 0xfffffe0053250100 [zio_read_intr_16]
101197 D - 0xfffffe0053250100 [zio_read_intr_15]
101196 D - 0xfffffe0053250100 [zio_read_intr_14]
101195 D - 0xfffffe0053250100 [zio_read_intr_13]
101194 D - 0xfffffe0053250100 [zio_read_intr_12]
101193 D - 0xfffffe0053250100 [zio_read_intr_11]
101192 D - 0xfffffe0053250100 [zio_read_intr_10]
101191 D - 0xfffffe0053250100 [zio_read_intr_9]
101190 D - 0xfffffe0053250100 [zio_read_intr_8]
101189 D - 0xfffffe0053250100 [zio_read_intr_7]
101188 D - 0xfffffe0053250100 [zio_read_intr_6]
101187 D - 0xfffffe0053250100 [zio_read_intr_5]
101186 D - 0xfffffe0053250100 [zio_read_intr_4]
101185 D - 0xfffffe0053250100 [zio_read_intr_3]
101184 D - 0xfffffe0053250100 [zio_read_intr_2]
101183 D - 0xfffffe0053250100 [zio_read_intr_1]
101182 D - 0xfffffe0053250100 [zio_read_intr_0]
101181 D - 0xfffffe0053250180 [zio_read_issue_7]
101180 D - 0xfffffe0053250180 [zio_read_issue_6]
101179 D - 0xfffffe0053250180 [zio_read_issue_5]
101178 D - 0xfffffe0053250180 [zio_read_issue_4]
101177 D - 0xfffffe0053250180 [zio_read_issue_3]
101176 D - 0xfffffe0053250180 [zio_read_issue_2]
101175 D - 0xfffffe0053250180 [zio_read_issue_1]
101174 D - 0xfffffe0053250180 [zio_read_issue_0]
101173 D - 0xfffffe0053250200 [zio_null_intr]
101172 D - 0xfffffe0053250280 [zio_null_issue]
100987 D - 0xfffffe0048cc9500 [zfs_vn_rele_taskq]
100986 D - 0xfffffe0048c72280 [zio_ioctl_intr]
100985 D - 0xfffffe0048c71a00 [zio_ioctl_issue]
100984 D - 0xfffffe0048dd0d00 [zio_claim_intr]
100983 D - 0xfffffe0048dd0680 [zio_claim_issue]
100982 D - 0xfffffe004a949080 [zio_free_intr]
100981 D - 0xfffffe0048b77d80 [zio_free_issue_99]
[99 more zio_free_issue_* threads deleted]
100881 D - 0xfffffe004a94a480
[zio_write_intr_high]
100880 D - 0xfffffe004a94a480
[zio_write_intr_high]
100879 D - 0xfffffe004a94a480
[zio_write_intr_high]
100878 D - 0xfffffe004a94a480
[zio_write_intr_high]
100877 D - 0xfffffe004a94a480
[zio_write_intr_high]
100876 D - 0xfffffe0048dd1180 [zio_write_intr_7]
100875 D - 0xfffffe0048dd1180 [zio_write_intr_6]
100874 D - 0xfffffe0048dd1180 [zio_write_intr_5]
100873 D - 0xfffffe0048dd1180 [zio_write_intr_4]
100872 D - 0xfffffe0048dd1180 [zio_write_intr_3]
100871 D - 0xfffffe0048dd1180 [zio_write_intr_2]
100870 D - 0xfffffe0048dd1180 [zio_write_intr_1]
100869 D - 0xfffffe0048dd1180 [zio_write_intr_0]
100868 D - 0xfffffe0048dd1100
[zio_write_issue_hig]
100867 D - 0xfffffe0048dd1100
[zio_write_issue_hig]
100866 D - 0xfffffe0048dd1100
[zio_write_issue_hig]
100865 D - 0xfffffe0048dd1100
[zio_write_issue_hig]
100864 D - 0xfffffe0048dd1100
[zio_write_issue_hig]
100863 D - 0xfffffe0048dd1080
[zio_write_issue_23]
100862 D - 0xfffffe0048dd1080
[zio_write_issue_22]
100861 D - 0xfffffe0048dd1080
[zio_write_issue_21]
100860 D - 0xfffffe0048dd1080
[zio_write_issue_20]
100859 D - 0xfffffe0048dd1080
[zio_write_issue_19]
100858 D - 0xfffffe0048dd1080
[zio_write_issue_18]
100857 D - 0xfffffe0048dd1080
[zio_write_issue_17]
100856 D - 0xfffffe0048dd1080
[zio_write_issue_16]
100855 D - 0xfffffe0048dd1080
[zio_write_issue_15]
100854 D - 0xfffffe0048dd1080
[zio_write_issue_14]
100853 D - 0xfffffe0048dd1080
[zio_write_issue_13]
100852 D - 0xfffffe0048dd1080
[zio_write_issue_12]
100851 D - 0xfffffe0048dd1080
[zio_write_issue_11]
100850 D - 0xfffffe0048dd1080
[zio_write_issue_10]
100849 D - 0xfffffe0048dd1080 [zio_write_issue_9]
100848 D - 0xfffffe0048dd1080 [zio_write_issue_8]
100847 D - 0xfffffe0048dd1080 [zio_write_issue_7]
100846 D - 0xfffffe0048dd1080 [zio_write_issue_6]
100845 D - 0xfffffe0048dd1080 [zio_write_issue_5]
100844 D - 0xfffffe0048dd1080 [zio_write_issue_4]
100843 D - 0xfffffe0048dd1080 [zio_write_issue_3]
100842 D - 0xfffffe0048dd1080 [zio_write_issue_2]
100841 D - 0xfffffe0048dd1080 [zio_write_issue_1]
100840 D - 0xfffffe0048dd1080 [zio_write_issue_0]
100839 D - 0xfffffe0048dd1000 [zio_read_intr_23]
100838 D - 0xfffffe0048dd1000 [zio_read_intr_22]
100837 D - 0xfffffe0048dd1000 [zio_read_intr_21]
100836 D - 0xfffffe0048dd1000 [zio_read_intr_20]
100835 D - 0xfffffe0048dd1000 [zio_read_intr_19]
100834 D - 0xfffffe0048dd1000 [zio_read_intr_18]
100833 D - 0xfffffe0048dd1000 [zio_read_intr_17]
100832 D - 0xfffffe0048dd1000 [zio_read_intr_16]
100831 D - 0xfffffe0048dd1000 [zio_read_intr_15]
100830 D - 0xfffffe0048dd1000 [zio_read_intr_14]
100829 D - 0xfffffe0048dd1000 [zio_read_intr_13]
100828 D - 0xfffffe0048dd1000 [zio_read_intr_12]
100827 D - 0xfffffe0048dd1000 [zio_read_intr_11]
100826 D - 0xfffffe0048dd1000 [zio_read_intr_10]
100825 D - 0xfffffe0048dd1000 [zio_read_intr_9]
100824 D - 0xfffffe0048dd1000 [zio_read_intr_8]
100823 D - 0xfffffe0048dd1000 [zio_read_intr_7]
100822 D - 0xfffffe0048dd1000 [zio_read_intr_6]
100821 D - 0xfffffe0048dd1000 [zio_read_intr_5]
100820 D - 0xfffffe0048dd1000 [zio_read_intr_4]
100819 D - 0xfffffe0048dd1000 [zio_read_intr_3]
100818 D - 0xfffffe0048dd1000 [zio_read_intr_2]
100817 D - 0xfffffe0048dd1000 [zio_read_intr_1]
100816 D - 0xfffffe0048dd1000 [zio_read_intr_0]
100815 D - 0xfffffe0048dd0e00 [zio_read_issue_7]
100814 D - 0xfffffe0048dd0e00 [zio_read_issue_6]
100813 D - 0xfffffe0048dd0e00 [zio_read_issue_5]
100812 D - 0xfffffe0048dd0e00 [zio_read_issue_4]
100811 D - 0xfffffe0048dd0e00 [zio_read_issue_3]
100810 D - 0xfffffe0048dd0e00 [zio_read_issue_2]
100809 D - 0xfffffe0048dd0e00 [zio_read_issue_1]
100808 D - 0xfffffe0048dd0e00 [zio_read_issue_0]
100807 D - 0xfffffe0048dd0600 [zio_null_intr]
100806 D - 0xfffffe0048dd0180 [zio_null_issue]
100594 D - 0xfffffe004a3bcc80 [zil_clean]
100591 D - 0xfffffe0048c65100 [zfs_vn_rele_taskq]
100590 D - 0xfffffe0048d5c280 [zio_ioctl_intr]
100589 D - 0xfffffe0048d5c300 [zio_ioctl_issue]
100588 D - 0xfffffe0048d5c380 [zio_claim_intr]
100587 D - 0xfffffe0048d5c400 [zio_claim_issue]
100586 D - 0xfffffe0048d5c480 [zio_free_intr]
100585 D - 0xfffffe0048d5c500 [zio_free_issue_99]
[99 more zio_free_issue_* threads deleted]
100485 D - 0xfffffe0048d5c580
[zio_write_intr_high]
100484 D - 0xfffffe0048d5c580
[zio_write_intr_high]
100483 D - 0xfffffe0048d5c580
[zio_write_intr_high]
100482 D - 0xfffffe0048d5c580
[zio_write_intr_high]
100481 D - 0xfffffe0048d5c580
[zio_write_intr_high]
100480 D - 0xfffffe0048d5c600 [zio_write_intr_7]
100479 D - 0xfffffe0048d5c600 [zio_write_intr_6]
100478 D - 0xfffffe0048d5c600 [zio_write_intr_5]
100477 D - 0xfffffe0048d5c600 [zio_write_intr_4]
100476 D - 0xfffffe0048d5c600 [zio_write_intr_3]
100475 D - 0xfffffe0048d5c600 [zio_write_intr_2]
100474 D - 0xfffffe0048d5c600 [zio_write_intr_1]
100473 D - 0xfffffe0048d5c600 [zio_write_intr_0]
100472 D - 0xfffffe0048d5c680
[zio_write_issue_hig]
100471 D - 0xfffffe0048d5c680
[zio_write_issue_hig]
100470 D - 0xfffffe0048d5c680
[zio_write_issue_hig]
100469 D - 0xfffffe0048d5c680
[zio_write_issue_hig]
100468 D - 0xfffffe0048d5c680
[zio_write_issue_hig]
100467 D - 0xfffffe0048d5c700
[zio_write_issue_23]
100466 D - 0xfffffe0048d5c700
[zio_write_issue_22]
100465 D - 0xfffffe0048d5c700
[zio_write_issue_21]
100464 D - 0xfffffe0048d5c700
[zio_write_issue_20]
100463 D - 0xfffffe0048d5c700
[zio_write_issue_19]
100462 D - 0xfffffe0048d5c700
[zio_write_issue_18]
100461 D - 0xfffffe0048d5c700
[zio_write_issue_17]
100460 D - 0xfffffe0048d5c700
[zio_write_issue_16]
100459 D - 0xfffffe0048d5c700
[zio_write_issue_15]
100458 D - 0xfffffe0048d5c700
[zio_write_issue_14]
100457 D - 0xfffffe0048d5c700
[zio_write_issue_13]
100456 D - 0xfffffe0048d5c700
[zio_write_issue_12]
100455 D - 0xfffffe0048d5c700
[zio_write_issue_11]
100454 D - 0xfffffe0048d5c700
[zio_write_issue_10]
100453 D - 0xfffffe0048d5c700 [zio_write_issue_9]
100452 D - 0xfffffe0048d5c700 [zio_write_issue_8]
100451 D - 0xfffffe0048d5c700 [zio_write_issue_7]
100450 D - 0xfffffe0048d5c700 [zio_write_issue_6]
100449 D - 0xfffffe0048d5c700 [zio_write_issue_5]
100448 D - 0xfffffe0048d5c700 [zio_write_issue_4]
100447 D - 0xfffffe0048d5c700 [zio_write_issue_3]
100446 D - 0xfffffe0048d5c700 [zio_write_issue_2]
100445 D - 0xfffffe0048d5c700 [zio_write_issue_1]
100444 D - 0xfffffe0048d5c700 [zio_write_issue_0]
100443 D - 0xfffffe0048d5c780 [zio_read_intr_23]
100442 D - 0xfffffe0048d5c780 [zio_read_intr_22]
100441 D - 0xfffffe0048d5c780 [zio_read_intr_21]
100440 D - 0xfffffe0048d5c780 [zio_read_intr_20]
100439 D - 0xfffffe0048d5c780 [zio_read_intr_19]
100438 D - 0xfffffe0048d5c780 [zio_read_intr_18]
100437 D - 0xfffffe0048d5c780 [zio_read_intr_17]
100436 D - 0xfffffe0048d5c780 [zio_read_intr_16]
100435 D - 0xfffffe0048d5c780 [zio_read_intr_15]
100434 D - 0xfffffe0048d5c780 [zio_read_intr_14]
100433 D - 0xfffffe0048d5c780 [zio_read_intr_13]
100432 D - 0xfffffe0048d5c780 [zio_read_intr_12]
100431 D - 0xfffffe0048d5c780 [zio_read_intr_11]
100430 D - 0xfffffe0048d5c780 [zio_read_intr_10]
100429 D - 0xfffffe0048d5c780 [zio_read_intr_9]
100428 D - 0xfffffe0048d5c780 [zio_read_intr_8]
100427 D - 0xfffffe0048d5c780 [zio_read_intr_7]
100426 D - 0xfffffe0048d5c780 [zio_read_intr_6]
100425 D - 0xfffffe0048d5c780 [zio_read_intr_5]
100424 D - 0xfffffe0048d5c780 [zio_read_intr_4]
100423 D - 0xfffffe0048d5c780 [zio_read_intr_3]
100422 D - 0xfffffe0048d5c780 [zio_read_intr_2]
100421 D - 0xfffffe0048d5c780 [zio_read_intr_1]
100420 D - 0xfffffe0048d5c780 [zio_read_intr_0]
100419 D - 0xfffffe0048d5c800 [zio_read_issue_7]
100418 D - 0xfffffe0048d5c800 [zio_read_issue_6]
100417 D - 0xfffffe0048d5c800 [zio_read_issue_5]
100416 D - 0xfffffe0048d5c800 [zio_read_issue_4]
100415 D - 0xfffffe0048d5c800 [zio_read_issue_3]
100414 D - 0xfffffe0048d5c800 [zio_read_issue_2]
100413 D - 0xfffffe0048d5c800 [zio_read_issue_1]
100412 D - 0xfffffe0048d5c800 [zio_read_issue_0]
100411 D - 0xfffffe0048d5c880 [zio_null_intr]
100410 D - 0xfffffe0048d5c900 [zio_null_issue]
100214 D - 0xfffffe00348bbc00 [system_taskq_23]
[23 more system_taskq_* threads deleted]
100190 D - 0xfffffe00348bbc80 [mca taskq]
100168 D - 0xfffffe0034092b00 [igb1 que]
100166 D - 0xfffffe0034092c80 [igb1 que]
100164 D - 0xfffffe0034092e00 [igb1 que]
100162 D - 0xfffffe0034092180 [igb1 que]
100160 D - 0xfffffe0034092300 [igb1 que]
100158 D - 0xfffffe0034092480 [igb1 que]
100156 D - 0xfffffe003408b300 [igb1 que]
100154 D - 0xfffffe003408b480 [igb1 que]
100151 D - 0xfffffe003405b500 [igb0 que]
100149 D - 0xfffffe003405b680 [igb0 que]
100147 D - 0xfffffe0034054580 [igb0 que]
100145 D - 0xfffffe003404d400 [igb0 que]
100143 D - 0xfffffe003404d580 [igb0 que]
100141 D - 0xfffffe003404d700 [igb0 que]
100139 D - 0xfffffe003404d880 [igb0 que]
100137 D - 0xfffffe003404da00 [igb0 que]
100113 D - 0xfffffe0027807300 [mps2 taskq]
100110 D - 0xfffffe0027697a80 [mps1 taskq]
100109 D - 0xfffffe002768a700 [ix1 linkq]
100107 D - 0xfffffe002768a800 [ix1 que]
100105 D - 0xfffffe002768a980 [ix1 que]
100103 D - 0xfffffe002768ab00 [ix1 que]
100101 D - 0xfffffe002768ac80 [ix1 que]
100099 D - 0xfffffe0027680b80 [ix1 que]
100097 D - 0xfffffe0027680d00 [ix1 que]
100095 D - 0xfffffe0027623680 [ix1 que]
100093 D - 0xfffffe0027623380 [ix1 que]
100091 D - 0xfffffe0027600480 [ix0 linkq]
100089 D - 0xfffffe0027600580 [ix0 que]
100087 D - 0xfffffe0027600700 [ix0 que]
100085 D - 0xfffffe0027527300 [ix0 que]
100083 D - 0xfffffe0027527480 [ix0 que]
100081 D - 0xfffffe0027527600 [ix0 que]
100079 D - 0xfffffe0027527780 [ix0 que]
100077 D - 0xfffffe0027527900 [ix0 que]
100075 D - 0xfffffe0027527a80 [ix0 que]
100071 D - 0xfffffe00274ffe00 [mps0 taskq]
100070 D - 0xfffffe00273fdb00 [kqueue taskq]
100069 D - 0xfffffe00273fdb80 [ffs_trim taskq]
100068 D - 0xfffffe00273fdc00 [acpi_task_2]
100067 D - 0xfffffe00273fdc00 [acpi_task_1]
100066 D - 0xfffffe00273fdc00 [acpi_task_0]
100062 D - 0xfffffe002743b280 [aiod_bio taskq]
100061 D zfsvfs-> 0xfffffe0048ff8138 [thread taskq]
100056 D - 0xfffffe002732d600 [firmware taskq]
100000 D sched 0xffffffff80de6d80 [swapper]
The stuck df(1) processes running as nobody were undoubtedly started
by munin-node, and seem to be related to my user's symptom (munin
graphs show no response for about half an hour after the user's
problem starts). It may not be a *true* deadlock, because over the
past few days, munin has been showing this problem at about the same
time of day, but the system always comes back (without a reboot) in
a little over half an hour.
Does anyone recognize this? If it happens again, which threads' stack
traces would be useful in diagnosing this?
-GAWollman
on 29/01/2013 05:21 Garrett Wollman said the following:> When > I restarted mountd, it hung waiting on rrl->rr_, but the system may > already have been deadlocked at that point. procstat reported: > > 87678 104365 mountd - mi_switch sleepq_wait _cv_wait rrw_enter zfs_root lookup namei vfs_donmount sys_nmount amd64_syscall Xfast_syscall...> If it happens againprocstat -kk -a -- Andriy Gapon