Jonathan Feally
2006-Nov-22 20:43 UTC
6-STABLE (6.2-PRE) and applications (named natd dhcpd) getting stuck in state zoneli (zone limit) - dynamic ipfw rules not working after time - vlans on em
Sorry to cross post, but the net list didn't help a couple weeks back on
this.
names, natd, and dhcpd have all been getting stuck in zoneli (zone
limit) since I upgraded to the box to stable about a month ago. It was
running a 6.1-STABLE before with out difficulty. Very little has changed
on the box. All the same applications, same ipfw rules for the most part
(just more rules for new customers). Most of the time the processes
cannot be killed. I did get lucky yesterday with dhcpd. It finally died
about an hour later. I was compiling the latest stable at the time. But
got up again today to find dhcpd in zoneli. Can someone please point me
in the correct direction to trouble shoot this problem. I don't really
know how to get a full dump of what a process is doing, so a quick what
to do and post back would be great.
More Info On the Setup:
This box is acting as a all-in-one router with traffic shaping.
It has a single em card with 10 tagged vlans on it routing the majority
of traffic from the customer vlan (vlan125) to the isp vlan (vlan901).
It also runs dhcpd and named to service the customers on vlan125. All of
the customer's have a set of ipfw rules like the following.
# ipfw queue 11206 show
q11206: weight 40 pipe 1206 384 KB 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp
0 tcp x.y.125.136/1925 a.b.247.137/9580 264461 111550268 0
0 0
# ipfw queue 21206 show
q21206: weight 60 pipe 1206 384 KB 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp
0 udp c.d.55.125/36049 x.y.125.136/17307 239325 38512205 1 76 0
# ipfw queue 31206 show
q31206: weight 21 pipe 10000 384 KB 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp
0 tcp x.y.125.136/1925 a.b.247.137/9580 265850 112136314 0
0 0
# ipfw queue 41206 show
q41206: weight 21 pipe 20000 384 KB 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp
0 udp c.d.55.125/36049 x.y.125.136/17307 239447 38467203 0 0 0
# ipfw pipe 1206 show
01206: 2.090 Mbit/s 0 ms 768 KB 0 queues (1 buckets) droptail
q11206: weight 40 pipe 1206 384 KB 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp
0 tcp x.y.125.136/1925 a.b.247.137/9580 267269 113411268 0
0 0
q21206: weight 60 pipe 1206 384 KB 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp
0 udp c.d.55.125/36049 x.y.125.136/17307 241121 38680715 0 0 0
# ipfw pipe 10000 show
10000: 20.224 Mbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail
q31206: weight 21 pipe 10000 384 KB 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp
0 tcp x.y.125.136/1925 a.b.247.137/9580 268208 113672991 0
0 0
# ipfw pipe 20000 show
20000: 20.224 Mbit/s 0 ms 50 sl. 0 queues (1 buckets) droptail
q41206: weight 21 pipe 20000 384 KB 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes
Pkt/Byte Drp
0 udp c.d.55.125/36049 x.y.125.136/17307 241392 38632811 0 0 0
# The 10000 and 20000 pipes are for the whole link, while the 1206 pipe
is the customers alloted bandwidth. Customer cannot exceed his
bandwidth, and if the whole connection is maxed out he would be weighted
at 21 against other customers competing for isp bandwidth. The more
bandwidth a customer pays for the bigger his weight in the main pipes.
#Customer rules (#10000-#49999)
19000 0 0 reset tcp from any 25,135,139,445 to x.y.125.136
19000 4 176 reset tcp from any to x.y.125.136 dst-port
25,135,139,445
19000 0 0 reset tcp from x.y.125.136 to any dst-port
25,135,139,445
19001 216130 87769342 queue 11206 ip from x.y.125.136 to any out
19002 197604 34923736 queue 21206 ip from any to x.y.125.136 out
19003 216635 87807692 queue 31206 ip from x.y.125.136 to any in
19004 197283 34831273 queue 41206 ip from any to x.y.125.136 in
19005 432765 175577034 allow ip from x.y.125.136 to any
19006 394887 69755009 allow ip from any to x.y.125.136
#Other keep state rules - the keep-state rules will stop working after a
while.
50000 3325 258473 allow udp from any to any dst-port 53
50000 3020 938086 allow udp from any 53 to any
50001 41462 28661128 divert 8668 ip from any to any via vlan901
50002 0 0 check-state
# Used to forward all un-provisioned customer machines to the local
httpd so their mac can be recorded.
51004 358 23712 fwd 127.0.0.1,8080 tcp from x.y.125.0/24 to any
dst-port 80 in keep-state
# Drop anybody not allowed above
51099 1158 141144 deny ip from x.y.125.0/24 to any
# Allow admin vlan1 to go anywhere
60003 257 50939 allow ip from 10.255.1.0/24 to any keep-state
$ Final drop
65000 430 44547 reset tcp from any to any
65001 1887 424765 deny ip from any to any
Below are some dumps of process states when it locks up
#NATD zoneli
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
0 0 0 0 96 0 0 0 - WLs ?? 0:00.00
[swapper]
0 1 0 0 8 0 772 368 wait ILs ?? 0:00.80
/sbin/init --
0 2 0 0 8 0 0 8 crypto DL ?? 0:00.00 [crypto]
0 3 0 0 8 0 0 8 crypto DL ?? 0:00.00
[crypto returns]
0 4 0 0 -8 0 0 8 - DL ?? 0:12.55
[g_event]
0 5 0 0 -8 0 0 8 - DL ?? 0:17.83 [g_up]
0 6 0 0 -8 0 0 8 - DL ?? 0:18.84 [g_down]
0 7 0 0 8 0 0 8 - DL ?? 0:00.00
[kqueue taskq]
0 8 0 0 8 0 0 8 - DL ?? 0:00.00
[acpi_task_0]
0 9 0 0 8 0 0 8 - DL ?? 0:00.00
[acpi_task_1]
0 10 0 133 171 0 0 8 - RL ?? 4032:07.99
[idle: cpu0]
0 11 0 5 -44 0 0 8 - WL ?? 215:09.39
[swi1: net]
0 12 0 0 -32 0 0 8 - WL ?? 195:44.05
[swi4: clock sio]
0 13 0 0 -36 0 0 8 - WL ?? 0:00.00
[swi3: vm]
0 14 0 0 -16 0 0 8 - DL ?? 11:00.18 [yarrow]
0 15 0 0 -40 0 0 8 - WL ?? 0:00.00
[swi2: cambio]
0 16 0 0 -24 0 0 8 - WL ?? 0:00.00
[swi6: task queue]
0 17 0 0 8 0 0 8 - DL ?? 0:00.00
[acpi_task_2]
0 18 0 0 -24 0 0 8 - WL ?? 0:00.00
[swi6: Giant taskq]
0 19 0 0 8 0 0 8 - DL ?? 0:00.00
[thread taskq]
0 20 0 0 -28 0 0 8 - WL ?? 0:00.00
[swi5: +]
0 21 0 0 -52 0 0 8 - WL ?? 0:00.00
[irq9: acpi0]
0 22 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq16: uhci0 uhci3]
0 23 0 0 8 0 0 8 usbevt DL ?? 0:00.02 [usb0]
0 24 0 0 8 0 0 8 usbtsk DL ?? 0:00.00
[usbtask]
0 25 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq19: uhci1]
0 26 0 0 8 0 0 8 usbevt DL ?? 0:00.01 [usb1]
0 27 0 0 -64 0 0 8 - WL ?? 0:08.08
[irq18: uhci2+]
0 28 0 0 8 0 0 8 usbevt DL ?? 0:00.02 [usb2]
0 29 0 0 8 0 0 8 usbevt DL ?? 0:00.01 [usb3]
0 30 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq23: ehci0]
0 31 0 0 8 0 0 8 usbevt DL ?? 0:00.02 [usb4]
0 32 0 1 -68 0 0 8 - DL ?? 46:33.63 [em0
taskq]
0 33 0 0 -68 0 0 8 - WL ?? 0:00.00
[irq20: fxp0]
0 34 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq14: ata0]
0 35 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq15: ata1]
0 36 0 0 -60 0 0 8 - WL ?? 0:00.00
[irq1: atkbd0]
0 37 0 0 -48 0 0 8 - WL ?? 0:00.00
[swi0: sio]
0 38 0 0 -60 0 0 8 - WL ?? 0:00.00
[irq7: ppc0]
0 39 0 0 -16 0 0 8 psleep DL ?? 0:00.29
[pagedaemon]
0 40 0 0 20 0 0 8 psleep DL ?? 0:00.00
[vmdaemon]
0 41 0 0 171 0 0 8 pgzero DL ?? 1:07.23
[pagezero]
0 42 0 0 -16 0 0 8 psleep DL ?? 0:01.50
[bufdaemon]
0 43 0 0 -4 0 0 8 vlruwt DL ?? 0:01.90 [vnlru]
0 44 0 0 20 0 0 8 syncer DL ?? 2:28.29 [syncer]
0 45 0 0 -16 0 0 8 sdflus DL ?? 0:03.68
[softdepflush]
0 46 0 0 -64 0 0 8 - DL ?? 0:19.45
[schedcpu]
0 163 1 167 20 0 1252 728 pause Is ?? 0:00.00
adjkerntz -i
0 1242 1 167 116 0 528 388 select Is ?? 0:00.00
/sbin/devd
0 1486 1 0 -16 0 1572 1112 zoneli Ds ?? 1:47.61
/sbin/natd -f /etc/natd.conf -n vlan901
0 1645 1 0 96 0 1404 1112 select Ss ?? 0:18.82
/usr/sbin/syslogd -l /var/run/log -l /var/named/var/run/log -c
53 1707 1 0 96 0 13552 12700 select Ss ?? 7:38.07
/usr/sbin/named -t /var/named -u bind
0 1717 1 0 96 0 1504 1216 select Is ?? 0:00.26
/usr/sbin/rpcbind
0 1772 1 176 118 0 1492 1256 select Is ?? 0:00.00
/usr/sbin/mountd -r
0 1774 1 176 118 0 1372 1104 select Is ?? 0:00.03
nfsd: master (nfsd)
0 1775 1774 176 4 0 1280 888 - I ?? 0:00.00
nfsd: server (nfsd)
0 1776 1774 176 4 0 1280 888 - I ?? 0:00.00
nfsd: server (nfsd)
0 1777 1774 176 4 0 1280 888 - I ?? 0:00.00
nfsd: server (nfsd)
0 1778 1774 176 4 0 1280 888 - I ?? 0:00.00
nfsd: server (nfsd)
0 1784 1 0 96 0 263584 1112 select Is ?? 0:00.20
/usr/sbin/rpc.statd
0 1789 1 0 96 0 1596 1364 select Is ?? 0:00.24
rpc.lockd: server (rpc.lockd)
1 1796 1789 176 4 0 1596 1364 nfsloc I ?? 0:00.00
rpc.lockd: client (rpc.lockd)
0 1808 1 0 96 0 3204 2508 select Ss ?? 0:01.08
/usr/local/sbin/racoon
0 1853 1 0 96 0 2924 1824 select Ss ?? 0:14.87
/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db
0 1873 1 0 96 0 1288 824 select Is ?? 0:00.36
/usr/sbin/usbd
0 1902 1 0 96 0 3520 2648 select Is ?? 0:13.45
/usr/sbin/sshd
0 1907 1 0 96 0 3508 2948 select Ss ?? 0:05.87
sendmail: accepting connections (sendmail)
25 1911 1 0 20 0 3384 2804 pause Is ?? 0:00.10
sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (se
0 1917 1 0 8 0 1388 1100 nanslp Is ?? 0:30.23
/usr/sbin/cron -s
0 6535 1 0 8 0 11708 9140 nanslp Ss ?? 0:10.04
/usr/local/apache2/bin/httpd -k start
0 6565 1 0 96 0 1468 1220 select Is ?? 0:00.00
/usr/sbin/inetd -wW -C 60
65534 18455 6535 0 20 0 12040 9488 lockf I ?? 0:00.02
/usr/local/apache2/bin/httpd -k start
65534 51018 6535 0 20 0 12136 9632 lockf I ?? 0:00.09
/usr/local/apache2/bin/httpd -k start
65534 51041 6535 0 4 0 12128 9624 kqread I ?? 0:00.12
/usr/local/apache2/bin/httpd -k start
65534 51042 6535 0 20 0 12064 9560 lockf I ?? 0:00.03
/usr/local/apache2/bin/httpd -k start
65534 51439 6535 0 20 0 12140 9632 lockf I ?? 0:00.08
/usr/local/apache2/bin/httpd -k start
65534 51462 6535 0 20 0 12196 9700 lockf I ?? 0:00.13
/usr/local/apache2/bin/httpd -k start
65534 51463 6535 0 20 0 12064 9552 lockf I ?? 0:00.10
/usr/local/apache2/bin/httpd -k start
65534 51476 6535 0 20 0 12060 9512 lockf I ?? 0:00.02
/usr/local/apache2/bin/httpd -k start
65534 51477 6535 0 20 0 12152 9648 lockf I ?? 0:00.11
/usr/local/apache2/bin/httpd -k start
65534 51479 6535 0 20 0 12048 9540 lockf I ?? 0:00.07
/usr/local/apache2/bin/httpd -k start
1001 82596 1 0 96 0 2732 2452 select Ss ?? 0:03.93
/usr/local/sbin/dhcpd -cf /usr/local/etc/dhcpd.conf -lf /var/db
0 88474 1902 0 4 0 6272 3276 sbwait Is ?? 0:00.04
sshd: vulture [priv] (sshd)
1002 88653 88474 0 96 0 6256 3280 select S ?? 0:00.23
sshd: vulture@ttyp0 (sshd)
0 6583 1 172 5 0 1344 952 ttyin Is+ v0 0:00.00
/usr/libexec/getty Pc ttyv0
0 6584 1 172 5 0 1344 952 ttyin Is+ v1 0:00.00
/usr/libexec/getty Pc ttyv1
0 6585 1 172 5 0 1344 952 ttyin Is+ v2 0:00.00
/usr/libexec/getty Pc ttyv2
0 6586 1 172 5 0 1344 952 ttyin Is+ v3 0:00.00
/usr/libexec/getty Pc ttyv3
0 6587 1 172 5 0 1344 952 ttyin Is+ v4 0:00.00
/usr/libexec/getty Pc ttyv4
0 6588 1 172 5 0 1344 952 ttyin Is+ v5 0:00.00
/usr/libexec/getty Pc ttyv5
0 6589 1 172 5 0 1344 952 ttyin Is+ v6 0:00.00
/usr/libexec/getty Pc ttyv6
0 6590 1 172 5 0 1344 952 ttyin Is+ v7 0:00.00
/usr/libexec/getty Pc ttyv7
0 1882 1 0 -58 0 3204 2832 bpf S con- 0:06.62
/usr/local/sbin/arpwatch -i vlan125 -f arp.vlan125.dat
1002 88657 88653 0 8 0 3196 2052 wait Is p0 0:00.01
-bash (bash)
0 88661 88657 0 8 0 1776 1416 wait I p0 0:00.01 su -
0 88662 88661 1 8 0 3232 2092 wait S p0 0:00.06 -su
(bash)
0 90806 88662 0 96 0 1500 988 - R+ p0 0:00.00 ps -axl
#Named zoneli
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME
COMMAND USER %CPU %MEM STARTED
0 10 0 269 171 0 0 8 - RL ?? 3841:42.30
[idle: cpu0] root 79.2 0.0 Sun05PM
0 57074 56659 87 106 0 4892 4472 select S+ v1 0:10.89
cvsup system/sta root 9.6 0.9 5:00PM
0 11 0 42 -44 0 0 8 - WL ?? 190:27.13
[swi1: net] root 4.8 0.0 Sun05PM
0 0 0 0 96 0 0 0 - WLs ?? 0:00.00
[swapper] root 0.0 0.0 Sun05PM
0 1 0 0 8 0 772 368 wait ILs ?? 0:00.08
/sbin/init -- root 0.0 0.1 Sun05PM
0 2 0 0 8 0 0 8 crypto DL ?? 0:00.00
[crypto] root 0.0 0.0 Sun05PM
0 3 0 0 8 0 0 8 crypto DL ?? 0:00.00
[crypto returns] root 0.0 0.0 Sun05PM
0 4 0 0 -8 0 0 8 - DL ?? 0:11.47
[g_event] root 0.0 0.0 Sun05PM
0 5 0 0 -8 0 0 8 - DL ?? 0:14.95
[g_up] root 0.0 0.0 Sun05PM
0 6 0 1 -8 0 0 8 - DL ?? 0:17.02
[g_down] root 0.0 0.0 Sun05PM
0 7 0 0 8 0 0 8 - DL ?? 0:00.00
[kqueue taskq] root 0.0 0.0 Sun05PM
0 8 0 0 8 0 0 8 - DL ?? 0:00.00
[acpi_task_0] root 0.0 0.0 Sun05PM
0 9 0 0 8 0 0 8 - DL ?? 0:00.00
[acpi_task_1] root 0.0 0.0 Sun05PM
0 12 0 1 -32 0 0 8 - WL ?? 148:47.86
[swi4: clock sio root 0.0 0.0 Sun05PM
0 13 0 0 -36 0 0 8 - WL ?? 0:00.00
[swi3: vm] root 0.0 0.0 Sun05PM
0 14 0 0 -16 0 0 8 - DL ?? 8:07.19
[yarrow] root 0.0 0.0 Sun05PM
0 15 0 0 -40 0 0 8 - WL ?? 0:00.00
[swi2: cambio] root 0.0 0.0 Sun05PM
0 16 0 0 -24 0 0 8 - WL ?? 0:00.00
[swi6: task queu root 0.0 0.0 Sun05PM
0 17 0 0 8 0 0 8 - DL ?? 0:00.00
[acpi_task_2] root 0.0 0.0 Sun05PM
0 18 0 1 -24 0 0 8 - WL ?? 0:00.56
[swi6: Giant tas root 0.0 0.0 Sun05PM
0 19 0 0 8 0 0 8 - DL ?? 0:00.00
[thread taskq] root 0.0 0.0 Sun05PM
0 20 0 0 -28 0 0 8 - WL ?? 0:00.00
[swi5: +] root 0.0 0.0 Sun05PM
0 21 0 0 -52 0 0 8 - WL ?? 0:00.00
[irq9: acpi0] root 0.0 0.0 Sun05PM
0 22 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq16: uhci0 uh root 0.0 0.0 Sun05PM
0 23 0 0 8 0 0 8 usbevt DL ?? 0:00.01
[usb0] root 0.0 0.0 Sun05PM
0 24 0 0 8 0 0 8 usbtsk DL ?? 0:00.00
[usbtask] root 0.0 0.0 Sun05PM
0 25 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq19: uhci1] root 0.0 0.0 Sun05PM
0 26 0 0 8 0 0 8 usbevt DL ?? 0:00.01
[usb1] root 0.0 0.0 Sun05PM
0 27 0 1 -64 0 0 8 - WL ?? 0:06.87
[irq18: uhci2+] root 0.0 0.0 Sun05PM
0 28 0 0 8 0 0 8 usbevt DL ?? 0:00.01
[usb2] root 0.0 0.0 Sun05PM
0 29 0 0 8 0 0 8 usbevt DL ?? 0:00.01
[usb3] root 0.0 0.0 Sun05PM
0 30 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq23: ehci0] root 0.0 0.0 Sun05PM
0 31 0 0 8 0 0 8 usbevt DL ?? 0:00.02
[usb4] root 0.0 0.0 Sun05PM
0 32 0 4 -68 0 0 8 - DL ?? 36:30.75 [em0
taskq] root 0.0 0.0 Sun05PM
0 33 0 0 -68 0 0 8 - WL ?? 0:00.00
[irq20: fxp0] root 0.0 0.0 Sun05PM
0 34 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq14: ata0] root 0.0 0.0 Sun05PM
0 35 0 0 -64 0 0 8 - WL ?? 0:00.00
[irq15: ata1] root 0.0 0.0 Sun05PM
0 36 0 0 -60 0 0 8 - WL ?? 0:00.04
[irq1: atkbd0] root 0.0 0.0 Sun05PM
0 37 0 0 -48 0 0 8 - WL ?? 0:00.00
[swi0: sio] root 0.0 0.0 Sun05PM
0 38 0 0 -60 0 0 8 - WL ?? 0:00.00
[irq7: ppc0] root 0.0 0.0 Sun05PM
0 39 0 0 -16 0 0 8 psleep DL ?? 0:00.26
[pagedaemon] root 0.0 0.0 Sun05PM
0 40 0 0 20 0 0 8 psleep DL ?? 0:00.00
[vmdaemon] root 0.0 0.0 Sun05PM
0 41 0 0 171 0 0 8 pgzero DL ?? 0:58.03
[pagezero] root 0.0 0.0 Sun05PM
0 42 0 0 -16 0 0 8 psleep DL ?? 0:01.37
[bufdaemon] root 0.0 0.0 Sun05PM
0 43 0 0 -4 0 0 8 vlruwt DL ?? 0:01.78
[vnlru] root 0.0 0.0 Sun05PM
0 44 0 0 20 0 0 8 syncer DL ?? 2:53.26
[syncer] root 0.0 0.0 Sun05PM
0 45 0 0 -16 0 0 8 sdflus DL ?? 0:03.31
[softdepflush] root 0.0 0.0 Sun05PM
0 46 0 0 -64 0 0 8 - DL ?? 0:17.67
[schedcpu] root 0.0 0.0 Sun05PM
0 163 1 167 20 0 1252 728 pause Is ?? 0:00.00
adjkerntz -i root 0.0 0.1 Sun05PM
0 1242 1 165 116 0 528 388 select Is ?? 0:00.00
/sbin/devd root 0.0 0.1 Sun05PM
0 1486 1 0 96 0 1568 1104 select Ss ?? 0:19.48
/sbin/natd -f /e root 0.0 0.2 Sun05PM
65534 1590 6647 0 20 0 11788 9236 lockf S ?? 0:00.04
/usr/local/apach nobody 0.0 1.8 3:14AM
0 1645 1 0 96 0 1408 1144 select Ss ?? 0:16.44
/usr/sbin/syslog root 0.0 0.2 Sun05PM
53 1707 1 0 -16 0 9296 8436 zoneli Ds ?? 5:48.13
/usr/sbin/named bind 0.0 1.6 Sun05PM
0 1717 1 0 96 0 1504 1216 select Is ?? 0:00.23
/usr/sbin/rpcbin root 0.0 0.2 Sun05PM
0 1772 1 164 116 0 1492 1256 select Is ?? 0:00.00
/usr/sbin/mountd root 0.0 0.2 Sun05PM
0 1774 1 164 116 0 1372 1104 select Is ?? 0:00.03
nfsd: master (nf root 0.0 0.2 Sun05PM
0 1775 1774 164 4 0 1280 888 - I ?? 0:00.00
nfsd: server (nf root 0.0 0.2 Sun05PM
0 1776 1774 164 4 0 1280 888 - I ?? 0:00.00
nfsd: server (nf root 0.0 0.2 Sun05PM
0 1777 1774 164 4 0 1280 888 - I ?? 0:00.00
nfsd: server (nf root 0.0 0.2 Sun05PM
0 1778 1774 164 4 0 1280 888 - I ?? 0:00.00
nfsd: server (nf root 0.0 0.2 Sun05PM
0 1784 1 0 96 0 263584 1112 select Is ?? 0:00.19
/usr/sbin/rpc.st root 0.0 0.2 Sun05PM
0 1789 1 0 96 0 1596 1364 select Is ?? 0:00.22
rpc.lockd: serve root 0.0 0.3 Sun05PM
1 1796 1789 164 4 0 1596 1364 nfsloc I ?? 0:00.00
rpc.lockd: clien daemon 0.0 0.3 Sun05PM
0 1808 1 0 96 0 3204 2436 select Ss ?? 0:01.30
/usr/local/sbin/ root 0.0 0.5 Sun05PM
0 2101 1 0 96 0 2924 1824 select Ss ?? 0:13.51
/usr/sbin/ntpd - root 0.0 0.4 Sun05PM
0 2121 1 0 96 0 1288 824 select Is ?? 0:00.33
/usr/sbin/usbd root 0.0 0.2 Sun05PM
0 2150 1 40 101 0 3520 2648 select Is ?? 0:01.11
/usr/sbin/sshd root 0.0 0.5 Sun05PM
0 2157 1 0 96 0 3508 2944 select Ss ?? 0:05.36
sendmail: accept root 0.0 0.6 Sun05PM
25 2161 1 0 20 0 3384 2804 pause Is ?? 0:00.09
sendmail: Queue smmsp 0.0 0.5 Sun05PM
0 2167 1 9 8 0 1388 1100 nanslp Ss ?? 0:27.45
/usr/sbin/cron - root 0.0 0.2 Sun05PM
65534 3605 6647 0 20 0 12148 9644 lockf S ?? 0:00.38
/usr/local/apach nobody 0.0 1.9 9:06PM
0 6647 1 0 8 0 11708 9140 nanslp Ss ?? 0:09.25
/usr/local/apach root 0.0 1.8 Sun05PM
0 6672 1 158 115 0 1468 1168 select Is ?? 0:00.00
/usr/sbin/inetd root 0.0 0.2 Sun05PM
65534 34952 6647 0 20 0 12340 9820 lockf S ?? 0:01.05
/usr/local/apach nobody 0.0 1.9 9:48PM
1001 43220 1 0 96 0 2708 2428 select Ss ?? 0:00.42
/usr/local/sbin/ dhcpd 0.0 0.5 3:46PM
65534 49660 6647 0 20 0 11788 9236 lockf S ?? 0:00.03
/usr/local/apach nobody 0.0 1.8 7:32AM
65534 49710 6647 0 20 0 11788 9236 lockf S ?? 0:00.02
/usr/local/apach nobody 0.0 1.8 7:32AM
65534 50579 6647 0 4 0 12132 9624 kqread S ?? 0:00.14
/usr/local/apach nobody 0.0 1.9 11:07PM
65534 51202 6647 0 20 0 12092 9596 lockf S ?? 0:00.13
/usr/local/apach nobody 0.0 1.9 11:10PM
0 57451 2167 0 -8 0 1388 1164 piperd S ?? 0:00.00
cron: running jo root 0.0 0.2 5:03PM
0 57452 2167 0 -8 0 1388 1164 piperd S ?? 0:00.00
cron: running jo root 0.0 0.2 5:03PM
0 57454 2167 0 -8 0 1388 1164 piperd S ?? 0:00.00
cron: running jo root 0.0 0.2 5:03PM
0 57457 57451 0 8 0 1708 1224 wait Ss ?? 0:00.00
/bin/sh -c /etc/ root 0.0 0.2 5:03PM
0 57458 57454 0 8 0 1708 1224 wait Ss ?? 0:00.00
/bin/sh -c /etc/ root 0.0 0.2 5:03PM
0 57466 57452 1 8 0 1708 1224 wait Ss ?? 0:00.00
/bin/sh -c /etc/ root 0.0 0.2 5:03PM
0 57484 57457 0 8 0 1704 1256 wait S ?? 0:00.00
/bin/sh /etc/pin root 0.0 0.2 5:03PM
0 57495 57466 2 8 0 1704 1256 wait S ?? 0:00.00
/bin/sh /etc/pin root 0.0 0.2 5:03PM
0 57509 57458 0 8 0 1704 1256 wait S ?? 0:00.00
/bin/sh /etc/pin root 0.0 0.2 5:03PM
0 57521 57495 2 8 0 1248 472 nanslp S ?? 0:00.00
sleep 21 root 0.0 0.1 5:03PM
0 57523 57484 0 8 0 1248 472 nanslp S ?? 0:00.00
sleep 22 root 0.0 0.1 5:03PM
0 57525 57509 0 8 0 1248 472 nanslp S ?? 0:00.00
sleep 20 root 0.0 0.1 5:03PM
65534 97950 6647 0 20 0 12128 9624 lockf S ?? 0:00.38
/usr/local/apach nobody 0.0 1.9 2:57AM
65534 98135 6647 0 20 0 12064 9568 lockf S ?? 0:00.22
/usr/local/apach nobody 0.0 1.9 2:58AM
65534 98319 6647 0 96 0 12260 9744 select S ?? 0:00.15
/usr/local/apach nobody 0.0 1.9 2:59AM
0 6690 1 0 8 0 1760 1512 wait Is v0 0:00.03
login [pam] (log root 0.0 0.3 Sun05PM
0 56648 6690 0 8 0 3216 2072 wait S v0 0:00.03
-bash (bash) root 0.0 0.4 4:58PM
0 57620 56648 1 96 0 1508 1032 - R+ v0 0:00.00 ps
-axeluwww root 0.0 0.2 5:03PM
0 6691 1 0 8 0 1740 1488 wait Is v1 0:00.02
login [pam] (log root 0.0 0.3 Sun05PM
0 56659 6691 0 8 0 3216 2072 wait I v1 0:00.04
-bash (bash) root 0.0 0.4 4:58PM
0 6692 1 0 8 0 1736 1460 wait Is v2 0:00.02
login [pam] (log root 0.0 0.3 Sun05PM
0 57198 6692 0 5 0 3212 2068 ttyin S+ v2 0:00.01
-bash (bash) root 0.0 0.4 5:01PM
0 6693 1 158 5 0 1344 952 ttyin Is+ v3 0:00.00
/usr/libexec/get root 0.0 0.2 Sun05PM
0 6694 1 158 5 0 1344 952 ttyin Is+ v4 0:00.00
/usr/libexec/get root 0.0 0.2 Sun05PM
0 6695 1 158 5 0 1344 952 ttyin Is+ v5 0:00.00
/usr/libexec/get root 0.0 0.2 Sun05PM
0 6696 1 158 5 0 1344 952 ttyin Is+ v6 0:00.00
/usr/libexec/get root 0.0 0.2 Sun05PM
0 6697 1 158 5 0 1344 952 ttyin Is+ v7 0:00.00
/usr/libexec/get root 0.0 0.2 Sun05PM
0 2130 1 0 -58 0 3216 2844 bpf S con- 0:06.37
/usr/local/sbin/ root 0.0 0.6 Sun05PM
Hopefully somebody can point me in the correct direction soon. dhcpd is
locking up within 12 hours now with the latest stable from yesterday. If
its the way I have the vlans all on 1 card, then I can fix that.
Thanks
-Jon
delphij@delphij.net
2006-Nov-23 12:24 UTC
6-STABLE (6.2-PRE) and applications (named natd dhcpd) getting stuckin state zoneli (zone limit) - dynamic ipfw rules not working after time- vlans on em
Hi, On Wed, 22 Nov 2006 12:41:04 -0800, Jonathan Feally <vulture@netvulture.com> wrote:> Sorry to cross post, but the net list didn't help a couple weeks back on > this. > > names, natd, and dhcpd have all been getting stuck in zoneli (zone > limit) since I upgraded to the box to stable about a month ago. It was > running a 6.1-STABLE before with out difficulty. Very little has changed > on the box. All the same applications, same ipfw rules for the most part > (just more rules for new customers). Most of the time the processes > cannot be killed. I did get lucky yesterday with dhcpd. It finally died > about an hour later. I was compiling the latest stable at the time. But > got up again today to find dhcpd in zoneli. Can someone please point me > in the correct direction to trouble shoot this problem. I don't really > know how to get a full dump of what a process is doing, so a quick what > to do and post back would be great.Will you please try the patch at: http://people.freebsd.org/~delphij/misc/patch-zonelimit-workaround To see if it gets your situation improved? Thanks in advance! Cheers,