Hi. It seems i have a deadlock on 6.2-PRERELEASE. This is squid server in accelerator mode. I can easily trigger it with a high rate of requests. Squid is locked on some "zoneli" state, i am not sure what it is. Also i can't KILL proccess even with SIGKILL. In addition one of sshd proccess is locked too. Is there any additional information that i could provide? last pid: 1197; load averages: 0.00, 0.00, 0.00 up 0+01:54:58 14:46:40 31 processes: 1 running, 29 sleeping, 1 zombie CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 704M Active, 629M Inact, 447M Wired, 12K Cache, 112M Buf, 2109M Free Swap: 4070M Total, 4070M Free PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 671 squid 1 -16 0 688M 688M zoneli 6:32 0.00% squid ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 680 root 1 96 0 6628K 4760K select 0:02 0.00% snmpd 1170 root 1 96 0 2332K 1588K RUN 0:00 0.00% top 698 root 1 -8 0 7768K 7288K piperd 0:00 0.00% perl5.8.8 634 root 1 96 0 2984K 1808K select 0:00 0.00% ntpd 362 _pflogd 1 -58 0 1600K 1144K bpf 0:00 0.00% pflogd 1097 quetzal 1 96 0 6220K 3220K select 0:00 0.00% sshd 709 root 1 96 0 3464K 2796K select 0:00 0.00% sendmail 1100 root 1 20 0 5036K 3064K pause 0:00 0.00% tcsh 551 root 1 96 0 1352K 996K select 0:00 0.00% syslogd 1085 root 1 4 0 6232K 3204K sbwait 0:00 0.00% sshd 1095 root 1 4 0 6232K 3204K sbwait 0:00 0.00% sshd 1088 quetzal 1 6 0 4724K 2952K ttywai 0:00 0.00% tcsh 719 root 1 8 0 1364K 1060K nanslp 0:00 0.00% cron 1098 quetzal 1 20 0 4704K 2932K pause 0:00 0.00% tcsh 1087 quetzal 1 -16 0 6220K 3220K zoneli 0:00 0.00% sshd ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 654 root 1 96 0 1264K 804K select 0:00 0.00% usbd 692 root 1 96 0 3504K 2656K select 0:00 0.00% sshd 713 smmsp 1 20 0 3364K 2728K pause 0:00 0.00% sendmail 358 root 1 4 0 1536K 1092K sbwait 0:00 0.00% pflogd 769 root 1 5 0 1320K 896K ttyin 0:00 0.00% getty 773 root 1 5 0 1320K 896K ttyin 0:00 0.00% getty 772 root 1 5 0 1320K 896K ttyin 0:00 0.00% getty 771 root 1 5 0 1320K 896K ttyin 0:00 0.00% getty 770 root 1 5 0 1320K 896K ttyin 0:00 0.00% getty 775 root 1 5 0 1320K 896K ttyin 0:00 0.00% getty 774 root 1 5 0 1320K 896K ttyin 0:00 0.00% getty 776 root 1 5 0 1320K 896K ttyin 0:00 0.00% getty 497 root 1 114 0 528K 388K select 0:00 0.00% devd 128 root 1 20 0 1228K 680K pause 0:00 0.00% adjkerntz Also there is some interesting fstat info: root@accel1:~# fstat -p 671 -v | head -n 40 can't read vnode at 0x0 for pid 671 can't read vnode at 0x0 for pid 671 can't read vnode at 0x0 for pid 671 can't read vnode at 0x0 for pid 671 can't read vnode at 0x0 for pid 671 USER CMD PID FD MOUNT INUM MODE SZ|DV R/W squid squid 671 root / 2 drwxr-xr-x 512 r squid squid 671 wd /usr 1908230 drwxr-x--- 512 r squid squid 671 text /usr 1887228 -r-xr-xr-x 638296 r squid squid 671 0 - - error - squid squid 671 1 - - error - squid squid 671 2 - - error - squid squid 671 3 - - error - squid squid 671 4 /var 47121 -rw-r--r-- 2935342 rw squid squid 671 5* internet dgram udp c96205a0 squid squid 671 6 /var 47131 -rw-r--r-- 48909168 w squid squid 671 7* pipe c9551198 <-> c9551250 3 rw squid squid 671 8 /cache 7 -rw-r--r-- 91506636 w squid squid 671 9* internet stream tcp d2f17ae0 squid squid 671 10* pipe c9551a48 <-> c9551990 0 rw squid squid 671 11* internet stream tcp c971e3a0 squid squid 671 12* internet dgram udp c9620000 squid squid 671 13 - - error - squid squid 671 14* internet stream tcp squid squid 671 15* internet stream tcp d6b211d0 squid squid 671 16* internet stream tcp cf29c740 squid squid 671 17* internet stream tcp d0c9cae0 squid squid 671 18* internet stream tcp c9ebc570 squid squid 671 19* internet stream tcp d49c9000 squid squid 671 20* internet stream tcp d262eae0 squid squid 671 21 /cache 4031491 -rw-r--r-- 2037934 r squid squid 671 22* internet stream tcp ca1941d0 squid squid 671 23* internet stream tcp d5330570 squid squid 671 24* internet stream tcp d53fe000 squid squid 671 25* internet stream tcp d53a83a0 squid squid 671 26* internet stream tcp squid squid 671 27* internet stream tcp d2657910 squid squid 671 28* internet stream tcp d964b740 squid squid 671 29* internet stream tcp squid squid 671 30* internet stream tcp squid squid 671 31* internet stream tcp ca1d31d0 squid squid 671 32* internet stream tcp d83203a0 squid squid 671 33* internet stream tcp ca24b910 squid squid 671 34 /cache 3984782 -rw-r--r-- 1981893 r squid squid 671 35* internet stream tcp d7c0b3a0 -- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ======================================================================
Hi, On Wed, 22 Nov 2006 21:55:49 +0200, Nikolay Pavlov <quetzal@zone3000.net> wrote:> Hi. > It seems i have a deadlock on 6.2-PRERELEASE. > This is squid server in accelerator mode. > I can easily trigger it with a high rate of requests. > Squid is locked on some "zoneli" state, i am not sure what it is. > Also i can't KILL proccess even with SIGKILL. > In addition one of sshd proccess is locked too.Would you please update to the latest RELENG_6 and apply this patch: http://people.freebsd.org/~delphij/misc/patch-zonelimit-workaround to see if things gets improved? Thanks in advance! Cheers,
Nikolay Pavlov wrote:> No luck at all. > patch-zonelim-drain-20061212 works for me as a previos one. > no panics, but still zoneli. > All this is very odd, because other two squid servers works > perfectly in the same loadbalancer with out any patches and > kernel panics. I think that the case with this server > is realy rare.Would you please give a vmstat -z output when the server stuck in the zonelim livelock? Thanks! Cheers, -- Xin LI <delphij@delphij.net> http://www.delphij.net/ FreeBSD - The Power to Serve! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 249 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20061212/0856eb06/signature.pgp
On Wednesday, 13 December 2006 at 3:02:40 +0800, LI Xin wrote:> Nikolay Pavlov wrote: > > No luck at all. > > patch-zonelim-drain-20061212 works for me as a previos one. > > no panics, but still zoneli. > > All this is very odd, because other two squid servers works > > perfectly in the same loadbalancer with out any patches and > > kernel panics. I think that the case with this server > > is realy rare. > > Would you please give a vmstat -z output when the server stuck in the > zonelim livelock? Thanks!130947/775/131722 mbufs in use (current/cache/total) 130859/213/131072/131072 mbuf clusters in use (current/cache/total/max) 130859/213 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 294454K/619K/295074K bytes allocated to network (current/cache/total) 0/493001/246499 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/4/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 83 calls to protocol drain routines ITEM SIZE LIMIT USED FREE REQUESTS UMA Kegs: 140, 0, 84, 12, 84 UMA Zones: 120, 0, 84, 6, 84 UMA Slabs: 64, 0, 941, 3, 2709 UMA RCntSlabs: 104, 0, 65536, 28, 65536 UMA Hash: 128, 0, 4, 26, 6 16 Bucket: 76, 0, 27, 23, 35 32 Bucket: 140, 0, 21, 7, 29 64 Bucket: 268, 0, 30, 26, 67 128 Bucket: 524, 0, 215, 457, 827387 VM OBJECT: 132, 0, 41860, 16, 72475 MAP: 192, 0, 7, 33, 7 KMAP ENTRY: 68, 57456, 121, 47, 97518 MAP ENTRY: 68, 0, 702, 362, 94390 PV ENTRY: 24, 2228360, 124664, 2646, 984278 DP fakepg: 72, 0, 0, 0, 0 mt_zone: 64, 0, 182, 54, 182 16: 16, 0, 3754, 306, 432518 32: 32, 0, 1794, 353, 116429 64: 64, 0, 3205, 3580, 206314 128: 128, 0, 1616, 1384, 371069 256: 256, 0, 368, 427, 22536 512: 512, 0, 1266, 30, 486094 1024: 1024, 0, 48, 84, 434408 2048: 2048, 0, 147, 61, 43326 4096: 4096, 0, 129, 22, 4942 Files: 72, 0, 2672, 2734, 481720 MAC labels: 20, 0, 59586, 1085, 622560 PROC: 536, 0, 71, 27, 1564 THREAD: 376, 0, 98, 22, 98 KSEGRP: 88, 0, 98, 62, 98 UPCALL: 44, 0, 0, 0, 0 VMSPACE: 296, 0, 28, 24, 1518 mbuf_packet: 256, 0, 131052, 20, 9510911 mbuf: 256, 0, 91, 559, 12694062 mbuf_cluster: 2048, 131072, 131072, 0, 136594 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0 g_bio: 132, 0, 0, 1160, 522435 ata_request: 204, 0, 0, 0, 0 ata_composite: 196, 0, 0, 0, 0 VNODE: 348, 0, 47312, 10, 52436 VNODEPOLL: 76, 0, 0, 0, 0 S VFS Cache: 68, 0, 39770, 46, 44847 L VFS Cache: 291, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 12, 205701 DIRHASH: 1024, 0, 1595, 229, 7023 NFSMOUNT: 480, 0, 1, 15, 1 NFSNODE: 536, 0, 16, 5, 16 PIPE: 408, 0, 6, 21, 664 KNOTE: 68, 0, 0, 112, 76 socket: 356, 131076, 3930, 1229, 186491 unpcb: 140, 131096, 12, 44, 170 ipq: 32, 4181, 0, 0, 0 udpcb: 180, 131076, 12, 32, 188 inpcb: 180, 131076, 3859, 1509, 186132 tcpcb: 464, 131072, 3859, 1237, 186132 tcptw: 48, 8268, 0, 546, 140117 syncache: 100, 15366, 2, 271, 165792 hostcache: 76, 15400, 3717, 33, 6425 tcpreass: 20, 8281, 0, 169, 1493 sackhole: 20, 0, 4, 334, 372356 ripcb: 180, 131076, 0, 0, 0 rtentry: 132, 0, 22, 36, 31 pfsrctrpl: 100, 15015, 0, 0, 0 pfrulepl: 604, 0, 9, 9, 9 pfstatepl: 260, 15000, 4626, 7134, 165422 pfaltqpl: 128, 0, 0, 0, 0 pfpooladdrpl: 68, 0, 2, 110, 2 pfrktable: 1240, 0, 4, 5, 8 pfrkentry: 156, 0, 5, 45, 5 pfrkentry2: 156, 0, 0, 0, 0 pffrent: 16, 203, 0, 203, 61 pffrag: 48, 0, 0, 156, 12 pffrcache: 48, 10062, 0, 0, 0 pffrcent: 12, 50141, 0, 0, 0 pfstatescrub: 28, 0, 0, 0, 0 pfiaddrpl: 92, 0, 0, 0, 0 pfospfen: 108, 0, 345, 51, 345 pfosfp: 28, 0, 188, 193, 188 SWAPMETA: 276, 121576, 0, 0, 0 Mountpoints: 740, 0, 7, 8, 7 FFS inode: 132, 0, 47253, 17, 52376 FFS1 dinode: 128, 0, 0, 0, 0 FFS2 dinode: 256, 0, 47253, 12, 52376> > Cheers, > -- > Xin LI <delphij@delphij.net> http://www.delphij.net/ > FreeBSD - The Power to Serve! >-- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ======================================================================