Ian Pratt
2006-Apr-19 23:12 UTC
RE: [Xen-devel] Hard lock of server - saved history from serial consolewithin
> Yes, an iptable rule for each IP for bandwidth accounting. > Also 4 rules in the raw table and 4 rules in the nat table > for port redirection. I don''t see how the actual iptables > rules could effect the kernel though.(!) iptables causes big changes to the way packets pass through the kernel. In particular, it sounds like you may be using connection tracking for NAT. There were lots of changes in this area for 2.6.16, one of which broke bridging for large UDP and ICMP datagrams. Seeing what iptables modules you have loaded may be interesting. It doesn''t look much like a xen issue, but I wouldn''t totally rule it out. Ian> >> It''s really nothing special... routing is actually less > complex than > >> bridging IMHO. > >> > >> Is this perhaps a 2.6.16 kernel bug? Are there any > dis-advantages to > >> upgrading to 2.6.16.9 or .10-rc? > > > > Please try .9 or .10-rc and let us know how you get on. > > > > Easier said than done. I have 31 nodes and 2 have rebooted > with this error so far. It''s especially hurtful as I just > upgraded all nodes to the changeset listed in the original > message yesterday. I''ll give it a shot, but I have > absolutely no idea which node will reboot next. > > > Ian > > > >> Thanks, > >> Matt > >> > >> Ian Pratt wrote: > >>> Please can you describe your networking setup. Judging by > the stack > >>> trace its non standard. > >>> > >>> Ian > >>> > >>>> -----Original Message----- > >>>> From: xen-devel-bounces@lists.xensource.com > >>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Matt > >>>> Ayres > >>>> Sent: 19 April 2006 23:07 > >>>> To: xen-devel@lists.xensource.com > >>>> Subject: [Xen-devel] Hard lock of server - saved history > >> from serial > >>>> consolewithin > >>>> > >>>> This was a hard lock... > >>>> > >>>> xen_changeset : Sun Apr 16 15:45:57 2006 +0100 > >>>> 9629:07408313ce33 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> --- TOP OF BUFFER --- > >>>> > >>>> Unable to handle kernel paging request at virtual > address d144a053 > >>>> > >>>> printing eip: > >>>> > >>>> c03e5b55 > >>>> > >>>> 09baa000 -> *pde = 00000000:ec867001 > >>>> > >>>> 0e467000 -> *pme = 00000000:eb346067 > >>>> > >>>> 0fb46000 -> *pte = 00000000:00000000 > >>>> > >>>> Oops: 0000 [#1] > >>>> > >>>> SMP > >>>> > >>>> Modules linked in: > >>>> > >>>> CPU: 0 > >>>> > >>>> EIP: 0061:[<c03e5b55>] Not tainted VLI > >>>> > >>>> EFLAGS: 00010282 (2.6.16-xen #4) > >>>> > >>>> EIP is at ipt_do_table+0xad/0x2d0 > >>>> > >>>> eax: 00000002 ebx: c804ca1e ecx: c804ca1e edx: 00000000 > >>>> > >>>> esi: d144a000 edi: 00004000 ebp: d144d9d0 esp: c0567dd0 > >>>> > >>>> ds: 007b es: 007b ss: 0069 > >>>> > >>>> Process swapper (pid: 0, threadinfo=c0566000 task=c04e5c80) > >>>> > >>>> Stack: <0>c051956c 00000000 c804ca1e 00000000 ce2f1800 cfdcb000 > >>>> d144a000 0000000 0 > >>>> > >>>> 00000001 00000000 c0567e70 80000000 c03aeb78 cfdcb000 > >>>> c03e6bd0 > >>>> c0567eb4 > >>>> 00000002 ce2f1800 cfdcb000 c0519540 00000000 c03a42f8 > >>>> 00000002 > >>>> c0567eb4 > >>>> Call Trace: > >>>> > >>>> PORT HISTORY> NEXT > >>>> > >>>> [<c03aeb78>] ip_forward_finish+0x0/0x36 > >>>> > >>>> [<c03e6bd0>] ipt_hook+0x1c/0x20 > >>>> > >>>> [<c03a42f8>] nf_iterate+0x2c/0x5e > >>>> > >>>> [<c03aeb78>] ip_forward_finish+0x0/0x36 > >>>> > >>>> [<c03aeb78>] ip_forward_finish+0x0/0x36 > >>>> > >>>> [<c03a43f3>] nf_hook_slow+0x3c/0xc3 > >>>> > >>>> [<c03aeb78>] ip_forward_finish+0x0/0x36 > >>>> > >>>> [<c03aed4c>] ip_forward+0x19e/0x22e > >>>> > >>>> [<c03aeb78>] ip_forward_finish+0x0/0x36 > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> Xen-devel mailing list > >>>> Xen-devel@lists.xensource.com > >>>> http://lists.xensource.com/xen-devel > >>>> > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Matt Ayres
2006-Apr-19 23:26 UTC
Re: [Xen-devel] Hard lock of server - saved history from serial consolewithin
Ian Pratt wrote:>> Yes, an iptable rule for each IP for bandwidth accounting. >> Also 4 rules in the raw table and 4 rules in the nat table >> for port redirection. I don''t see how the actual iptables >> rules could effect the kernel though. > > (!) iptables causes big changes to the way packets pass through the > kernel. > > In particular, it sounds like you may be using connection tracking for > NAT. There were lots of changes in this area for 2.6.16, one of which > broke bridging for large UDP and ICMP datagrams. > > Seeing what iptables modules you have loaded may be interesting. It > doesn''t look much like a xen issue, but I wouldn''t totally rule it out.Most all are compiled in. I do use connection tracking. I doubt it is a Xen problem anymore. I am upgrading all kernels to 2.6.16.9 to see if that fixes it. If not I''ll have to try 2.6.17-rcX (whatever is the latest). I tried applying it and got many rejects so I don''t know how advanced of a kernel hacker I''d have to be to get that working. Thanks, Matt _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christian Limpach
2006-Apr-22 01:29 UTC
Re: [Xen-devel] Hard lock of server - saved history from serial consolewithin
On 4/20/06, Matt Ayres <matta@tektonic.net> wrote:> > > Ian Pratt wrote: > >> Yes, an iptable rule for each IP for bandwidth accounting. > >> Also 4 rules in the raw table and 4 rules in the nat table > >> for port redirection. I don''t see how the actual iptables > >> rules could effect the kernel though. > > > > (!) iptables causes big changes to the way packets pass through the > > kernel. > > > > In particular, it sounds like you may be using connection tracking for > > NAT. There were lots of changes in this area for 2.6.16, one of which > > broke bridging for large UDP and ICMP datagrams. > > > > Seeing what iptables modules you have loaded may be interesting. It > > doesn''t look much like a xen issue, but I wouldn''t totally rule it out. > > Most all are compiled in. I do use connection tracking. I doubt it is a > Xen problem anymore. I am upgrading all kernels to 2.6.16.9 to see if > that fixes it. If not I''ll have to try 2.6.17-rcX (whatever is the > latest). I tried applying it and got many rejects so I don''t know how > advanced of a kernel hacker I''d have to be to get that working.You can get 2.6.17-rc2 from: http://xenbits.xensource.com/ext/linux-2.6.rc-xen.hg Re 2.6.16.9: If you drop the patch into patches/linux-2.6.16, you should check if it changes any of the files in the sparse tree and/or any of the original files which we copied (-xen suffix). You''d then want to apply the changes to the files in the sparse tree as well. christian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel