Steven Timm
2009-May-01 19:06 UTC
[Xen-users] Attempt to allocate order 5 skbuff. Increase MAX_SKBUFF_ORDER
Running redhat-clone xen 3.0.3 (really patched 3.1.2) with kernel-xen-2.6.18-128.1.6.el5xen both on dom0 (64-bit) and domU (32 bit). The domU in question is a Squid server. Actually I have 2 such domU on different physical hardware, same setup, and they are both having the same trouble. In the /var/log/messages there is 2009-05-01T11:00:23-05:00 s_sys@fg3x3.fnal.gov kernel: Attempt to allocate order 5 skbuff. Increase MAX_SKBUFF_ORDER. This error is happening continuously, hundreds of times per minute. Under the previous kernel I was running, which was the 2.6.18-xen kernel from the xen.org xen 3.1.0 tarball, I had other problems-- squid: page allocation fa ilure. order:5, mode:0x20 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c015b2b6>] __alloc_page s+0x216/0x300 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c01772cd>] kmem_getpage s+0x3d/0xe0 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c017824c>] cache_grow+0 xdc/0x200 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c017851c>] cache_alloc_ refill+0x1ac/0x200 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c01789fd>] __kmalloc+0x ad/0xc0 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c029d4e0>] __alloc_skb+ 0x50/0x110 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c029dd5a>] skb_copy+0x2 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c02be1cc>] skb_make_wri table+0x3c/0xd0 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<ee52e718>] manip_pkt+0x 28/0x100 [ip_nat] 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<ee52e867>] ip_nat_packe t+0x77/0xa0 [ip_nat] 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<ee526550>] ip_nat_fn+0x 90/0x230 [iptable_nat] 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<ee5202bb>] ipt_do_table +0x28b/0x350 [ip_tables] 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<ee5267fb>] ip_nat_out+0 x5b/0xe0 [iptable_nat] 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c02c7470>] ip_finish_ou tput+0x0/0x1f0 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c02be055>] nf_iterate+0 x55/0x90 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c02c7470>] ip_finish_ou tput+0x0/0x1f0 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c02c7470>] ip_finish_ou tput+0x0/0x1f0 2009-04-28T05:17:53-05:00 s_sys@fg3x3.fnal.gov kernel: [<c02be0f6>] nf_hook_slow +0x66/0x100 and then a dump of the kernel memory info. xm top shows this domu using up to 100% of cpu. Any idea what is wrong? Some googling found a bugzilla ticket about this in Fedora 6 but it was closed with instructions to reopen if it happens again in Fedora 7. Steve Timm ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steven Timm
2009-May-06 01:33 UTC
RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase MAX_SKBUFF_ORDER
>> very few ports from accessing our site. >> >> I am not sure why the iptables_nat module would be loaded >> because we are not using NAT in our network configuration, at least we >> are >> not intending to do so. > > Well if you don''t need it then just try and remove the NAT module using "modprobe -r iptable_nat". And see if that makes any difference. >Can''t remove it, get the message "module is in use".. not sure by what.> > >>> Now the issue about MAX_SKBUFF_ORDER should only show up when you are >> trying to send large packets. Do you use jumbo frames or something like >> that? What MTU sizes are set for the interfaces? As far as I know the >> message you get means that Xen is trying to allocate a buffer for the >> packet to send, but the packet size is too big for the buffer >> allocator. >>> >> [root@fg3x3 ~]# ifconfig >> eth0 Link encap:Ethernet HWaddr 00:16:3E:05:03:03 >> inet addr:131.225.107.144 Bcast:131.225.107.255 >> Mask:255.255.255.0 >> inet6 addr: fe80::216:3eff:fe05:303/64 Scope:Link >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:2971214615 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:1576876803 errors:0 dropped:0 overruns:0 >> carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:2428856680 (2.2 GiB) TX bytes:4068069258 (3.7 GiB) >> >> No--no jumbo frames anywhere. MTU size is the standard 1500. > > This is on all Dom0/DomU frontend and backend interfaces?That''s right.> > > > >>> In general, you can configure the Xen kernel to use a Xen-specific >> buffer allocator, or the kernel''s default buffer allocator. There is a >> kernel configuration option for that and it is called >> CONFIG_XEN_SKBUFF. You could try and switch that off, and recompile the >> kernel. >>> >> So CONFIG_XEN_SKBUFF is by default on? >CONFIG_XEN_SKBUFF is on in my config. There is no MAX_SKBUFF_ORDER parameter anywhere in my source tree, much less the config file. Steve Timm> I don''t know for sure. It depends on your distro and whatever kernel you use. You can check this in your kernel config files or eventually by looking at /proc/config.gz. It depends on the Xen version etc, I don''t know that by heart. > > Also, the value of MAX_SKBUFF_ORDER depends on your kernel configuration and memory options. > > >>> Looking at your megasas error message, I suspect that this is simply >> because your system is running under high load and it is probably >> running out of memory in various places. Maybe that is also the root >> cause for the skbuff problem. What does Xen tell you about the memory >> utilization, and how do you allocate your memory to DomU/Dom0? >>> >> This is a dell poweredge 2950 with 16 GB of RAM overall. 2GB for dom0, >> [root@fermigrid3 ~]# free >> total used free shared buffers >> cached >> Mem: 2156544 1218036 938508 0 242084 >> 434092 >> -/+ buffers/cache: 541860 1614684 >> Swap: 24579440 0 24579440 >> >> 4GB for this domU. >> >> [root@fg3x3 ~]# free >> total used free shared buffers >> cached >> Mem: 4096176 4073044 23132 0 33024 >> 3130072 >> -/+ buffers/cache: 909948 3186228 >> Swap: 8191992 48 8191944 >> >> 4 other domU''s on this system but they are mostly idle. > > Can you post an output of /proc/slabinfo of your DomU/Dom0 under high load? > > >Kernel config and slabinfo are posted. -- ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fischer, Anna
2009-May-06 05:46 UTC
RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase MAX_SKBUFF_ORDER
> Subject: RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase > MAX_SKBUFF_ORDER > > >> very few ports from accessing our site. > >> > >> I am not sure why the iptables_nat module would be loaded > >> because we are not using NAT in our network configuration, at least > we > >> are > >> not intending to do so. > > > > Well if you don''t need it then just try and remove the NAT module > using "modprobe -r iptable_nat". And see if that makes any difference. > > > > Can''t remove it, get the message > "module is in use".. not sure by what.Do you have any rules in the NAT table? E.g. check "iptables -t nat -L". Then remove those rules and try removing the module again. I doubt that the NAT module is the core of your problem though.> >>> Now the issue about MAX_SKBUFF_ORDER should only show up when you > are > >> trying to send large packets. Do you use jumbo frames or something > like > >> that? What MTU sizes are set for the interfaces? As far as I know > the > >> message you get means that Xen is trying to allocate a buffer for > the > >> packet to send, but the packet size is too big for the buffer > >> allocator. > >>> > >> [root@fg3x3 ~]# ifconfig > >> eth0 Link encap:Ethernet HWaddr 00:16:3E:05:03:03 > >> inet addr:131.225.107.144 Bcast:131.225.107.255 > >> Mask:255.255.255.0 > >> inet6 addr: fe80::216:3eff:fe05:303/64 Scope:Link > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> RX packets:2971214615 errors:0 dropped:0 overruns:0 > frame:0 > >> TX packets:1576876803 errors:0 dropped:0 overruns:0 > >> carrier:0 > >> collisions:0 txqueuelen:1000 > >> RX bytes:2428856680 (2.2 GiB) TX bytes:4068069258 (3.7 > GiB) > >> > >> No--no jumbo frames anywhere. MTU size is the standard 1500. > > > > This is on all Dom0/DomU frontend and backend interfaces? > > That''s right. > > > > > > > > > > >>> In general, you can configure the Xen kernel to use a Xen-specific > >> buffer allocator, or the kernel''s default buffer allocator. There is > a > >> kernel configuration option for that and it is called > >> CONFIG_XEN_SKBUFF. You could try and switch that off, and recompile > the > >> kernel. > >>> > >> So CONFIG_XEN_SKBUFF is by default on? > > > CONFIG_XEN_SKBUFF is on in my config. > There is no MAX_SKBUFF_ORDER parameter anywhere in my source tree, > much less the config file.MAX_SKUFF_ORDER is not a configuration option. It is part of the Dom0/DomU kernel code. Your posted kernel config is from your Dom0? You said before that you are running a 64-bit Dom0. You need to check the CONFIG_XEN_SKBUFF option in the Dom0 config. I am in general wondering if you might have issues with your DomU/Dom0 configuration. How did you install those kernels? Did you install them using the distro? Did you compile them yourself? I assume you also run a 64-bit hypervisor? If it is easy for you to recompile DomU/Dom0 kernels then you could try and recompile with CONFIG_XEN_SKBUFF, CONFIG_HAVE_ARCH_ALLOC_SKB and CONFIG_HAVE_ARCH_DEV_ALLOC_SKB disabled, and see if it makes any difference. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Robert Dunkley
2009-May-06 07:03 UTC
RE: [Xen-users] Attempt to allocate order 5 skbuff. IncreaseMAX_SKBUFF_ORDER
I get this same error on bootup only, I''m using a Centos 5.2 Dom0 and the Xen 3.31 Centos RPMs. I suspect it is related to Infiniband/IPOIB (OFED 1.3.1) and the 32Kb MTU I use but I never found a solution. Please let me know if you find any kind of solution. Thanks, Rob -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Fischer, Anna Sent: 06 May 2009 06:47 To: Steven Timm Cc: xen-users@lists.xensource.com Subject: RE: [Xen-users] Attempt to allocate order 5 skbuff. IncreaseMAX_SKBUFF_ORDER> Subject: RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase > MAX_SKBUFF_ORDER > > >> very few ports from accessing our site. > >> > >> I am not sure why the iptables_nat module would be loaded > >> because we are not using NAT in our network configuration, at least > we > >> are > >> not intending to do so. > > > > Well if you don''t need it then just try and remove the NAT module > using "modprobe -r iptable_nat". And see if that makes any difference. > > > > Can''t remove it, get the message > "module is in use".. not sure by what.Do you have any rules in the NAT table? E.g. check "iptables -t nat -L". Then remove those rules and try removing the module again. I doubt that the NAT module is the core of your problem though.> >>> Now the issue about MAX_SKBUFF_ORDER should only show up when you > are > >> trying to send large packets. Do you use jumbo frames or something > like > >> that? What MTU sizes are set for the interfaces? As far as I know > the > >> message you get means that Xen is trying to allocate a buffer for > the > >> packet to send, but the packet size is too big for the buffer > >> allocator. > >>> > >> [root@fg3x3 ~]# ifconfig > >> eth0 Link encap:Ethernet HWaddr 00:16:3E:05:03:03 > >> inet addr:131.225.107.144 Bcast:131.225.107.255 > >> Mask:255.255.255.0 > >> inet6 addr: fe80::216:3eff:fe05:303/64 Scope:Link > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> RX packets:2971214615 errors:0 dropped:0 overruns:0 > frame:0 > >> TX packets:1576876803 errors:0 dropped:0 overruns:0 > >> carrier:0 > >> collisions:0 txqueuelen:1000 > >> RX bytes:2428856680 (2.2 GiB) TX bytes:4068069258 (3.7 > GiB) > >> > >> No--no jumbo frames anywhere. MTU size is the standard 1500. > > > > This is on all Dom0/DomU frontend and backend interfaces? > > That''s right. > > > > > > > > > > >>> In general, you can configure the Xen kernel to use a Xen-specific > >> buffer allocator, or the kernel''s default buffer allocator. Thereis> a > >> kernel configuration option for that and it is called > >> CONFIG_XEN_SKBUFF. You could try and switch that off, and recompile > the > >> kernel. > >>> > >> So CONFIG_XEN_SKBUFF is by default on? > > > CONFIG_XEN_SKBUFF is on in my config. > There is no MAX_SKBUFF_ORDER parameter anywhere in my source tree, > much less the config file.MAX_SKUFF_ORDER is not a configuration option. It is part of the Dom0/DomU kernel code. Your posted kernel config is from your Dom0? You said before that you are running a 64-bit Dom0. You need to check the CONFIG_XEN_SKBUFF option in the Dom0 config. I am in general wondering if you might have issues with your DomU/Dom0 configuration. How did you install those kernels? Did you install them using the distro? Did you compile them yourself? I assume you also run a 64-bit hypervisor? If it is easy for you to recompile DomU/Dom0 kernels then you could try and recompile with CONFIG_XEN_SKBUFF, CONFIG_HAVE_ARCH_ALLOC_SKB and CONFIG_HAVE_ARCH_DEV_ALLOC_SKB disabled, and see if it makes any difference. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users The SAQ Group Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ SAQ is the trading name of SEMTEC Limited. Registered in England & Wales Company Number: 06481952 http://www.saqnet.co.uk AS29219 SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. ISPA Member Find us in http://www.thebestof.co.uk/petersfield _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fischer, Anna
2009-May-06 07:28 UTC
RE: [Xen-users] Attempt to allocate order 5 skbuff. IncreaseMAX_SKBUFF_ORDER
I don''t know the detailed implementation but as I said to Steven I believe this issue should only show up when you are trying to allocate too large buffers. I would recommend to post this to xen-devel.> -----Original Message----- > From: Robert Dunkley [mailto:Robert@saq.co.uk] > Sent: 06 May 2009 00:04 > To: Fischer, Anna > Cc: xen-users@lists.xensource.com > Subject: RE: [Xen-users] Attempt to allocate order 5 skbuff. > IncreaseMAX_SKBUFF_ORDER > > I get this same error on bootup only, I''m using a Centos 5.2 Dom0 and > the Xen 3.31 Centos RPMs. I suspect it is related to Infiniband/IPOIB > (OFED 1.3.1) and the 32Kb MTU I use but I never found a solution. > Please > let me know if you find any kind of solution. > > Thanks, > > Rob > -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Fischer, > Anna > Sent: 06 May 2009 06:47 > To: Steven Timm > Cc: xen-users@lists.xensource.com > Subject: RE: [Xen-users] Attempt to allocate order 5 skbuff. > IncreaseMAX_SKBUFF_ORDER > > > Subject: RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase > > MAX_SKBUFF_ORDER > > > > >> very few ports from accessing our site. > > >> > > >> I am not sure why the iptables_nat module would be loaded > > >> because we are not using NAT in our network configuration, at > least > > we > > >> are > > >> not intending to do so. > > > > > > Well if you don''t need it then just try and remove the NAT module > > using "modprobe -r iptable_nat". And see if that makes any > difference. > > > > > > > Can''t remove it, get the message > > "module is in use".. not sure by what. > > Do you have any rules in the NAT table? E.g. check "iptables -t nat - > L". > Then remove those rules and try removing the module again. I doubt that > the NAT module is the core of your problem though. > > > > >>> Now the issue about MAX_SKBUFF_ORDER should only show up when you > > are > > >> trying to send large packets. Do you use jumbo frames or something > > like > > >> that? What MTU sizes are set for the interfaces? As far as I know > > the > > >> message you get means that Xen is trying to allocate a buffer for > > the > > >> packet to send, but the packet size is too big for the buffer > > >> allocator. > > >>> > > >> [root@fg3x3 ~]# ifconfig > > >> eth0 Link encap:Ethernet HWaddr 00:16:3E:05:03:03 > > >> inet addr:131.225.107.144 Bcast:131.225.107.255 > > >> Mask:255.255.255.0 > > >> inet6 addr: fe80::216:3eff:fe05:303/64 Scope:Link > > >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > >> RX packets:2971214615 errors:0 dropped:0 overruns:0 > > frame:0 > > >> TX packets:1576876803 errors:0 dropped:0 overruns:0 > > >> carrier:0 > > >> collisions:0 txqueuelen:1000 > > >> RX bytes:2428856680 (2.2 GiB) TX bytes:4068069258 (3.7 > > GiB) > > >> > > >> No--no jumbo frames anywhere. MTU size is the standard 1500. > > > > > > This is on all Dom0/DomU frontend and backend interfaces? > > > > That''s right. > > > > > > > > > > > > > > > > >>> In general, you can configure the Xen kernel to use a Xen- > specific > > >> buffer allocator, or the kernel''s default buffer allocator. There > is > > a > > >> kernel configuration option for that and it is called > > >> CONFIG_XEN_SKBUFF. You could try and switch that off, and > recompile > > the > > >> kernel. > > >>> > > >> So CONFIG_XEN_SKBUFF is by default on? > > > > > CONFIG_XEN_SKBUFF is on in my config. > > There is no MAX_SKBUFF_ORDER parameter anywhere in my source tree, > > much less the config file. > > MAX_SKUFF_ORDER is not a configuration option. It is part of the > Dom0/DomU kernel code. > > Your posted kernel config is from your Dom0? You said before that you > are running a 64-bit Dom0. You need to check the CONFIG_XEN_SKBUFF > option in the Dom0 config. I am in general wondering if you might have > issues with your DomU/Dom0 configuration. How did you install those > kernels? Did you install them using the distro? Did you compile them > yourself? I assume you also run a 64-bit hypervisor? > > If it is easy for you to recompile DomU/Dom0 kernels then you could try > and recompile with CONFIG_XEN_SKBUFF, CONFIG_HAVE_ARCH_ALLOC_SKB and > CONFIG_HAVE_ARCH_DEV_ALLOC_SKB disabled, and see if it makes any > difference. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > The SAQ Group > > Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ > SAQ is the trading name of SEMTEC Limited. Registered in England & > Wales > Company Number: 06481952 > > http://www.saqnet.co.uk AS29219 > > SAQ Group Delivers high quality, honestly priced communication and I.T. > services to UK Business. > > Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : > Transit : Backups : Managed Networks : Remote Support. > > ISPA Member > > Find us in http://www.thebestof.co.uk/petersfield_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steven Timm
2009-May-06 14:18 UTC
RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase MAX_SKBUFF_ORDER
>>> Well if you don''t need it then just try and remove the NAT module >> using "modprobe -r iptable_nat". And see if that makes any difference. >>> >> >> Can''t remove it, get the message >> "module is in use".. not sure by what. > > Do you have any rules in the NAT table? E.g. check "iptables -t nat -L". Then remove those rules and try removing the module again. I doubt that the NAT module is the core of your problem though.Yes it turns out we did. This server is an LVS "real server" back end and Horm''s transparent proxy is in use. that''s what the NAT is being used for.>>> >> CONFIG_XEN_SKBUFF is on in my config. >> There is no MAX_SKBUFF_ORDER parameter anywhere in my source tree, >> much less the config file. > > MAX_SKBUFF_ORDER is not a configuration option. It is part of the Dom0/DomU kernel code.I have grepped the whole kernel code from the kernel which I''m running, which is unmodified from the redhat SRPMS as delivered with version 5, update 3. 2.6.18-128.1.6.el5xen. I don''t find the string MAX_SKBUFF_ORDER anywhere in there. Which source file should it be in?> > Your posted kernel config is from your Dom0? You said before that you are running a 64-bit Dom0. You need to check the CONFIG_XEN_SKBUFF option in the Dom0 config. I am in general wondering if you might have issues with your DomU/Dom0 configuration. How did you install those kernels? Did you install them using the distro? Did you compile them yourself? I assume you also run a 64-bit hypervisor?Yes running 64-bit dom0 and hypervisor. the kernel config is actually the same for both dom0 and domU as in redhattish xen you actually run the same kernel-xen in both. Only difference is that it is 64-bit for the dom0 and 32-bit for the domU.> > If it is easy for you to recompile DomU/Dom0 kernels then you could try and recompile with CONFIG_XEN_SKBUFF, CONFIG_HAVE_ARCH_ALLOC_SKB and CONFIG_HAVE_ARCH_DEV_ALLOC_SKB disabled, and see if it makes any difference. >We think we see how to get rid of the NAT so we will try to do that first, and then will recompile the kernel as mentioned above if necessary. Government sites like ours don''t like running custom kernels.. in fact we just completed a process to get back in sync with the distro after running a custom xen kernel for 1.5 years, which is how we noticed this problem, although it was going on with our custom kernel too and we just never noticed it at the time. Steve Timm -- ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fischer, Anna
2009-May-06 14:57 UTC
RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase MAX_SKBUFF_ORDER
> Subject: RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase > MAX_SKBUFF_ORDER > > >>> Well if you don''t need it then just try and remove the NAT module > >> using "modprobe -r iptable_nat". And see if that makes any > difference. > >>> > >> > >> Can''t remove it, get the message > >> "module is in use".. not sure by what. > > > > Do you have any rules in the NAT table? E.g. check "iptables -t nat - > L". Then remove those rules and try removing the module again. I doubt > that the NAT module is the core of your problem though. > > Yes it turns out we did. This server is an LVS "real server" back > end and Horm''s transparent proxy is in use. that''s > what the NAT is being used for. > > >>> > >> CONFIG_XEN_SKBUFF is on in my config. > >> There is no MAX_SKBUFF_ORDER parameter anywhere in my source tree, > >> much less the config file. > > > > MAX_SKBUFF_ORDER is not a configuration option. It is part of the > Dom0/DomU kernel code. > > I have grepped the whole kernel code from the kernel which I''m > running, which is unmodified from the redhat SRPMS as delivered with > version 5, update 3. 2.6.18-128.1.6.el5xen. I don''t find > the string MAX_SKBUFF_ORDER anywhere in there. Which source file > should > it be in? > > > > > > Your posted kernel config is from your Dom0? You said before that you > are running a 64-bit Dom0. You need to check the CONFIG_XEN_SKBUFF > option in the Dom0 config. I am in general wondering if you might have > issues with your DomU/Dom0 configuration. How did you install those > kernels? Did you install them using the distro? Did you compile them > yourself? I assume you also run a 64-bit hypervisor? > > Yes running 64-bit dom0 and hypervisor. the kernel config > is actually the same for both dom0 and domU as in redhattish xen > you actually run the same kernel-xen in both. Only difference is > that it is 64-bit for the dom0 and 32-bit for the domU.How can you have the same kernel configuration for a 32-bit and a 64-bit system ? Have you edited those yourself and then compiled the kernels ? There are a lot of dependencies around the CONFIG_X86_64, CONFIG_X86_32 etc defines. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steven Timm
2009-May-06 15:01 UTC
RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase MAX_SKBUFF_ORDER
>>> Your posted kernel config is from your Dom0? You said before that you >> are running a 64-bit Dom0. You need to check the CONFIG_XEN_SKBUFF >> option in the Dom0 config. I am in general wondering if you might have >> issues with your DomU/Dom0 configuration. How did you install those >> kernels? Did you install them using the distro? Did you compile them >> yourself? I assume you also run a 64-bit hypervisor? >> >> Yes running 64-bit dom0 and hypervisor. the kernel config >> is actually the same for both dom0 and domU as in redhattish xen >> you actually run the same kernel-xen in both. Only difference is >> that it is 64-bit for the dom0 and 32-bit for the domU. > > How can you have the same kernel configuration for a 32-bit and a 64-bit system ? Have you edited those yourself and then compiled the kernels ? There are a lot of dependencies around the CONFIG_X86_64, CONFIG_X86_32 etc defines. >All right, it is not exactly the same. The one I posted was the domU config. Now attached is the dom0 config, from the 64-bit dom0. CONFIG_XEN_SKBUFF is "Y" in the dom0 too. Steve timm -- ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steven Timm
2009-May-06 16:32 UTC
RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase MAX_SKBUFF_ORDER
Both of these domU''s are back up now in a different configuration which does not use NAT and does not have the iptable_nat module loaded, and things are looking much better. So far we have not seen the MAX_SKBUFF_ORDER problem at all. We will continue to monitor. Thanks for your help. Steve Timm On Wed, 6 May 2009, Fischer, Anna wrote:>> Subject: RE: [Xen-users] Attempt to allocate order 5 skbuff. Increase >> MAX_SKBUFF_ORDER >> >>>>> Well if you don''t need it then just try and remove the NAT module >>>> using "modprobe -r iptable_nat". And see if that makes any >> difference. >>>>> >>>> >>>> Can''t remove it, get the message >>>> "module is in use".. not sure by what. >>> >>> Do you have any rules in the NAT table? E.g. check "iptables -t nat - >> L". Then remove those rules and try removing the module again. I doubt >> that the NAT module is the core of your problem though. >> >> Yes it turns out we did. This server is an LVS "real server" back >> end and Horm''s transparent proxy is in use. that''s >> what the NAT is being used for. >> >>>>> >>>> CONFIG_XEN_SKBUFF is on in my config. >>>> There is no MAX_SKBUFF_ORDER parameter anywhere in my source tree, >>>> much less the config file. >>> >>> MAX_SKBUFF_ORDER is not a configuration option. It is part of the >> Dom0/DomU kernel code. >> >> I have grepped the whole kernel code from the kernel which I''m >> running, which is unmodified from the redhat SRPMS as delivered with >> version 5, update 3. 2.6.18-128.1.6.el5xen. I don''t find >> the string MAX_SKBUFF_ORDER anywhere in there. Which source file >> should >> it be in? >> >> >>> >>> Your posted kernel config is from your Dom0? You said before that you >> are running a 64-bit Dom0. You need to check the CONFIG_XEN_SKBUFF >> option in the Dom0 config. I am in general wondering if you might have >> issues with your DomU/Dom0 configuration. How did you install those >> kernels? Did you install them using the distro? Did you compile them >> yourself? I assume you also run a 64-bit hypervisor? >> >> Yes running 64-bit dom0 and hypervisor. the kernel config >> is actually the same for both dom0 and domU as in redhattish xen >> you actually run the same kernel-xen in both. Only difference is >> that it is 64-bit for the dom0 and 32-bit for the domU. > > How can you have the same kernel configuration for a 32-bit and a 64-bit system ? Have you edited those yourself and then compiled the kernels ? There are a lot of dependencies around the CONFIG_X86_64, CONFIG_X86_32 etc defines. >-- ------------------------------------------------------------------ Steven C. Timm, Ph.D (630) 840-8525 timm@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users