Hey, all -- Earlier this afternoon, we started experiencing slowness accessing DomU machines from Dom0, via the network. Network access to Dom0 is unaffected, and console access to DomU machines is fine. /var/log/messages on DomU shows this: Sep 22 14:43:15 devxen0 kernel: printk: 4 messages suppressed. Sep 22 14:43:15 devxen0 kernel: xen_net: Memory squeeze in netback driver. Sep 22 14:43:20 devxen0 kernel: printk: 4 messages suppressed. Sep 22 14:43:20 devxen0 kernel: xen_net: Memory squeeze in netback driver. Anyone have suggestions for how to relieve the squeeze? Machine/version info: devxen0:~ # uname -a Linux devxen0 2.6.16.13-4-xen #1 SMP Wed May 3 04:53:23 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux devxen0:~ # rpm -qa | grep xen-3 xen-3.0.2_09656-4 (I know that there isn''t much info here, but I don''t know what to provide, I''m afraid...) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, all I''m getting the following in my /var/log/messages file on Dom0: Oct 18 11:00:36 devxen0 kernel: printk: 4 messages suppressed. Oct 18 11:00:36 devxen0 kernel: xen_net: Memory squeeze in netback driver. Which would make sense to me if there were a memory crunch on the box, or in Dom0 alone...but Dom0 has about 12GB of memory availble: devxen0:~ # xm list Name ID Mem(MiB) VCPUs State Time(s) Build5-8-0-km 2 448 1 -b---- 6307.1 Build5-8-0-km-DealsFilter 10 448 1 -b---- 33.6 Build5-8-0-km-GDS 4 448 1 -b---- 372.6 Build5-8-2-zj 8 448 1 -b---- 1696.0 Build5-8-2-zj-gds 9 448 1 r----- 24591.1 Dale-GDS 7 448 1 -b---- 1759.5 Dale-Web 6 448 1 -b---- 2135.7 Domain-0 0 12849 4 r----- 5269.6 mail 1 128 1 -b---- 294.2 Any thoughts/suggestions on what I can do to convince the netback driver to be a little happier with its life? Xen version is 3.0.2_09656-4, running on SUSE Linux 10.1. Kernel is 2.6.16.13-4-xen, stock from the SUSE 10.1 install. Additionally, I''ve got "dom0_mem=1024M" specified on the kernel command line at boot time. (I''m not sure if that parameter actually does anything, but it doesn''t seem to be hurting anything, so...) Thanks! - Ian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Anyone have thoughts on this? I''ve googled around, and haven''t come up with anything that I haven''t tried already. And, it severely limits the effectiveness of the Xen environment, because I''m now running a machine that appears unable to use more than 25% of the available resources...> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Marlier, Ian > Sent: Wednesday, October 18, 2006 11:08 AM > To: xen-users@lists.xensource.com > Subject: [Xen-users] "Memory squeeze in netback driver" > > Hi, all > > I''m getting the following in my /var/log/messages file on Dom0: > Oct 18 11:00:36 devxen0 kernel: printk: 4 messages suppressed. > Oct 18 11:00:36 devxen0 kernel: xen_net: Memory squeeze in > netback driver. > > Which would make sense to me if there were a memory crunch on > the box, or in Dom0 alone...but Dom0 has about 12GB of memory > availble: > devxen0:~ # xm list > Name ID Mem(MiB) VCPUs State Time(s) > Build5-8-0-km 2 448 1 -b---- 6307.1 > Build5-8-0-km-DealsFilter 10 448 1 -b---- 33.6 > Build5-8-0-km-GDS 4 448 1 -b---- 372.6 > Build5-8-2-zj 8 448 1 -b---- 1696.0 > Build5-8-2-zj-gds 9 448 1 r----- 24591.1 > Dale-GDS 7 448 1 -b---- 1759.5 > Dale-Web 6 448 1 -b---- 2135.7 > Domain-0 0 12849 4 r----- 5269.6 > mail 1 128 1 -b---- 294.2 > > Any thoughts/suggestions on what I can do to convince the > netback driver to be a little happier with its life? > > Xen version is 3.0.2_09656-4, running on SUSE Linux 10.1. > Kernel is 2.6.16.13-4-xen, stock from the SUSE 10.1 install. > > Additionally, I''ve got "dom0_mem=1024M" specified on the > kernel command line at boot time. (I''m not sure if that > parameter actually does anything, but it doesn''t seem to be > hurting anything, so...) > > Thanks! > > - Ian > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Marlier, Ian > Sent: 19 October 2006 17:28 > To: Marlier, Ian; xen-users@lists.xensource.com > Subject: RE: [Xen-users] "Memory squeeze in netback driver" > > Anyone have thoughts on this? I''ve googled around, and > haven''t come up > with anything that I haven''t tried already. > > And, it severely limits the effectiveness of the Xen environment, > because I''m now running a machine that appears unable to use more than > 25% of the available resources...If you look at linux-<ver>-xen/drivers/xen/netback/netback.c around line 572 (this isn''t the latest source, but I don''t think it''s changed much in the last week or so), you''ll see where the error message comes from. I''m not sure what the code does tho'' - it''s got something to do with the hypercall XENMEM_increase_reservation, but I haven''t got time to look at what that does. dom0_mem shouldn''t be added to the vmlinuz line, but to the xen.gz line - which I think is the reason your dom0 is showing 12GB. Not sure if that''s got anything to do with the above problem tho''. -- Mats> > > > > -----Original Message----- > > From: xen-users-bounces@lists.xensource.com > > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > > Marlier, Ian > > Sent: Wednesday, October 18, 2006 11:08 AM > > To: xen-users@lists.xensource.com > > Subject: [Xen-users] "Memory squeeze in netback driver" > > > > Hi, all > > > > I''m getting the following in my /var/log/messages file on Dom0: > > Oct 18 11:00:36 devxen0 kernel: printk: 4 messages suppressed. > > Oct 18 11:00:36 devxen0 kernel: xen_net: Memory squeeze in > > netback driver. > > > > Which would make sense to me if there were a memory crunch on > > the box, or in Dom0 alone...but Dom0 has about 12GB of memory > > availble: > > devxen0:~ # xm list > > Name ID Mem(MiB) VCPUs State Time(s) > > Build5-8-0-km 2 448 1 -b---- 6307.1 > > Build5-8-0-km-DealsFilter 10 448 1 -b---- 33.6 > > Build5-8-0-km-GDS 4 448 1 -b---- 372.6 > > Build5-8-2-zj 8 448 1 -b---- 1696.0 > > Build5-8-2-zj-gds 9 448 1 r----- 24591.1 > > Dale-GDS 7 448 1 -b---- 1759.5 > > Dale-Web 6 448 1 -b---- 2135.7 > > Domain-0 0 12849 4 r----- 5269.6 > > mail 1 128 1 -b---- 294.2 > > > > Any thoughts/suggestions on what I can do to convince the > > netback driver to be a little happier with its life? > > > > Xen version is 3.0.2_09656-4, running on SUSE Linux 10.1. > > Kernel is 2.6.16.13-4-xen, stock from the SUSE 10.1 install. > > > > Additionally, I''ve got "dom0_mem=1024M" specified on the > > kernel command line at boot time. (I''m not sure if that > > parameter actually does anything, but it doesn''t seem to be > > hurting anything, so...) > > > > Thanks! > > > > - Ian > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: Petersson, Mats [mailto:Mats.Petersson@amd.com] > Sent: Thursday, October 19, 2006 1:09 PM > To: Marlier, Ian; xen-users@lists.xensource.com > Subject: RE: [Xen-users] "Memory squeeze in netback driver" > > > > > -----Original Message----- > > From: xen-users-bounces@lists.xensource.com > > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Marlier, > > Ian > > Sent: 19 October 2006 17:28 > > To: Marlier, Ian; xen-users@lists.xensource.com > > Subject: RE: [Xen-users] "Memory squeeze in netback driver" > > > > Anyone have thoughts on this? I''ve googled around, and > haven''t come > > up with anything that I haven''t tried already. > > > > And, it severely limits the effectiveness of the Xen environment, > > because I''m now running a machine that appears unable to > use more than > > 25% of the available resources... > > If you look at linux-<ver>-xen/drivers/xen/netback/netback.c > around line > 572 (this isn''t the latest source, but I don''t think it''s > changed much in the last week or so), you''ll see where the > error message comes from. > I''m not sure what the code does tho'' - it''s got something to > do with the hypercall XENMEM_increase_reservation, but I > haven''t got time to look at what that does. > > dom0_mem shouldn''t be added to the vmlinuz line, but to the > xen.gz line > - which I think is the reason your dom0 is showing 12GB. Not > sure if that''s got anything to do with the above problem tho''.Mats, perfect! Moving the dom0_mem parameter to the kernel line in my grub config was all that was needed. I feel a little silly for not having figured that one out myself, though... :-/ I''m not going to pretend that I understand what the code you pointed to is doing -- I know a bit of C, but it would take me a good long while to walk that back. I would bet, though, that it has to do with the chunk of memory allocated to Dom0 by the balloon driver when a new machine is created (for what, I dunno, but...). We started seeing problems when the total amount of memory allocated was at or just above 4GB, so maybe Dom0 just didn''t have DMA-able memory available to allocate itself? Or, at least, that makes sense to my growing-but-not-yet-grown understanding, and seems to fit with the symptoms and the solution. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
[...]> > dom0_mem shouldn''t be added to the vmlinuz line, but to the > > xen.gz line[...] Hi, what''s the name of the document where all those boot options are explained, and, BTW: Why is the max mem of Dom0 limited by boot option, while the min mem seens to be limited in the xend config file? Or did I get it wrong? Regards, Ulrich _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Harald Koenig
2006-Nov-21 15:29 UTC
[Xen-devel] Re: [Xen-users] "Memory squeeze in netback driver"
Hi, today my XEN server run into the same problem: many kernel messages in dom0 xen_net: Memory squeeze in netback driver. printk: 4 messages suppressed. and all 13 domU networks are dead. this is a dual-dual-core-operon (4 CPUs) running SUSE 10.1 in dom0 (xen 3.0.2_09763-0.8 from SUSE) with 8GB ram and Broadcom BCM5704 Gigabit Ethernet on Supermicro H8DA8. is there any chance to avoid rebooting dom0 ? is it possible to avoid this problem in the future ? On Oct 19, Petersson, Mats wrote:> dom0_mem shouldn''t be added to the vmlinuz line, but to the xen.gz line > - which I think is the reason your dom0 is showing 12GB. Not sure if > that''s got anything to do with the above problem tho''.I don''t think so because in my setup I''m passing dom0_mem to xen: kernel /boot/xen.gz dom0_mem=500M com1=38400,8n1 console=vga,com1 sync_console module /boot/vmlinuz-xen root=/dev/sda1 showopts console=ttyS0,38400 console=tty0 but checking the time stamps in syslog when those messages started continuesly (there are some few "Memory squeeze in netback" events from time to time) it looks like this might got triggered by the creation of a new domU which caused the dom0 being shrinked from 500 MB to ~290 MB (dom0-min-mem is 196). shortly after that, dom0 was reset to 500 MB. here is a short histogram on the number of "Memory squeeze in netback driver" messages per hour (with those "printk: XXX messages suppressed" added correctly) day HH count Nov 20 16 1 Nov 20 17 1 Nov 20 18 350 Nov 20 19 273 Nov 20 20 277 Nov 20 21 268 Nov 20 22 250 Nov 20 23 275 Nov 21 00 1195 Nov 21 01 3600 Nov 21 02 3600 Nov 21 03 3600 Nov 21 04 3595 Nov 21 05 3600 ... around 18 o''clock (6 pm;) the dom0 memory resize happend. looks like this was an initial trigger for the problem. the external connection got lost around 00:45 am this moring, which matches the increase of those netback messages from about every 15 secs to every second. do you need any more information ? on xen-devel I found only two mails about "Memory squeeze in netback driver" from Isaku Yamahata <yamahata@valinux.co.jp> with a patch which removes that kernel printk(): Attachment #2: 9897:5fcebfab4e31_netback_autotranslate_mode.patch # HG changeset patch # User yamahata@valinux.co.jp # Node ID 5fcebfab4e317f1db0b0379ac4be99d8cdb9b2b2 # Parent a19cc748469e99eae7f289f3beb03206982b9ba2 change semantics of grant transfers for vp guests so that the operation automatically gets you a fresh page at the same pseudo-physical address as Keir suggested. PATCHNAME: netback_autotranslate_mode any ideas or comments ? thanks, Harald Koenig -- "I hope to die ___ _____ before I *have* to use Microsoft Word.", 0--,| /OOOOOOO\ Donald E. Knuth, 02-Oct-2001 in Tuebingen. <_/ / /OOOOOOOOOOO\ \ \/OOOOOOOOOOOOOOO\ \ OOOOOOOOOOOOOOOOO|// Harald Koenig \/\/\/\/\/\/\/\/\/ science+computing ag // / \\ \ koenig@science-computing.de ^^^^^ ^^^^^ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel