Erik Hensema
2006-Jun-06 18:47 UTC
[Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
Hi everybody,
When I start more than about 3 domU''s the following messages are
logged in dom0:
Jun 6 18:30:03 thebe kernel: printk: 18 messages suppressed.
Jun 6 18:30:03 thebe kernel: xen_net: Memory squeeze in netback
driver.
Jun 6 18:30:03 thebe last message repeated 9 times
This progressively gets worse when I start more and more domU''s, up
until the point some or all domU''s lose their network connectivety.
When I google on the error message, a few pages turn up, but no real
sollutions.
I''ll try to describe my setup in detail:
the hardware:
AMD Opteron 144
4 GB ram
Dual Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet
controllers
It''s Aplus hardware from supermicro.
the software:
Suse Linux 10.1, default kernel: 2.6.16.13-4-xen
Xen version 3.0.2_09656-4 (abuild@suse.de) (gcc version 4.1.0 (SUSE
Linux)) Tue May 2 01:39:27 UTC 2006
I''ve got two bridges: xenbr0 and xenbr1:
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no vif0.0
peth0
vif1.0
(all vifX.0)
xenbr1 8000.feffffffffff no vif0.1
peth1
vif1.1
(all vifX.1)
So basically I''ve got two seperate lans, available to all domains. No
NAT. One lan is an internal vlan, the other lan is the public
internet.
Each domain runs an iptables firewall, including domain0. The firewall
is your basic run-of-the-mill iptables firewall, rules are matching
eth0 and eth1, no special modules are used.
domain0''s got plenty of free ram when the memory squeeze happens:
thebe:~ # free -m
total used free shared buffers cached
Mem: 1506 646 859 0 290 101
-/+ buffers/cache: 255 1250
Swap: 4102 0 4102
--
Erik Hensema (erik@hensema.net)
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Anthony Liguori
2006-Jun-06 21:43 UTC
[Xen-devel] Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
This is a rather curious path in the code involving rate limiting that
I''m not all that familiar with. Does it help is you create the domains
with a large maxmem than memory? For instance:
maxmem = 512
memory = 256
Perhaps someone with more experience in the rate limiting code can
explain why this message is only printed when it''s enabled? To save
some work, here''s the relevant bits from netback.c:net_rx_action():
if (!xen_feature(XENFEAT_auto_translated_physmap)) {
/* Memory squeeze? Back off for an arbitrary while. */
if ((new_mfn = alloc_mfn()) == 0) {
if ( net_ratelimit() )
WPRINTK("Memory squeeze in netback "
"driver.\n");
mod_timer(&net_timer, jiffies + HZ);
skb_queue_head(&rx_queue, skb);
break;
}
Does the rate limiting code cause increase_reservation to fail if
you''ve
exceeded your limit?
Regards,
Anthony Liguori
Erik Hensema wrote:> Hi everybody,
>
> When I start more than about 3 domU''s the following messages are
> logged in dom0:
>
> Jun 6 18:30:03 thebe kernel: printk: 18 messages suppressed.
> Jun 6 18:30:03 thebe kernel: xen_net: Memory squeeze in netback
> driver.
> Jun 6 18:30:03 thebe last message repeated 9 times
>
> This progressively gets worse when I start more and more domU''s,
up
> until the point some or all domU''s lose their network
connectivety.
>
> When I google on the error message, a few pages turn up, but no real
> sollutions.
>
> I''ll try to describe my setup in detail:
>
> the hardware:
>
> AMD Opteron 144
> 4 GB ram
> Dual Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet
> controllers
>
> It''s Aplus hardware from supermicro.
>
> the software:
>
> Suse Linux 10.1, default kernel: 2.6.16.13-4-xen
> Xen version 3.0.2_09656-4 (abuild@suse.de) (gcc version 4.1.0 (SUSE
> Linux)) Tue May 2 01:39:27 UTC 2006
>
> I''ve got two bridges: xenbr0 and xenbr1:
> bridge name bridge id STP enabled interfaces
> xenbr0 8000.feffffffffff no vif0.0
> peth0
> vif1.0
> (all vifX.0)
> xenbr1 8000.feffffffffff no vif0.1
> peth1
> vif1.1
> (all vifX.1)
>
> So basically I''ve got two seperate lans, available to all domains.
No
> NAT. One lan is an internal vlan, the other lan is the public
> internet.
> Each domain runs an iptables firewall, including domain0. The firewall
> is your basic run-of-the-mill iptables firewall, rules are matching
> eth0 and eth1, no special modules are used.
>
> domain0''s got plenty of free ram when the memory squeeze happens:
> thebe:~ # free -m
> total used free shared buffers cached
> Mem: 1506 646 859 0 290 101
> -/+ buffers/cache: 255 1250
> Swap: 4102 0 4102
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Erik Hensema
2006-Jun-07 15:52 UTC
[Xen-devel] Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
On Tuesday 06 June 2006 23:43, Anthony Liguori wrote:> This is a rather curious path in the code involving rate limiting > that I''m not all that familiar with. Does it help is you create > the domains with a large maxmem than memory? For instance: > > maxmem = 512 > memory = 256Yes, it does seem to help. I''ve recreated almost all domains with 16 MB ''headroom'', now I can create domains up until the point I run out of loop devices ;-) (6 domU''s running ATM) I''ll reboot tonight (or shutdown all domains and reload the loop module) and try to start more domains. -- Erik Hensema (erik@hensema.net) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori
2006-Jun-07 20:43 UTC
[Xen-devel] Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
Erik Hensema wrote:> On Tuesday 06 June 2006 23:43, Anthony Liguori wrote: > >> This is a rather curious path in the code involving rate limiting >> that I''m not all that familiar with. Does it help is you create >> the domains with a large maxmem than memory? For instance: >> >> maxmem = 512 >> memory = 256 >> > > Yes, it does seem to help. I''ve recreated almost all domains with 16 > MB ''headroom'', now I can create domains up until the point I run out > of loop devices ;-) (6 domU''s running ATM) > I''ll reboot tonight (or shutdown all domains and reload the loop > module) and try to start more domains. >I suspected it would but I was hoping it wouldn''t. Do this mean that the netback driver needs to be able to increase it''s reservation? If so, should we be enforcing (in the tools) that memory < maxmem? Any thoughts Keir? Regards, Anthony Liguori _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Jun-08 07:30 UTC
[Xen-devel] Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
On 7 Jun 2006, at 21:43, Anthony Liguori wrote:> I suspected it would but I was hoping it wouldn''t. Do this mean that > the netback driver needs to be able to increase it''s reservation? If > so, should we be enforcing (in the tools) that memory < maxmem? Any > thoughts Keir?The kernel should be able to do it for itself. Drivers inform how much ''headroom'' they need via balloon_update_driver_allowance(). Balloon driver then strives to keep its memory allocation below maxmem by that amount. Possibly the balloon driver needs a kick when allocation fails in teh driver, to expand the balloon a little. Another alternative is for drivers to incr/decr reservation via another balloon driver interface, giving another hook point for the balloon driver to do something sensible. So, in short, tools may be able to screw things for a short while (and maybe crash things entirely, if maxmem is reduced drastically with no prior warning (update of memory target)). But the balloon driver is expected to recover except in that kind of unreasonable circumstance. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kanwar Ranbir Sandhu
2006-Jun-14 03:51 UTC
Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
On Tue, 2006-06-06 at 20:47 +0200, Erik Hensema wrote:> When I start more than about 3 domU''s the following messages are > logged in dom0: > > Jun 6 18:30:03 thebe kernel: printk: 18 messages suppressed. > Jun 6 18:30:03 thebe kernel: xen_net: Memory squeeze in netback > driver. > Jun 6 18:30:03 thebe last message repeated 9 timesI''m seeing this error as soon I as start a second domU. Like you, I''ve found no useful information, at all. I took a peek at the source code, which was a mistake: now I''m blind. It''s a head scratcher, all right. I have no idea what it means, so I''m reduced to tinkering with the config files, including lowering memory usage for my one working domU. Haven''t tried it yet, though. Anybody else run into this? Tips? Regards, Ranbir -- Kanwar Ranbir Sandhu Linux 2.6.16-1.2111_FC4 i686 GNU/Linux 23:47:43 up 17:17, 4 users, load average: 2.33, 0.78, 0.39 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Paddy Vishnubhatt
2006-Jun-14 03:58 UTC
RE: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
I found this link and found it to be useful, hope this helps you: http://lists.xensource.com/archives/html/xen-users/2006-02/msg00077.html -- -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Kanwar Ranbir Sandhu Sent: Tuesday, June 13, 2006 8:51 PM To: xen-users@lists.xensource.com Subject: Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver. On Tue, 2006-06-06 at 20:47 +0200, Erik Hensema wrote:> When I start more than about 3 domU''s the following messages are > logged in dom0: > > Jun 6 18:30:03 thebe kernel: printk: 18 messages suppressed. > Jun 6 18:30:03 thebe kernel: xen_net: Memory squeeze in netback > driver. > Jun 6 18:30:03 thebe last message repeated 9 timesI''m seeing this error as soon I as start a second domU. Like you, I''ve found no useful information, at all. I took a peek at the source code, which was a mistake: now I''m blind. It''s a head scratcher, all right. I have no idea what it means, so I''m reduced to tinkering with the config files, including lowering memory usage for my one working domU. Haven''t tried it yet, though. Anybody else run into this? Tips? Regards, Ranbir -- Kanwar Ranbir Sandhu Linux 2.6.16-1.2111_FC4 i686 GNU/Linux 23:47:43 up 17:17, 4 users, load average: 2.33, 0.78, 0.39 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Kanwar Ranbir Sandhu
2006-Jun-14 13:24 UTC
Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
On Tue, 2006-13-06 at 23:51 -0400, Kanwar Ranbir Sandhu wrote:> It''s a head scratcher, all right. I have no idea what it means, so I''m > reduced to tinkering with the config files, including lowering memory > usage for my one working domU. Haven''t tried it yet, though.My second domU is up and running now. I had forgotten to copy the modules for the xen linux kernel from dom0 to the new domU. So, I did that, and lowered the memory allocated to the first domU. The second domU booted without any problems. I''m not sure which one of the above fixed it for me, but I''d recommend looking into both. HTH, Ranbir -- Kanwar Ranbir Sandhu Linux 2.6.16-1.2111_FC4 i686 GNU/Linux 09:16:29 up 1 day, 2:46, 2 users, load average: 0.11, 0.11, 0.05 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Kanwar Ranbir Sandhu
2006-Jun-15 04:51 UTC
RE: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.
On Tue, 2006-13-06 at 20:58 -0700, Paddy Vishnubhatt wrote:> I found this link and found it to be useful, hope this helps you: > > http://lists.xensource.com/archives/html/xen-users/2006-02/msg00077.htmlThanks for the link. However, I already resolved the problem by lowering the memory allocated to the first domU. After I did that, the second domU started up and the "memory squeeze..." error stopped. I''ll use the recommendation in the posting anyway and set dom0''s max memory at boot time. Again, thanks for the link! Regards, Ranbir -- Kanwar Ranbir Sandhu Linux 2.6.16-1.2111_FC4 i686 GNU/Linux 00:48:42 up 1 day, 18:18, 4 users, load average: 0.15, 0.22, 0.14 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users