Xen users - Nov 2006 - memory issues with FC5.2200 kernels

hiya,

I''ve just upgraded a production Xen to FC5.2200 + the Xen userspace
stuff and its exhibiting some nasty problems. A couple of hours uptime
is followed by all domU networking stopping.

Here''s dmesg in dom0:

xen_net: Memory squeeze in netback driver.
printk: 4 messages suppressed.
xen_net: Memory squeeze in netback driver.
printk: 4 messages suppressed.

And xm dmesg:

(XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
(XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
(XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
(XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
(XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)

Googling has shown inconclusive results - a couple of possible fixes but no
explanation
why (and no consistent fix.)

Any ideas?

Thanks,


Adrian

I should add this from xm dmesg:

 Xen version 3.0-unstable (brewbuilder@build.redhat.com) (gcc version 4.1.1
20060525 (Red Hat 4.1.1-1)) Sat Oct 14 16:56:23 EDT 2006
 Latest ChangeSet: unavailable 

(XEN) Command line: /xen.gz-2.6.18-1.2200.fc5
(XEN) Physical RAM map:
(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000007eff0000 (usable)
(XEN)  000000007eff0000 - 000000007eff3000 (ACPI NVS)
(XEN)  000000007eff3000 - 000000007f000000 (ACPI data)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ffff0000 - 0000000100000000 (reserved)
(XEN) System RAM: 2031MB (2080316kB)
(XEN) Xen heap: 9MB (10108kB)
(XEN) PAE disabled.


Thanks,


adrian


On Tue, Nov 07, 2006, Adrian Chadd wrote:
> hiya,
> 
> I''ve just upgraded a production Xen to FC5.2200 + the Xen
userspace
> stuff and its exhibiting some nasty problems. A couple of hours uptime
> is followed by all domU networking stopping.
> 
> Here''s dmesg in dom0:
> 
> xen_net: Memory squeeze in netback driver.
> printk: 4 messages suppressed.
> xen_net: Memory squeeze in netback driver.
> printk: 4 messages suppressed.
> 
> And xm dmesg:
> 
> (XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
> (XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
> (XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
> (XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
> (XEN) (file=memory.c, line=66) Could not allocate order=0 extent: id=0
memflags=0 (0 of 63)
> 
> Googling has shown inconclusive results - a couple of possible fixes but no
explanation
> why (and no consistent fix.)
> 
> Any ideas?
> 
> Thanks,
> 
> 
> Adrian
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> Adrian Chadd
> Sent: 07 November 2006 13:37
> To: fedora-xen@redhat.com
> Cc: xen-users@lists.xensource.com
> Subject: [Xen-users] memory issues with FC5.2200 kernels
> 
> hiya,
> 
> I''ve just upgraded a production Xen to FC5.2200 + the Xen
userspace
> stuff and its exhibiting some nasty problems. A couple of hours uptime
> is followed by all domU networking stopping.
> 
> Here''s dmesg in dom0:
> 
> xen_net: Memory squeeze in netback driver.
> printk: 4 messages suppressed.
> xen_net: Memory squeeze in netback driver.
> printk: 4 messages suppressed.
> 
> And xm dmesg:
> 
> (XEN) (file=memory.c, line=66) Could not allocate order=0 
> extent: id=0 memflags=0 (0 of 63)
> (XEN) (file=memory.c, line=66) Could not allocate order=0 
> extent: id=0 memflags=0 (0 of 63)
> (XEN) (file=memory.c, line=66) Could not allocate order=0 
> extent: id=0 memflags=0 (0 of 63)
> (XEN) (file=memory.c, line=66) Could not allocate order=0 
> extent: id=0 memflags=0 (0 of 63)
> (XEN) (file=memory.c, line=66) Could not allocate order=0 
> extent: id=0 memflags=0 (0 of 63)
It is trying to allocate one page in "increase_reservation", and
failing... 

I don''t know WHY this is failing (I suspect the heap may have a memory
leak, perhaps)?

What is your DomU doing when you see this problem?

--
Mats
> 
> Googling has shown inconclusive results - a couple of 
> possible fixes but no explanation
> why (and no consistent fix.)
> 
> Any ideas?
> 
> Thanks,
> 
> 
> Adrian
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 
>

On Tue, Nov 07, 2006, Petersson, Mats wrote:
> It is trying to allocate one page in "increase_reservation", and
> failing... 
the domU or dom0? 
> I don''t know WHY this is failing (I suspect the heap may have a
memory
> leak, perhaps)?
> 
> What is your DomU doing when you see this problem?
No idea. I''ve got half a dozen domU''s here all running various
tasks each
(smtp, dns, www, etc.) so its difficult to pin down what each is doing.



Adrian

> -----Original Message-----
> From: Adrian Chadd [mailto:adrian@creative.net.au] 
> Sent: 07 November 2006 15:26
> To: Petersson, Mats
> Cc: fedora-xen@redhat.com; xen-users@lists.xensource.com
> Subject: Re: [Xen-users] memory issues with FC5.2200 kernels
> 
> On Tue, Nov 07, 2006, Petersson, Mats wrote:
> 
> > It is trying to allocate one page in "increase_reservation",
and
> > failing... 
> 
> the domU or dom0? 
The one with "id=0", so Dom0.
> 
> > I don''t know WHY this is failing (I suspect the heap may 
> have a memory
> > leak, perhaps)?
> > 
> > What is your DomU doing when you see this problem?
> 
> No idea. I''ve got half a dozen domU''s here all running 
> various tasks each
> (smtp, dns, www, etc.) so its difficult to pin down what each 
> is doing.
That makes debugging it quite difficult... 

Perhaps you can set up a machine for testing to find out what''s causing
it, if there''s some pattern to it, etc, etc. 

--
Mats
> 
> 
> 
> Adrian
> 
> 
> 
>

On Tue, Nov 07, 2006, Petersson, Mats wrote:
> > the domU or dom0? 
> 
> The one with "id=0", so Dom0.
Duh. :)
> > No idea. I''ve got half a dozen domU''s here all
running
> > various tasks each
> > (smtp, dns, www, etc.) so its difficult to pin down what each 
> > is doing.
> 
> That makes debugging it quite difficult... 
> 
> Perhaps you can set up a machine for testing to find out what''s
causing
> it, if there''s some pattern to it, etc, etc. 
I''ve got two test Xen servers which were running FC5.2200 Xen just
fine.
I''ve been bitten by previous networking bugs with the FC5 kernels
(specifically kernel panics in the netblk stuff) which people have noticed
but I can''t see any fixes for. I hadn''t seen any panics during
my testing
over two weeks so I moved this one production server over.



Adrian