Daniel P. Berrange
2007-Apr-26 23:12 UTC
[Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
I''m testing on the Xen 3.0.5 rc3 (#14934) hypervisor and userspace. On
the
whole it looks pretty good - the only significant problem I seeing is that
when booting a fullyvirt guest, the Domain-0 is not being ballooned down
to make room for its memory.
[2007-04-26 18:03:31 2654] ERROR (xmlrpclib2:178) Internal error handling
xend.domain.create
Traceback (most recent call last):
File "/usr/lib64/python2.5/site-packages/xen/util/xmlrpclib2.py",
line 131, in _marshaled_dispatch
response = self._dispatch(method, params)
File "/usr/lib64/python2.5/SimpleXMLRPCServer.py", line 415, in
_dispatch
return func(*params)
File
"/usr/lib64/python2.5/site-packages/xen/xend/server/XMLRPCServer.py",
line 72, in domain_create
info = XendDomain.instance().domain_create(config)
File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomain.py",
line 914, in domain_create
dominfo = XendDomainInfo.create(config)
File
"/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
81, in create
vm.start()
File
"/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
395, in start
XendTask.log_progress(0, 30, self._constructDomain)
File "/usr/lib64/python2.5/site-packages/xen/xend/XendTask.py", line
209, in log_progress
retval = func(*args, **kwds)
File
"/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line
1430, in _constructDomain
hvm = int(hvm))
Error: (12, ''Cannot allocate memory'')
What is interesting is that the start() method does:
XendTask.log_progress(0, 30, self._constructDomain)
XendTask.log_progress(31, 60, self._initDomain)
The _constructDomain call is failing in xc.domain_create() - the code which
deals with ballooning down Domain-0 is not even run yet - that lives in
the _initDomain method. So HVM domain creation is failing before XenD has
even had a chance to do ballooning. If I manually use ''xm
mem-set'' to
balloon down Domain-0 ahead of time, everything works just fine.
I''m at a lose to figure out how to solve this without major
re-arranging
of the order of _constructDomain & _initDomain work. Perhaps someone can
figure out an easier fix ?
This is on the x86_64 architecture and the guest config is:
name = "rhel3i386"
builder = "hvm"
memory = "500"
disk = [ ''file:/xen/rhel3i386.img,hda,w'', ]
vif = [ ''type=ioemu, mac=00:16:3e:0c:e4:e3, bridge=virbr0'', ]
uuid = "06017e0c-4f74-d1d0-017c-def49abd8c9c"
device_model = "/usr/lib64/xen/bin/qemu-dm"
kernel = "/usr/lib/xen/boot/hvmloader"
vnc=1
vncunused=1
apic=1
acpi=0
pae=1
vcpus=1
serial = "pty"
on_reboot = ''restart''
on_crash = ''restart''
The domain-0 host info is
# xm info
host : celery.virt.boston.redhat.com
release : 2.6.20-2925.5.fc7ehabkosttestbuild2xen
version : #1 SMP Thu Apr 26 16:43:56 EDT 2007
machine : x86_64
nr_cpus : 2
nr_nodes : 1
sockets_per_node : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2211
hw_caps :
178bfbff:ebd3fbff:00000000:00000010:00002001:00000000:0000001f
total_memory : 4095
free_memory : 0
xen_major : 3
xen_minor : 0
xen_extra : .5-rc3-2925.5.f
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
cc_compiler : gcc version 4.1.2 20070424 (Red Hat 4.1.2-11)
cc_compile_by : brewbuilder
cc_compile_domain : build.redhat.com
cc_compile_date : Thu Apr 26 16:40:09 EDT 2007
xend_config_format : 4
And there is nothing being reported by ''xm dmesg'' when the
failures
occur.
I''ve not had a chance to try & reproduce this on i386 yet.
Hopefully will
do that tomorrrow.
Regards,
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2007-Apr-27 06:53 UTC
Re: [Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
On 27/4/07 00:12, "Daniel P. Berrange" <berrange@redhat.com> wrote:> The _constructDomain call is failing in xc.domain_create() - the code which > deals with ballooning down Domain-0 is not even run yet - that lives in > the _initDomain method. So HVM domain creation is failing before XenD has > even had a chance to do ballooning. If I manually use ''xm mem-set'' to > balloon down Domain-0 ahead of time, everything works just fine. > > I''m at a lose to figure out how to solve this without major re-arranging > of the order of _constructDomain & _initDomain work. Perhaps someone can > figure out an easier fix ?I don''t think domain_create() allocates very many pages, although it''s possible it''s making a default allocation of shadow memory. Assuming not, it may be sufficient to make auto-balloon leave a bit of slack memory rather than try to account for all available memory down to the last page. A few pages might turn out to suffice. Does this problem occur with the very first domain you try to create after booting the machine? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2007-Apr-27 13:26 UTC
Re: [Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
On Fri, Apr 27, 2007 at 07:53:24AM +0100, Keir Fraser wrote:> On 27/4/07 00:12, "Daniel P. Berrange" <berrange@redhat.com> wrote: > > > The _constructDomain call is failing in xc.domain_create() - the code which > > deals with ballooning down Domain-0 is not even run yet - that lives in > > the _initDomain method. So HVM domain creation is failing before XenD has > > even had a chance to do ballooning. If I manually use ''xm mem-set'' to > > balloon down Domain-0 ahead of time, everything works just fine. > > > > I''m at a lose to figure out how to solve this without major re-arranging > > of the order of _constructDomain & _initDomain work. Perhaps someone can > > figure out an easier fix ? > > I don''t think domain_create() allocates very many pages, although it''s > possible it''s making a default allocation of shadow memory. Assuming not, it > may be sufficient to make auto-balloon leave a bit of slack memory rather > than try to account for all available memory down to the last page. A few > pages might turn out to suffice. Does this problem occur with the very first > domain you try to create after booting the machine?It seems to occurr with any HVM guest I try to create, unless there was already some free mem (from starting & stopping a previous dom). I''m testing with this very simple hack which seems to make it work - just ensure the balloon driver frees a 10 MB chunk right at the start. Hardcoding is fine because when we later get to the _initDomain() method we''ll definitely be freeing up much more than 10 MB for the HVM guests''s actual mem requirements. --- xen-3.0.5-testing.hg-rc3-14934/tools/python/xen/xend/XendDomainInfo.py 2007-04-26 19:30:11.000000000 -0400 +++ xen-3.0.5-testing.hg-rc3-14934.new/tools/python/xen/xend/XendDomainInfo.py 2007-04-26 19:36:06.000000000 -0400 @@ -1422,6 +1422,8 @@ class XendDomainInfo: raise VmError("HVM guest support is unavailable: is VT/AMD-V " "supported by your CPU and enabled in your " "BIOS?") + # Hack to pre-reserve some mem for basic HVM setup + balloon.free(10*1024) self.domid = xc.domain_create( domid = 0, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Apr-27 13:56 UTC
Re: [Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
On 27/4/07 14:26, "Daniel P. Berrange" <berrange@redhat.com> wrote:> It seems to occurr with any HVM guest I try to create, unless there was > already > some free mem (from starting & stopping a previous dom). I''m testing with this > very simple hack which seems to make it work - just ensure the balloon driver > frees a 10 MB chunk right at the start. Hardcoding is fine because when we > later get to the _initDomain() method we''ll definitely be freeing up much > more than 10 MB for the HVM guests''s actual mem requirements.We do most testing with a sensible ''dom0_mem='' so we wouldn''t see this. The hack looks fine. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2007-Apr-27 14:09 UTC
Re: [Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
On Fri, Apr 27, 2007 at 02:56:48PM +0100, Keir Fraser wrote:> On 27/4/07 14:26, "Daniel P. Berrange" <berrange@redhat.com> wrote: > > > It seems to occurr with any HVM guest I try to create, unless there was > > already > > some free mem (from starting & stopping a previous dom). I''m testing with this > > very simple hack which seems to make it work - just ensure the balloon driver > > frees a 10 MB chunk right at the start. Hardcoding is fine because when we > > later get to the _initDomain() method we''ll definitely be freeing up much > > more than 10 MB for the HVM guests''s actual mem requirements. > > We do most testing with a sensible ''dom0_mem='' so we wouldn''t see this.Well ''sensible'' depends on your target audience :-) In the desktop/laptop arena users like their Dom0 to have all the memory which isn''t used by active guests - even to the extent we''ve had bugs reported asking us to make XenD automatically balloon dom0 back up after a guest shuts down ! Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Apr-27 14:27 UTC
Re: [Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
On 27/4/07 15:09, "Daniel P. Berrange" <berrange@redhat.com> wrote:>> We do most testing with a sensible ''dom0_mem='' so we wouldn''t see this. > > Well ''sensible'' depends on your target audience :-) In the desktop/laptop > arena users like their Dom0 to have all the memory which isn''t used by > active guests - even to the extent we''ve had bugs reported asking us to > make XenD automatically balloon dom0 back up after a guest shuts down !Oh yes. If dom0 is your client OS and virtualisation is a secondary activity on your box then this makes perfect sense! -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Charles Coffing
2007-Apr-30 18:59 UTC
Re: [Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
On Fri, Apr 27, 2007 at 7:26 AM, "Daniel P. Berrange" <berrange@redhat.com> wrote:> On Fri, Apr 27, 2007 at 07:53:24AM +0100, Keir Fraser wrote: >> On 27/4/07 00:12, "Daniel P. Berrange" <berrange@redhat.com> wrote: >> >> > The _constructDomain call is failing in xc.domain_create() - the code which >> > deals with ballooning down Domain-0 is not even run yet - that lives in >> > the _initDomain method. So HVM domain creation is failing before XenD has >> > even had a chance to do ballooning. If I manually use ''xm mem-set'' to >> > balloon down Domain-0 ahead of time, everything works just fine. >> > >> > I''m at a lose to figure out how to solve this without major re-arranging >> > of the order of _constructDomain & _initDomain work. Perhaps someone can >> > figure out an easier fix ? >> >> I don''t think domain_create() allocates very many pages, although it''s >> possible it''s making a default allocation of shadow memory. Assuming not, it >> may be sufficient to make auto-balloon leave a bit of slack memory rather >> than try to account for all available memory down to the last page. A few >> pages might turn out to suffice. Does this problem occur with the very first >> domain you try to create after booting the machine? > > It seems to occurr with any HVM guest I try to create, unless there was > already > some free mem (from starting & stopping a previous dom). I''m testing with > this > very simple hack which seems to make it work - just ensure the balloon driver > frees a 10 MB chunk right at the start. Hardcoding is fine because when we > later get to the _initDomain() method we''ll definitely be freeing up much > more than 10 MB for the HVM guests''s actual mem requirements.Keir, The problem Daniel fixed for fully virtual affects paravirtual, also. We have received bug reports from ISVs of this problem in both the PV and FV cases. I traced code paths and discovered that FV attempts to allocate the most memory during the initial domain creation; 64-bit PV allocates the next most; 32-bit PV allocates the least. So failures for PV are less common, but not impossible. Would you please de-indent the added lines, so both PV and FV guests have the 2 MB slack? I''ve been testing such a patch for about a month in our SLES 10 SP1 betas (based on xen-3.0.4), and it works well. Thanks.> --- xen-3.0.5-testing.hg-rc3-14934/tools/python/xen/xend/XendDomainInfo.py > 2007-04-26 19:30:11.000000000 -0400 > +++ xen-3.0.5-testing.hg-rc3-14934.new/tools/python/xen/xend/XendDomainInfo.py > 2007-04-26 19:36:06.000000000 -0400 > @@ -1422,6 +1422,8 @@ class XendDomainInfo: > raise VmError("HVM guest support is unavailable: is VT/AMD-V > " > "supported by your CPU and enabled in your " > "BIOS?") > + # Hack to pre-reserve some mem for basic HVM setup > + balloon.free(10*1024) > > self.domid = xc.domain_create( > domid = 0, > > > Dan._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2007-Apr-30 19:11 UTC
Re: [Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
On Mon, Apr 30, 2007 at 02:59:08PM -0400, Charles Coffing wrote:> On Fri, Apr 27, 2007 at 7:26 AM, "Daniel P. Berrange" <berrange@redhat.com> wrote: > > On Fri, Apr 27, 2007 at 07:53:24AM +0100, Keir Fraser wrote: > >> On 27/4/07 00:12, "Daniel P. Berrange" <berrange@redhat.com> wrote: > >> > >> > The _constructDomain call is failing in xc.domain_create() - the code which > >> > deals with ballooning down Domain-0 is not even run yet - that lives in > >> > the _initDomain method. So HVM domain creation is failing before XenD has > >> > even had a chance to do ballooning. If I manually use ''xm mem-set'' to > >> > balloon down Domain-0 ahead of time, everything works just fine. > >> > > >> > I''m at a lose to figure out how to solve this without major re-arranging > >> > of the order of _constructDomain & _initDomain work. Perhaps someone can > >> > figure out an easier fix ? > >> > >> I don''t think domain_create() allocates very many pages, although it''s > >> possible it''s making a default allocation of shadow memory. Assuming not, it > >> may be sufficient to make auto-balloon leave a bit of slack memory rather > >> than try to account for all available memory down to the last page. A few > >> pages might turn out to suffice. Does this problem occur with the very first > >> domain you try to create after booting the machine? > > > > It seems to occurr with any HVM guest I try to create, unless there was > > already > > some free mem (from starting & stopping a previous dom). I''m testing with > > this > > very simple hack which seems to make it work - just ensure the balloon driver > > frees a 10 MB chunk right at the start. Hardcoding is fine because when we > > later get to the _initDomain() method we''ll definitely be freeing up much > > more than 10 MB for the HVM guests''s actual mem requirements. > > > Keir, > > The problem Daniel fixed for fully virtual affects paravirtual, also. > > We have received bug reports from ISVs of this problem in both the PV > and FV cases. I traced code paths and discovered that FV attempts to > allocate the most memory during the initial domain creation; 64-bit PV > allocates the next most; 32-bit PV allocates the least. So failures > for PV are less common, but not impossible.Indeed - in testing on 64-bit PV over the weekend I saw perhaps 1 in 5 PV creation attempts fail with the not enough memory bug. So it does look like the patch needs to be de-indented to fix PV too. Regards, Dan -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Apr-30 21:55 UTC
Re: [Xen-devel] 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory
On 30/4/07 20:11, "Daniel P. Berrange" <berrange@redhat.com> wrote:>> We have received bug reports from ISVs of this problem in both the PV >> and FV cases. I traced code paths and discovered that FV attempts to >> allocate the most memory during the initial domain creation; 64-bit PV >> allocates the next most; 32-bit PV allocates the least. So failures >> for PV are less common, but not impossible. > > Indeed - in testing on 64-bit PV over the weekend I saw perhaps 1 in 5 > PV creation attempts fail with the not enough memory bug. So it does look > like the patch needs to be de-indented to fix PV too.It''s weird that this is only cropping up now. Still, the workaround is pretty harmless. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel