Kai Meyer
2008-Mar-26 14:54 UTC
[Xen-devel] [Xen-users] xend (12, ''Cannot allocate memory'')
I''m getting the error:
xend (12, ''Cannot allocate memory'')
From the command line after the Dom-0 goes through a period of heavy
memory usage. I don''t have any hard numbers on how to duplicate it, but
I''ll give you the steps I''ve gone through to duplicate the
problem.
First, some information about my setup. It''s a new setup, and has been
in heavy development for the past few months. I''m new to Xen as of this
project we started in January. As far as I can tell, this problem has
existed since we started, and may have even happened before but we were
too new to think it wasn''t our fault (which it may still be....)
I''ve
found the same error addressed here:
http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921
The ''hack'' to fix this problem is already built into my
XendDomainInfo.py script already, so I''m a little more than lost.
[root@xen1 ~]# xm info
host : xen1.fiber.net
release : 2.6.18-53.1.13.el5xen
version : #1 SMP Tue Feb 12 13:33:07 EST 2008
machine : x86_64
nr_cpus : 8
nr_nodes : 1
sockets_per_node : 2
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 1866
hw_caps :
bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
total_memory : 16382
free_memory : 2
xen_major : 3
xen_minor : 1
xen_extra : .0-53.1.13.el5
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
cc_compiler : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
cc_compile_by : mockbuild
cc_compile_domain :
cc_compile_date : Tue Feb 12 12:55:35 EST 2008
xend_config_format : 2
Here''s the error in action:
[root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image
Using config file "./centos-5.1_core_image".
Error: (12, ''Cannot allocate memory'')
[root@xen1 centos-5.1_core_image]#
Here is the output from xend.log that occurs when I get this error:
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
(XendDomainInfo:200) XendDomainInfo.create([''vm'',
[''name'',
''centos-5.1_core_image''], [''memory'', 4024],
[''maxmem'', 4024],
[''on_poweroff'', ''destroy''],
[''on_reboot'', ''restart''],
[''on_crash'',
''restart''], [''vcpus'', 6],
[''image'', [''hvm'',
[''kernel'',
''/usr/lib/xen/boot/hvmloader''],
[''device_model'',
''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1],
[''vcpus'', 6], [''boot'',
''cda''], [''serial'', ''pty''],
[''vnc'', 1], [''vncunused'', 1],
[''xauthority'',
''/root/.Xauthority''], [''keymap'',
''en-us'']]], [''device'',
[''vbd'',
[''uname'',
''file:/xen/domains2/centos-5.1_core_image/new_disk.img''],
[''dev'', ''hda''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''file:/xen/domains2/centos-5.1_core_image/swap.img''],
[''dev'', ''hdb''],
[''mode'', ''w'']]],
[''device'', [''vif'',
[''bridge'', ''xenbr1''],
[''mac'',
''00:16:3e:4c:c0:01''], [''type'',
''ioemu'']]]])
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
(XendDomainInfo:306) parseConfig: config is [''vm'',
[''name'',
''centos-5.1_core_image''], [''memory'', 4024],
[''maxmem'', 4024],
[''on_poweroff'', ''destroy''],
[''on_reboot'', ''restart''],
[''on_crash'',
''restart''], [''vcpus'', 6],
[''image'', [''hvm'',
[''kernel'',
''/usr/lib/xen/boot/hvmloader''],
[''device_model'',
''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1],
[''vcpus'', 6], [''boot'',
''cda''], [''serial'', ''pty''],
[''vnc'', 1], [''vncunused'', 1],
[''xauthority'',
''/root/.Xauthority''], [''keymap'',
''en-us'']]], [''device'',
[''vbd'',
[''uname'',
''file:/xen/domains2/centos-5.1_core_image/new_disk.img''],
[''dev'', ''hda''], [''mode'',
''w'']]], [''device'', [''vbd'',
[''uname'',
''file:/xen/domains2/centos-5.1_core_image/swap.img''],
[''dev'', ''hdb''],
[''mode'', ''w'']]],
[''device'', [''vif'',
[''bridge'', ''xenbr1''],
[''mac'',
''00:16:3e:4c:c0:01''], [''type'',
''ioemu'']]]]
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
(XendDomainInfo:411) parseConfig: result is {''shadow_memory'':
None,
''start_time'': None, ''uuid'': None,
''on_crash'': ''restart'',
''on_reboot'':
''restart'', ''localtime'': None,
''image'': [''hvm'',
[''kernel'',
''/usr/lib/xen/boot/hvmloader''],
[''device_model'',
''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1],
[''vcpus'', 6], [''boot'',
''cda''], [''serial'', ''pty''],
[''vnc'', 1], [''vncunused'', 1],
[''xauthority'',
''/root/.Xauthority''], [''keymap'',
''en-us'']], ''on_poweroff'':
''destroy'',
''bootloader_args'': None, ''cpus'': None,
''name'': ''centos-5.1_core_image'',
''backend'': [], ''vcpus'': 6,
''cpu_weight'': None, ''features'': None,
''vcpu_avail'': None, ''memory'': 4024,
''device'': [(''vbd'', [''vbd'',
[''uname'',
''file:/xen/domains2/centos-5.1_core_image/new_disk.img''],
[''dev'',
''hda''], [''mode'', ''w'']]),
(''vbd'', [''vbd'', [''uname'',
''file:/xen/domains2/centos-5.1_core_image/swap.img''],
[''dev'', ''hdb''],
[''mode'', ''w'']]), (''vif'',
[''vif'', [''bridge'',
''xenbr1''], [''mac'',
''00:16:3e:4c:c0:01''], [''type'',
''ioemu'']])], ''bootloader'': None,
''cpu'':
None, ''maxmem'': 4024}
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
(XendDomainInfo:1349) XendDomainInfo.construct: None
[2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB
free; need 2048; done.
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR
(XendDomainInfo:212) Domain construction failed
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 204, in create
vm.construct()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 1369, in construct
hvm = hvm)
Error: (12, ''Cannot allocate memory'')
Here''s the configuration file being used:
name = "centos-5.1_core_image"
maxmem = 4024
memory = 4024
vcpus = 6
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
boot = "cda"
pae = 1
acpi = 0
apic = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
device_model = "/usr/lib64/xen/bin/qemu-dm"
sdl = 0
vnc = 1
vncunused = 1
keymap = "en-us"
disk = [
''file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w'' ,
''file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w'' ]
vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ]
serial = "pty"
At the time the error occured, my memory looked like this:
[root@xen1 centos-5.1_core_image]# free
total used free shared buffers cached
Mem: 15845376 15821080 24296 0 113596 15081468
-/+ buffers/cache: 626016 15219360
Swap: 4192956 4 4192952
The only way I''ve been able to duplicate this error is by making
multiple copies of the 6GB disk images (created with dd, so ls shows 6GB
but du shows 2GB) for different virtual machines and booting them up. I
can get up to about 8 new virtual machines roughly, before I start
seeing my memory error. If you''re familiar with IBM''s
''xen_deploy.pl''
script, I''ve modified it slightly to meet our needs, and launch our
systems with that tool. It''s basic functionality is to read in a xen
configuration file you want to use as a template, and output a modified
one based on arguments you pass to the scripts, and then create a floppy
disk image and copy the image from the template config file, and then
boot the new system.
I''ve read in previous emails that this error occurs because of a bug in
the hvmloader, and look for an update. That email response was for a 3.0
version of xen, and I''m on 3.1.
What other information can I provide to help diagnose this problem?
-Kai Meyer
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2008-Mar-26 15:00 UTC
Re: [Xen-devel] [Xen-users] xend (12, ''Cannot allocate memory'')
Given you are running dom0 with plenty of swap, it''s most likely that Xen has run out of memory to allocate to your new domU, rather than the allocation failure coming from within dom0. Probably auto-ballooning is screwing up somehow (that''s the bit of xend that gives away memory back to Xen to make sure it has enough memory to create your new domU). Sorry I don''t have a precise diagnosis for you, but I think looking at Xen''s allocator is at least the right direction for you to take. ''xm info'' tells you how much memory Xen thinks it has. -- Keir On 26/3/08 14:54, "Kai Meyer" <kai@unixlords.com> wrote:> I''m getting the error: > xend (12, ''Cannot allocate memory'') > From the command line after the Dom-0 goes through a period of heavy > memory usage. I don''t have any hard numbers on how to duplicate it, but > I''ll give you the steps I''ve gone through to duplicate the problem. > > First, some information about my setup. It''s a new setup, and has been > in heavy development for the past few months. I''m new to Xen as of this > project we started in January. As far as I can tell, this problem has > existed since we started, and may have even happened before but we were > too new to think it wasn''t our fault (which it may still be....) I''ve > found the same error addressed here: > http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921 > The ''hack'' to fix this problem is already built into my > XendDomainInfo.py script already, so I''m a little more than lost. > > [root@xen1 ~]# xm info > host : xen1.fiber.net > release : 2.6.18-53.1.13.el5xen > version : #1 SMP Tue Feb 12 13:33:07 EST 2008 > machine : x86_64 > nr_cpus : 8 > nr_nodes : 1 > sockets_per_node : 2 > cores_per_socket : 4 > threads_per_core : 1 > cpu_mhz : 1866 > hw_caps : > bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001 > total_memory : 16382 > free_memory : 2 > xen_major : 3 > xen_minor : 1 > xen_extra : .0-53.1.13.el5 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > cc_compiler : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) > cc_compile_by : mockbuild > cc_compile_domain : > cc_compile_date : Tue Feb 12 12:55:35 EST 2008 > xend_config_format : 2 > > > Here''s the error in action: > [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image > Using config file "./centos-5.1_core_image". > Error: (12, ''Cannot allocate memory'') > [root@xen1 centos-5.1_core_image]# > > > Here is the output from xend.log that occurs when I get this error: > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG > (XendDomainInfo:200) XendDomainInfo.create([''vm'', [''name'', > ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], > [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', > ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', > ''/usr/lib/xen/boot/hvmloader''], [''device_model'', > ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', > ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', > ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', > [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], > [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', > ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], > [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', > ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]]) > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG > (XendDomainInfo:306) parseConfig: config is [''vm'', [''name'', > ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], > [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', > ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', > ''/usr/lib/xen/boot/hvmloader''], [''device_model'', > ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', > ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', > ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', > [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], > [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', > ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], > [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', > ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]] > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG > (XendDomainInfo:411) parseConfig: result is {''shadow_memory'': None, > ''start_time'': None, ''uuid'': None, ''on_crash'': ''restart'', ''on_reboot'': > ''restart'', ''localtime'': None, ''image'': [''hvm'', [''kernel'', > ''/usr/lib/xen/boot/hvmloader''], [''device_model'', > ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', > ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', > ''/root/.Xauthority''], [''keymap'', ''en-us'']], ''on_poweroff'': ''destroy'', > ''bootloader_args'': None, ''cpus'': None, ''name'': ''centos-5.1_core_image'', > ''backend'': [], ''vcpus'': 6, ''cpu_weight'': None, ''features'': None, > ''vcpu_avail'': None, ''memory'': 4024, ''device'': [(''vbd'', [''vbd'', [''uname'', > ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], [''dev'', > ''hda''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', > ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], > [''mode'', ''w'']]), (''vif'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', > ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']])], ''bootloader'': None, ''cpu'': > None, ''maxmem'': 4024} > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG > (XendDomainInfo:1349) XendDomainInfo.construct: None > [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB > free; need 2048; done. > [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR > (XendDomainInfo:212) Domain construction failed > Traceback (most recent call last): > File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", > line 204, in create > vm.construct() > File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", > line 1369, in construct > hvm = hvm) > Error: (12, ''Cannot allocate memory'') > > > Here''s the configuration file being used: > name = "centos-5.1_core_image" > maxmem = 4024 > memory = 4024 > vcpus = 6 > builder = "hvm" > kernel = "/usr/lib/xen/boot/hvmloader" > boot = "cda" > pae = 1 > acpi = 0 > apic = 0 > on_poweroff = "destroy" > on_reboot = "restart" > on_crash = "restart" > device_model = "/usr/lib64/xen/bin/qemu-dm" > sdl = 0 > vnc = 1 > vncunused = 1 > keymap = "en-us" > disk = [ ''file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w'' , > ''file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w'' ] > vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ] > serial = "pty" > > At the time the error occured, my memory looked like this: > [root@xen1 centos-5.1_core_image]# free > total used free shared buffers cached > Mem: 15845376 15821080 24296 0 113596 15081468 > -/+ buffers/cache: 626016 15219360 > Swap: 4192956 4 4192952 > > The only way I''ve been able to duplicate this error is by making > multiple copies of the 6GB disk images (created with dd, so ls shows 6GB > but du shows 2GB) for different virtual machines and booting them up. I > can get up to about 8 new virtual machines roughly, before I start > seeing my memory error. If you''re familiar with IBM''s ''xen_deploy.pl'' > script, I''ve modified it slightly to meet our needs, and launch our > systems with that tool. It''s basic functionality is to read in a xen > configuration file you want to use as a template, and output a modified > one based on arguments you pass to the scripts, and then create a floppy > disk image and copy the image from the template config file, and then > boot the new system. > > I''ve read in previous emails that this error occurs because of a bug in > the hvmloader, and look for an update. That email response was for a 3.0 > version of xen, and I''m on 3.1. > > What other information can I provide to help diagnose this problem? > > -Kai Meyer > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kai Meyer
2008-Mar-26 15:25 UTC
Re: [Xen-devel] [Xen-users] xend (12, ''Cannot allocate memory'')
Well, that''s no so great. It thinks I have barely a fraction of what I''m supposed to have available. Free shows 10GB available, xm info shows 1 MB available. If I stop a virtual machine, the memory in xminfo increases by the amount of memory used by that xen machine, which is to be expected. Is there a way to force Xen to refresh that number, or let me force an override? -Kai Meyer Keir Fraser wrote:> Given you are running dom0 with plenty of swap, it''s most likely that Xen > has run out of memory to allocate to your new domU, rather than the > allocation failure coming from within dom0. Probably auto-ballooning is > screwing up somehow (that''s the bit of xend that gives away memory back to > Xen to make sure it has enough memory to create your new domU). > > Sorry I don''t have a precise diagnosis for you, but I think looking at Xen''s > allocator is at least the right direction for you to take. ''xm info'' tells > you how much memory Xen thinks it has. > > -- Keir > > On 26/3/08 14:54, "Kai Meyer" <kai@unixlords.com> wrote: > > >> I''m getting the error: >> xend (12, ''Cannot allocate memory'') >> From the command line after the Dom-0 goes through a period of heavy >> memory usage. I don''t have any hard numbers on how to duplicate it, but >> I''ll give you the steps I''ve gone through to duplicate the problem. >> >> First, some information about my setup. It''s a new setup, and has been >> in heavy development for the past few months. I''m new to Xen as of this >> project we started in January. As far as I can tell, this problem has >> existed since we started, and may have even happened before but we were >> too new to think it wasn''t our fault (which it may still be....) I''ve >> found the same error addressed here: >> http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921 >> The ''hack'' to fix this problem is already built into my >> XendDomainInfo.py script already, so I''m a little more than lost. >> >> [root@xen1 ~]# xm info >> host : xen1.fiber.net >> release : 2.6.18-53.1.13.el5xen >> version : #1 SMP Tue Feb 12 13:33:07 EST 2008 >> machine : x86_64 >> nr_cpus : 8 >> nr_nodes : 1 >> sockets_per_node : 2 >> cores_per_socket : 4 >> threads_per_core : 1 >> cpu_mhz : 1866 >> hw_caps : >> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001 >> total_memory : 16382 >> free_memory : 2 >> xen_major : 3 >> xen_minor : 1 >> xen_extra : .0-53.1.13.el5 >> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 >> hvm-3.0-x86_32p hvm-3.0-x86_64 >> xen_pagesize : 4096 >> platform_params : virt_start=0xffff800000000000 >> xen_changeset : unavailable >> cc_compiler : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >> cc_compile_by : mockbuild >> cc_compile_domain : >> cc_compile_date : Tue Feb 12 12:55:35 EST 2008 >> xend_config_format : 2 >> >> >> Here''s the error in action: >> [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image >> Using config file "./centos-5.1_core_image". >> Error: (12, ''Cannot allocate memory'') >> [root@xen1 centos-5.1_core_image]# >> >> >> Here is the output from xend.log that occurs when I get this error: >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >> (XendDomainInfo:200) XendDomainInfo.create([''vm'', [''name'', >> ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], >> [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', >> ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', >> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >> ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', >> [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], >> [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', >> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >> [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]]) >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >> (XendDomainInfo:306) parseConfig: config is [''vm'', [''name'', >> ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], >> [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', >> ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', >> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >> ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', >> [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], >> [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', >> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >> [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]] >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >> (XendDomainInfo:411) parseConfig: result is {''shadow_memory'': None, >> ''start_time'': None, ''uuid'': None, ''on_crash'': ''restart'', ''on_reboot'': >> ''restart'', ''localtime'': None, ''image'': [''hvm'', [''kernel'', >> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >> ''/root/.Xauthority''], [''keymap'', ''en-us'']], ''on_poweroff'': ''destroy'', >> ''bootloader_args'': None, ''cpus'': None, ''name'': ''centos-5.1_core_image'', >> ''backend'': [], ''vcpus'': 6, ''cpu_weight'': None, ''features'': None, >> ''vcpu_avail'': None, ''memory'': 4024, ''device'': [(''vbd'', [''vbd'', [''uname'', >> ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], [''dev'', >> ''hda''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', >> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >> [''mode'', ''w'']]), (''vif'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']])], ''bootloader'': None, ''cpu'': >> None, ''maxmem'': 4024} >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >> (XendDomainInfo:1349) XendDomainInfo.construct: None >> [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB >> free; need 2048; done. >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR >> (XendDomainInfo:212) Domain construction failed >> Traceback (most recent call last): >> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >> line 204, in create >> vm.construct() >> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >> line 1369, in construct >> hvm = hvm) >> Error: (12, ''Cannot allocate memory'') >> >> >> Here''s the configuration file being used: >> name = "centos-5.1_core_image" >> maxmem = 4024 >> memory = 4024 >> vcpus = 6 >> builder = "hvm" >> kernel = "/usr/lib/xen/boot/hvmloader" >> boot = "cda" >> pae = 1 >> acpi = 0 >> apic = 0 >> on_poweroff = "destroy" >> on_reboot = "restart" >> on_crash = "restart" >> device_model = "/usr/lib64/xen/bin/qemu-dm" >> sdl = 0 >> vnc = 1 >> vncunused = 1 >> keymap = "en-us" >> disk = [ ''file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w'' , >> ''file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w'' ] >> vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ] >> serial = "pty" >> >> At the time the error occured, my memory looked like this: >> [root@xen1 centos-5.1_core_image]# free >> total used free shared buffers cached >> Mem: 15845376 15821080 24296 0 113596 15081468 >> -/+ buffers/cache: 626016 15219360 >> Swap: 4192956 4 4192952 >> >> The only way I''ve been able to duplicate this error is by making >> multiple copies of the 6GB disk images (created with dd, so ls shows 6GB >> but du shows 2GB) for different virtual machines and booting them up. I >> can get up to about 8 new virtual machines roughly, before I start >> seeing my memory error. If you''re familiar with IBM''s ''xen_deploy.pl'' >> script, I''ve modified it slightly to meet our needs, and launch our >> systems with that tool. It''s basic functionality is to read in a xen >> configuration file you want to use as a template, and output a modified >> one based on arguments you pass to the scripts, and then create a floppy >> disk image and copy the image from the template config file, and then >> boot the new system. >> >> I''ve read in previous emails that this error occurs because of a bug in >> the hvmloader, and look for an update. That email response was for a 3.0 >> version of xen, and I''m on 3.1. >> >> What other information can I provide to help diagnose this problem? >> >> -Kai Meyer >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Mar-26 15:29 UTC
Re: [Xen-devel] [Xen-users] xend (12, ''Cannot allocate memory'')
Depending on your usage scenario for dom0 itself, you could could turn off auto-ballooning entirely (it can be done by modifying xend-config.sxp) and limit dom0''s initial memory allocation on Xen''s command line (e.g., modifying Xen boot parameters in your GRUB menu file). -- Keir On 26/3/08 15:25, "Kai Meyer" <kai@unixlords.com> wrote:> > Well, that''s no so great. It thinks I have barely a fraction of what I''m > supposed to have available. Free shows 10GB available, xm info shows 1 > MB available. If I stop a virtual machine, the memory in xminfo > increases by the amount of memory used by that xen machine, which is to > be expected. > > Is there a way to force Xen to refresh that number, or let me force an > override? > > -Kai Meyer > > Keir Fraser wrote: >> Given you are running dom0 with plenty of swap, it''s most likely that Xen >> has run out of memory to allocate to your new domU, rather than the >> allocation failure coming from within dom0. Probably auto-ballooning is >> screwing up somehow (that''s the bit of xend that gives away memory back to >> Xen to make sure it has enough memory to create your new domU). >> >> Sorry I don''t have a precise diagnosis for you, but I think looking at Xen''s >> allocator is at least the right direction for you to take. ''xm info'' tells >> you how much memory Xen thinks it has. >> >> -- Keir >> >> On 26/3/08 14:54, "Kai Meyer" <kai@unixlords.com> wrote: >> >> >>> I''m getting the error: >>> xend (12, ''Cannot allocate memory'') >>> From the command line after the Dom-0 goes through a period of heavy >>> memory usage. I don''t have any hard numbers on how to duplicate it, but >>> I''ll give you the steps I''ve gone through to duplicate the problem. >>> >>> First, some information about my setup. It''s a new setup, and has been >>> in heavy development for the past few months. I''m new to Xen as of this >>> project we started in January. As far as I can tell, this problem has >>> existed since we started, and may have even happened before but we were >>> too new to think it wasn''t our fault (which it may still be....) I''ve >>> found the same error addressed here: >>> http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921 >>> The ''hack'' to fix this problem is already built into my >>> XendDomainInfo.py script already, so I''m a little more than lost. >>> >>> [root@xen1 ~]# xm info >>> host : xen1.fiber.net >>> release : 2.6.18-53.1.13.el5xen >>> version : #1 SMP Tue Feb 12 13:33:07 EST 2008 >>> machine : x86_64 >>> nr_cpus : 8 >>> nr_nodes : 1 >>> sockets_per_node : 2 >>> cores_per_socket : 4 >>> threads_per_core : 1 >>> cpu_mhz : 1866 >>> hw_caps : >>> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001 >>> total_memory : 16382 >>> free_memory : 2 >>> xen_major : 3 >>> xen_minor : 1 >>> xen_extra : .0-53.1.13.el5 >>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 >>> hvm-3.0-x86_32p hvm-3.0-x86_64 >>> xen_pagesize : 4096 >>> platform_params : virt_start=0xffff800000000000 >>> xen_changeset : unavailable >>> cc_compiler : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>> cc_compile_by : mockbuild >>> cc_compile_domain : >>> cc_compile_date : Tue Feb 12 12:55:35 EST 2008 >>> xend_config_format : 2 >>> >>> >>> Here''s the error in action: >>> [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image >>> Using config file "./centos-5.1_core_image". >>> Error: (12, ''Cannot allocate memory'') >>> [root@xen1 centos-5.1_core_image]# >>> >>> >>> Here is the output from xend.log that occurs when I get this error: >>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>> (XendDomainInfo:200) XendDomainInfo.create([''vm'', [''name'', >>> ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], >>> [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', >>> ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', >>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>> ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', >>> [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], >>> [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', >>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>> [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]]) >>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>> (XendDomainInfo:306) parseConfig: config is [''vm'', [''name'', >>> ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], >>> [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', >>> ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', >>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>> ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', >>> [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], >>> [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', >>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>> [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]] >>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>> (XendDomainInfo:411) parseConfig: result is {''shadow_memory'': None, >>> ''start_time'': None, ''uuid'': None, ''on_crash'': ''restart'', ''on_reboot'': >>> ''restart'', ''localtime'': None, ''image'': [''hvm'', [''kernel'', >>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>> ''/root/.Xauthority''], [''keymap'', ''en-us'']], ''on_poweroff'': ''destroy'', >>> ''bootloader_args'': None, ''cpus'': None, ''name'': ''centos-5.1_core_image'', >>> ''backend'': [], ''vcpus'': 6, ''cpu_weight'': None, ''features'': None, >>> ''vcpu_avail'': None, ''memory'': 4024, ''device'': [(''vbd'', [''vbd'', [''uname'', >>> ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], [''dev'', >>> ''hda''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', >>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>> [''mode'', ''w'']]), (''vif'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']])], ''bootloader'': None, ''cpu'': >>> None, ''maxmem'': 4024} >>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>> (XendDomainInfo:1349) XendDomainInfo.construct: None >>> [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB >>> free; need 2048; done. >>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR >>> (XendDomainInfo:212) Domain construction failed >>> Traceback (most recent call last): >>> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >>> line 204, in create >>> vm.construct() >>> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >>> line 1369, in construct >>> hvm = hvm) >>> Error: (12, ''Cannot allocate memory'') >>> >>> >>> Here''s the configuration file being used: >>> name = "centos-5.1_core_image" >>> maxmem = 4024 >>> memory = 4024 >>> vcpus = 6 >>> builder = "hvm" >>> kernel = "/usr/lib/xen/boot/hvmloader" >>> boot = "cda" >>> pae = 1 >>> acpi = 0 >>> apic = 0 >>> on_poweroff = "destroy" >>> on_reboot = "restart" >>> on_crash = "restart" >>> device_model = "/usr/lib64/xen/bin/qemu-dm" >>> sdl = 0 >>> vnc = 1 >>> vncunused = 1 >>> keymap = "en-us" >>> disk = [ ''file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w'' , >>> ''file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w'' ] >>> vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ] >>> serial = "pty" >>> >>> At the time the error occured, my memory looked like this: >>> [root@xen1 centos-5.1_core_image]# free >>> total used free shared buffers cached >>> Mem: 15845376 15821080 24296 0 113596 15081468 >>> -/+ buffers/cache: 626016 15219360 >>> Swap: 4192956 4 4192952 >>> >>> The only way I''ve been able to duplicate this error is by making >>> multiple copies of the 6GB disk images (created with dd, so ls shows 6GB >>> but du shows 2GB) for different virtual machines and booting them up. I >>> can get up to about 8 new virtual machines roughly, before I start >>> seeing my memory error. If you''re familiar with IBM''s ''xen_deploy.pl'' >>> script, I''ve modified it slightly to meet our needs, and launch our >>> systems with that tool. It''s basic functionality is to read in a xen >>> configuration file you want to use as a template, and output a modified >>> one based on arguments you pass to the scripts, and then create a floppy >>> disk image and copy the image from the template config file, and then >>> boot the new system. >>> >>> I''ve read in previous emails that this error occurs because of a bug in >>> the hvmloader, and look for an update. That email response was for a 3.0 >>> version of xen, and I''m on 3.1. >>> >>> What other information can I provide to help diagnose this problem? >>> >>> -Kai Meyer >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kai Meyer
2008-Mar-26 16:16 UTC
Re: [Xen-devel] [Xen-users] xend (12, ''Cannot allocate memory'')
Just to reiterate so I understand: You think that xend is not ''balooning'' Dom-0''s memory correctly, and your suggestion is to make modifications to both xend-config.sxp and the grub.conf files to statically set Dom-0''s memory allocation? I''ve heard people talk about grub and options you can set for the hypervisor in the grub.conf file. Do you have some documentation for me to read? For the xend-config.sxp file, I have already set: (dom0-min-mem 256) Dom-0 could survive on 256 or 512 MB of memory. I would probably set it to 512 if I''m going to statically set it for now. Thanks for the info, it''s been days since I posted to the Xen-users list, and never got a response. -Kai Meyer Keir Fraser wrote:> Depending on your usage scenario for dom0 itself, you could could turn off > auto-ballooning entirely (it can be done by modifying xend-config.sxp) and > limit dom0''s initial memory allocation on Xen''s command line (e.g., > modifying Xen boot parameters in your GRUB menu file). > > -- Keir > > On 26/3/08 15:25, "Kai Meyer" <kai@unixlords.com> wrote: > > >> Well, that''s no so great. It thinks I have barely a fraction of what I''m >> supposed to have available. Free shows 10GB available, xm info shows 1 >> MB available. If I stop a virtual machine, the memory in xminfo >> increases by the amount of memory used by that xen machine, which is to >> be expected. >> >> Is there a way to force Xen to refresh that number, or let me force an >> override? >> >> -Kai Meyer >> >> Keir Fraser wrote: >> >>> Given you are running dom0 with plenty of swap, it''s most likely that Xen >>> has run out of memory to allocate to your new domU, rather than the >>> allocation failure coming from within dom0. Probably auto-ballooning is >>> screwing up somehow (that''s the bit of xend that gives away memory back to >>> Xen to make sure it has enough memory to create your new domU). >>> >>> Sorry I don''t have a precise diagnosis for you, but I think looking at Xen''s >>> allocator is at least the right direction for you to take. ''xm info'' tells >>> you how much memory Xen thinks it has. >>> >>> -- Keir >>> >>> On 26/3/08 14:54, "Kai Meyer" <kai@unixlords.com> wrote: >>> >>> >>> >>>> I''m getting the error: >>>> xend (12, ''Cannot allocate memory'') >>>> From the command line after the Dom-0 goes through a period of heavy >>>> memory usage. I don''t have any hard numbers on how to duplicate it, but >>>> I''ll give you the steps I''ve gone through to duplicate the problem. >>>> >>>> First, some information about my setup. It''s a new setup, and has been >>>> in heavy development for the past few months. I''m new to Xen as of this >>>> project we started in January. As far as I can tell, this problem has >>>> existed since we started, and may have even happened before but we were >>>> too new to think it wasn''t our fault (which it may still be....) I''ve >>>> found the same error addressed here: >>>> http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921 >>>> The ''hack'' to fix this problem is already built into my >>>> XendDomainInfo.py script already, so I''m a little more than lost. >>>> >>>> [root@xen1 ~]# xm info >>>> host : xen1.fiber.net >>>> release : 2.6.18-53.1.13.el5xen >>>> version : #1 SMP Tue Feb 12 13:33:07 EST 2008 >>>> machine : x86_64 >>>> nr_cpus : 8 >>>> nr_nodes : 1 >>>> sockets_per_node : 2 >>>> cores_per_socket : 4 >>>> threads_per_core : 1 >>>> cpu_mhz : 1866 >>>> hw_caps : >>>> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001 >>>> total_memory : 16382 >>>> free_memory : 2 >>>> xen_major : 3 >>>> xen_minor : 1 >>>> xen_extra : .0-53.1.13.el5 >>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 >>>> hvm-3.0-x86_32p hvm-3.0-x86_64 >>>> xen_pagesize : 4096 >>>> platform_params : virt_start=0xffff800000000000 >>>> xen_changeset : unavailable >>>> cc_compiler : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>> cc_compile_by : mockbuild >>>> cc_compile_domain : >>>> cc_compile_date : Tue Feb 12 12:55:35 EST 2008 >>>> xend_config_format : 2 >>>> >>>> >>>> Here''s the error in action: >>>> [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image >>>> Using config file "./centos-5.1_core_image". >>>> Error: (12, ''Cannot allocate memory'') >>>> [root@xen1 centos-5.1_core_image]# >>>> >>>> >>>> Here is the output from xend.log that occurs when I get this error: >>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>>> (XendDomainInfo:200) XendDomainInfo.create([''vm'', [''name'', >>>> ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], >>>> [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', >>>> ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', >>>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>>> ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', >>>> [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], >>>> [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', >>>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>>> [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]]) >>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>>> (XendDomainInfo:306) parseConfig: config is [''vm'', [''name'', >>>> ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], >>>> [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', >>>> ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', >>>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>>> ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', >>>> [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], >>>> [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', >>>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>>> [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]] >>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>>> (XendDomainInfo:411) parseConfig: result is {''shadow_memory'': None, >>>> ''start_time'': None, ''uuid'': None, ''on_crash'': ''restart'', ''on_reboot'': >>>> ''restart'', ''localtime'': None, ''image'': [''hvm'', [''kernel'', >>>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>>> ''/root/.Xauthority''], [''keymap'', ''en-us'']], ''on_poweroff'': ''destroy'', >>>> ''bootloader_args'': None, ''cpus'': None, ''name'': ''centos-5.1_core_image'', >>>> ''backend'': [], ''vcpus'': 6, ''cpu_weight'': None, ''features'': None, >>>> ''vcpu_avail'': None, ''memory'': 4024, ''device'': [(''vbd'', [''vbd'', [''uname'', >>>> ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], [''dev'', >>>> ''hda''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', >>>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>>> [''mode'', ''w'']]), (''vif'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']])], ''bootloader'': None, ''cpu'': >>>> None, ''maxmem'': 4024} >>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>>> (XendDomainInfo:1349) XendDomainInfo.construct: None >>>> [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB >>>> free; need 2048; done. >>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR >>>> (XendDomainInfo:212) Domain construction failed >>>> Traceback (most recent call last): >>>> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >>>> line 204, in create >>>> vm.construct() >>>> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >>>> line 1369, in construct >>>> hvm = hvm) >>>> Error: (12, ''Cannot allocate memory'') >>>> >>>> >>>> Here''s the configuration file being used: >>>> name = "centos-5.1_core_image" >>>> maxmem = 4024 >>>> memory = 4024 >>>> vcpus = 6 >>>> builder = "hvm" >>>> kernel = "/usr/lib/xen/boot/hvmloader" >>>> boot = "cda" >>>> pae = 1 >>>> acpi = 0 >>>> apic = 0 >>>> on_poweroff = "destroy" >>>> on_reboot = "restart" >>>> on_crash = "restart" >>>> device_model = "/usr/lib64/xen/bin/qemu-dm" >>>> sdl = 0 >>>> vnc = 1 >>>> vncunused = 1 >>>> keymap = "en-us" >>>> disk = [ ''file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w'' , >>>> ''file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w'' ] >>>> vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ] >>>> serial = "pty" >>>> >>>> At the time the error occured, my memory looked like this: >>>> [root@xen1 centos-5.1_core_image]# free >>>> total used free shared buffers cached >>>> Mem: 15845376 15821080 24296 0 113596 15081468 >>>> -/+ buffers/cache: 626016 15219360 >>>> Swap: 4192956 4 4192952 >>>> >>>> The only way I''ve been able to duplicate this error is by making >>>> multiple copies of the 6GB disk images (created with dd, so ls shows 6GB >>>> but du shows 2GB) for different virtual machines and booting them up. I >>>> can get up to about 8 new virtual machines roughly, before I start >>>> seeing my memory error. If you''re familiar with IBM''s ''xen_deploy.pl'' >>>> script, I''ve modified it slightly to meet our needs, and launch our >>>> systems with that tool. It''s basic functionality is to read in a xen >>>> configuration file you want to use as a template, and output a modified >>>> one based on arguments you pass to the scripts, and then create a floppy >>>> disk image and copy the image from the template config file, and then >>>> boot the new system. >>>> >>>> I''ve read in previous emails that this error occurs because of a bug in >>>> the hvmloader, and look for an update. That email response was for a 3.0 >>>> version of xen, and I''m on 3.1. >>>> >>>> What other information can I provide to help diagnose this problem? >>>> >>>> -Kai Meyer >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >>>> >>>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >>> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Mar-26 18:04 UTC
Re: [Xen-devel] [Xen-users] xend (12, ''Cannot allocate memory'')
Yes. So, if you are happy for dom0 to always run with a static amount of memory then: Modify xend-config.sxp to have: (dom0-min-mem 0) Modify /boot/grub/menu.lst to include dom0_mem=256M (for example) on the Xen command line. You''ll now have all the memory reported as free in ''xm info'' to play with, and xend will not screw you around. -- Keir On 26/3/08 16:16, "Kai Meyer" <kai@unixlords.com> wrote:> Just to reiterate so I understand: > You think that xend is not ''balooning'' Dom-0''s memory correctly, and > your suggestion is to make modifications to both xend-config.sxp and the > grub.conf files to statically set Dom-0''s memory allocation? > > I''ve heard people talk about grub and options you can set for the > hypervisor in the grub.conf file. Do you have some documentation for me > to read? > > For the xend-config.sxp file, I have already set: > (dom0-min-mem 256) > Dom-0 could survive on 256 or 512 MB of memory. I would probably set it > to 512 if I''m going to statically set it for now. > > Thanks for the info, it''s been days since I posted to the Xen-users > list, and never got a response. > > -Kai Meyer > > Keir Fraser wrote: >> Depending on your usage scenario for dom0 itself, you could could turn off >> auto-ballooning entirely (it can be done by modifying xend-config.sxp) and >> limit dom0''s initial memory allocation on Xen''s command line (e.g., >> modifying Xen boot parameters in your GRUB menu file). >> >> -- Keir >> >> On 26/3/08 15:25, "Kai Meyer" <kai@unixlords.com> wrote: >> >> >>> Well, that''s no so great. It thinks I have barely a fraction of what I''m >>> supposed to have available. Free shows 10GB available, xm info shows 1 >>> MB available. If I stop a virtual machine, the memory in xminfo >>> increases by the amount of memory used by that xen machine, which is to >>> be expected. >>> >>> Is there a way to force Xen to refresh that number, or let me force an >>> override? >>> >>> -Kai Meyer >>> >>> Keir Fraser wrote: >>> >>>> Given you are running dom0 with plenty of swap, it''s most likely that Xen >>>> has run out of memory to allocate to your new domU, rather than the >>>> allocation failure coming from within dom0. Probably auto-ballooning is >>>> screwing up somehow (that''s the bit of xend that gives away memory back to >>>> Xen to make sure it has enough memory to create your new domU). >>>> >>>> Sorry I don''t have a precise diagnosis for you, but I think looking at >>>> Xen''s >>>> allocator is at least the right direction for you to take. ''xm info'' tells >>>> you how much memory Xen thinks it has. >>>> >>>> -- Keir >>>> >>>> On 26/3/08 14:54, "Kai Meyer" <kai@unixlords.com> wrote: >>>> >>>> >>>> >>>>> I''m getting the error: >>>>> xend (12, ''Cannot allocate memory'') >>>>> From the command line after the Dom-0 goes through a period of heavy >>>>> memory usage. I don''t have any hard numbers on how to duplicate it, but >>>>> I''ll give you the steps I''ve gone through to duplicate the problem. >>>>> >>>>> First, some information about my setup. It''s a new setup, and has been >>>>> in heavy development for the past few months. I''m new to Xen as of this >>>>> project we started in January. As far as I can tell, this problem has >>>>> existed since we started, and may have even happened before but we were >>>>> too new to think it wasn''t our fault (which it may still be....) I''ve >>>>> found the same error addressed here: >>>>> http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921 >>>>> The ''hack'' to fix this problem is already built into my >>>>> XendDomainInfo.py script already, so I''m a little more than lost. >>>>> >>>>> [root@xen1 ~]# xm info >>>>> host : xen1.fiber.net >>>>> release : 2.6.18-53.1.13.el5xen >>>>> version : #1 SMP Tue Feb 12 13:33:07 EST 2008 >>>>> machine : x86_64 >>>>> nr_cpus : 8 >>>>> nr_nodes : 1 >>>>> sockets_per_node : 2 >>>>> cores_per_socket : 4 >>>>> threads_per_core : 1 >>>>> cpu_mhz : 1866 >>>>> hw_caps : >>>>> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001 >>>>> total_memory : 16382 >>>>> free_memory : 2 >>>>> xen_major : 3 >>>>> xen_minor : 1 >>>>> xen_extra : .0-53.1.13.el5 >>>>> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 >>>>> hvm-3.0-x86_32p hvm-3.0-x86_64 >>>>> xen_pagesize : 4096 >>>>> platform_params : virt_start=0xffff800000000000 >>>>> xen_changeset : unavailable >>>>> cc_compiler : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >>>>> cc_compile_by : mockbuild >>>>> cc_compile_domain : >>>>> cc_compile_date : Tue Feb 12 12:55:35 EST 2008 >>>>> xend_config_format : 2 >>>>> >>>>> >>>>> Here''s the error in action: >>>>> [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image >>>>> Using config file "./centos-5.1_core_image". >>>>> Error: (12, ''Cannot allocate memory'') >>>>> [root@xen1 centos-5.1_core_image]# >>>>> >>>>> >>>>> Here is the output from xend.log that occurs when I get this error: >>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>>>> (XendDomainInfo:200) XendDomainInfo.create([''vm'', [''name'', >>>>> ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], >>>>> [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', >>>>> ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', >>>>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>>>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>>>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>>>> ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', >>>>> [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], >>>>> [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', >>>>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>>>> [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>>>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]]) >>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>>>> (XendDomainInfo:306) parseConfig: config is [''vm'', [''name'', >>>>> ''centos-5.1_core_image''], [''memory'', 4024], [''maxmem'', 4024], >>>>> [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''on_crash'', >>>>> ''restart''], [''vcpus'', 6], [''image'', [''hvm'', [''kernel'', >>>>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>>>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>>>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>>>> ''/root/.Xauthority''], [''keymap'', ''en-us'']]], [''device'', [''vbd'', >>>>> [''uname'', ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], >>>>> [''dev'', ''hda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', >>>>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>>>> [''mode'', ''w'']]], [''device'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>>>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']]]] >>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>>>> (XendDomainInfo:411) parseConfig: result is {''shadow_memory'': None, >>>>> ''start_time'': None, ''uuid'': None, ''on_crash'': ''restart'', ''on_reboot'': >>>>> ''restart'', ''localtime'': None, ''image'': [''hvm'', [''kernel'', >>>>> ''/usr/lib/xen/boot/hvmloader''], [''device_model'', >>>>> ''/usr/lib64/xen/bin/qemu-dm''], [''pae'', 1], [''vcpus'', 6], [''boot'', >>>>> ''cda''], [''serial'', ''pty''], [''vnc'', 1], [''vncunused'', 1], [''xauthority'', >>>>> ''/root/.Xauthority''], [''keymap'', ''en-us'']], ''on_poweroff'': ''destroy'', >>>>> ''bootloader_args'': None, ''cpus'': None, ''name'': ''centos-5.1_core_image'', >>>>> ''backend'': [], ''vcpus'': 6, ''cpu_weight'': None, ''features'': None, >>>>> ''vcpu_avail'': None, ''memory'': 4024, ''device'': [(''vbd'', [''vbd'', [''uname'', >>>>> ''file:/xen/domains2/centos-5.1_core_image/new_disk.img''], [''dev'', >>>>> ''hda''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', >>>>> ''file:/xen/domains2/centos-5.1_core_image/swap.img''], [''dev'', ''hdb''], >>>>> [''mode'', ''w'']]), (''vif'', [''vif'', [''bridge'', ''xenbr1''], [''mac'', >>>>> ''00:16:3e:4c:c0:01''], [''type'', ''ioemu'']])], ''bootloader'': None, ''cpu'': >>>>> None, ''maxmem'': 4024} >>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >>>>> (XendDomainInfo:1349) XendDomainInfo.construct: None >>>>> [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB >>>>> free; need 2048; done. >>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR >>>>> (XendDomainInfo:212) Domain construction failed >>>>> Traceback (most recent call last): >>>>> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >>>>> line 204, in create >>>>> vm.construct() >>>>> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >>>>> line 1369, in construct >>>>> hvm = hvm) >>>>> Error: (12, ''Cannot allocate memory'') >>>>> >>>>> >>>>> Here''s the configuration file being used: >>>>> name = "centos-5.1_core_image" >>>>> maxmem = 4024 >>>>> memory = 4024 >>>>> vcpus = 6 >>>>> builder = "hvm" >>>>> kernel = "/usr/lib/xen/boot/hvmloader" >>>>> boot = "cda" >>>>> pae = 1 >>>>> acpi = 0 >>>>> apic = 0 >>>>> on_poweroff = "destroy" >>>>> on_reboot = "restart" >>>>> on_crash = "restart" >>>>> device_model = "/usr/lib64/xen/bin/qemu-dm" >>>>> sdl = 0 >>>>> vnc = 1 >>>>> vncunused = 1 >>>>> keymap = "en-us" >>>>> disk = [ ''file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w'' , >>>>> ''file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w'' ] >>>>> vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ] >>>>> serial = "pty" >>>>> >>>>> At the time the error occured, my memory looked like this: >>>>> [root@xen1 centos-5.1_core_image]# free >>>>> total used free shared buffers cached >>>>> Mem: 15845376 15821080 24296 0 113596 15081468 >>>>> -/+ buffers/cache: 626016 15219360 >>>>> Swap: 4192956 4 4192952 >>>>> >>>>> The only way I''ve been able to duplicate this error is by making >>>>> multiple copies of the 6GB disk images (created with dd, so ls shows 6GB >>>>> but du shows 2GB) for different virtual machines and booting them up. I >>>>> can get up to about 8 new virtual machines roughly, before I start >>>>> seeing my memory error. If you''re familiar with IBM''s ''xen_deploy.pl'' >>>>> script, I''ve modified it slightly to meet our needs, and launch our >>>>> systems with that tool. It''s basic functionality is to read in a xen >>>>> configuration file you want to use as a template, and output a modified >>>>> one based on arguments you pass to the scripts, and then create a floppy >>>>> disk image and copy the image from the template config file, and then >>>>> boot the new system. >>>>> >>>>> I''ve read in previous emails that this error occurs because of a bug in >>>>> the hvmloader, and look for an update. That email response was for a 3.0 >>>>> version of xen, and I''m on 3.1. >>>>> >>>>> What other information can I provide to help diagnose this problem? >>>>> >>>>> -Kai Meyer >>>>> >>>>> _______________________________________________ >>>>> Xen-devel mailing list >>>>> Xen-devel@lists.xensource.com >>>>> http://lists.xensource.com/xen-devel >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >>>> >>>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >> >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Possibly Parallel Threads
- xend (12, ''Cannot allocate memory'')
- Live migration and DRBD
- xm create: Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
- Error: (22, ''Invalid argument'')
- Re: xm create: Error: Device 0 (vif) could not be connected. Hotplug scripts not working. One more question.