thr3ads.net - CentOS virt - [CentOS-virt] Cannot allocate Memory [Jun 2016]

If this information is useful, please help other people find it:
Share via:

Shaun Reitan

2016-Jun-22 17:45 UTC

[CentOS-virt] Cannot allocate Memory

Any of you guys ever seen an issue with Xen 4.4 were xm cannot create a 
guest because of what looks like an issue allocating memory even though 
xm info shows like 5x the amount of free memory needed? We are still 
unfortunately still using xm... it's on my list, i know..

We've had this happen on a couple hosts now.  Only way to resolve seams 
to be rebooting the host.  I'm going to update the host to latest Xen 
4.4 now hoping this is a old bug.


Here's from xen logs

[2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:105) 
XendDomainInfo.create(['vm', ['name', 'xxx'],
['memory', 2048],
['on_xend_start', 'ignore'], ['on_xend_stop',
'ignore'], ['vcpus', 2],
['oos', 1], ['image', ['linux', ['kernel', 
'/kernels/vmlinux-2.6.18.8-4'], ['videoram', 4],
['args',
'root=/dev/xvda ro xencons=tty console=tty1 '], ['tsc_mode', 0],
['nomigrate', 0]]], ['s3_integrity', 1], ['device',
['vbd', ['uname',
'phy:vg/fs_6818'], ['dev', 'xvda'], ['mode',
'w']]], ['device', ['vbd',
['uname', 'phy:vg/fs_6819'], ['dev', 'xvdb'],
['mode', 'w']]],
['device', ['vif', ['rate', '40mb/s'],
['mac', 'FE:FD:48:01:F1:E7']]]])
[2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:2504) 
XendDomainInfo.constructDomain
[2016-06-22 09:13:50 1958] DEBUG (balloon:187) Balloon: 7602632 KiB 
free; need 16384; done.
[2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:2566) (12, 'Cannot 
allocate memory')
Traceback (most recent call last):
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 2561, in _constructDomain
     target = self.info.target())
Error: (12, 'Cannot allocate memory')
[2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:490) VM start failed
Traceback (most recent call last):
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 475, in start
     XendTask.log_progress(0, 30, self._constructDomain)
   File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
line
209, in log_progress
     retval = func(*args, **kwds)
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 2572, in _constructDomain
     raise VmError(failmsg)
VmError: Creating domain failed: name=xxx
[2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:110) Domain 
construction failed
Traceback (most recent call last):
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 108, in create
     vm.start()
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 475, in start
     XendTask.log_progress(0, 30, self._constructDomain)
   File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
line
209, in log_progress
     retval = func(*args, **kwds)
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 2572, in _constructDomain
     raise VmError(failmsg)
VmError: Creating domain failed: name=xxx

--
Shaun
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20160622/3c6df3fd/attachment-0006.html>

George Dunlap

2016-Jun-24 11:28 UTC

head link

[CentOS-virt] Cannot allocate Memory

On Wed, Jun 22, 2016 at 6:45 PM, Shaun Reitan <shaun.reitan at
ndchost.com> wrote:> Any of you guys ever seen an issue with Xen 4.4 were xm cannot create a
> guest because of what looks like an issue allocating memory even though xm
> info shows like 5x the amount of free memory needed? We are still
> unfortunately still using xm... it's on my list, i know..
>
> We've had this happen on a couple hosts now.  Only way to resolve seams
to
> be rebooting the host.  I'm going to update the host to latest Xen 4.4
now
> hoping this is a old bug.
xend hasn't had much love in years, so it's fairly unlikely that this
has been fixed.
> Here's from xen logs
>
> [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:105)
> XendDomainInfo.create(['vm', ['name', 'xxx'],
['memory', 2048],
> ['on_xend_start', 'ignore'], ['on_xend_stop',
'ignore'], ['vcpus', 2],
> ['oos', 1], ['image', ['linux', ['kernel',
'/kernels/vmlinux-2.6.18.8-4'],
> ['videoram', 4], ['args', 'root=/dev/xvda ro
xencons=tty console=tty1 '],
> ['tsc_mode', 0], ['nomigrate', 0]]],
['s3_integrity', 1], ['device', ['vbd',
> ['uname', 'phy:vg/fs_6818'], ['dev',
'xvda'], ['mode', 'w']]], ['device',
> ['vbd', ['uname', 'phy:vg/fs_6819'],
['dev', 'xvdb'], ['mode', 'w']]],
> ['device', ['vif', ['rate', '40mb/s'],
['mac', 'FE:FD:48:01:F1:E7']]]])
> [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:2504)
> XendDomainInfo.constructDomain
> [2016-06-22 09:13:50 1958] DEBUG (balloon:187) Balloon: 7602632 KiB free;
> need 16384; done.
> [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:2566) (12, 'Cannot
allocate
> memory')
> Traceback (most recent call last):
>   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line
> 2561, in _constructDomain
>     target = self.info.target())
> Error: (12, 'Cannot allocate memory')
> [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:490) VM start failed
> Traceback (most recent call last):
>   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line
> 475, in start
>     XendTask.log_progress(0, 30, self._constructDomain)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
line 209,
> in log_progress
>     retval = func(*args, **kwds)
>   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line
> 2572, in _constructDomain
>     raise VmError(failmsg)
> VmError: Creating domain failed: name=xxx
> [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:110) Domain construction
> failed
> Traceback (most recent call last):
>   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line
> 108, in create
>     vm.start()
>   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line
> 475, in start
>     XendTask.log_progress(0, 30, self._constructDomain)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
line 209,
> in log_progress
>     retval = func(*args, **kwds)
>   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line
> 2572, in _constructDomain
>     raise VmError(failmsg)
> VmError: Creating domain failed: name=xxx
If you haven't decided to switch to xl, would you mind reposting this
question to xen-users?  If you do, please also include the output of
"xl info" after the failure.

The logs tell us that xend is asking dom0 to free up some memory to
use to create the guest.  My guess is that there's a slight mismatch
between how much memory xend things needs to be freed and how much
memory actually needs freeing.

Thanks,
 -George

Pasi Kärkkäinen

2016-Jun-26 16:30 UTC

head link

[CentOS-virt] Cannot allocate Memory

On Wed, Jun 22, 2016 at 05:45:05PM +0000, Shaun Reitan
wrote:>    Any of you guys ever seen an issue with Xen 4.4 were xm cannot create a
>    guest because of what looks like an issue allocating memory even though
xm
>    info shows like 5x the amount of free memory needed? We are still
>    unfortunately still using xm... it's on my list, i know..
>    We've had this happen on a couple hosts now.  Only way to resolve
seams to
>    be rebooting the host.  I'm going to update the host to latest Xen
4.4 now
>    hoping this is a old bug.
>    Here's from xen logs
>    [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:105)
>    XendDomainInfo.create(['vm', ['name', 'xxx'],
['memory', 2048],
>    ['on_xend_start', 'ignore'], ['on_xend_stop',
'ignore'], ['vcpus', 2],
>    ['oos', 1], ['image', ['linux',
['kernel', '/kernels/vmlinux-2.6.18.8-4'],
>    ['videoram', 4], ['args', 'root=/dev/xvda ro
xencons=tty console=tty1 '],
>    ['tsc_mode', 0], ['nomigrate', 0]]],
['s3_integrity', 1], ['device',
>    ['vbd', ['uname', 'phy:vg/fs_6818'],
['dev', 'xvda'], ['mode', 'w']]],
>    ['device', ['vbd', ['uname',
'phy:vg/fs_6819'], ['dev', 'xvdb'], ['mode',
>    'w']]], ['device', ['vif', ['rate',
'40mb/s'], ['mac',
>    'FE:FD:48:01:F1:E7']]]])
>    [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:2504)
>    XendDomainInfo.constructDomain
>    [2016-06-22 09:13:50 1958] DEBUG (balloon:187) Balloon: 7602632 KiB
free;
>    need 16384; done.
>
Hmm.. so you're using (dom0) memory ballooning? You might want to disable
it.
Things worked better without ballooning (at least with xend).


-- Pasi
>    [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:2566) (12, 'Cannot
>    allocate memory')
>    Traceback (most recent call last):
>      File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>    line 2561, in _constructDomain
>        target = self.info.target())
>    Error: (12, 'Cannot allocate memory')
>    [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:490) VM start failed
>    Traceback (most recent call last):
>      File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>    line 475, in start
>        XendTask.log_progress(0, 30, self._constructDomain)
>      File
"/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line
>    209, in log_progress
>        retval = func(*args, **kwds)
>      File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>    line 2572, in _constructDomain
>        raise VmError(failmsg)
>    VmError: Creating domain failed: name=xxx
>    [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:110) Domain
construction
>    failed
>    Traceback (most recent call last):
>      File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>    line 108, in create
>        vm.start()
>      File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>    line 475, in start
>        XendTask.log_progress(0, 30, self._constructDomain)
>      File
"/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line
>    209, in log_progress
>        retval = func(*args, **kwds)
>      File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>    line 2572, in _constructDomain
>        raise VmError(failmsg)
>    VmError: Creating domain failed: name=xxx
>    --
>    Shaun

Shaun Reitan

2016-Jun-30 23:23 UTC

head link

[CentOS-virt] Cannot allocate Memory

We are planing on moving to xl but have not done so yet.

Our xend configs are untouched so they should match what they were when 
the xen-* packages were installed.

We force dom0 to 1024MB of memory via grub.

On a couple servers were we've seen this issue we've had 16+ GB of 
memory free and the domU was requesting something small like 2GB.

--
Shaun



------ Original Message ------
From: "George Dunlap" <dunlapg at umich.edu>
To: "Shaun Reitan" <shaun.reitan at ndchost.com>;
"Discussion about the
virtualization on CentOS" <centos-virt at centos.org>
Sent: 6/24/2016 4:28:53 AM
Subject: Re: [CentOS-virt] Cannot allocate Memory
>On Wed, Jun 22, 2016 at 6:45 PM, Shaun Reitan 
><shaun.reitan at ndchost.com> wrote:
>>  Any of you guys ever seen an issue with Xen 4.4 were xm cannot create 
>>a
>>  guest because of what looks like an issue allocating memory even 
>>though xm
>>  info shows like 5x the amount of free memory needed? We are still
>>  unfortunately still using xm... it's on my list, i know..
>>
>>  We've had this happen on a couple hosts now.  Only way to resolve 
>>seams to
>>  be rebooting the host.  I'm going to update the host to latest Xen
>>4.4 now
>>  hoping this is a old bug.
>
>xend hasn't had much love in years, so it's fairly unlikely that
this
>has been fixed.
>
>>  Here's from xen logs
>>
>>  [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:105)
>>  XendDomainInfo.create(['vm', ['name', 'xxx'],
['memory', 2048],
>>  ['on_xend_start', 'ignore'], ['on_xend_stop',
'ignore'], ['vcpus',
>>2],
>>  ['oos', 1], ['image', ['linux',
['kernel',
>>'/kernels/vmlinux-2.6.18.8-4'],
>>  ['videoram', 4], ['args', 'root=/dev/xvda ro
xencons=tty console=tty1
>>'],
>>  ['tsc_mode', 0], ['nomigrate', 0]]],
['s3_integrity', 1], ['device',
>>['vbd',
>>  ['uname', 'phy:vg/fs_6818'], ['dev',
'xvda'], ['mode', 'w']]],
>>['device',
>>  ['vbd', ['uname', 'phy:vg/fs_6819'],
['dev', 'xvdb'], ['mode',
>>'w']]],
>>  ['device', ['vif', ['rate', '40mb/s'],
['mac',
>>'FE:FD:48:01:F1:E7']]]])
>>  [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:2504)
>>  XendDomainInfo.constructDomain
>>  [2016-06-22 09:13:50 1958] DEBUG (balloon:187) Balloon: 7602632 KiB 
>>free;
>>  need 16384; done.
>>  [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:2566) (12,
'Cannot
>>allocate
>>  memory')
>>  Traceback (most recent call last):
>>    File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line
>>  2561, in _constructDomain
>>      target = self.info.target())
>>  Error: (12, 'Cannot allocate memory')
>>  [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:490) VM start failed
>>  Traceback (most recent call last):
>>    File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line
>>  475, in start
>>      XendTask.log_progress(0, 30, self._constructDomain)
>>    File
"/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
>>line 209,
>>  in log_progress
>>      retval = func(*args, **kwds)
>>    File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line
>>  2572, in _constructDomain
>>      raise VmError(failmsg)
>>  VmError: Creating domain failed: name=xxx
>>  [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:110) Domain 
>>construction
>>  failed
>>  Traceback (most recent call last):
>>    File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line
>>  108, in create
>>      vm.start()
>>    File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line
>>  475, in start
>>      XendTask.log_progress(0, 30, self._constructDomain)
>>    File
"/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
>>line 209,
>>  in log_progress
>>      retval = func(*args, **kwds)
>>    File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line
>>  2572, in _constructDomain
>>      raise VmError(failmsg)
>>  VmError: Creating domain failed: name=xxx
>
>If you haven't decided to switch to xl, would you mind reposting this
>question to xen-users?  If you do, please also include the output of
>"xl info" after the failure.
>
>The logs tell us that xend is asking dom0 to free up some memory to
>use to create the guest.  My guess is that there's a slight mismatch
>between how much memory xend things needs to be freed and how much
>memory actually needs freeing.
>
>Thanks,
>  -George

Shaun Reitan

2016-Jun-30 23:24 UTC

head link

[CentOS-virt] Cannot allocate Memory

We're using the stock configuration, the dom0 is limited to 1024MB via 
grub.



------ Original Message ------
From: "Pasi K?rkk?inen" <pasik at iki.fi>
To: "Shaun Reitan" <shaun.reitan at ndchost.com>;
"Discussion about the
virtualization on CentOS" <centos-virt at centos.org>
Sent: 6/26/2016 9:30:57 AM
Subject: Re: [CentOS-virt] Cannot allocate Memory
>On Wed, Jun 22, 2016 at 05:45:05PM +0000, Shaun Reitan wrote:
>>     Any of you guys ever seen an issue with Xen 4.4 were xm cannot 
>>create a
>>     guest because of what looks like an issue allocating memory even 
>>though xm
>>     info shows like 5x the amount of free memory needed? We are still
>>     unfortunately still using xm... it's on my list, i know..
>>     We've had this happen on a couple hosts now.  Only way to
resolve
>>seams to
>>     be rebooting the host.  I'm going to update the host to latest
Xen
>>4.4 now
>>     hoping this is a old bug.
>>     Here's from xen logs
>>     [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:105)
>>     XendDomainInfo.create(['vm', ['name',
'xxx'], ['memory', 2048],
>>     ['on_xend_start', 'ignore'],
['on_xend_stop', 'ignore'], ['vcpus',
>>2],
>>     ['oos', 1], ['image', ['linux',
['kernel',
>>'/kernels/vmlinux-2.6.18.8-4'],
>>     ['videoram', 4], ['args', 'root=/dev/xvda ro
xencons=tty
>>console=tty1 '],
>>     ['tsc_mode', 0], ['nomigrate', 0]]],
['s3_integrity', 1],
>>['device',
>>     ['vbd', ['uname', 'phy:vg/fs_6818'],
['dev', 'xvda'], ['mode',
>>'w']]],
>>     ['device', ['vbd', ['uname',
'phy:vg/fs_6819'], ['dev', 'xvdb'],
>>['mode',
>>     'w']]], ['device', ['vif', ['rate',
'40mb/s'], ['mac',
>>     'FE:FD:48:01:F1:E7']]]])
>>     [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:2504)
>>     XendDomainInfo.constructDomain
>>     [2016-06-22 09:13:50 1958] DEBUG (balloon:187) Balloon: 7602632 
>>KiB free;
>>     need 16384; done.
>>
>
>Hmm.. so you're using (dom0) memory ballooning? You might want to 
>disable it.
>Things worked better without ballooning (at least with xend).
>
>
>-- Pasi
>
>>     [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:2566) (12, 
>>'Cannot
>>     allocate memory')
>>     Traceback (most recent call last):
>>       File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>>     line 2561, in _constructDomain
>>         target = self.info.target())
>>     Error: (12, 'Cannot allocate memory')
>>     [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:490) VM start 
>>failed
>>     Traceback (most recent call last):
>>       File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>>     line 475, in start
>>         XendTask.log_progress(0, 30, self._constructDomain)
>>       File
"/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
>>line
>>     209, in log_progress
>>         retval = func(*args, **kwds)
>>       File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>>     line 2572, in _constructDomain
>>         raise VmError(failmsg)
>>     VmError: Creating domain failed: name=xxx
>>     [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:110) Domain 
>>construction
>>     failed
>>     Traceback (most recent call last):
>>       File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>>     line 108, in create
>>         vm.start()
>>       File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>>     line 475, in start
>>         XendTask.log_progress(0, 30, self._constructDomain)
>>       File
"/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
>>line
>>     209, in log_progress
>>         retval = func(*args, **kwds)
>>       File 
>>"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
>>     line 2572, in _constructDomain
>>         raise VmError(failmsg)
>>     VmError: Creating domain failed: name=xxx
>>     --
>>     Shaun
>

Maybe Matching Threads

Search for more maybe matching threads

CentOS virt - Jun 2016 - Cannot allocate Memory

[CentOS-virt] Cannot allocate Memory

[CentOS-virt] Cannot allocate Memory

[CentOS-virt] Cannot allocate Memory

[CentOS-virt] Cannot allocate Memory

[CentOS-virt] Cannot allocate Memory

Maybe Matching Threads