Xen users - Apr 2008 - xen dom0 32 bits memory is limited to 16GB?

Hi all,
I''m reading that xen dom0 32 bits is limited to 16GB of ram (see
http://www.centos.org/modules/newbb/viewtopic.php?topic_id=12690)

Is it true? How can I remove this limit?

Thanks

Marco

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Marco Strullato wrote:
> Hi all,
> I''m reading that xen dom0 32 bits is limited to 16GB of ram (see
> http://www.centos.org/modules/newbb/viewtopic.php?topic_id=12690)
>
> Is it true? How can I remove this limit?
>
> Thanks
>
> Marco
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>   
That''s a 32bit limit, not an OS limit. The only way around this is to 
use 64bit a OS :)

-- 

Kind Regards
Rudi Ahlers
CEO, SoftDux

Web:   http://www.SoftDux.com
Check out my technical blog, http://blog.softdux.com for Linux or other
technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting stuff


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I don''t think so: the linux kernel supports pae and pae can map up to
64 GB of ram: read here
http://en.wikipedia.org/wiki/Physical_Address_Extension
Moreover, if you build the linux kernel you''ll find the same: into
processor types and features the high memory supports can map to 64
GB.



Marco

2008/4/29 Rudi Ahlers <Rudi@softdux.com>:
> Marco Strullato wrote:
>
> >
> >
> >
> > Hi all,
> > I''m reading that xen dom0 32 bits is limited to 16GB of ram
(see
> > http://www.centos.org/modules/newbb/viewtopic.php?topic_id=12690)
> >
> > Is it true? How can I remove this limit?
> >
> > Thanks
> >
> > Marco
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
> >
> >
> >
>  That''s a 32bit limit, not an OS limit. The only way around this
is to use
> 64bit a OS :)
>
>  --
>
>  Kind Regards
>  Rudi Ahlers
>  CEO, SoftDux
>
>  Web:   http://www.SoftDux.com
>  Check out my technical blog, http://blog.softdux.com for Linux or other
> technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting
> stuff
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

That is limited due to 32 bit OS

On Tue, Apr 29, 2008 at 4:38 PM, Marco Strullato
<marco.strullato@gmail.com>
wrote:
> Hi all,
> I''m reading that xen dom0 32 bits is limited to 16GB of ram (see
> http://www.centos.org/modules/newbb/viewtopic.php?topic_id=12690)
>
> Is it true? How can I remove this limit?
>
> Thanks
>
> Marco
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>


-- 
Regards--
Rishi Pathak
National PARAM Supercomputing Facility
Center for Development of Advanced Computing(C-DAC)
Pune University Campus,Ganesh Khind Road
Pune-Maharastra


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Marco Strullato wrote:
> I don''t think so: the linux kernel supports pae and pae can map up
to
> 64 GB of ram: read here
> http://en.wikipedia.org/wiki/Physical_Address_Extension
> Moreover, if you build the linux kernel you''ll find the same: into
> processor types and features the high memory supports can map to 64
> GB.
>
>
>
> Marco
>
> 2008/4/29 Rudi Ahlers <Rudi@softdux.com>:
>   
>> Marco Strullato wrote:
>>
>>     
>>>
>>> Hi all,
>>> I''m reading that xen dom0 32 bits is limited to 16GB of
ram (see
>>> http://www.centos.org/modules/newbb/viewtopic.php?topic_id=12690)
>>>
>>> Is it true? How can I remove this limit?
>>>
>>> Thanks
>>>
>>> Marco
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> Xen-users@lists.xensource.com
>>> http://lists.xensource.com/xen-users
>>>
>>>
>>>
>>>       
>>  That''s a 32bit limit, not an OS limit. The only way around
this is to use
>> 64bit a OS :)
>>
>>  --
>>
>>  Kind Regards
>>  Rudi Ahlers
>>  CEO, SoftDux
>>
>>  Web:   http://www.SoftDux.com
>>  Check out my technical blog, http://blog.softdux.com for Linux or
other
>> technical stuff, or visit http://www.WebHostingTalk.co.za for Web
Hosting
>> stuff
>>
>>
>>     
Ok, so I answered before I checked the facts :)

This could be related to grub, as per this page: 
https://bugzilla.redhat.com/show_bug.cgi?id=250299#c5


I know I had the same problem when upgrading from 2GB to 4GB RAM. SO, I 
just installed the updated grub rpm, and reinstalled it on the HDD as such:

[root@venus ~]# history | grep grub
  489  vi /etc/grub.conf
  789  vi /boot/grub/menu.lst
  881  history  | grep grub
  903  grub-install /dev/sda
  905  more /boot/grub/menu.lst
  912  rpm -Uvh grub-0.97-14.x86_64.rpm
  913  rpm -e grub-0.97-14.x86_64
  914  rpm -Uvh grub-0.97-14.x86_64.rpm
  915  grub-install /dev/sda


The last 2 (or 4) options are what you''re looking for. It updates the 
grub RPM, and then reinstalls grub on the boot partition of /dev/sda




-- 

Kind Regards
Rudi Ahlers
CEO, SoftDux

Web:   http://www.SoftDux.com
Check out my technical blog, http://blog.softdux.com for Linux or other
technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting stuff


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Apr 29, 2008 at 7:08 AM, Marco Strullato
<marco.strullato@gmail.com> wrote:
> Hi all,
>  I''m reading that xen dom0 32 bits is limited to 16GB of ram (see
>  http://www.centos.org/modules/newbb/viewtopic.php?topic_id=12690)
>
>  Is it true? How can I remove this limit?
The ultimate limit should be 64GB using PAE enabled kernels.  I''ve
done this before, but there are some things you should know...

First, I''ve found that ultimately, you really should limit dom0 to
about 1-4GB or so.  The main purpose for dom0 is to handle all the
domUs.  I''ve run into many conditions before where the code that
balloons dom0 memory to free RAM for domUs can fail, causing the
machine to become unstable or to crash, or on the less catastrophic
end, simply cause Xen to stop allocating RAM to domUs.

Second,  I''ve run 32bit Xen w/ 32bit OSs and I have never seen them
stable at 64GB in any case.  Normally, I can get to about 48GB (I
think that''s the support limit for Red Hat Enterprise and SLES).  So
anything more than, say 32GB to be safe should probably be limited to
48-54GB max.  But that''s just been my experience with Xen on large
systems.

another thing you need to consider, if your system is NUMA capable, is
to enable NUMA (numa=on) to help with performance, especially on
multi-core large memory configurations.

Cheers
Jeff

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi all,

I''ve just downloaded the last grub from cvs, built and set up with the
option that allows me to specify the "mem" parameter. So I set this
parameter to 25G or 25600.
After boot I confirm that nothing has changed The system "see" only
16GB of ram.

So I excluded the boot loader, the kernel  that is "pae" with high
memory support. So the other part is xen. I believe there''s a limit
somewhere so that a 32 bit dom0 can handle a maximun of 16GB.... but
where? I want to avoid this limit!

Thanks

Marco

2008/4/29 Rudi Ahlers <Rudi@softdux.com>:
>
> Marco Strullato wrote:
>
> > I don''t think so: the linux kernel supports pae and pae can
map up to
> > 64 GB of ram: read here
> > http://en.wikipedia.org/wiki/Physical_Address_Extension
> > Moreover, if you build the linux kernel you''ll find the same:
into
> > processor types and features the high memory supports can map to 64
> > GB.
> >
> >
> >
> > Marco
> >
> > 2008/4/29 Rudi Ahlers <Rudi@softdux.com>:
> >
> >
> > > Marco Strullato wrote:
> > >
> > >
> > >
> > > >
> > > > Hi all,
> > > > I''m reading that xen dom0 32 bits is limited to
16GB of ram (see
> > > >
http://www.centos.org/modules/newbb/viewtopic.php?topic_id=12690)
> > > >
> > > > Is it true? How can I remove this limit?
> > > >
> > > > Thanks
> > > >
> > > > Marco
> > > >
> > > > _______________________________________________
> > > > Xen-users mailing list
> > > > Xen-users@lists.xensource.com
> > > > http://lists.xensource.com/xen-users
> > > >
> > > >
> > > >
> > > >
> > > >
> > >  That''s a 32bit limit, not an OS limit. The only way
around this is to
> use
> > > 64bit a OS :)
> > >
> > >  --
> > >
> > >  Kind Regards
> > >  Rudi Ahlers
> > >  CEO, SoftDux
> > >
> > >  Web:   http://www.SoftDux.com
> > >  Check out my technical blog, http://blog.softdux.com for Linux
or other
> > > technical stuff, or visit http://www.WebHostingTalk.co.za for Web
> Hosting
> > > stuff
> > >
> > >
> > >
> > >
> >
>  Ok, so I answered before I checked the facts :)
>
>  This could be related to grub, as per this page:
> https://bugzilla.redhat.com/show_bug.cgi?id=250299#c5
>
>
>  I know I had the same problem when upgrading from 2GB to 4GB RAM. SO, I
> just installed the updated grub rpm, and reinstalled it on the HDD as such:
>
>  [root@venus ~]# history | grep grub
>   489  vi /etc/grub.conf
>   789  vi /boot/grub/menu.lst
>   881  history  | grep grub
>   903  grub-install /dev/sda
>   905  more /boot/grub/menu.lst
>   912  rpm -Uvh grub-0.97-14.x86_64.rpm
>   913  rpm -e grub-0.97-14.x86_64
>   914  rpm -Uvh grub-0.97-14.x86_64.rpm
>   915  grub-install /dev/sda
>
>
>  The last 2 (or 4) options are what you''re looking for. It updates
the grub
> RPM, and then reinstalls grub on the boot partition of /dev/sda
>
>
>
>
>  --
>
>
>  Kind Regards
>  Rudi Ahlers
>  CEO, SoftDux
>
>  Web:   http://www.SoftDux.com
>  Check out my technical blog, http://blog.softdux.com for Linux or other
> technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting
> stuff
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Apr 29, 2008 at 01:08:47PM +0200, Marco Strullato wrote:
Hi,
> I''m reading that xen dom0 32 bits is limited to 16GB of ram (see
> http://www.centos.org/modules/newbb/viewtopic.php?topic_id=12690)
> 
> Is it true? How can I remove this limit?
Yes, that''s true. 64-bits Xen hypervisor doesn''t have this
limit.

You should use:
1) 64bits Xen + 64bits dom0 kernel, userland 64- or 32-bits;
2) 64bits Xen + 32bits dom0 kernel, userland 32bits

Second option requires quite recent Xen and dom0 kernel, but I don''t
know exact versions.

Regards,
Kupson
-- 
Great software without the knowledge to run it is pretty useless.
(Linux Gazette #1)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>  You should use:
>  1) 64bits Xen + 64bits dom0 kernel, userland 64- or 32-bits;
>  2) 64bits Xen + 32bits dom0 kernel, userland 32bits
So you suggest me to install a 64 bit xen version into a 32 bit dom0?
I think it won''t run but I''ll try.

Thanks

Marco

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Marco Strullato wrote:
> Hi all,
>
> I''ve just downloaded the last grub from cvs, built and set up with
the
> option that allows me to specify the "mem" parameter. So I set
this
> parameter to 25G or 25600.
> After boot I confirm that nothing has changed The system "see"
only 16GB of ram.
>
> So I excluded the boot loader, the kernel  that is "pae" with
high
> memory support. So the other part is xen. I believe there''s a
limit
> somewhere so that a 32 bit dom0 can handle a maximun of 16GB.... but
> where? I want to avoid this limit!
>
> Thanks
>
> Marco
>
> 2008/4/29 Rudi Ahlers <Rudi@softdux.com>:
>   
>
rpm -Uvh grub-0.97-14.x86_64.rpm
grub-install /dev/sda


That''s all you need todo

-- 

Kind Regards
Rudi Ahlers
CEO, SoftDux

Web:   http://www.SoftDux.com
Check out my technical blog, http://blog.softdux.com for Linux or other
technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting stuff


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Unluckly this is not enough: I''ve just removed grub and reinstalled
unsuccessfully.

Thanks

Marco

2008/4/29 Rudi Ahlers <Rudi@softdux.com>:
> Marco Strullato wrote:
>
> > Hi all,
> >
> >
> > I''ve just downloaded the last grub from cvs, built and set up
with the
> > option that allows me to specify the "mem" parameter. So I
set this
> > parameter to 25G or 25600.
> > After boot I confirm that nothing has changed The system
"see" only 16GB
> of ram.
> >
> > So I excluded the boot loader, the kernel  that is "pae"
with high
> > memory support. So the other part is xen. I believe there''s a
limit
> > somewhere so that a 32 bit dom0 can handle a maximun of 16GB.... but
> > where? I want to avoid this limit!
> >
> > Thanks
> >
> > Marco
> >
> > 2008/4/29 Rudi Ahlers <Rudi@softdux.com>:
> >
> >
>
>  rpm -Uvh grub-0.97-14.x86_64.rpm
>  grub-install /dev/sda
>
>
>  That''s all you need todo
>
>  --
>
>
>  Kind Regards
>  Rudi Ahlers
>  CEO, SoftDux
>
>  Web:   http://www.SoftDux.com
>  Check out my technical blog, http://blog.softdux.com for Linux or other
> technical stuff, or visit http://www.WebHostingTalk.co.za for Web Hosting
> stuff
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tried. Dont'' work (obviously)

Marco

2008/4/29 Marco Strullato
<marco.strullato@gmail.com>:
> >  You should use:
>  >  1) 64bits Xen + 64bits dom0 kernel, userland 64- or 32-bits;
>  >  2) 64bits Xen + 32bits dom0 kernel, userland 32bits
>
>  So you suggest me to install a 64 bit xen version into a 32 bit dom0?
>  I think it won''t run but I''ll try.
>
>  Thanks
>
>  Marco
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Apr 29, 2008 at 03:49:39PM +0200, Marco Strullato wrote:
Hi,
> > >  You should use:
> >  >  1) 64bits Xen + 64bits dom0 kernel, userland 64- or 32-bits;
> >  >  2) 64bits Xen + 32bits dom0 kernel, userland 32bits
> >  So you suggest me to install a 64 bit xen version into a 32 bit dom0?
> >  I think it won''t run but I''ll try.
> Tried. Dont'' work (obviously)
Did you as well test first solution?

What Xen & dom0 kernel version have you tried?

Kupson
-- 
Great software without the knowledge to run it is pretty useless.
(Linux Gazette #1)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I've already tried the first solution but it has a big issue: with a
64 bit configuration (dom0 @64bits and kernel @64 bits, xen 3.2 built
from the src.rpm from the xen web site) I can not save nor migrate
domUs running at 32 bits.
I get this error:

[2008-04-28 14:00:16 2283] ERROR (XendCheckpoint:141) Save failed on
domain TEST (2).
Traceback (most recent call last):
  File
"/usr/lib64/python2.5/site-packages/xen/xend/XendCheckpoint.py",
line 109, in save
    forkHelper(cmd, fd, saveInputHandler, False)
  File
"/usr/lib64/python2.5/site-packages/xen/xend/XendCheckpoint.py",
line 351, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 4 2 0 0 0 failed

This is probably due to the different architecture (32 and 64 bits)
from the dom0 to domUs: I suppose, although the documentation says the
opposite, that xen dom0 @64 bits can not handle correctly many
operations. I mean, with 54 bits dom0 I can start/stop a 32 bit domU
but I can not use a very big number of features.

See the faq:
http://wiki.xensource.com/xenwiki/XenFaq#head-5f7176b3909cb0382cece43a6a8fc25a3a114e93

Marco

2008/4/29 Rafał Kupka
<rkupka+Listy.Xen@pronet.com.pl>:
> On Tue, Apr 29, 2008 at 03:49:39PM +0200, Marco Strullato wrote:
>  Hi,
>
> > > >  You should use:
>  > >  >  1) 64bits Xen + 64bits dom0 kernel, userland 64- or
32-bits;
>  > >  >  2) 64bits Xen + 32bits dom0 kernel, userland 32bits
>
>  > >  So you suggest me to install a 64 bit xen version into a 32 bit
dom0?
>  > >  I think it won't run but I'll try.
>
>  > Tried. Dont' work (obviously)
>
>  Did you as well test first solution?
>
>  What Xen & dom0 kernel version have you tried?
>
>
>
>  Kupson
>  --
>  Great software without the knowledge to run it is pretty useless.
>  (Linux Gazette #1)
>
>  _______________________________________________
>  Xen-users mailing list
>  Xen-users@lists.xensource.com
>  http://lists.xensource.com/xen-users
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Marco Strullato wrote:
> 
> I''ve already tried the first solution but it has a big issue: with
a
> 64 bit configuration (dom0 @64bits and kernel @64 bits, xen 3.2 built
> from the src.rpm from the xen web site) I can not save nor migrate
> domUs running at 32 bits.
> I get this error:
> 
> [2008-04-28 14:00:16 2283] ERROR (XendCheckpoint:141) Save failed on
> domain TEST (2).
> Traceback (most recent call last):
>   File 
> "/usr/lib64/python2.5/site-packages/xen/xend/XendCheckpoint.py",
> line 109, in save
>     forkHelper(cmd, fd, saveInputHandler, False)
>   File 
> "/usr/lib64/python2.5/site-packages/xen/xend/XendCheckpoint.py",
> line 351, in forkHelper
>     raise XendError("%s failed" % string.join(cmd))
> XendError: /usr/lib64/xen/bin/xc_save 4 2 0 0 0 failed
That shouldn''t happen. Maybe a bug in the latest Xen? Maybe a
misconfiguration in dom0?
> This is probably due to the different architecture (32 and 64 bits)
> from the dom0 to domUs: I suppose, although the documentation says the
> opposite, that xen dom0 @64 bits can not handle correctly many
> operations. I mean, with 54 bits dom0 I can start/stop a 32 bit domU
> but I can not use a very big number of features.
> 
> See the faq: 
>
http://wiki.xensource.com/xenwiki/XenFaq#head-5f7176b3909cb0382cece43a6a8fc25a3a114e93
The wiki page describes the compatibility matrix, but doesn''t mention
that there is a limited feature set.

I can verify that there is a problem with Xen 3.2 running the 64-bit
hypervisor with a 32-bit dom0, but that''s the only problem I have
encountered so far.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>  That shouldn''t happen. Maybe a bug in the latest Xen? Maybe a
>  misconfiguration in dom0?
I believe this is a bug: I''ll send a bug to the bugtraq system in a
few time. However I think it''s not a misconfiguration of the dom0:
I''ve verified the problem with xen 3.1 on fc8 64 bit, xen 3.2 on fc8
64 bit, xen 3.2 in centos 5 64 bit.

>  The wiki page describes the compatibility matrix, but doesn''t
mention
>  that there is a limited feature set.
I suggest the xen manteiner to update the compatibility matrix  with
each function..

Thanks

Marco

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> >  >  1) 64bits Xen + 64bits dom0 kernel, userland 64- or 32-bits;
> >  >  2) 64bits Xen + 32bits dom0 kernel, userland 32bits
> >
> >  So you suggest me to install a 64 bit xen version into a 32 bit dom0?
> >  I think it won''t run but I''ll try.
You should be able to boot a 32-bit dom0 with a 64-bit Xen.  Xen''s 
compatibility code for 32-bit guests does support dom0.  In fact, there are a 
number of advantages to running a 32-bit dom0 on a 64-bit Xen.

If you run a 32-bit dom0 OS then I think you''ll be able to save /
restore /
migrate 32-bit domUs on that system OK.

Cheers,
Mark

-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users