Fedora xen - Jul 2007 - Xen only showing 16GB of ram out of 28GB

I installed FC6 and FC7 and the limit exists on both even though FC7
was supposed to be using a 64bit hypervisor which supports 32bit guests
(which isn''t the case). After various linux installs (including CentOS)
and different configs (even replacing the hypervisor with its 64bit
counterpart) I wasn''t able to get a system showing all 28GB of ram AND
work with 32bit guests.

Out of desperation I bought "Virtualization with Xen" on Amazon for
more insight. Although the book isn''t too helpful since at least half
of it if not more, is advertising the commercial XenSource (why would I
buy an expensive Xen book if I wanted to use the newb commercial
software?) I was able to figure out how to install the latest version
of Xen 3.1 from source. I got it to boot perfectly and was expecting to
see all of my 28GB of ram when I ran top but to my dismay it only
showed 16GB once again...

Can someone please help me and tell me what am I doing wrong. Or am I
going about this the wrong way? Basically I want to see all of my 28GB
of ram and run 32bit guests. I read everywhere this can be done but it
never works. Thank you to anyone who can help me solve this. If money
is the problem, please say so and that can be arranged.

Marco Jorge

On Tue, Jul 17, 2007 at 05:32:11AM -0400, thewird wrote:
> I installed FC6 and FC7 and the limit exists on both even though FC7
> was supposed to be using a 64bit hypervisor which supports 32bit guests
> (which isn''t the case). After various linux installs (including
CentOS)
> and different configs (even replacing the hypervisor with its 64bit
> counterpart) I wasn''t able to get a system showing all 28GB of ram
AND
> work with 32bit guests.
> 
> Out of desperation I bought "Virtualization with Xen" on Amazon
for
> more insight. Although the book isn''t too helpful since at least
half
> of it if not more, is advertising the commercial XenSource (why would I
> buy an expensive Xen book if I wanted to use the newb commercial
> software?) I was able to figure out how to install the latest version
> of Xen 3.1 from source. I got it to boot perfectly and was expecting to
> see all of my 28GB of ram when I ran top but to my dismay it only
> showed 16GB once again...
What architecture did you compile with Xen 3.1 ?  Because of the tricks
Xen does on i386, even if you have PAE enabled you stil won''t see more
than 16 GB of RAM. I believe you need 64-bit HV to see all. You may or
may not also need a 64-bit Dom0 - I don''t know for sure since
I''m not
lucky enough to have more than 16 GB of RAM :-(
> Can someone please help me and tell me what am I doing wrong. Or am I
> going about this the wrong way? Basically I want to see all of my 28GB
> of ram and run 32bit guests. I read everywhere this can be done but it
> never works. Thank you to anyone who can help me solve this. If money
> is the problem, please say so and that can be arranged.
We''re working on a kernl update for Fedora that will hopefully make it
possible to really run 32-bit guests on 64-bit hypervisor.  If you did
a compile from upstream Xen 3.1 kenrel this should be working for you
already though.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

thewird wrote:
> I installed FC6 and FC7 and the limit exists on both even though FC7
> was supposed to be using a 64bit hypervisor which supports 32bit guests
> (which isn''t the case).
WorksForMe[tm].  Test box with a bunch of systems installed, among them
fedora7 32bit and fedora7 64bit.  Booting the 32bit fedora dom0 with the
64bit hypervisor (aka /boot/xen.gz-<version>) from the 64bit
installation works just fine.

If you got it right "xm dmesg" looks like this:
[ ... ]
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0xc1000000 -> 0xc15e2000
[ ... ]

The limitation is that the dom0 kernel backend drivers don''t support
32-on-64, so you can''t mix 32bit and 64bit domains.  As long as dom0
and
all guests are identical it works.

cheers,
  Gerd

Hi,
> than 16 GB of RAM. I believe you need 64-bit HV to see all. You may or
> may not also need a 64-bit Dom0 - I don''t know for sure since
I''m not
> lucky enough to have more than 16 GB of RAM :-(
Dom0 can be 32bit (and must be in fedora7 if you want run 32bit guests
and don''t wait for the kernel update).

With a 32bit hypervisor everything must fit into the hypervisor address
space hole.  With a 64bit hypervisor only the m2p table must fit into
the address space hole.  Thus even 32bit guests can see much more memory
when running on a 64bit hypervisor.  The limit is far beyond 64GB (was
some odd number, 166GB IIRC), although I''d be prepared to run into bugs
when crossing the 64GB border ...

cheers,

  Gerd

> > I installed FC6 and FC7 and the limit exists on both even though
> FC7
> > was supposed to be using a 64bit hypervisor which supports 32bit
> guests
> > (which isn''t the case).
> 
> WorksForMe[tm].  Test box with a bunch of systems installed, among
> them
> fedora7 32bit and fedora7 64bit.  Booting the 32bit fedora dom0 with
> the
> 64bit hypervisor (aka /boot/xen.gz-<version>) from the 64bit
> installation works just fine.
> 
> If you got it right "xm dmesg" looks like this:
> [ ... ]
> (XEN)  Xen  kernel: 64-bit, lsb, compat32
> (XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0xc1000000 -> 0xc15e2000
> [ ... ]
> 
> The limitation is that the dom0 kernel backend drivers don''t
support
> 32-on-64, so you can''t mix 32bit and 64bit domains.  As long as
dom0
> and
> all guests are identical it works.
> 
> cheers,
>   Gerd
> 
I''ve tried this before. I installed FC7 32bit and replaced the xen.gz
with its 64bit version. Sure, it boots fine but when I try to boot
up/install a 32bit guest, the guest just hangs or pauses and will not
run even though the dom0 seems to be working just fine.

Marco Jorge