Xen users - Jun 2007 - Xen 3.1 - 32 vs 64 bit hypervisor

I''ve been experimenting with Xen 3.1.

My requirements are to run lots 32 bit paravirt domains on large
memory hosts (8GB or more).

Previously I''ve used a 32 bit Xen with PAE hypervisor which works
well.  However I seen from the announcement that you can use a 64 bit
hypervisor and still run 32 bit PAE domains.  I''ve tried this and it
works as advertised!

What I''m wondering is, will it be best for me to run a 32 bit PAE Xen
hypervisor or a 64 bit hypervisor?  What is the tradeoff between the
two? Which is likely to be more efficient and which more stable?

Thanks

Nick
-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I''ve been experimenting with Xen 3.1.
>
> My requirements are to run lots 32 bit paravirt domains on large
> memory hosts (8GB or more).
>
> Previously I''ve used a 32 bit Xen with PAE hypervisor which works
> well.  However I seen from the announcement that you can use a 64 bit
> hypervisor and still run 32 bit PAE domains.  I''ve tried this and
it
> works as advertised!
Awesome :-)
> What I''m wondering is, will it be best for me to run a 32 bit PAE
Xen
> hypervisor or a 64 bit hypervisor?  What is the tradeoff between the
> two? Which is likely to be more efficient and which more stable?
Stability-wise, 32-on-64 is a new feature so it will have received less real 
world abuse testing.  There may be teething troubles.  Performance-wise...  
Again, the combination is less tested, so it''s not inconceivable that
there
will be teething troubles.

That said, in principle it ought to be alright.  32-on-64 even fully supports 
glibc''s TLS, which even 32-on-32 doesn''t (this is because of
slightly arcane
implementation details in x86.  Really.  The mind boggles.).

If you''re running a 64-bit Xen, I think you also have the potential to
run
64-bit paravirt guests, and 64-bit HVM guests in addition to all the stuff 
you could run on 32-bit Xen.

At some point it should be possible to run a 32-bit dom0, but I''m not
sure if
that''s allowed yet.

cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Jun 08, 2007 at 03:36:02PM +0100, Mark Williamson
wrote:
> > I''ve been experimenting with Xen 3.1.
> >
> > My requirements are to run lots 32 bit paravirt domains on large
> > memory hosts (8GB or more).
> >
> > Previously I''ve used a 32 bit Xen with PAE hypervisor which
works
> > well.  However I seen from the announcement that you can use a 64 bit
> > hypervisor and still run 32 bit PAE domains.  I''ve tried this
and it
> > works as advertised!
> 
> Awesome :-)
> 
> > What I''m wondering is, will it be best for me to run a 32 bit
PAE Xen
> > hypervisor or a 64 bit hypervisor?  What is the tradeoff between the
> > two? Which is likely to be more efficient and which more stable?
> 
> Stability-wise, 32-on-64 is a new feature so it will have received less
real
> world abuse testing.  There may be teething troubles.  Performance-wise...
> Again, the combination is less tested, so it''s not inconceivable
that there
> will be teething troubles.
> 
> That said, in principle it ought to be alright.  32-on-64 even fully
supports
> glibc''s TLS, which even 32-on-32 doesn''t (this is because
of slightly arcane
> implementation details in x86.  Really.  The mind boggles.).
I won''t ask as I don''t need my brain exploded on Friday
afternoon ;-)
> If you''re running a 64-bit Xen, I think you also have the
potential to run
> 64-bit paravirt guests, and 64-bit HVM guests in addition to all the stuff 
> you could run on 32-bit Xen.
Yes that is certainly of future interest.
> At some point it should be possible to run a 32-bit dom0, but I''m
not sure if
> that''s allowed yet.
Well that is how I tested it so I guess it is possible ;-)

Thanks for your useful comments!

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > That said, in principle it ought to be alright.  32-on-64 even fully
> > supports glibc''s TLS, which even 32-on-32 doesn''t
(this is because of
> > slightly arcane implementation details in x86.  Really.  The mind
> > boggles.).
>
> I won''t ask as I don''t need my brain exploded on Friday
afternoon ;-)
Well, it''s really very simple.  Basically what happens is *bang* *bang*
*splat*[1]

OK, maybe you were right :-)
> > If you''re running a 64-bit Xen, I think you also have the
potential to
> > run 64-bit paravirt guests, and 64-bit HVM guests in addition to all
the
> > stuff you could run on 32-bit Xen.
>
> Yes that is certainly of future interest.
>
> > At some point it should be possible to run a 32-bit dom0, but
I''m not
> > sure if that''s allowed yet.
>
> Well that is how I tested it so I guess it is possible ;-)
Well, more particularly, I don''t know if a 32-bit dom0 will boot 64-bit
guests, which may or may not matter for you.  I think it''s on the
roadmap.

32-bit PV guests potentially are actually faster on 64-bit than 64-bit PV 
guests.  Due to <more arcane x86isms> I believe the system call interface
can
be faster for a 32-bit PV guest than a 64-bit guest.  So conceivably
it''s
more efficient to run a 32-bit dom0 (maybe it won''t *actually* matter,
I
don''t know).

Cheers,
Mark

[1] Actually, it''s very simple as long as you don''t expect it
to sound
remotely reasonable.  It *is* x86 after all :-)  Xen wants to be in the top 
part of the address space.  32-bit glibc wants to mess with the top of the 
address space in order to implement TLS.  On 32-bit hosts this conflicts.  On 
a 64-bit host, Xen stashes itself at the top of the virtual address space; 
because it''s a 64-bit host this is waaaaaaaaaaaay higher than a 32-bit
guest
is capable of addressing.  Therefore, it''s fine to let glibc play with
the
top of the 32-bit space - it just can''t get at Xen :-)

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Jun 08, 2007 at 05:38:33PM +0100, Mark Williamson
wrote:
> > > That said, in principle it ought to be alright.  32-on-64 even
fully
> > > supports glibc''s TLS, which even 32-on-32
doesn''t (this is because of
> > > slightly arcane implementation details in x86.  Really.  The mind
> > > boggles.).
> >
> > I won''t ask as I don''t need my brain exploded on
Friday afternoon ;-)
> 
> Well, it''s really very simple.  Basically what happens is *bang*
*bang*
> *splat*[1]
> 
> OK, maybe you were right :-)
> 
> > > If you''re running a 64-bit Xen, I think you also have
the potential to
> > > run 64-bit paravirt guests, and 64-bit HVM guests in addition to
all the
> > > stuff you could run on 32-bit Xen.
> >
> > Yes that is certainly of future interest.
> >
> > > At some point it should be possible to run a 32-bit dom0, but
I''m not
> > > sure if that''s allowed yet.
> >
> > Well that is how I tested it so I guess it is possible ;-)
> 
> Well, more particularly, I don''t know if a 32-bit dom0 will boot
64-bit
> guests, which may or may not matter for you.  I think it''s on the
roadmap.
Pretty sure it will. Back before 3.1.0 was released I was running 32-bit
Dom0 with 64-bit HV. IIRC I booted a 64-bit guest too. The only dep between
dom0 and guest is the front/back drivers & they''re now 64-bit
invariant
so AFAICT it shouldn''t matter whether Dom0 is 32 or 64-bit wrt to
guests,
as long as the HV is 64-bit.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 2007-06-08 at 17:38 +0100, Mark Williamson wrote:
> 32-bit PV guests potentially are actually faster on 64-bit than 64-bit PV 
> guests.  Due to <more arcane x86isms> I believe the system call
interface can
> be faster for a 32-bit PV guest than a 64-bit guest.  So conceivably
it''s
> more efficient to run a 32-bit dom0 (maybe it won''t *actually*
matter, I
> don''t know).
I''m now thoroughly confused. You mean, if you could run a 64 bit guest
/
32 bit hypervisor, right? 
> [1] Actually, it''s very simple as long as you don''t
expect it to sound
> remotely reasonable.  It *is* x86 after all :-)  Xen wants to be in the top
> part of the address space.  32-bit glibc wants to mess with the top of the 
> address space in order to implement TLS.  On 32-bit hosts this conflicts. 
On
> a 64-bit host, Xen stashes itself at the top of the virtual address space; 
> because it''s a 64-bit host this is waaaaaaaaaaaay higher than a
32-bit guest
> is capable of addressing.  Therefore, it''s fine to let glibc play
with the
> top of the 32-bit space - it just can''t get at Xen :-)
That''s why I''m wondering what you meant above, or does this
completely
change if you run a 32 bit hypervisor with a 64 bit guest?

So if a 64 bit tree fell in 32 bit address space ... , oh sod it never
mind. Just tell us when it works and we''ll say "Thank you."

Best,
--Tim


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 2007-06-08 at 17:38 +0100, Mark Williamson
wrote:
> > > That said, in principle it ought to be alright.  32-on-64 even
fully
> > > supports glibc''s TLS, which even 32-on-32
doesn''t (this is because of
> > > slightly arcane implementation details in x86.  Really.  The mind
> > > boggles.).
> >
> > I won''t ask as I don''t need my brain exploded on
Friday afternoon ;-)
> 
> Well, it''s really very simple.  Basically what happens is *bang*
*bang*
> *splat*[1]
I just caught Daniels other reply, now it makes sense. Those on this
list don''t commonly build dom-0 apart from the HV, that''s why
I got
confused. It just doesn''t stick in my head that the two build apart
since the Makefile just passes the same arch type to both.

You guys are used to building the two apart. Once you remember that you
can this all makes perfect sense. 

Hope that helps un-confuse the next person who asks.

Best,
--Tim



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users