Xen devel - Dec 2007 - Question about implementation of 32-bit guests on 64-bit hypervisor (IDT-related)

In a recent conversation one of my coworkers raised a concern about
memory limitations when running 32-bit guests on top of the 64-bit
hypervisor.  At this point the discussion is academic; I don''t know
when/if we''ll ever be able to get system resources to test it, to see
if
the concerns that he expressed are real.  So I decided to post this in
hope of getting comments from the people who worked in this area.  (I
apologize in advance if this is not a proper usage of this mailing list.
In that case, which mailing list would be correct?)

The discussion went as follows.  It seems that there are two ways to
support a 32-bit guest on top of a 64-bit Monitor. We''re not familiar
enough with the Xen code to know which Xen uses.  The first method
involves running the guest in compatibility mode (64-bit environment
that executes code the same way as a 32-bit processor), and the second
method is to run the guest in native 32-bit mode.

If the guest is run in compatibility mode, the processor will use 64-bit
page tables, which will allow the guest RAM to be placed anywhere within
the physical address space (on current processors, this is 0-1TB). This
seems like the ideal solution, but unfortunately there is a catch. If
the processor is in compatibility mode, the IDT and interrupt handler
functions must all be 64-bit compatible. Since the 32-bit guest will
attempt to load a 32-bit IDT, the hypervisor would have to substitute a
64-bit IDT. This seems to be extremely dangerous, because the
substituted IDT would have to work exactly like the guest''s original
32-bit IDT, and there is a possibility that any kernel updates or
service packs to the guest OS would require changes in the substituted
64-bit IDT. 

The alternative is to run the guest in native 32-bit mode. This would
allow the 32-bit IDT provided by the guest OS to be used. However, this
also means that the page tables that are in effect when the guest runs
(the Shadow Page Table) must be either plain 32-bit, or PAE 32-bit.  PAE
seems like an obvious choice, since it is safe to use PAE "under the
hood" even if the guest is non-paging or non-PAE. The problem with this
solution is that PAE page tables only have enough space in the entries
to designate a maximum hardware physical address of 64GB. (Our systems
go substantially larger.)  So the hardware physical locations of the
Guest RAM for all 32-bit guests would have to be located within the
first 64GB.

My questions are:

(1)  Which approach does Xen use?  Is it one of the above, or yet
something else?

(2)  What are the configuration restrictions, if any, for running 32-bit
guests on top of the 64-bit hypervisor?

Thanks for your time and assistance.

    - Bruce

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/12/07 14:05, "Vessey, Bruce A" <Bruce.Vessey@unisys.com>
wrote:
> (1)  Which approach does Xen use?  Is it one of the above, or yet
> something else?
32-bit PV guests are run in compatibility mode. 32-bit HVM guests are run in
legacy mode, inside a VMX or SVM container.
> (2)  What are the configuration restrictions, if any, for running 32-bit
> guests on top of the 64-bit hypervisor?
32-bit PV guests are restricted to using the low 128GB of physical memory.
There''s not even that restriction for 32-bit HVM guests.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > (2)  What are the configuration restrictions, if any, for running
32-
> bit
> > guests on top of the 64-bit hypervisor?
> 
> 32-bit PV guests are restricted to using the low 128GB of physical
> memory.
> There''s not even that restriction for 32-bit HVM guests.
We could even use shadow translate mode to get around the 128GB
restriction, though at some performance cost. I guess you''d switch the
guest to shadow mode if you''ve run out of memory below 128GB.

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > > guests on top of the 64-bit hypervisor?
> >
> > 32-bit PV guests are restricted to using the low 128GB of physical
> > memory.
> > There''s not even that restriction for 32-bit HVM guests.
>
> We could even use shadow translate mode to get around the 128GB
> restriction, though at some performance cost. I guess you''d switch
the
> guest to shadow mode if you''ve run out of memory below 128GB.
That''s not possible right now, though, right?  Shadowing PV guests was
still
broken IIRC?

Cheers,
Mark


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks to all of you for your answers.  I think we''ll have some more
questions, but we need to figure out what they are first.  

          - Bruce



-----Original Message-----
From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On Behalf Of Mark
Williamson
Sent: Friday, December 07, 2007 11:52 AM
To: xen-devel@lists.xensource.com
Cc: Ian Pratt; Keir Fraser; Vessey, Bruce A
Subject: Re: [Xen-devel] Question about implementation of 32-bit guests
on64-bit hypervisor (IDT-related)
> > > guests on top of the 64-bit hypervisor?
> >
> > 32-bit PV guests are restricted to using the low 128GB of physical 
> > memory.
> > There''s not even that restriction for 32-bit HVM guests.
>
> We could even use shadow translate mode to get around the 128GB 
> restriction, though at some performance cost. I guess you''d switch
the
> guest to shadow mode if you''ve run out of memory below 128GB.
That''s not possible right now, though, right?  Shadowing PV guests was
still broken IIRC?

Cheers,
Mark


--
Dave: Just a question. What use is a unicyle with no seat?  And no
pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I said there would probably be more questions.  :-)

We have some confusion regarding the assignment of hardware physical
addresses to the HVMs.  If all 32-bit HVMs are run in legacy mode, then
they would be restricted to using 32-bit PAE shadow page tables.  The
maximum physical addressing of such tables is limited to 64GB, so this
implies that all 32-bit HVMs must be located within the first 64GB of
hardware physical memory.  However, your previous responses indicated
that 32-bit HVMs do not have such a restriction.  So our conclusion is
that 32-bit HVMs running in legacy mode using 32-bit PAE page tables are
able to address hardware physical memory above the 64GB limit.  Is this
correct?  Could we create two separate 32-bit HVM guests, assign each of
them 64GB of memory, and have them run concurrently without any problems
(assuming that the platform has sufficient memory installed)?

Thanks again,
Bruce


-----Original Message-----
From: Vessey, Bruce A 
Sent: Friday, December 07, 2007 1:27 PM
To: ''Mark Williamson''; xen-devel@lists.xensource.com
Cc: Ian Pratt; Keir Fraser
Subject: RE: [Xen-devel] Question about implementation of 32-bit guests
on64-bit hypervisor (IDT-related)

Thanks to all of you for your answers.  I think we''ll have some more
questions, but we need to figure out what they are first.  

          - Bruce



-----Original Message-----
From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On Behalf Of Mark
Williamson
Sent: Friday, December 07, 2007 11:52 AM
To: xen-devel@lists.xensource.com
Cc: Ian Pratt; Keir Fraser; Vessey, Bruce A
Subject: Re: [Xen-devel] Question about implementation of 32-bit guests
on64-bit hypervisor (IDT-related)
> > > guests on top of the 64-bit hypervisor?
> >
> > 32-bit PV guests are restricted to using the low 128GB of physical 
> > memory.
> > There''s not even that restriction for 32-bit HVM guests.
>
> We could even use shadow translate mode to get around the 128GB 
> restriction, though at some performance cost. I guess you''d switch
the
> guest to shadow mode if you''ve run out of memory below 128GB.
That''s not possible right now, though, right?  Shadowing PV guests was
still broken IIRC?

Cheers,
Mark


--
Dave: Just a question. What use is a unicyle with no seat?  And no
pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/12/07 20:47, "Vessey, Bruce A" <Bruce.Vessey@unisys.com>
wrote:
> We have some confusion regarding the assignment of hardware physical
> addresses to the HVMs.  If all 32-bit HVMs are run in legacy mode, then
> they would be restricted to using 32-bit PAE shadow page tables.  The
> maximum physical addressing of such tables is limited to 64GB, so this
> implies that all 32-bit HVMs must be located within the first 64GB of
> hardware physical memory.  However, your previous responses indicated
> that 32-bit HVMs do not have such a restriction.  So our conclusion is
> that 32-bit HVMs running in legacy mode using 32-bit PAE page tables are
> able to address hardware physical memory above the 64GB limit.  Is this
> correct?  Could we create two separate 32-bit HVM guests, assign each of
> them 64GB of memory, and have them run concurrently without any problems
> (assuming that the platform has sufficient memory installed)?
Yes. I believe that the Intel manuals are incorrect in stating that PAE
pagetables are restricted to 36-bit addressing. Processors which support
long mode have their physical address size advertised in CPUID, and I''m
pretty sure that addresses up to that size can be poked into 8-byte
pagetable entries whether the pagetable format is 64-bit-mode or pae-mode.
AMD state explicitly in their manual that PAE pagetables can address up to
52 bits, just like 64-bit pagetables, and that this is the architectural
limit. Furthermore, you guys (Unisys) have done testing on big memory ES7000
systems (>128GB), and those are Intel boxes -- and I expect some of your
testing has been 32-bit HVM guests? Given we allocate larger addresses
first, this would confirm that Intel really does allow addresses >64GB in
PAE pagetables in practice.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/12/07 22:06, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
> Yes. I believe that the Intel manuals are incorrect in stating that PAE
> pagetables are restricted to 36-bit addressing. Processors which support
> long mode have their physical address size advertised in CPUID, and
I''m
> pretty sure that addresses up to that size can be poked into 8-byte
> pagetable entries whether the pagetable format is 64-bit-mode or pae-mode.
> AMD state explicitly in their manual that PAE pagetables can address up to
> 52 bits, just like 64-bit pagetables, and that this is the architectural
> limit. Furthermore, you guys (Unisys) have done testing on big memory
ES7000
> systems (>128GB), and those are Intel boxes -- and I expect some of your
> testing has been 32-bit HVM guests? Given we allocate larger addresses
> first, this would confirm that Intel really does allow addresses >64GB
in
> PAE pagetables in practice.
Although Intel SDM Vol 3A Section 3.8 makes it look like PAE only supports
36-bit addressing, the manual simply hasn''t been updated properly.
There are
various references elsewhere in Vol 3 that make it clear that PAE addressing
is extended on Intel64 processors that have more address lines. The clearest
statement I can find is the second footnote of Section 22.3.1.6 in Vol 3B
(the latest Vol 3B, dated Nov 2007). I''m glad Intel don''t hide
these details
away. ;-)

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 7/12/07 22:06, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
> 
> > Yes. I believe that the Intel manuals are incorrect in stating that
PAE
> > pagetables are restricted to 36-bit addressing. Processors which
support
> > long mode have their physical address size advertised in CPUID, and
I''m
> > pretty sure that addresses up to that size can be poked into 8-byte
> > pagetable entries whether the pagetable format is 64-bit-mode or
pae-mode.
> > AMD state explicitly in their manual that PAE pagetables can address
up to
> > 52 bits, just like 64-bit pagetables, and that this is the
architectural
> > limit. Furthermore, you guys (Unisys) have done testing on big
memory ES7000
> > systems (>128GB), and those are Intel boxes -- and I expect some of
your
> > testing has been 32-bit HVM guests? Given we allocate larger
addresses
> > first, this would confirm that Intel really does allow addresses
>64GB in
> > PAE pagetables in practice.
> 
> Although Intel SDM Vol 3A Section 3.8 makes it look like PAE only
supports
> 36-bit addressing, the manual simply hasn''t been updated properly.
There are
> various references elsewhere in Vol 3 that make it clear that PAE
addressing
> is extended on Intel64 processors that have more address lines. The
clearest
> statement I can find is the second footnote of Section 22.3.1.6 in Vol
3B
> (the latest Vol 3B, dated Nov 2007). I''m glad Intel don''t
hide these
details
> away. ;-)
You should be able to find this in SDM Vol 3A:
3.8.1 Enhanced Legacy PAE Paging
On Intel 64 processors, the page directory pointer entry supports
physical address size of the underlying implementation (reported by
CPUID.80000008H). Legacy PAE enabled paging [see Section 3.8.2, "Linear
Address Translation With PAE Enabled (4-KByte Pages)" and Section 3.8.3,
"Linear Address Translation With PAE Enabled (2-MByte Pages)"] can
address physical memory greater than 64-GByte if the implementation''s
physical address size is greater than 36 bits.
> 
>  -- Keir
> 
Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8/12/07 05:54, "Nakajima, Jun" <jun.nakajima@intel.com>
wrote:
> You should be able to find this in SDM Vol 3A:
> 3.8.1 Enhanced Legacy PAE Paging
> On Intel 64 processors, the page directory pointer entry supports
> physical address size of the underlying implementation (reported by
> CPUID.80000008H). Legacy PAE enabled paging [see Section 3.8.2,
"Linear
> Address Translation With PAE Enabled (4-KByte Pages)" and Section
3.8.3,
> "Linear Address Translation With PAE Enabled (2-MByte Pages)"]
can
> address physical memory greater than 64-GByte if the
implementation''s
> physical address size is greater than 36 bits.
Ah, I missed the most obvious one!

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel