Xen devel - Sep 2006 - PAE, Elf headers, and Extended CR3 registers

Folks --

I''m trying to figure out why it makes sense to assume that if the elf  
header notes are present and they indicate the use of PAE then an  
extended CR3 register is present.

     /*
      * If we have ELF notes then PAE=yes implies that we must support
      * the extended cr3 syntax. Otherwise we need to find the
      * [extended-cr3] syntax in the __xen_guest string.
      */
     dsi->pae_kernel = PAEKERN_no;
     if ( dsi->__elfnote_section )
     {
         p = xen_elfnote_string(dsi, XEN_ELFNOTE_PAE_MODE);
         if ( p != NULL && strncmp(p, "yes", 3) == 0 )
             dsi->pae_kernel = PAEKERN_extended_cr3;

     }
     else
     {
         p = xen_guest_lookup(dsi, XEN_ELFNOTE_PAE_MODE);
         if ( p != NULL && strncmp(p, "yes", 3) == 0 )
         {
             dsi->pae_kernel = PAEKERN_yes;
             if ( !strncmp(p+4, "[extended-cr3]", 14) )
                 dsi->pae_kernel = PAEKERN_extended_cr3;
         }
     }

Can someone explain why the assumption declared in the comment and  
the first if () clause makes sense?

I would have expected the above to have been written as:

     dsi->pae_kernel = PAEKERN_no;
     p = NULL;
     if ( dsi->__elfnote_section )
     {
         p = xen_elfnote_string(dsi, XEN_ELFNOTE_PAE_MODE);
     }
     else
     {
         p = xen_guest_lookup(dsi, XEN_ELFNOTE_PAE_MODE);
     }

     if ( p != NULL && strncmp(p, "yes", 3) == 0 )
     {
         dsi->pae_kernel = PAEKERN_yes;
         if ( !strncmp(p+4, "[extended-cr3]", 14) )
             dsi->pae_kernel = PAEKERN_extended_cr3;
     }

This would ignore whether the "yes" came from the elf note or the  
guest string.

-- Randy

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14/9/06 08:40, "Randy Thelen" <rthelen@netapp.com> wrote:
> I''m trying to figure out why it makes sense to assume that if the
elf
> header notes are present and they indicate the use of PAE then an
> extended CR3 register is present.
[extended-cr3] was only needed to provide backward compatibility for old
guests that do not grok the extended cr3 format. None of those older guests
support this new Elf notes format (since the new format is much newer than
the introduction of extended-cr3 format). So if the guest uses Elf notes
then it is implicit that it understands extended cr3 format and there is no
need to state that explicitly.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> [extended-cr3] was only needed to provide backward compatibility  
> for old
> guests that do not grok the extended cr3 format. None of those  
> older guests
> support this new Elf notes format (since the new format is much  
> newer than
> the introduction of extended-cr3 format). So if the guest uses Elf  
> notes
> then it is implicit that it understands extended cr3 format and  
> there is no
> need to state that explicitly.
Understand the extended cr3 format?  Do you mean hardware or  
virtualized cr3?

I''m confused.  Did I draw an incorrect assumption from this code  
(from xc_linux_build.c):

     /* First allocate page for page dir. */
     ppt_alloc = (vpt_start - dsi_v_start) >> PAGE_SHIFT;

     if ( pae_mode == PAEKERN_extended_cr3 )
     {
         ctxt->vm_assist |= (1UL << VMASST_TYPE_pae_extended_cr3);
     }
     else if ( page_array[ppt_alloc] > 0xfffff )
     {
         nmfn = xc_make_page_below_4G(xc_handle, dom, page_array 
[ppt_alloc]);
         /* Error handling removed */
         page_array[ppt_alloc] = nmfn;
     }

In the case of PAEKERN_extended_cr3, a 64 bit cr3 register is assumed  
and thus any page in the memory space (up to 64GB) is a valid page  
directory.  Whereas with a x86-class processor, only a 32 bit CR3  
register is present and so the page directory has to be in the low 4GB.

??  What am I missing?

I''m trying to get FreeBSD 6.0 in 32 bit mode + Xen 3.0 support  
running in on a 32 bit Xen w/ PAE and I just want to understand the  
cr3 register from Xen''s point of view.

-- Randy

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

''Extended CR3'' means that, on PAE, we fold the higher-order
bits of the page
directory address into the low-order bits of the 32-bit virtual CR3 value.

See the xen_pfn_to_cr3() and xen_cr3_to_pfn() macros in
xen/include/public/arch-x86_64.h.

In general the guest OS does not manipulate CR3 values at the hypercall
interface, so this is not an issue (e.g., the hypercall for switching
pagetable base pointer takes a page frame number). The one exception in
Linux was that SMP booting required writing a CR3 value into a vcpu_context
structure. So that''s the only place we use those two conversion macros.

 -- Keir


On 14/9/06 21:37, "Randy Thelen" <rthelen@netapp.com> wrote:
> In the case of PAEKERN_extended_cr3, a 64 bit cr3 register is assumed
> and thus any page in the memory space (up to 64GB) is a valid page
> directory.  Whereas with a x86-class processor, only a 32 bit CR3
> register is present and so the page directory has to be in the low 4GB.
> 
> ??  What am I missing?
> 
> I''m trying to get FreeBSD 6.0 in 32 bit mode + Xen 3.0 support
> running in on a 32 bit Xen w/ PAE and I just want to understand the
> cr3 register from Xen''s point of view.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> ''Extended CR3'' means that, on PAE, we fold the
higher-order bits of
> the page
> directory address into the low-order bits of the 32-bit virtual CR3  
> value.
>
> See the xen_pfn_to_cr3() and xen_cr3_to_pfn() macros in
> xen/include/public/arch-x86_64.h.
>
> In general the guest OS does not manipulate CR3 values at the  
> hypercall
> interface, so this is not an issue (e.g., the hypercall for switching
> pagetable base pointer takes a page frame number). The one  
> exception in
> Linux was that SMP booting required writing a CR3 value into a  
> vcpu_context
> structure. So that''s the only place we use those two conversion  
> macros.
Thanks, Keir.

Interesting encoding of the virtual CR3 value:

#define xen_pfn_to_cr3(pfn) (((unsigned)(pfn) << 12) | ((unsigned) 
(pfn) >> 20))

This comment in the Xen code clarifies my thinking on this:

/* Only PDPTs above 4GB boundary need to be shadowed in low memory. */
#define l3tab_needs_shadow(mfn) ((mfn) >= 0x100000)

My concern had been that ''extended CR3'' implied hardware
support for
a 64 bit CR3 register.  Instead it means that I shouldn''t really care  
if the mfn of my Page Table Directory Pointer is above 4GB, Xen will  
handle the condition for me.  And, the way I''ll decode the mfn, if  
necessary (which I would agree should be rare), would be with those  
macros.

-- Randy

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 15/9/06 8:40 pm, "Randy Thelen" <rthelen@netapp.com> wrote:
> My concern had been that ''extended CR3'' implied hardware
support for
> a 64 bit CR3 register.  Instead it means that I shouldn''t really
care
> if the mfn of my Page Table Directory Pointer is above 4GB, Xen will
> handle the condition for me.  And, the way I''ll decode the mfn, if
> necessary (which I would agree should be rare), would be with those
> macros.
Yes, that''s correct. You *must* decode/encode any CR3 value passed
to/from
Xen using those macros or you simply will not work properly on >4GB systems.
Luckily, as I pointed out for Linux, this will probably only be an issue you
come across when initialising secondary VCPUs.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel