thr3ads.net - Xen devel - [DOCS/PATCH] http://wiki.xen.org/wiki/X86_Paravirtualised_Memory

If this information is useful, please help other people find it:
Share via:

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

[DOCS/PATCH] http://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management changes

These changes would allow the above mention Wiki to reference to
more materials. This is what I had in mind to add to the Wiki:

diff --git a/intro.txt b/intro.txt
index b902b1e..72d3ca1 100644
--- a/intro.txt
+++ b/intro.txt
@@ -339,6 +339,188 @@ into the page table base register but will not be
explicitly pinned.
 The initial virtual and pseudo-physical layout of a new guest is
 described
[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
here].
 
+
+At the start of the guest, the kernel image is read and the ELF section is
+parsed. The hypervisor looks a specific section that has the string ‘Xen’
+(or “xen”) to find out: where the kernel expects its virtual base address,
+what type of hypervisor it can work with, certain features the kernel image
+can support, and the location of the hypercall. There are two variants of this:
+
+; a). The lagacy ASCIIZ string with all of the keys concatenated. Each key
+being a string and the equal sign with the value also being an string
+(numeric values are typed as string). The delimiter is a comma.
+The key can be up to 32 characters and the value up to 128 characters.
+For example:
+
+
GUEST_OS=Mini-OS,XEN_VER=xen-3.0,VIRT_BASE=0x0,ELF_PADDR_OFFSET=0x0,HYPERCALL_PAGE=0x2,LOADER=generic
+
+; b). A “.note.Xen” section in ELF header conforming to the ELF .note” format.
+This structure is a 4-byte aligned structure. First section is an numerical
+key (aligned to 4 bytes); followed by either a string or a numerical value
+(again, aligned to 4 bytes). The values can up to any length, if the key is
+assumed to a string. If it is a numerical value, it is a long
+(64-bit value - which means 8 bytes).
+
+The parameters and its purpose are explained in
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,elfnote.h.html#incontents_elfnote
here].
+
+And the XEN_ELF_FEATURES values are explained in
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,features.h.html#incontents_elfnote_features
here].
+
+For example, if the ELF values were as so:
+
+{| border="1" cellpadding="2" cellspacing="0"
+ | Name of Xen ELF entry
+ | Contents
+ |-
+ | XEN_ELFNOTE_GUEST_OS (6)
+ | linux
+ |-
+ | XEN_ELFNOTE_GUEST_VERSION (7)
+ | 2.6
+ |-
+ | XEN_ELFNOTE_XEN_VERSION (5)
+ | xen-3.0
+ |-
+ | XEN_ELFNOTE_VIRT_BASE (3)
+ | 0xffffffff80000000
+ |-
+ | XEN_ELFNOTE_ENTRY (1)
+ | 0xffffffff81899200
+ |-
+ | XEN_ELFNOTE_HYPERCALL_PAGE (2)
+ | 0xffffffff81001000
+ |-
+ | XEN_ELFNOTE_FEATURES (10)
+ | !writable_page_tables|pae_pgdir_above_4gb
+ |-
+ | XEN_ELFNOTE_PAE_MODE (9)
+ | yes
+ |-
+ | XEN_ELFNOTE_LOADER (8)
+ | generic
+ |-
+ | XEN_ELFNOTE_SUSPEND_CANCEL (14)
+ | 1
+ |-
+ | XEN_ELFNOTE_HV_START_LOW
+ | 0xffff800000000000
+ |-
+ | XEN_ELFNOTE_PADDR_OFFSET
+ | 0
+ |}
+
+With that setup, the hypervisor constructs an initial page table that spans the
+region from virtual start address up (0xffffffff80000000) to the end of the
+p2m map.
+
+For example, the layout (which then is going to be feed in
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday]
+can look as so:
+
+{| border="1" cellpadding="2" cellspacing="0"
+ | Page Frame (PFN)
+ | contents
+ |-
+ | 0x0
+ | location of
[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday_shared
struct shared_info]
+ |-
+ | 0x1000
+ | location of the kernel
+ |-
+ | 0x1001
+ | location of the hypercall within the kernel
+ |-
+ | 0x1E3E
+ | ramdisk
+ |-
+ | 0xFC69
+ | phys2mach (P2M) - an array of machine frame numbers. The total size of this
array is dependent on the nr_pages in
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
struct start_info]
+and the architecture of the guest (each entry is four bytes
+under 32-bit kernels and eight bytes under 64-bit kernels).
+ |-
+ | 0xFCE9
+ | location of 
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
start_info structure]
+ |-
+ | 0xFCEA
+ | location of
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,xs_wire.h.html#incontents_xenstore_struct
XenStore structure].
+Also refer to XXX docs/misc/xenstore.txt
+ |-
+ | 0xFCEB
+ | Depending on the type of the guest (initial domain or subsequent domains),
it can either be the
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday_dom0_console
dom0_vga_console_info structure]
+or 
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,console.h.html
XenConsole structure].
+The parameters defining this location are in the 
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
start_info structure]
+ |-
+ | 0xFCEC
+ | bootstrap page tables. These pagetables are loaded in the guest at startup
+and cover from 0x0 up to 0xFD6f (the bootstack).
+ |-
+ | 0xFD6F
+ | bootstrap stack.
+ |}
+
+In terms of virtual addresses, these example PFNs are mapped to virtual
addresses
+as follow:
+{| border="1" cellpadding="2" cellspacing="0"
+ | Virtual Address
+ | contents
+ |-
+ | 0xffffffff80000000
+ | location of
[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday_shared
struct shared_info]
+ |-
+ | 0xffffffff81000000
+ | location of the kernel
+ |-
+ | 0xffffffff81001000
+ | location of the hypercall within the kernel
+ |-
+ | 0xffffffff81e3e000 
+ | ramdisk
+ |-
+ | 0xffffffff8fc69000
+ | phys2mach (P2M) - an array of machine frame numbers. The total size of this
array is dependent on the nr_pages in
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
struct start_info]
+and the architecture of the guest (each entry is four bytes
+under 32-bit kernels and eight bytes under 64-bit kernels).
+ |-
+ | 0xffffffff8fce9000
+ | location of 
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
start_info structure]
+ |-
+ | 0xffffffff8fcea000
+ | location of
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,xs_wire.h.html#incontents_xenstore_struct
XenStore structure].
+Also refer to XXX docs/misc/xenstore.txt
+ |-
+ | 0xffffffff8fceb000
+ | Depending on the type of the guest (initial domain or subsequent domains),
it can either be the
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday_dom0_console
dom0_vga_console_info structure]
+or 
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,console.h.html
XenConsole structure].
+The parameters defining this location are in the 
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
start_info structure]
+ |-
+ | 0xffffffff8fcfc000
+ | bootstrap page tables. These pagetables are loaded in the guest at startup
+and cover from 0x0 up to 0xFD6f (the bootstack).
+ |-
+ | 0xffffffff8fd6f000
+ | bootstrap stack.
+ |}
+
+When the guest is launched, per
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday]
+explanation, the register %esi contains the virtual address to the
+start_info_t (0xffffffff8fce9000), the %cr3 points to the beginning of the
+bootstrap page-tables (0xffffffff8fcfc000), and the %esp points to the
+bootstrap stack (0xffffffff8fd6f000).
+
 = Virtual Address Space  
 Xen enforces certain restrictions on the virtual addresses which are


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

head link

[PATCH 1/8] docs: Document the ELF notes

Mark-up for inclusion of generated docs.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/include/public/elfnote.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/include/public/elfnote.h b/xen/include/public/elfnote.h
index 42d76da..3c57a06 100644
--- a/xen/include/public/elfnote.h
+++ b/xen/include/public/elfnote.h
@@ -28,6 +28,8 @@
 #define __XEN_PUBLIC_ELFNOTE_H__
 
 /*
+ * `incontents 200 elfnotes ELF notes
+ *
  * The notes should live in a PT_NOTE segment and have "Xen" in the
  * name field.
  *
-- 
1.7.11.7

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

head link

[PATCH 2/8] docs: Provide some examples of the various ELF values.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/include/public/elfnote.h | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/xen/include/public/elfnote.h b/xen/include/public/elfnote.h
index 3c57a06..74223dd 100644
--- a/xen/include/public/elfnote.h
+++ b/xen/include/public/elfnote.h
@@ -48,14 +48,19 @@
 /*
  * The virtual address of the entry point (numeric).
  *
+ * Example is 0xffffffff81899200.
+ *
  * LEGACY: VIRT_ENTRY
  */
 #define XEN_ELFNOTE_ENTRY          1
 
 /* The virtual address of the hypercall transfer page (numeric).
  *
+ * Example is 0xffffffff81001000 (but not for legacy)
+ *
  * LEGACY: HYPERCALL_PAGE. (n.b. legacy value is a physical page
  * number not a virtual address)
+ *
  */
 #define XEN_ELFNOTE_HYPERCALL_PAGE 2
 
@@ -63,6 +68,8 @@
  *
  * Defaults to 0.
  *
+ * Example is 0xffffffff80000000.
+ *
  * LEGACY: VIRT_BASE
  */
 #define XEN_ELFNOTE_VIRT_BASE      3
@@ -82,6 +89,8 @@
 /*
  * The version of Xen that we work with (string).
  *
+ * Example is "xen-3.0"
+ *
  * LEGACY: XEN_VER
  */
 #define XEN_ELFNOTE_XEN_VERSION    5
@@ -89,6 +98,7 @@
 /*
  * The name of the guest operating system (string).
  *
+ * Example is "linux"
  * LEGACY: GUEST_OS
  */
 #define XEN_ELFNOTE_GUEST_OS       6
@@ -96,6 +106,8 @@
 /*
  * The version of the guest operating system (string).
  *
+ * Example is "2.6"
+ *
  * LEGACY: GUEST_VER
  */
 #define XEN_ELFNOTE_GUEST_VERSION  7
@@ -103,6 +115,8 @@
 /*
  * The loader type (string).
  *
+ * Example is "generic"
+ *
  * LEGACY: LOADER
  */
 #define XEN_ELFNOTE_LOADER         8
@@ -130,6 +144,9 @@
  * features.h, without the "XENFEAT_" prefix) separated by
''|''
  * characters. If a feature is required for the kernel to function
  * then the feature name must be preceded by a ''!'' character.
+ * For the full list, please look in "features.h"
+ *
+ * Example are: "!writable_page_tables|pae_pgdir_above_4gb"
  *
  * LEGACY: FEATURES
  */
@@ -149,6 +166,8 @@
  * This must not be set higher than HYPERVISOR_VIRT_START. Its presence
  * also indicates to the hypervisor that the kernel can deal with the
  * hole starting at a higher address.
+ *
+ * Example is  0xffff800000000000.
  */
 #define XEN_ELFNOTE_HV_START_LOW  12
 
-- 
1.7.11.7

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

head link

[PATCH 3/8] docs: Document the ELF_FEATURES entry

Mark-up for inclusion of generated docs.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/include/public/features.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/xen/include/public/features.h b/xen/include/public/features.h
index b4533cc..a7ddf27 100644
--- a/xen/include/public/features.h
+++ b/xen/include/public/features.h
@@ -28,6 +28,12 @@
 #define __XEN_PUBLIC_FEATURES_H__
 
 /*
+ * `incontents 200 elfnotes_features XEN_ELFNOTE_FEATURES
+ *
+ * The list of all the features the guest supports.
+ */
+
+/*
  * If set, the guest does not need to write-protect its pagetables, and can
  * update them via direct writes.
  */
-- 
1.7.11.7

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

head link

[PATCH 4/8] docs: Add some extra details to the ELF note.

Such as how they are composed (ASCII or ASCIZ).
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/include/public/elfnote.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/xen/include/public/elfnote.h b/xen/include/public/elfnote.h
index 74223dd..5205c9d 100644
--- a/xen/include/public/elfnote.h
+++ b/xen/include/public/elfnote.h
@@ -38,6 +38,8 @@
  *
  * LEGACY indicated the fields in the legacy __xen_guest string which
  * this a note type replaces.
+ *
+ * String values (for non-legacy) are ASCIZ type.
  */
 
 /*
@@ -57,6 +59,7 @@
 /* The virtual address of the hypercall transfer page (numeric).
  *
  * Example is 0xffffffff81001000 (but not for legacy)
+ * For LEGACY an example is 0x2
  *
  * LEGACY: HYPERCALL_PAGE. (n.b. legacy value is a physical page
  * number not a virtual address)
@@ -179,6 +182,9 @@
 
 /*
  * Whether or not the guest supports cooperative suspend cancellation.
+ * This is a numeric value.
+ *
+ * Default is 0
  */
 #define XEN_ELFNOTE_SUSPEND_CANCEL 14
 
-- 
1.7.11.7

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

head link

[PATCH 5/8] docs: Document the shared structure.

Mark-up for inclusion of generated docs.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/include/public/xen.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
index 5593066..e1748d1 100644
--- a/xen/include/public/xen.h
+++ b/xen/include/public/xen.h
@@ -622,6 +622,7 @@ typedef struct vcpu_info vcpu_info_t;
 #endif
 
 /*
+ * `incontents 200 startofday_shared Start-of-day shared data structure
  * Xen/kernel shared data -- pointer provided in start_info.
  *
  * This structure is defined to be both smaller than a page, and the
-- 
1.7.11.7

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

head link

[PATCH 6/8] docs: Document the dom0_vga_console_info structure.

Mark-up for inclusion of generated docs.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/include/public/xen.h | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
index e1748d1..c50bbfc 100644
--- a/xen/include/public/xen.h
+++ b/xen/include/public/xen.h
@@ -779,7 +779,15 @@ struct xen_multiboot_mod_list
     /* Unused, must be zero */
     uint32_t pad;
 };
-
+/*
+ * `incontents 200 startofday_dom0_console Dom9_console
+ *
+ * The console structure in start_info.console.dom0
+ *
+ * The contents off this structure follows mostly the Linux kernel
+ * bootparam structure and includes a variety of information required
+ * to have a working VGA/VESA console.
+ */
 typedef struct dom0_vga_console_info {
     uint8_t video_type; /* DOM0_VGA_CONSOLE_??? */
 #define XEN_VGATYPE_TEXT_MODE_3 0x03
-- 
1.7.11.7

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

head link

[PATCH 7/8] docs: Document start_info changes in Xen 4.2.

The git commit 7a9d7646307b7c872b8dbd7546579acd3b54223d "x86/32-on-64:
adjust Dom0 initial page table layout" fixes a bug in the reported
value of pt_base versus what is stored in the %cr3 register. This
documents this in the start of the world header note.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/include/public/xen.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
index c50bbfc..1685317 100644
--- a/xen/include/public/xen.h
+++ b/xen/include/public/xen.h
@@ -705,6 +705,13 @@ typedef struct shared_info shared_info_t;
  *  8. There is guaranteed to be at least 512kB padding after the final
  *     bootstrap element. If necessary, the bootstrap virtual region is
  *     extended by an extra 4MB to ensure this.
+ *
+ * Note: Prior to 7a9d7646307b7c872b8dbd7546579acd3b54223d ("x86/32-on-64:
+ * adjust Dom0 initial page table layout") the 3.e) contained two
different
+ * values with a 64-bit hypervisor and a 32-bit initial domain kernel. The
+ * pt_base pointed to the L4 (setup by the hypervisor and not used by
+ * the guest) and the %cr3 pointed to the L3. This meant an difference of
+ * one page.
  */
 
 #define MAX_GUEST_CMDLINE 1024
-- 
1.7.11.7

Konrad Rzeszutek Wilk

2013-Jan-28 18:32 UTC

head link

[PATCH 8/8] docs: Document the XenBus structure.

Mark-up for inclusion of generated docs.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/include/public/io/xs_wire.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/xen/include/public/io/xs_wire.h b/xen/include/public/io/xs_wire.h
index 7e454c4..20863f7 100644
--- a/xen/include/public/io/xs_wire.h
+++ b/xen/include/public/io/xs_wire.h
@@ -103,7 +103,10 @@ enum xs_watch_type
     XS_WATCH_TOKEN
 };
 
-/* Inter-domain shared memory communications. */
+/*
+ * `incontents 150 xenstore_struct XenStore structure layout
+ *
+ * Inter-domain shared memory communications. */
 #define XENSTORE_RING_SIZE 1024
 typedef uint32_t XENSTORE_RING_IDX;
 #define MASK_XENSTORE_IDX(idx) ((idx) & (XENSTORE_RING_SIZE-1))
-- 
1.7.11.7

Ian Campbell

2013-Jan-29 09:16 UTC

head link

Re: [PATCH 1/8] docs: Document the ELF notes

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> Mark-up for inclusion of generated docs.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Campbell <ian.campbel@citrix.com>

Ian Campbell

2013-Jan-29 09:23 UTC

head link

Re: [PATCH 2/8] docs: Provide some examples of the various ELF values.

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  xen/include/public/elfnote.h | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/xen/include/public/elfnote.h b/xen/include/public/elfnote.h
> index 3c57a06..74223dd 100644
> --- a/xen/include/public/elfnote.h
> +++ b/xen/include/public/elfnote.h
> @@ -48,14 +48,19 @@
>  /*
>   * The virtual address of the entry point (numeric).
>   *
> + * Example is 0xffffffff81899200.
I''m not convinced that many of these add much, especially these
"random
number" ones. It would be better if it could link to some suitable
documentation about the context of the field (or perhaps you are about
to add some such documentation later in this series).
> @@ -82,6 +89,8 @@
>  /*
>   * The version of Xen that we work with (string).
>   *
> + * Example is "xen-3.0"
I think this one is effectively "must always be xen-3.0" since the API
has been frozen since 3.0.

Ian.

Ian Campbell

2013-Jan-29 09:23 UTC

head link

Re: [PATCH 3/8] docs: Document the ELF_FEATURES entry

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> Mark-up for inclusion of generated docs.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/include/public/features.h | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/xen/include/public/features.h b/xen/include/public/features.h
> index b4533cc..a7ddf27 100644
> --- a/xen/include/public/features.h
> +++ b/xen/include/public/features.h
> @@ -28,6 +28,12 @@
>  #define __XEN_PUBLIC_FEATURES_H__
>  
>  /*
> + * `incontents 200 elfnotes_features XEN_ELFNOTE_FEATURES
> + *
> + * The list of all the features the guest supports.
You could add "and/or requires" since there is provision for Must Have
features too.

Ian Campbell

2013-Jan-29 09:26 UTC

head link

Re: [PATCH 4/8] docs: Add some extra details to the ELF note.

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> Such as how they are composed (ASCII or ASCIZ).
Are any of them actually non-null terminated ASCII?
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  xen/include/public/elfnote.h | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/xen/include/public/elfnote.h b/xen/include/public/elfnote.h
> index 74223dd..5205c9d 100644
> --- a/xen/include/public/elfnote.h
> +++ b/xen/include/public/elfnote.h
> @@ -38,6 +38,8 @@
>   *
>   * LEGACY indicated the fields in the legacy __xen_guest string which
>   * this a note type replaces.
> + *
> + * String values (for non-legacy) are ASCIZ type.
I''d be tempted to spell this out as "NULL terminated ASCII".
>   */
>  
>  /*
> @@ -57,6 +59,7 @@
>  /* The virtual address of the hypercall transfer page (numeric).
>   *
>   * Example is 0xffffffff81001000 (but not for legacy)
> + * For LEGACY an example is 0x2
A better example (but see my comment on patch 2) would be the pfn
corresponding to the non-legacy address, which I suppose is 0x81001 or
something.

Ian Campbell

2013-Jan-29 09:26 UTC

head link

Re: [PATCH 5/8] docs: Document the shared structure.

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> Mark-up for inclusion of generated docs.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/include/public/xen.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> index 5593066..e1748d1 100644
> --- a/xen/include/public/xen.h
> +++ b/xen/include/public/xen.h
> @@ -622,6 +622,7 @@ typedef struct vcpu_info vcpu_info_t;
>  #endif
>  
>  /*
> + * `incontents 200 startofday_shared Start-of-day shared data structure
>   * Xen/kernel shared data -- pointer provided in start_info.
>   *
>   * This structure is defined to be both smaller than a page, and the

Ian Campbell

2013-Jan-29 09:31 UTC

head link

Re: [PATCH 6/8] docs: Document the dom0_vga_console_info structure.

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> Mark-up for inclusion of generated docs.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  xen/include/public/xen.h | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> index e1748d1..c50bbfc 100644
> --- a/xen/include/public/xen.h
> +++ b/xen/include/public/xen.h
> @@ -779,7 +779,15 @@ struct xen_multiboot_mod_list
>      /* Unused, must be zero */
>      uint32_t pad;
>  };
> -
> +/*
> + * `incontents 200 startofday_dom0_console Dom9_console
> + *
> + * The console structure in start_info.console.dom0
> + *
> + * The contents off this structure follows mostly the Linux kernel
> + * bootparam structure and includes a variety of information required
> + * to have a working VGA/VESA console.
Would it be useful to either link to the bootparam structure information
or to explain from first principals? I suppose the existing fields are
actually reasonably well commented.

NB, ./arch/x86/include/uapi/asm/bootparam.h doesn''t seem to have a
literal struct bootparam so I guess you are referring to something else?
There is a struct boot_params but it doesn''t really match what is here.
There is a struct screen_info in Linux but it isn''t a good match for
this Xen structure either.

Are you sure it mostly follows a Linux bootparam structure? Maybe it
would be easiest to just say:

        This structure includes a variety of information required to
        have a working VGA/VESA console.
        
?

Ian Campbell

2013-Jan-29 09:37 UTC

head link

Re: [PATCH 7/8] docs: Document start_info changes in Xen 4.2.

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> The git commit 7a9d7646307b7c872b8dbd7546579acd3b54223d "x86/32-on-64:
> adjust Dom0 initial page table layout" fixes a bug in the reported
> value of pt_base versus what is stored in the %cr3 register. This
> documents this in the start of the world header note.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  xen/include/public/xen.h | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> index c50bbfc..1685317 100644
> --- a/xen/include/public/xen.h
> +++ b/xen/include/public/xen.h
> @@ -705,6 +705,13 @@ typedef struct shared_info shared_info_t;
>   *  8. There is guaranteed to be at least 512kB padding after the final
>   *     bootstrap element. If necessary, the bootstrap virtual region is
>   *     extended by an extra 4MB to ensure this.
> + *
> + * Note: Prior to 7a9d7646307b7c872b8dbd7546579acd3b54223d
("x86/32-on-64:
> + * adjust Dom0 initial page table layout") the 3.e) contained two
different
> + * values with a 64-bit hypervisor and a 32-bit initial domain kernel. The
> + * pt_base pointed to the L4 (setup by the hypervisor and not used by
> + * the guest) and the %cr3 pointed to the L3. This meant an difference of
> + * one page.
>   */
7a9d7646307b7c872b8dbd7546579acd3b54223d is not a Xen revision. I think
you meant 25833:bb85bbccb1c9. I assume this was a bug fix, in which case
I would say "Prior to Xen commit xxx:yyyy ("x86/32-on...") a bug
caused
the pt_base (3.e above) to contain ..."

It would be useful to explain clearly the after case too, which I
suppose is that on 32-on-64 the pt_base now points to what the guest
kernel should consider to be the base? Would it be relevant to explain
how to detect this situation and what to do about it?

The difference of one page was just an implementation artefact, rather
than a meaningful semantic difference? Unless it is relevant to
detecting the situation I would be inclined not to do it.

It should also be made clearer that this affected only 32-on-64 dom0
kernels and not 32-on-64 domU or 64-bit kernels of any colour.

Ian.

Ian Campbell

2013-Jan-29 09:38 UTC

head link

Re: [PATCH 8/8] docs: Document the XenBus structure.

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> Mark-up for inclusion of generated docs.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  xen/include/public/io/xs_wire.h | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/include/public/io/xs_wire.h
b/xen/include/public/io/xs_wire.h
> index 7e454c4..20863f7 100644
> --- a/xen/include/public/io/xs_wire.h
> +++ b/xen/include/public/io/xs_wire.h
> @@ -103,7 +103,10 @@ enum xs_watch_type
>      XS_WATCH_TOKEN
>  };
>  
> -/* Inter-domain shared memory communications. */
> +/*
> + * `incontents 150 xenstore_struct XenStore structure layout
"Xenstore wire protocol" perhaps?
> + *
> + * Inter-domain shared memory communications. */
>  #define XENSTORE_RING_SIZE 1024
>  typedef uint32_t XENSTORE_RING_IDX;
>  #define MASK_XENSTORE_IDX(idx) ((idx) & (XENSTORE_RING_SIZE-1))

Ian Campbell

2013-Jan-29 10:05 UTC

head link

Re: [DOCS/PATCH] http://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management changes

On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk
wrote:> These changes would allow the above mention Wiki to reference to
> more materials. This is what I had in mind to add to the Wiki:
> 
> diff --git a/intro.txt b/intro.txt
> index b902b1e..72d3ca1 100644
> --- a/intro.txt
> +++ b/intro.txt
> @@ -339,6 +339,188 @@ into the page table base register but will not be
explicitly pinned.
>  The initial virtual and pseudo-physical layout of a new guest is
>  described
[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
here].
>  
> +
> +At the start of the guest, the kernel image is read and the ELF section is
"When the guest is started...." (or "built")
> +parsed. The hypervisor looks a specific section that has the string ‘Xen’
> +(or “xen”) to find out:
This is an ELF PT_NOTE program header (as opposed to a SHT_NOTE
section), right? I'm not 100% sure of the ELF terminology but it would
be good to be accurate (especially since PHDRS and Sections are
different things in ELF).

In ELF parlance the string is called the "name".

Is the name really "Xen" or "xen", or is it specifically one
or the
other? I think it is "Xen" only, and note that it is NULL terminated.
>  where the kernel expects its virtual base address,
> +what type of hypervisor it can work with, certain features the kernel
image
> +can support, and the location of the hypercall. There are two variants of
this:                                                 ^page

(NB: A description of the hypercall page would be useful somewhere)
> +
> +; a). The lagacy ASCIIZ string with all of the keys concatenated. Each key
"legacy"

I think I would put the modern ELF note thing first and the legacy one
second, to give prominence to the ELF notes.
> +being a string and the equal sign with the value also being an string
> +(numeric values are typed as string). The delimiter is a comma.
Are they always in hex or is decimal acceptable?
> +The key can be up to 32 characters and the value up to 128 characters.
> +For example:
> +
> +
GUEST_OS=Mini-OS,XEN_VER=xen-3.0,VIRT_BASE=0x0,ELF_PADDR_OFFSET=0x0,HYPERCALL_PAGE=0x2,LOADER=generic
> +
> +; b). A “.note.Xen” section in ELF header conforming to the ELF .note”
format.
Extra " towards the end there?

I don't think the .note.Xen thing is part of the standard interface.
Looking at elf_xen_parse in libelf-dominfo.c it is the PT_NOTE which
matters.

(AIUI PHDRS are a loading time thing, Sections are a linking time thing)
> +This structure is a 4-byte aligned structure. First section is an
numerical
> +key (aligned to 4 bytes);
The type field is part of the ELF spec I think. Whereas the specific
type values and the the rest (the descr) is defined by Xen as its
interpretation of things in the Xen namespace.
>  followed by either a string or a numerical value
> +(again, aligned to 4 bytes). The values can up to any length, if the key
is
> +assumed to a string. If it is a numerical value, it is a long
> +(64-bit value - which means 8 bytes).
Could use a link to the ELF spec somewhere here.
> +
> +The parameters and its purpose are explained in
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,elfnote.h.html#incontents_elfnote
here].
> +
> +And the XEN_ELF_FEATURES values are explained in
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,features.h.html#incontents_elfnote_features
here].
> +
> +For example, if the ELF values were as so:
> +
> +{| border="1" cellpadding="2"
cellspacing="0"
> + | Name of Xen ELF entry
> + | Contents
> + |-
> + | XEN_ELFNOTE_GUEST_OS (6)
> + | linux
> + |-
> + | XEN_ELFNOTE_GUEST_VERSION (7)
> + | 2.6
> + |-
> + | XEN_ELFNOTE_XEN_VERSION (5)
> + | xen-3.0
> + |-
> + | XEN_ELFNOTE_VIRT_BASE (3)
> + | 0xffffffff80000000
> + |-
> + | XEN_ELFNOTE_ENTRY (1)
> + | 0xffffffff81899200
> + |-
> + | XEN_ELFNOTE_HYPERCALL_PAGE (2)
> + | 0xffffffff81001000
> + |-
> + | XEN_ELFNOTE_FEATURES (10)
> + | !writable_page_tables|pae_pgdir_above_4gb
> + |-
> + | XEN_ELFNOTE_PAE_MODE (9)
> + | yes
> + |-
> + | XEN_ELFNOTE_LOADER (8)
> + | generic
> + |-
> + | XEN_ELFNOTE_SUSPEND_CANCEL (14)
> + | 1
> + |-
> + | XEN_ELFNOTE_HV_START_LOW
> + | 0xffff800000000000
> + |-
> + | XEN_ELFNOTE_PADDR_OFFSET
> + | 0
> + |}
> +
> +With that setup, the hypervisor constructs an initial page table that
spans the
> +region from virtual start address up (0xffffffff80000000) to the end of
the
> +p2m map.
> +
> +For example, the layout (which then is going to be feed in
Not sure what you mean by "to be feed in". "which is used to
initialise
the start of day structure <link>" ?

>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday]
> +can look as so:
Does xen make any guarantee about the pfn space layout at start of day?
I thought it was just the virtual space and the fact that pfn space was
contiguous..

I think an example is OK but you need to make it clear that this is just
an example and not part of the ABI.
> +
> +{| border="1" cellpadding="2"
cellspacing="0"
> + | Page Frame (PFN)
> + | contents
> + |-
> + | 0x0
> + | location of
[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday_shared
struct shared_info]
The "Start-of-day memory layout" bit of the interface doesn't say
anything about the location of struct shared_info so I don't think you
can take this one as read?
> + |-
> + | 0x1000
> + | location of the kernel
> + |-
> + | 0x1001
> + | location of the hypercall within the kernel
> + |-
> + | 0x1E3E
This assumes kernel and ramdisk of particular sizes, might be worth
specifying and also being explicit about the fact that this is an
example layout given those inputs.
> + | ramdisk
> + |-
> + | 0xFC69
> + | phys2mach (P2M) - an array of machine frame numbers. The total size of
this array is dependent on the nr_pages in
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
struct start_info]
> +and the architecture of the guest (each entry is four bytes
> +under 32-bit kernels and eight bytes under 64-bit kernels).
> + |-
> + | 0xFCE9
> + | location of 
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
start_info structure]
> + |-
> + | 0xFCEA
> + | location of
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,xs_wire.h.html#incontents_xenstore_struct
XenStore structure].
> +Also refer to XXX docs/misc/xenstore.txt
http://xenbits.xen.org/docs/unstable/misc/xenstore.txt
> + |-
> + | 0xFCEB
> + | Depending on the type of the guest (initial domain or subsequent
domains), it can either be the
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday_dom0_console
dom0_vga_console_info structure]
> +or 
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,console.h.html
XenConsole structure].
> +The parameters defining this location are in the 
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
start_info structure]
> + |-
> + | 0xFCEC
> + | bootstrap page tables. These pagetables are loaded in the guest at
startup
> +and cover from 0x0 up to 0xFD6f (the bootstack).
> + |-
> + | 0xFD6F
> + | bootstrap stack.
> + |}
> +
> +In terms of virtual addresses, these example PFNs are mapped to virtual
addresses
> +as follow:
This repeats the above quite a bit, perhaps a single table with columns
for both physical and virtual address would work better?
> +{| border="1" cellpadding="2"
cellspacing="0"
> + | Virtual Address
> + | contents
> + |-
> + | 0xffffffff80000000
> + | location of
[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday_shared
struct shared_info]
> + |-
> + | 0xffffffff81000000
> + | location of the kernel
> + |-
> + | 0xffffffff81001000
> + | location of the hypercall within the kernel
> + |-
> + | 0xffffffff81e3e000 
> + | ramdisk
> + |-
> + | 0xffffffff8fc69000
> + | phys2mach (P2M) - an array of machine frame numbers. The total size of
this array is dependent on the nr_pages in
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
struct start_info]
> +and the architecture of the guest (each entry is four bytes
> +under 32-bit kernels and eight bytes under 64-bit kernels).
> + |-
> + | 0xffffffff8fce9000
> + | location of 
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
start_info structure]
> + |-
> + | 0xffffffff8fcea000
> + | location of
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,xs_wire.h.html#incontents_xenstore_struct
XenStore structure].
> +Also refer to XXX docs/misc/xenstore.txt
> + |-
> + | 0xffffffff8fceb000
> + | Depending on the type of the guest (initial domain or subsequent
domains), it can either be the
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday_dom0_console
dom0_vga_console_info structure]
> +or 
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,console.h.html
XenConsole structure].
> +The parameters defining this location are in the 
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday
start_info structure]
> + |-
> + | 0xffffffff8fcfc000
> + | bootstrap page tables. These pagetables are loaded in the guest at
startup
> +and cover from 0x0 up to 0xFD6f (the bootstack).
> + |-
> + | 0xffffffff8fd6f000
> + | bootstrap stack.
> + |}
> +
> +When the guest is launched, per
>
+[http://xenbits.xen.org/docs/unstable/hypercall/include,public,xen.h.html#incontents_startofday]
> +explanation, the register %esi contains the virtual address to the
> +start_info_t (0xffffffff8fce9000), the %cr3 points to the beginning of the
> +bootstrap page-tables (0xffffffff8fcfc000), and the %esp points to the
> +bootstrap stack (0xffffffff8fd6f000).
> +
>  = Virtual Address Space >  
>  Xen enforces certain restrictions on the virtual addresses which are
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Konrad Rzeszutek Wilk

2013-Jan-30 17:10 UTC

head link

Re: [PATCH 6/8] docs: Document the dom0_vga_console_info structure.

On Tue, Jan 29, 2013 at 09:31:38AM +0000, Ian Campbell
wrote:> On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk wrote:
> > Mark-up for inclusion of generated docs.
> > 
> > Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > ---
> >  xen/include/public/xen.h | 10 +++++++++-
> >  1 file changed, 9 insertions(+), 1 deletion(-)
> > 
> > diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> > index e1748d1..c50bbfc 100644
> > --- a/xen/include/public/xen.h
> > +++ b/xen/include/public/xen.h
> > @@ -779,7 +779,15 @@ struct xen_multiboot_mod_list
> >      /* Unused, must be zero */
> >      uint32_t pad;
> >  };
> > -
> > +/*
> > + * `incontents 200 startofday_dom0_console Dom9_console
> > + *
> > + * The console structure in start_info.console.dom0
> > + *
> > + * The contents off this structure follows mostly the Linux kernel
> > + * bootparam structure and includes a variety of information required
> > + * to have a working VGA/VESA console.
> 
> Would it be useful to either link to the bootparam structure information
> or to explain from first principals? I suppose the existing fields are
> actually reasonably well commented.
> 
> NB, ./arch/x86/include/uapi/asm/bootparam.h doesn''t seem to have a
> literal struct bootparam so I guess you are referring to something else?
> There is a struct boot_params but it doesn''t really match what is
here.
> There is a struct screen_info in Linux but it isn''t a good match
for
> this Xen structure either.
> 
> Are you sure it mostly follows a Linux bootparam structure? Maybe it
No. I confused it with another structure that is retrievied via the
XENPF_firmware_info hypercall.
> would be easiest to just say:
> 
>         This structure includes a variety of information required to
>         have a working VGA/VESA console.
>         Perect.
> ?
> 
>

Konrad Rzeszutek Wilk

2013-Jan-30 17:23 UTC

head link

Re: [PATCH 7/8] docs: Document start_info changes in Xen 4.2.

On Tue, Jan 29, 2013 at 09:37:00AM +0000, Ian Campbell
wrote:> On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk wrote:
> > The git commit 7a9d7646307b7c872b8dbd7546579acd3b54223d
"x86/32-on-64:
> > adjust Dom0 initial page table layout" fixes a bug in the
reported
> > value of pt_base versus what is stored in the %cr3 register. This
> > documents this in the start of the world header note.
> > 
> > Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > ---
> >  xen/include/public/xen.h | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> > index c50bbfc..1685317 100644
> > --- a/xen/include/public/xen.h
> > +++ b/xen/include/public/xen.h
> > @@ -705,6 +705,13 @@ typedef struct shared_info shared_info_t;
> >   *  8. There is guaranteed to be at least 512kB padding after the
final
> >   *     bootstrap element. If necessary, the bootstrap virtual region
is
> >   *     extended by an extra 4MB to ensure this.
> > + *
> > + * Note: Prior to 7a9d7646307b7c872b8dbd7546579acd3b54223d
("x86/32-on-64:
> > + * adjust Dom0 initial page table layout") the 3.e) contained
two different
> > + * values with a 64-bit hypervisor and a 32-bit initial domain
kernel. The
> > + * pt_base pointed to the L4 (setup by the hypervisor and not used by
> > + * the guest) and the %cr3 pointed to the L3. This meant an
difference of
> > + * one page.
> >   */
> 
> 7a9d7646307b7c872b8dbd7546579acd3b54223d is not a Xen revision. I think
> you meant 25833:bb85bbccb1c9. I assume this was a bug fix, in which case
> I would say "Prior to Xen commit xxx:yyyy ("x86/32-on...") a
bug caused
> the pt_base (3.e above) to contain ..."
> 
> It would be useful to explain clearly the after case too, which I
> suppose is that on 32-on-64 the pt_base now points to what the guest
> kernel should consider to be the base? Would it be relevant to explain
> how to detect this situation and what to do about it?
> 
> The difference of one page was just an implementation artefact, rather
> than a meaningful semantic difference? Unless it is relevant to
> detecting the situation I would be inclined not to do it.
It can be construed either way. If you look at:

 *      e. bootstrap page tables         [pt_base, CR3 (x86)]                   

It mentions _both_ sources of information - the register and the
contents of pt_base. It could imply that both of them have the same
value (if you read that as an enumeration of where this value is
contained). But if you read the '','' as an
''or'', then it is not a
semantic diference as they both could contain drastically diferent
values.

Implementation wise, we get is that the value that CR3 is different than what:
    unsigned long pt_base;      /* VIRTUAL address of page directory.  */   

has. In other words, the pt_base is suspect and should be carefully
compared to the %cr3. If they are different, then the user should _NOT_
touch the pages in between pt_base and %cr3 and mark them as unusable.
> 
> It should also be made clearer that this affected only 32-on-64 dom0
> kernels and not 32-on-64 domU or 64-bit kernels of any colour.
Right. How about this:

commit 489682893ffc8bbbbb12ee820defac17cce762d0
Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date:   Mon Jan 28 13:20:29 2013 -0500

    docs: Document start_info changes in Xen 4.2.
    
    The  25833:bb85bbccb1c9. "x86/32-on-64: adjust Dom0 initial page table
layout"
    fixes a bug in the reported value of pt_base versus what is stored in
    the %cr3 register. This documents this in the start of the world header
note.
    
    Also make it crystal clear that pt_base == %cr3.
    
    Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
index c50bbfc..3f51cd6 100644
--- a/xen/include/public/xen.h
+++ b/xen/include/public/xen.h
@@ -693,7 +693,7 @@ typedef struct shared_info shared_info_t;
  *      c. list of allocated page frames [mfn_list, nr_pages]
  *         (unless relocated due to XEN_ELFNOTE_INIT_P2M)
  *      d. start_info_t structure        [register ESI (x86)]
- *      e. bootstrap page tables         [pt_base, CR3 (x86)]
+ *      e. bootstrap page tables         [pt_base and CR3 (x86)]
  *      f. bootstrap stack               [register ESP (x86)]
  *  4. Bootstrap elements are packed together, but each is 4kB-aligned.
  *  5. The initial ram disk may be omitted.
@@ -705,6 +705,15 @@ typedef struct shared_info shared_info_t;
  *  8. There is guaranteed to be at least 512kB padding after the final
  *     bootstrap element. If necessary, the bootstrap virtual region is
  *     extended by an extra 4MB to ensure this.
+ *
+ * Note: Prior to 25833:bb85bbccb1c9. ("x86/32-on-64 adjust Dom0 initial
page
+ * table layout") a bug caused the pt_base (3.e above) to contain a
different
+ * value than the CR3 register. This only manifested itself on 32-on-64 dom0
+ * kernels and not 32-on-64 domU or 64-bit kernels of any colour. The
+ * pt_base pointed to the L4 (setup by the hypervisor and not used by
+ * the guest) and the %cr3 pointed to the L3. This meant an difference of
+ * one page. The guest should _NOT_ use the pages from pt_base to %cr3
+ * for anything and mark them as reserved/unused.
  */
 
 #define MAX_GUEST_CMDLINE 1024

Ian Campbell

2013-Jan-31 09:58 UTC

head link

Re: [PATCH 7/8] docs: Document start_info changes in Xen 4.2.

On Wed, 2013-01-30 at 17:23 +0000, Konrad Rzeszutek Wilk
wrote:> On Tue, Jan 29, 2013 at 09:37:00AM +0000, Ian Campbell wrote:
> > On Mon, 2013-01-28 at 18:32 +0000, Konrad Rzeszutek Wilk wrote:
> > > The git commit 7a9d7646307b7c872b8dbd7546579acd3b54223d
"x86/32-on-64:
> > > adjust Dom0 initial page table layout" fixes a bug in the
reported
> > > value of pt_base versus what is stored in the %cr3 register. This
> > > documents this in the start of the world header note.
> > > 
> > > Signed-off-by: Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com>
> > > ---
> > >  xen/include/public/xen.h | 7 +++++++
> > >  1 file changed, 7 insertions(+)
> > > 
> > > diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> > > index c50bbfc..1685317 100644
> > > --- a/xen/include/public/xen.h
> > > +++ b/xen/include/public/xen.h
> > > @@ -705,6 +705,13 @@ typedef struct shared_info shared_info_t;
> > >   *  8. There is guaranteed to be at least 512kB padding after
the final
> > >   *     bootstrap element. If necessary, the bootstrap virtual
region is
> > >   *     extended by an extra 4MB to ensure this.
> > > + *
> > > + * Note: Prior to 7a9d7646307b7c872b8dbd7546579acd3b54223d
("x86/32-on-64:
> > > + * adjust Dom0 initial page table layout") the 3.e)
contained two different
> > > + * values with a 64-bit hypervisor and a 32-bit initial domain
kernel. The
> > > + * pt_base pointed to the L4 (setup by the hypervisor and not
used by
> > > + * the guest) and the %cr3 pointed to the L3. This meant an
difference of
> > > + * one page.
> > >   */
> > 
> > 7a9d7646307b7c872b8dbd7546579acd3b54223d is not a Xen revision. I
think
> > you meant 25833:bb85bbccb1c9. I assume this was a bug fix, in which
case
> > I would say "Prior to Xen commit xxx:yyyy
("x86/32-on...") a bug caused
> > the pt_base (3.e above) to contain ..."
> > 
> > It would be useful to explain clearly the after case too, which I
> > suppose is that on 32-on-64 the pt_base now points to what the guest
> > kernel should consider to be the base? Would it be relevant to explain
> > how to detect this situation and what to do about it?
> > 
> > The difference of one page was just an implementation artefact, rather
> > than a meaningful semantic difference? Unless it is relevant to
> > detecting the situation I would be inclined not to do it.
> 
> It can be construed either way. If you look at:
> 
>  *      e. bootstrap page tables         [pt_base, CR3 (x86)]
> 
> It mentions _both_ sources of information - the register and the
> contents of pt_base. It could imply that both of them have the same
> value (if you read that as an enumeration of where this value is
> contained).
This comment is written from the context of a 32-bit hypervisor, where
this would be true. I think it''s just become inaccurate on 32-on-64 due
to the trick we play with grafting the guest 3-level tables into a
4-level table.
>  But if you read the '','' as an ''or'',
then it is not a
> semantic diference as they both could contain drastically diferent
> values.
> 
> Implementation wise, we get is that the value that CR3 is different than
what:
>     unsigned long pt_base;      /* VIRTUAL address of page directory.  */
> 
> has. In other words, the pt_base is suspect and should be carefully
> compared to the %cr3. If they are different, then the user should _NOT_
> touch the pages in between pt_base and %cr3 and mark them as unusable.
> 
> > 
> > It should also be made clearer that this affected only 32-on-64 dom0
> > kernels and not 32-on-64 domU or 64-bit kernels of any colour.
> 
> Right. How about this:
> 
> commit 489682893ffc8bbbbb12ee820defac17cce762d0
> Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Date:   Mon Jan 28 13:20:29 2013 -0500
> 
>     docs: Document start_info changes in Xen 4.2.
>     
>     The  25833:bb85bbccb1c9. "x86/32-on-64: adjust Dom0 initial page
table layout"
>     fixes a bug in the reported value of pt_base versus what is stored in
>     the %cr3 register. This documents this in the start of the world header
note.
>     
>     Also make it crystal clear that pt_base == %cr3.
>     
>     Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> 
> diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
> index c50bbfc..3f51cd6 100644
> --- a/xen/include/public/xen.h
> +++ b/xen/include/public/xen.h
> @@ -693,7 +693,7 @@ typedef struct shared_info shared_info_t;
>   *      c. list of allocated page frames [mfn_list, nr_pages]
>   *         (unless relocated due to XEN_ELFNOTE_INIT_P2M)
>   *      d. start_info_t structure        [register ESI (x86)]
> - *      e. bootstrap page tables         [pt_base, CR3 (x86)]
> + *      e. bootstrap page tables         [pt_base and CR3 (x86)]
>   *      f. bootstrap stack               [register ESP (x86)]
>   *  4. Bootstrap elements are packed together, but each is 4kB-aligned.
>   *  5. The initial ram disk may be omitted.
> @@ -705,6 +705,15 @@ typedef struct shared_info shared_info_t;
>   *  8. There is guaranteed to be at least 512kB padding after the final
>   *     bootstrap element. If necessary, the bootstrap virtual region is
>   *     extended by an extra 4MB to ensure this.
> + *
> + * Note: Prior to 25833:bb85bbccb1c9. ("x86/32-on-64 adjust Dom0
initial page
> + * table layout") a bug caused the pt_base (3.e above) to contain a
different
> + * value than the CR3 register.
I don''t think this is right, the bug was that the L4 table was in guest
space at all and that pt_base pointed to it, wasn''t it? The fix was to
remove the L4 from guest space altogether,

Remember that this is a 32-on-64 guest so the 64-bit hypervisor should
be providing the illusion of running on the 32-bit hypervisor, at least
so far as it is able (and perhaps the guest reading %cr3 direct is one
of those places where it can''t)

Perhaps Jan can correct our misconceptions on this one.
>  This only manifested itself on 32-on-64 dom0
> + * kernels and not 32-on-64 domU or 64-bit kernels of any colour. The
> + * pt_base pointed to the L4 (setup by the hypervisor and not used by
> + * the guest) and the %cr3 pointed to the L3.
That sounds backwards, if actual %cr3 pointed to L3 things wouldn''t
work
-- the processor is in long mode (on a compat segment) and so needs 4
level pages.
>  This meant an difference of
> + * one page. The guest should _NOT_ use the pages from pt_base to %cr3
> + * for anything and mark them as reserved/unused.
>   */
>  
>  #define MAX_GUEST_CMDLINE 1024

Xen devel - Jan 2013 - [DOCS/PATCH] http://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management changes

[DOCS/PATCH] http://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management changes

[PATCH 1/8] docs: Document the ELF notes

[PATCH 2/8] docs: Provide some examples of the various ELF values.

[PATCH 3/8] docs: Document the ELF_FEATURES entry

[PATCH 4/8] docs: Add some extra details to the ELF note.

[PATCH 5/8] docs: Document the shared structure.

[PATCH 6/8] docs: Document the dom0_vga_console_info structure.

[PATCH 7/8] docs: Document start_info changes in Xen 4.2.

[PATCH 8/8] docs: Document the XenBus structure.

Re: [PATCH 1/8] docs: Document the ELF notes

Re: [PATCH 2/8] docs: Provide some examples of the various ELF values.

Re: [PATCH 3/8] docs: Document the ELF_FEATURES entry

Re: [PATCH 4/8] docs: Add some extra details to the ELF note.

Re: [PATCH 5/8] docs: Document the shared structure.

Re: [PATCH 6/8] docs: Document the dom0_vga_console_info structure.

Re: [PATCH 7/8] docs: Document start_info changes in Xen 4.2.

Re: [PATCH 8/8] docs: Document the XenBus structure.

Re: [DOCS/PATCH] http://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management changes

Re: [PATCH 6/8] docs: Document the dom0_vga_console_info structure.

Re: [PATCH 7/8] docs: Document start_info changes in Xen 4.2.

Re: [PATCH 7/8] docs: Document start_info changes in Xen 4.2.