thr3ads.net - Xen devel - [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs [Feb 2013]

If this information is useful, please help other people find it:
Share via:

Stefan Bader

2013-Feb-14 17:11 UTC

[PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On a machine that could not cover all its RAM with MTRR ranges,
a crash on boot as dom0 was caused by dom0 trying to create
kernel mapping tables for the clipped range.

(XEN) WARNING: MTRRs do not cover all of memory.
(XEN) Truncating RAM from 9109504kB to 9043968kB
...
(XEN)  0000000228000000 - 000000022c000000 (unusable)
...
[    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
[    0.000000]  0228000000 - 022c000000 page 4k
[    0.000000] kernel direct mapping tables up to 22c000000 @
               1e97d8000-1e97fa000
(XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
(XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
(XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
      type 1000000000000000: caf=8000000000000003 taf=1000000000000001
(XEN) mm.c:2985:d0 Error while pinning mfn 81de

Setting the range in E820 to E280_UNUSABLE seems ambigous as
this is the same type that gets used for memory to be used only
as guest memory (using dom0_mem=).

To avoid this, the clipped memory should be dropped completely
from E820 (as it is done if setting the memory type fails).
This is currently restricted to only the case of memory not
coverable by MTRRs (which could be tested). Maybe it should
be done in other cases, too.

BugLink: http://bugs.launchpad.net/bugs/1111470

Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 xen/arch/x86/e820.c |   13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/e820.c b/xen/arch/x86/e820.c
index 55fe0d6..8dfe427 100644
--- a/xen/arch/x86/e820.c
+++ b/xen/arch/x86/e820.c
@@ -372,7 +372,7 @@ static unsigned long __init find_max_pfn(void)
     return max_pfn;
 }
 
-static void __init clip_to_limit(uint64_t limit, char *warnmsg)
+static void __init clip_to_limit(uint64_t limit, char *warnmsg, int drop)
 {
     int i;
     char _warnmsg[160];
@@ -394,7 +394,8 @@ static void __init clip_to_limit(uint64_t limit, char
*warnmsg)
             uint64_t, old_limit, e820.map[i].addr + e820.map[i].size);
 
         /* We try to convert clipped RAM areas to E820_UNUSABLE. */
-        if ( e820_change_range_type(&e820, max(e820.map[i].addr, limit),
+        if ( !drop &&
+             e820_change_range_type(&e820, max(e820.map[i].addr, limit),
                                     e820.map[i].addr + e820.map[i].size,
                                     E820_RAM, E820_UNUSABLE) )
             continue;
@@ -524,7 +525,7 @@ static void __init machine_specific_memory_setup(
     (void)copy_e820_map(raw, nr);
 
     if ( opt_mem )
-        clip_to_limit(opt_mem, NULL);
+        clip_to_limit(opt_mem, NULL, 0);
 
     if ( opt_availmem )
     {
@@ -534,7 +535,7 @@ static void __init machine_specific_memory_setup(
         if ( size > opt_availmem )
             clip_to_limit(
                 e820.map[i-1].addr + e820.map[i-1].size - (size-opt_availmem),
-                NULL);
+                NULL, 0);
     }
 
     mpt_limit = ((RDWR_MPT_VIRT_END - RDWR_MPT_VIRT_START)
@@ -545,13 +546,13 @@ static void __init machine_specific_memory_setup(
         mpt_limit = ro_mpt_limit;
     clip_to_limit(mpt_limit,
                   "Only the first %lu GB of the physical "
-                  "memory map can be accessed by Xen.");
+                  "memory map can be accessed by Xen.", 0);
 
     reserve_dmi_region();
 
     top_of_ram = mtrr_top_of_ram();
     if ( top_of_ram )
-        clip_to_limit(top_of_ram, "MTRRs do not cover all of
memory.");
+        clip_to_limit(top_of_ram, "MTRRs do not cover all of
memory.", 1);
 }
 
 /* This function relies on the passed in e820->map[] being sorted. */
-- 
1.7.9.5

Jan Beulich

2013-Feb-15 09:19 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

>>> On 14.02.13 at 18:11, Stefan Bader
<stefan.bader@canonical.com> wrote:
> On a machine that could not cover all its RAM with MTRR ranges,
> a crash on boot as dom0 was caused by dom0 trying to create
> kernel mapping tables for the clipped range.
> 
> (XEN) WARNING: MTRRs do not cover all of memory.
> (XEN) Truncating RAM from 9109504kB to 9043968kB
> ...
> (XEN)  0000000228000000 - 000000022c000000 (unusable)
> ...
> [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
> [    0.000000]  0228000000 - 022c000000 page 4k
> [    0.000000] kernel direct mapping tables up to 22c000000 @
>                1e97d8000-1e97fa000
> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
>       type 1000000000000000: caf=8000000000000003 taf=1000000000000001
> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
> 
> Setting the range in E820 to E280_UNUSABLE seems ambigous as
> this is the same type that gets used for memory to be used only
> as guest memory (using dom0_mem=).
Since when is E820_UNUSABLE memory being used as guest
memory? If that''s indeed the case, that''s the bug to fix. The
above data to me shows, however, that the range above
228000000 is considered invalid. So then the question is why the
kernel tries to map that memory in the first place (the hypervisor
rejecting the attempt, despite Dom0 being privileged, seems
correct to me, as the range is also known not to be MMIO).
> To avoid this, the clipped memory should be dropped completely
> from E820 (as it is done if setting the memory type fails).
> This is currently restricted to only the case of memory not
> coverable by MTRRs (which could be tested). Maybe it should
> be done in other cases, too.
No, that''s wrong. When dropping the range completely, the
Dom0 kernel then could allocate MMIO resources in that address
range, and the devices using those MMIO resources then would
not work.

Bottom line - I think this needs to be fixed in the kernel.

Jan
> BugLink: http://bugs.launchpad.net/bugs/1111470 
> 
> Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  xen/arch/x86/e820.c |   13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/x86/e820.c b/xen/arch/x86/e820.c
> index 55fe0d6..8dfe427 100644
> --- a/xen/arch/x86/e820.c
> +++ b/xen/arch/x86/e820.c
> @@ -372,7 +372,7 @@ static unsigned long __init find_max_pfn(void)
>      return max_pfn;
>  }
>  
> -static void __init clip_to_limit(uint64_t limit, char *warnmsg)
> +static void __init clip_to_limit(uint64_t limit, char *warnmsg, int drop)
>  {
>      int i;
>      char _warnmsg[160];
> @@ -394,7 +394,8 @@ static void __init clip_to_limit(uint64_t limit, char 
> *warnmsg)
>              uint64_t, old_limit, e820.map[i].addr + e820.map[i].size);
>  
>          /* We try to convert clipped RAM areas to E820_UNUSABLE. */
> -        if ( e820_change_range_type(&e820, max(e820.map[i].addr,
limit),
> +        if ( !drop &&
> +             e820_change_range_type(&e820, max(e820.map[i].addr,
limit),
>                                      e820.map[i].addr + e820.map[i].size,
>                                      E820_RAM, E820_UNUSABLE) )
>              continue;
> @@ -524,7 +525,7 @@ static void __init machine_specific_memory_setup(
>      (void)copy_e820_map(raw, nr);
>  
>      if ( opt_mem )
> -        clip_to_limit(opt_mem, NULL);
> +        clip_to_limit(opt_mem, NULL, 0);
>  
>      if ( opt_availmem )
>      {
> @@ -534,7 +535,7 @@ static void __init machine_specific_memory_setup(
>          if ( size > opt_availmem )
>              clip_to_limit(
>                  e820.map[i-1].addr + e820.map[i-1].size -
(size-opt_availmem),
> -                NULL);
> +                NULL, 0);
>      }
>  
>      mpt_limit = ((RDWR_MPT_VIRT_END - RDWR_MPT_VIRT_START)
> @@ -545,13 +546,13 @@ static void __init machine_specific_memory_setup(
>          mpt_limit = ro_mpt_limit;
>      clip_to_limit(mpt_limit,
>                    "Only the first %lu GB of the physical "
> -                  "memory map can be accessed by Xen.");
> +                  "memory map can be accessed by Xen.", 0);
>  
>      reserve_dmi_region();
>  
>      top_of_ram = mtrr_top_of_ram();
>      if ( top_of_ram )
> -        clip_to_limit(top_of_ram, "MTRRs do not cover all of
memory.");
> +        clip_to_limit(top_of_ram, "MTRRs do not cover all of
memory.", 1);
>  }
>  
>  /* This function relies on the passed in e820->map[] being sorted. */
> -- 
> 1.7.9.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org 
> http://lists.xen.org/xen-devel

Stefan Bader

2013-Feb-15 10:34 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On 15.02.2013 10:19, Jan Beulich wrote:>>>> On 14.02.13 at 18:11, Stefan Bader
<stefan.bader@canonical.com> wrote:
>> On a machine that could not cover all its RAM with MTRR ranges,
>> a crash on boot as dom0 was caused by dom0 trying to create
>> kernel mapping tables for the clipped range.
>>
>> (XEN) WARNING: MTRRs do not cover all of memory.
>> (XEN) Truncating RAM from 9109504kB to 9043968kB
>> ...
>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
>> ...
>> [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
>> [    0.000000]  0228000000 - 022c000000 page 4k
>> [    0.000000] kernel direct mapping tables up to 22c000000 @
>>                1e97d8000-1e97fa000
>> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space 00228000
>> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
>> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
>>       type 1000000000000000: caf=8000000000000003 taf=1000000000000001
>> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
>>
>> Setting the range in E820 to E280_UNUSABLE seems ambigous as
>> this is the same type that gets used for memory to be used only
>> as guest memory (using dom0_mem=).
> 
> Since when is E820_UNUSABLE memory being used as guest
> memory? If that''s indeed the case, that''s the bug to fix.
The
> above data to me shows, however, that the range above
> 228000000 is considered invalid. So then the question is why the
> kernel tries to map that memory in the first place (the hypervisor
> rejecting the attempt, despite Dom0 being privileged, seems
> correct to me, as the range is also known not to be MMIO).
That seems to be done for a while now... On my testbox, when using dom0_mem=:

(XEN)  00000000dfe9e000 - 00000000dfea0000 type 9
(XEN)  00000000dfea0000 - 00000000dfeb2000 (ACPI data)
(XEN)  00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
(XEN)  00000000dfee0000 - 00000000f0000000 (reserved)
(XEN)  00000000ffe00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000820000000 (usable)

In dom0:
[    0.000000] Xen: [mem 0x00000000dfe9e000-0x00000000dfe9ffff] type 9
[    0.000000] Xen: [mem 0x00000000dfea0000-0x00000000dfeb1fff] ACPI data
[    0.000000] Xen: [mem 0x00000000dfeb2000-0x00000000dfedffff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000dfee0000-0x00000000efffffff] reserved
[    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] Xen: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x00000002e063ffff] usable
[    0.000000] Xen: [mem 0x00000002e0640000-0x000000081fffffff] unusable

[    0.000000] init_memory_mapping: [mem 0x100000000-0x2e063ffff]
[    0.000000]  [mem 0x100000000-0x2e063ffff] page 4k
[    0.000000] kernel direct mapping tables up to 0x2e063ffff @ [mem
0x3f163000-0x4006ffff]

[   10.288549] e820: reserve RAM buffer [mem 0xdfe90000-0xdfffffff]
[   10.288555] e820: reserve RAM buffer [mem 0x2e0640000-0x2e3ffffff]

There the majority of memory appears as E820_UNUSABLE for dom0 and is used for
guests. Ok, in that case not trying to do any kernel mappings...
Hm, not sure whether I should be worried about that first reserve RAM buffer
covering the type 9 (whatever that is... I think it appeared when I upgraded the
BIOS to get IOMMU back after xsa-36) range...

It seems to be arch/x86/xen/setup.c:xen_memory_setup() that changes things to
unusable...

-Stefan> 
>> To avoid this, the clipped memory should be dropped completely
>> from E820 (as it is done if setting the memory type fails).
>> This is currently restricted to only the case of memory not
>> coverable by MTRRs (which could be tested). Maybe it should
>> be done in other cases, too.
> 
> No, that''s wrong. When dropping the range completely, the
> Dom0 kernel then could allocate MMIO resources in that address
> range, and the devices using those MMIO resources then would
> not work.
> 
> Bottom line - I think this needs to be fixed in the kernel.
> 
> Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Jan Beulich

2013-Feb-15 11:13 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

>>> On 15.02.13 at 11:34, Stefan Bader
<stefan.bader@canonical.com> wrote:
> On 15.02.2013 10:19, Jan Beulich wrote:
>>>>> On 14.02.13 at 18:11, Stefan Bader
<stefan.bader@canonical.com> wrote:
>>> On a machine that could not cover all its RAM with MTRR ranges,
>>> a crash on boot as dom0 was caused by dom0 trying to create
>>> kernel mapping tables for the clipped range.
>>>
>>> (XEN) WARNING: MTRRs do not cover all of memory.
>>> (XEN) Truncating RAM from 9109504kB to 9043968kB
>>> ...
>>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
>>> ...
>>> [    0.000000] init_memory_mapping:
0000000228000000-000000022c000000
>>> [    0.000000]  0228000000 - 022c000000 page 4k
>>> [    0.000000] kernel direct mapping tables up to 22c000000 @
>>>                1e97d8000-1e97fa000
>>> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space
00228000
>>> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
>>> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8) for
>>>       type 1000000000000000: caf=8000000000000003
taf=1000000000000001
>>> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
>>>
>>> Setting the range in E820 to E280_UNUSABLE seems ambigous as
>>> this is the same type that gets used for memory to be used only
>>> as guest memory (using dom0_mem=).
>> 
>> Since when is E820_UNUSABLE memory being used as guest
>> memory? If that''s indeed the case, that''s the bug to
fix. The
>> above data to me shows, however, that the range above
>> 228000000 is considered invalid. So then the question is why the
>> kernel tries to map that memory in the first place (the hypervisor
>> rejecting the attempt, despite Dom0 being privileged, seems
>> correct to me, as the range is also known not to be MMIO).
> 
> That seems to be done for a while now... On my testbox, when using 
> dom0_mem=:
> 
> (XEN)  00000000dfe9e000 - 00000000dfea0000 type 9
> (XEN)  00000000dfea0000 - 00000000dfeb2000 (ACPI data)
> (XEN)  00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
> (XEN)  00000000dfee0000 - 00000000f0000000 (reserved)
> (XEN)  00000000ffe00000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 0000000820000000 (usable)
All that counts is what you see above.
> In dom0:
> [    0.000000] Xen: [mem 0x00000000dfe9e000-0x00000000dfe9ffff] type 9
> [    0.000000] Xen: [mem 0x00000000dfea0000-0x00000000dfeb1fff] ACPI data
> [    0.000000] Xen: [mem 0x00000000dfeb2000-0x00000000dfedffff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000dfee0000-0x00000000efffffff] reserved
> [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
> [    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff] reserved
> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> [    0.000000] Xen: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
> [    0.000000] Xen: [mem 0x0000000100000000-0x00000002e063ffff] usable
> [    0.000000] Xen: [mem 0x00000002e0640000-0x000000081fffffff] unusable
What Dom0 does to its representation of the E820 map is of no
concern to the hypervisor. So are we in agreement then that the
hypervisor code is fine without any changes?

Jan

Stefan Bader

2013-Feb-15 11:31 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On 15.02.2013 12:13, Jan Beulich wrote:>>>> On 15.02.13 at 11:34, Stefan Bader
<stefan.bader@canonical.com> wrote:
>> On 15.02.2013 10:19, Jan Beulich wrote:
>>>>>> On 14.02.13 at 18:11, Stefan Bader
<stefan.bader@canonical.com> wrote:
>>>> On a machine that could not cover all its RAM with MTRR ranges,
>>>> a crash on boot as dom0 was caused by dom0 trying to create
>>>> kernel mapping tables for the clipped range.
>>>>
>>>> (XEN) WARNING: MTRRs do not cover all of memory.
>>>> (XEN) Truncating RAM from 9109504kB to 9043968kB
>>>> ...
>>>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
>>>> ...
>>>> [    0.000000] init_memory_mapping:
0000000228000000-000000022c000000
>>>> [    0.000000]  0228000000 - 022c000000 page 4k
>>>> [    0.000000] kernel direct mapping tables up to 22c000000 @
>>>>                1e97d8000-1e97fa000
>>>> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space
00228000
>>>> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
>>>> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn 1e97d8)
for
>>>>       type 1000000000000000: caf=8000000000000003
taf=1000000000000001
>>>> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
>>>>
>>>> Setting the range in E820 to E280_UNUSABLE seems ambigous as
>>>> this is the same type that gets used for memory to be used only
>>>> as guest memory (using dom0_mem=).
>>>
>>> Since when is E820_UNUSABLE memory being used as guest
>>> memory? If that''s indeed the case, that''s the bug
to fix. The
>>> above data to me shows, however, that the range above
>>> 228000000 is considered invalid. So then the question is why the
>>> kernel tries to map that memory in the first place (the hypervisor
>>> rejecting the attempt, despite Dom0 being privileged, seems
>>> correct to me, as the range is also known not to be MMIO).
>>
>> That seems to be done for a while now... On my testbox, when using 
>> dom0_mem=:
>>
>> (XEN)  00000000dfe9e000 - 00000000dfea0000 type 9
>> (XEN)  00000000dfea0000 - 00000000dfeb2000 (ACPI data)
>> (XEN)  00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
>> (XEN)  00000000dfee0000 - 00000000f0000000 (reserved)
>> (XEN)  00000000ffe00000 - 0000000100000000 (reserved)
>> (XEN)  0000000100000000 - 0000000820000000 (usable)
> 
> All that counts is what you see above.
> 
>> In dom0:
>> [    0.000000] Xen: [mem 0x00000000dfe9e000-0x00000000dfe9ffff] type 9
>> [    0.000000] Xen: [mem 0x00000000dfea0000-0x00000000dfeb1fff] ACPI
data
>> [    0.000000] Xen: [mem 0x00000000dfeb2000-0x00000000dfedffff] ACPI
NVS
>> [    0.000000] Xen: [mem 0x00000000dfee0000-0x00000000efffffff]
reserved
>> [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff]
reserved
>> [    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff]
reserved
>> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff]
reserved
>> [    0.000000] Xen: [mem 0x00000000ffe00000-0x00000000ffffffff]
reserved
>> [    0.000000] Xen: [mem 0x0000000100000000-0x00000002e063ffff] usable
>> [    0.000000] Xen: [mem 0x00000002e0640000-0x000000081fffffff]
unusable
> 
> What Dom0 does to its representation of the E820 map is of no
> concern to the hypervisor. So are we in agreement then that the
> hypervisor code is fine without any changes?
Yes, if we also agree that using unusable for something that is then used is
wrong. So lets drop the xen patch and I try to figure fixing the kernel.

-Stefan




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Konrad Rzeszutek Wilk

2013-Feb-15 13:47 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On Fri, Feb 15, 2013 at 11:13:52AM +0000, Jan Beulich
wrote:> >>> On 15.02.13 at 11:34, Stefan Bader
<stefan.bader@canonical.com> wrote:
> > On 15.02.2013 10:19, Jan Beulich wrote:
> >>>>> On 14.02.13 at 18:11, Stefan Bader
<stefan.bader@canonical.com> wrote:
> >>> On a machine that could not cover all its RAM with MTRR
ranges,
> >>> a crash on boot as dom0 was caused by dom0 trying to create
> >>> kernel mapping tables for the clipped range.
> >>>
> >>> (XEN) WARNING: MTRRs do not cover all of memory.
> >>> (XEN) Truncating RAM from 9109504kB to 9043968kB
> >>> ...
> >>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
> >>> ...
> >>> [    0.000000] init_memory_mapping:
0000000228000000-000000022c000000
> >>> [    0.000000]  0228000000 - 022c000000 page 4k
> >>> [    0.000000] kernel direct mapping tables up to 22c000000 @
> >>>                1e97d8000-1e97fa000
> >>> (XEN) mm.c:825:d0 Non-privileged (0) attempt to map I/O space
00228000
> >>> (XEN) mm.c:1222:d0 Failure in alloc_l1_table: entry 0
> >>> (XEN) mm.c:2177:d0 Error while validating mfn 81de (pfn
1e97d8) for
> >>>       type 1000000000000000: caf=8000000000000003
taf=1000000000000001
> >>> (XEN) mm.c:2985:d0 Error while pinning mfn 81de
> >>>
> >>> Setting the range in E820 to E280_UNUSABLE seems ambigous as
> >>> this is the same type that gets used for memory to be used
only
> >>> as guest memory (using dom0_mem=).
> >> 
> >> Since when is E820_UNUSABLE memory being used as guest
> >> memory? If that''s indeed the case, that''s the
bug to fix. The
> >> above data to me shows, however, that the range above
> >> 228000000 is considered invalid. So then the question is why the
> >> kernel tries to map that memory in the first place (the hypervisor
> >> rejecting the attempt, despite Dom0 being privileged, seems
> >> correct to me, as the range is also known not to be MMIO).
B/c it gets the E820 from the hypervisor, which shows that area as
E820_UNUSABLE. And dom0 (or rather, the generic memory code) ends up
creating pagetables for it.
> > 
> > That seems to be done for a while now... On my testbox, when using 
> > dom0_mem=:
> > 
> > (XEN)  00000000dfe9e000 - 00000000dfea0000 type 9
> > (XEN)  00000000dfea0000 - 00000000dfeb2000 (ACPI data)
> > (XEN)  00000000dfeb2000 - 00000000dfee0000 (ACPI NVS)
> > (XEN)  00000000dfee0000 - 00000000f0000000 (reserved)
> > (XEN)  00000000ffe00000 - 0000000100000000 (reserved)
> > (XEN)  0000000100000000 - 0000000820000000 (usable)
> 
> All that counts is what you see above.
> 
> > In dom0:
> > [    0.000000] Xen: [mem 0x00000000dfe9e000-0x00000000dfe9ffff] type 9
> > [    0.000000] Xen: [mem 0x00000000dfea0000-0x00000000dfeb1fff] ACPI
data
> > [    0.000000] Xen: [mem 0x00000000dfeb2000-0x00000000dfedffff] ACPI
NVS
> > [    0.000000] Xen: [mem 0x00000000dfee0000-0x00000000efffffff]
reserved
> > [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff]
reserved
> > [    0.000000] Xen: [mem 0x00000000fec20000-0x00000000fec20fff]
reserved
> > [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff]
reserved
> > [    0.000000] Xen: [mem 0x00000000ffe00000-0x00000000ffffffff]
reserved
> > [    0.000000] Xen: [mem 0x0000000100000000-0x00000002e063ffff] usable
> > [    0.000000] Xen: [mem 0x00000002e0640000-0x000000081fffffff]
unusable
> 
> What Dom0 does to its representation of the E820 map is of no
> concern to the hypervisor. So are we in agreement then that the
> hypervisor code is fine without any changes?
I am not clear on what the right fix is. Dom0 could interpret E820_UNUSABLE
as memory to be considered as E820_RESERVED - but then I am wondering what
to do with hotplug memory. As in, does the BIOS mark memory that can be
hotplugged
as E820_RESERVED or E820_UNUSABLE? So if we mark the memory as E820_RESERVED
and the hotplug memory code expects that - will we confuse it as there are now
new ranges of it?

Perhaps dom0 should just eviscerate that range altogether. But then it will
be considered as MMIO region and that will lead to trouble with i915 at least.

Keir Fraser

2013-Feb-15 14:00 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On 15/02/2013 13:47, "Konrad Rzeszutek Wilk"
<konrad.wilk@oracle.com> wrote:
>> What Dom0 does to its representation of the E820 map is of no
>> concern to the hypervisor. So are we in agreement then that the
>> hypervisor code is fine without any changes?
> 
> I am not clear on what the right fix is. Dom0 could interpret E820_UNUSABLE
> as memory to be considered as E820_RESERVED - but then I am wondering what
> to do with hotplug memory. As in, does the BIOS mark memory that can be
> hotplugged
> as E820_RESERVED or E820_UNUSABLE? So if we mark the memory as
E820_RESERVED
> and the hotplug memory code expects that - will we confuse it as there are
now
> new ranges of it?
> 
> Perhaps dom0 should just eviscerate that range altogether. But then it will
> be considered as MMIO region and that will lead to trouble with i915 at
least.
I don''t think E820_RESERVED is specifically for hotplug use only, and
we
could use it here in place of E820_UNUSABLE. That may be more correct, since
E820_UNUSABLE is specifically intended for RAM in which errors have been
detected.

 -- Keir

Jan Beulich

2013-Feb-15 14:52 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

>>> On 15.02.13 at 14:47, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>> >> Since when is E820_UNUSABLE memory being used as guest
>> >> memory? If that''s indeed the case, that''s
the bug to fix. The
>> >> above data to me shows, however, that the range above
>> >> 228000000 is considered invalid. So then the question is why
the
>> >> kernel tries to map that memory in the first place (the
hypervisor
>> >> rejecting the attempt, despite Dom0 being privileged, seems
>> >> correct to me, as the range is also known not to be MMIO).
> 
> B/c it gets the E820 from the hypervisor, which shows that area as
> E820_UNUSABLE. And dom0 (or rather, the generic memory code) ends up
> creating pagetables for it.
That would be wrong even on native, and I don''t see how that
would happen: memblock_add() gets called from
memblock_x86_fill() only for E820_RAM and E820_RESERVED_KERN
ranges.

Jan

Konrad Rzeszutek Wilk

2013-Feb-15 16:40 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On Fri, Feb 15, 2013 at 02:52:41PM +0000, Jan Beulich
wrote:> >>> On 15.02.13 at 14:47, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> >> >> Since when is E820_UNUSABLE memory being used as guest
> >> >> memory? If that''s indeed the case,
that''s the bug to fix. The
> >> >> above data to me shows, however, that the range above
> >> >> 228000000 is considered invalid. So then the question is
why the
> >> >> kernel tries to map that memory in the first place (the
hypervisor
> >> >> rejecting the attempt, despite Dom0 being privileged,
seems
> >> >> correct to me, as the range is also known not to be
MMIO).
> > 
> > B/c it gets the E820 from the hypervisor, which shows that area as
> > E820_UNUSABLE. And dom0 (or rather, the generic memory code) ends up
> > creating pagetables for it.
> 
> That would be wrong even on native, and I don''t see how that
> would happen: memblock_add() gets called from
> memblock_x86_fill() only for E820_RAM and E820_RESERVED_KERN
> ranges.
Hm, the bug report shows that the ranges (which are E820_UNUSABLE)
do get called with init_memory_mapping.

Stefan Bader

2013-Feb-15 16:45 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On 15.02.2013 17:40, Konrad Rzeszutek Wilk wrote:> On Fri, Feb 15, 2013 at 02:52:41PM +0000, Jan Beulich wrote:
>>>>> On 15.02.13 at 14:47, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>>>>>> Since when is E820_UNUSABLE memory being used as guest
>>>>>> memory? If that''s indeed the case,
that''s the bug to fix. The
>>>>>> above data to me shows, however, that the range above
>>>>>> 228000000 is considered invalid. So then the question
is why the
>>>>>> kernel tries to map that memory in the first place (the
hypervisor
>>>>>> rejecting the attempt, despite Dom0 being privileged,
seems
>>>>>> correct to me, as the range is also known not to be
MMIO).
>>>
>>> B/c it gets the E820 from the hypervisor, which shows that area as
>>> E820_UNUSABLE. And dom0 (or rather, the generic memory code) ends
up
>>> creating pagetables for it.
>>
>> That would be wrong even on native, and I don''t see how that
>> would happen: memblock_add() gets called from
>> memblock_x86_fill() only for E820_RAM and E820_RESERVED_KERN
>> ranges.
> 
> Hm, the bug report shows that the ranges (which are E820_UNUSABLE)
> do get called with init_memory_mapping.
> Not sure it makes the difference but keep in mind that the report is about a 3.2
kernel. They initially claimed that 3.5 works, but then some comments seem to
say that was when using dom0_mem= which would in that case work around the bug.
Maybe time to go back and ask whether a recent kernel without dom0_mem on the
same machine still crashes...




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

William Dauchy

2013-Feb-22 13:08 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On Fri, Feb 15, 2013 at 5:45 PM, Stefan Bader
<stefan.bader@canonical.com> wrote:> Not sure it makes the difference but keep in mind that the report is about
a 3.2
> kernel. They initially claimed that 3.5 works, but then some comments seem
to
> say that was when using dom0_mem= which would in that case work around the
bug.
> Maybe time to go back and ask whether a recent kernel without dom0_mem on
the
> same machine still crashes...
I have the same issue on a dom0 3.4.x (3.4.33 in my case)

-- 
William

Stefan Bader

2013-Feb-22 13:22 UTC

head link

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

On 22.02.2013 14:08, William Dauchy wrote:> On Fri, Feb 15, 2013 at 5:45 PM, Stefan Bader
> <stefan.bader@canonical.com> wrote:
>> Not sure it makes the difference but keep in mind that the report is
about a 3.2
>> kernel. They initially claimed that 3.5 works, but then some comments
seem to
>> say that was when using dom0_mem= which would in that case work around
the bug.
>> Maybe time to go back and ask whether a recent kernel without dom0_mem
on the
>> same machine still crashes...
> 
> I have the same issue on a dom0 3.4.x (3.4.33 in my case)
> Hrm, while my reporter claims the affected machine boots with kernel 3.3 and
later. Something is wrong with the picture... somewhere... :/

-Stefan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Xen devel - Feb 2013 - [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

[PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs

Re: [PATCH] xen/x86: Force removal of memory range when not covered by MTRRs