thr3ads.net - Xen devel - Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box [Jul 2013]

If this information is useful, please help other people find it:
Share via:

Wei Liu

2013-Jul-08 10:31 UTC

Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Hardware:
Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz
8GB Ram

Kernel: stock Debian Whezzy 3.2.0-4-amd64

Xen: tested with 4.1.5, 4.2.2, unstable

All have the same problem and same log. Serial log attached.

Vanilla 3.10 kernel works.

================== __  __            _  _   _  _                      _        _
_
 \ \/ /___ _ __   | || | | || |     _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ ''_ \  | || |_| || |_ __| | | | ''_ \/ __| __/ _` |
''_ \| |/ _ \
  /  \  __/ | | | |__   _|__   _|__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_|    |_|(_) |_|     \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                          
(XEN) Xen version 4.4-unstable (weil@uk.xensource.com) (gcc (Debian 4.7.2-5)
4.7.2) debug=y Fri Jul  5 18:37:48 BST 2013
(XEN) Latest ChangeSet: Thu Jul 4 14:58:55 2013 +0100 git:1adea4d-dirty
(XEN) Console output is synchronous.
(XEN) Bootloader: GRUB 1.99-27+deb7u1
(XEN) Command line: placeholder loglvl=all guest_loglvl=all com1=115200,8n1
console=com1,vga console_to_ring sync_console
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) WARNING: MTRRs do not cover all of memory.
(XEN) Truncating RAM from 9109504kB to 9043968kB
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000008f000 (usable)
(XEN)  000000000008f000 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000ce69a000 (usable)
(XEN)  00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
(XEN)  00000000ce6f1000 - 00000000cf5fb000 (usable)
(XEN)  00000000cf5fb000 - 00000000cf608000 (reserved)
(XEN)  00000000cf608000 - 00000000cf6a5000 (usable)
(XEN)  00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
(XEN)  00000000cf6aa000 - 00000000cf6ab000 (usable)
(XEN)  00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
(XEN)  00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
(XEN)  00000000cf6ff000 - 00000000cf700000 (usable)
(XEN)  00000000cf700000 - 00000000d0000000 (reserved)
(XEN)  00000000fff00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000228000000 (usable)
(XEN)  0000000228000000 - 000000022c000000 (unusable)
(XEN) ACPI: RSDP 000FE020, 0014 (r0 INTEL )
(XEN) ACPI: RSDT CF6FD038, 0050 (r1 INTEL  DG965SS       685       1000013)
(XEN) ACPI: FACP CF6FC000, 0074 (r1 INTEL  DG965SS       685 MSFT  1000013)
(XEN) ACPI: DSDT CF6F7000, 40E9 (r1 INTEL  DG965SS       685 MSFT  1000013)
(XEN) ACPI: FACS CF6AB000, 0040
(XEN) ACPI: APIC CF6F6000, 0078 (r1 INTEL  DG965SS       685 MSFT  1000013)
(XEN) ACPI: WDDT CF6F5000, 0040 (r1 INTEL  DG965SS       685 MSFT  1000013)
(XEN) ACPI: MCFG CF6F4000, 003C (r1 INTEL  DG965SS       685 MSFT  1000013)
(XEN) ACPI: ASF! CF6F3000, 00A6 (r32 INTEL  DG965SS       685 MSFT  1000013)
(XEN) ACPI: HPET CF6F2000, 0038 (r1 INTEL  DG965SS       685 MSFT  1000013)
(XEN) ACPI: SSDT CF6A9000, 01BC (r1 INTEL     CpuPm      685 MSFT  1000013)
(XEN) ACPI: SSDT CF6A8000, 0175 (r1 INTEL   Cpu0Ist      685 MSFT  1000013)
(XEN) ACPI: SSDT CF6A7000, 0175 (r1 INTEL   Cpu1Ist      685 MSFT  1000013)
(XEN) ACPI: SSDT CF6A6000, 0175 (r1 INTEL   Cpu2Ist      685 MSFT  1000013)
(XEN) ACPI: SSDT CF6A5000, 0175 (r1 INTEL   Cpu3Ist      685 MSFT  1000013)
(XEN) System RAM: 8053MB (8247112kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000000228000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fe200
(XEN) DMI 2.4 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: SLEEP INFO: pm1x_cnt[404,0], pm1x_evt[400,0]
(XEN) ACPI:             wakeup_vec[cf6ab00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 6:15 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
(XEN) Processor #1 6:15 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)               
[97/2068]
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) ERST table was not found
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 4 CPUs (2 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 376 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2131.250 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:717: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1 extended
MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base f0000000 segment 0000 buses 00 - 7f
(XEN) PCI: Not using MCFG for segment 0000 bus 00-7f
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) mwait-idle: does not run on family 6 model 15
(XEN) VMX: Supported advanced features:
(XEN)  - APIC TPR shadow
(XEN)  - MSR direct-access bitmap
(XEN) HVM: ASIDs disabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) not detected
(XEN) Brought up 2 CPUs
(XEN) HPET: 3 timers (0 will be used for broadcast)
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x553000
(XEN) elf_parse_binary: phdr: paddr=0x1600000 memsz=0x950e0
(XEN) elf_parse_binary: phdr: paddr=0x1696000 memsz=0x14400
(XEN) elf_parse_binary: phdr: paddr=0x16ab000 memsz=0x292000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x193d000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff816ab200
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb"
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0x0
(XEN)     virt_offset      = 0xffffffff80000000
(XEN)     virt_kstart      = 0xffffffff81000000
(XEN)     virt_kend        = 0xffffffff8193d000
(XEN)     virt_entry       = 0xffffffff816ab200
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x193d000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000218000000->000000021c000000 (1980949 pages to be
allocated)
(XEN)  Init. ramdisk: 0000000226293000->0000000227fffc00
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff8193d000
(XEN)  Init. ramdisk: ffffffff8193d000->ffffffff836a9c00
(XEN)  Phys-Mach map: ffffffff836aa000->ffffffff845f5c10
(XEN)  Start info:    ffffffff845f6000->ffffffff845f64b4
(XEN)  Page tables:   ffffffff845f7000->ffffffff8461e000
(XEN)  Boot stack:    ffffffff8461e000->ffffffff8461f000
(XEN)  TOTAL:         ffffffff80000000->ffffffff84800000
(XEN)  ENTRY ADDRESS: ffffffff816ab200
(XEN) Dom0 has maximum 2 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff81553000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81600000 -> 0xffffffff816950e0
(XEN) elf_load_binary: phdr 2 at 0xffffffff81696000 -> 0xffffffff816aa400
(XEN) elf_load_binary: phdr 3 at 0xffffffff816ab000 -> 0xffffffff81729000
(XEN) Scrubbing Free RAM: .done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) **********************************************
(XEN) ******* WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) ******* This option is intended to aid debugging of Xen by ensuring
(XEN) ******* that all output is synchronously delivered on the serial line.
(XEN) ******* However it can introduce SIGNIFICANT latencies and affect
(XEN) ******* timekeeping. It is NOT recommended for production use!
(XEN) **********************************************
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times
to switch input to Xen)
(XEN) Freed 272kB init memory.
mapping kernel into physical memory
Xen: setup ISA identity maps
about to get started...
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org) (gcc
version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.46-1
[    0.000000] Command line: placeholder
root=UUID=f00ccdf7-bfa4-46cf-b94e-8c8ed2bcc5d6 ro console=tty0 console=hvc0
earlyprintk=xen
[    0.000000] Freeing  8f-100 pfn range: 113 pages freed
[    0.000000] Freeing  ce69a-ce6f1 pfn range: 87 pages freed
[    0.000000] Freeing  cf5fb-cf608 pfn range: 13 pages freed
[    0.000000] Freeing  cf6a5-cf6aa pfn range: 5 pages freed
[    0.000000] Freeing  cf6ab-cf6ff pfn range: 84 pages freed
[    0.000000] Freeing  cf700-100000 pfn range: 198912 pages freed
[    0.000000] Released 199214 pages of unused memory
[    0.000000] Set 215598 page(s) to 1-1 mapping
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 000000000008f000 (usable)
[    0.000000]  Xen: 000000000008f000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 00000000ce69a000 (usable)
[    0.000000]  Xen: 00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
[    0.000000]  Xen: 00000000ce6f1000 - 00000000cf5fb000 (usable)
[    0.000000]  Xen: 00000000cf5fb000 - 00000000cf608000 (reserved)
[    0.000000]  Xen: 00000000cf608000 - 00000000cf6a5000 (usable)
[    0.000000]  Xen: 00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
[    0.000000]  Xen: 00000000cf6aa000 - 00000000cf6ab000 (usable)
[    0.000000]  Xen: 00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
[    0.000000]  Xen: 00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
[    0.000000]  Xen: 00000000cf6ff000 - 00000000cf700000 (usable)
[    0.000000]  Xen: 00000000cf700000 - 00000000d0000000 (reserved)
[    0.000000]  Xen: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  Xen: 00000000fff00000 - 0000000100000000 (reserved)
[    0.000000]  Xen: 0000000100000000 - 0000000228000000 (usable)
[    0.000000]  Xen: 0000000228000000 - 000000022c000000 (unusable)
[    0.000000] bootconsole [xenboot0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x228000 max_arch_pfn = 0x400000000
[    0.000000] last_pfn = 0xcf700 max_arch_pfn = 0x400000000
[    0.000000] init_memory_mapping: 0000000000000000-00000000cf700000
[    0.000000] init_memory_mapping: 0000000100000000-0000000228000000
[    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
(XEN) mm.c:783:d0 Non-privileged (0) attempt to map I/O space 00228000
(XEN) mm.c:1219:d0 Failure in alloc_l1_table: entry 0
(XEN) mm.c:2096:d0 Error while validating mfn 226221 (pfn 1e9760) for type
1000000000000000: caf=8000000000000003 taf=1000000000000001
(XEN) mm.c:2992:d0 Error while pinning mfn 226221
(XEN) traps.c:455:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0
[ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.4-unstable  x86_64  debug=y  Tainted:    C ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff8134654c>]
(XEN) RFLAGS: 0000000000000286   EM: 1   CONTEXT: pv guest
(XEN) rax: ffffffffffffffff   rbx: ffff8801e9760000   rcx: ffffffff81809000
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000deadbeef
(XEN) rbp: ffffffffff4b8a00   rsp: ffffffff81601c18   r8:  0000000000000000
(XEN) r9:  00000000deadbeef   r10: 00000000deadbeef   r11: 0000000000000000
(XEN) r12: ffffffffff4b8a00   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 000000022c000000   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 0000000219605000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81601c18:
(XEN)    ffffffff81809000 0000000000000000 ffffffff8134654c 000000010000e030
(XEN)    0000000000010086 ffffffff81601c58 000000000000e02b ffffffff81346548
(XEN)    ffff8801e9760000 ffffffff00000000 0000000000226221 00000000001e9760
(XEN)    00000000deadbeef ffffffff8102d973 000000000fc40fc4 0000000228000000
(XEN)    0000000000000000 ffffffff8134483a ffffffffff4f8000 0000000228200000
(XEN)    0000000000000140 8000000000000163 00000140cf4b9000 8000000000000163
(XEN)    ffffffffff4b8000 80000000000001e3 ffffffffff478040 00000001e9760000
(XEN)    000000022c000000 0000000228000000 ffffffffff4b8000 0000000000000000
(XEN)    000000022c000000 0000000000000000 000000022c000000 ffffffff81344985
(XEN)    0000000000000002 8000000000000163 0000000000000008 0000000000000000
(XEN)    0000000000000000 ffffffffff478000 ffff88022c000000 0000000000000000
(XEN)    0000000000000000 ffffffffff478000 ffff88022c000000 0000000000000000
(XEN)    0000000000000000 ffff88022c000000 0000008000000000 ffffffff81344b76
(XEN)    000000022c000000 000000022c000000 ffff880228000000 ffff880228000000
(XEN)    ffffffff81601e28 ffffffff81601de8 ffffffff816c978c 0000000000000000
(XEN)    0000000000000001 ffffffff81601e58 0000000000000001 ffffffff814c8b77
(XEN)    000000022c000000 ffffffff81331ec0 ffff00205d303030 0000000228000000
(XEN)    0000000000000000 0000000228000000 000000022c000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

Jan Beulich

2013-Jul-08 11:29 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

>>> On 08.07.13 at 12:31, Wei Liu <wei.liu2@citrix.com> wrote:
$subject is probably the wrong way round, since ...
> (XEN)  0000000000000000 - 000000000008f000 (usable)
> (XEN)  000000000008f000 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 00000000ce69a000 (usable)
> (XEN)  00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
> (XEN)  00000000ce6f1000 - 00000000cf5fb000 (usable)
> (XEN)  00000000cf5fb000 - 00000000cf608000 (reserved)
> (XEN)  00000000cf608000 - 00000000cf6a5000 (usable)
> (XEN)  00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
> (XEN)  00000000cf6aa000 - 00000000cf6ab000 (usable)
> (XEN)  00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
> (XEN)  00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
> (XEN)  00000000cf6ff000 - 00000000cf700000 (usable)
> (XEN)  00000000cf700000 - 00000000d0000000 (reserved)
> (XEN)  00000000fff00000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 0000000228000000 (usable)
> (XEN)  0000000228000000 - 000000022c000000 (unusable)
.. this and ...
> [    0.000000]  Xen: 0000000000000000 - 000000000008f000 (usable)
> [    0.000000]  Xen: 000000000008f000 - 0000000000100000 (reserved)
> [    0.000000]  Xen: 0000000000100000 - 00000000ce69a000 (usable)
> [    0.000000]  Xen: 00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
> [    0.000000]  Xen: 00000000ce6f1000 - 00000000cf5fb000 (usable)
> [    0.000000]  Xen: 00000000cf5fb000 - 00000000cf608000 (reserved)
> [    0.000000]  Xen: 00000000cf608000 - 00000000cf6a5000 (usable)
> [    0.000000]  Xen: 00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
> [    0.000000]  Xen: 00000000cf6aa000 - 00000000cf6ab000 (usable)
> [    0.000000]  Xen: 00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
> [    0.000000]  Xen: 00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
> [    0.000000]  Xen: 00000000cf6ff000 - 00000000cf700000 (usable)
> [    0.000000]  Xen: 00000000cf700000 - 00000000d0000000 (reserved)
> [    0.000000]  Xen: 00000000fee00000 - 00000000fee01000 (reserved)
> [    0.000000]  Xen: 00000000fff00000 - 0000000100000000 (reserved)
> [    0.000000]  Xen: 0000000100000000 - 0000000228000000 (usable)
> [    0.000000]  Xen: 0000000228000000 - 000000022c000000 (unusable)
... this show that the kernel should be well aware that it shouldn''t
map (or use in any other way) this region.
> [    0.000000] init_memory_mapping: 0000000000000000-00000000cf700000
> [    0.000000] init_memory_mapping: 0000000100000000-0000000228000000
> [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
Yet it does, ...
> (XEN) mm.c:783:d0 Non-privileged (0) attempt to map I/O space 00228000
> (XEN) mm.c:1219:d0 Failure in alloc_l1_table: entry 0
> (XEN) mm.c:2096:d0 Error while validating mfn 226221 (pfn 1e9760) for type 
> 1000000000000000: caf=8000000000000003 taf=1000000000000001
> (XEN) mm.c:2992:d0 Error while pinning mfn 226221
... and Xen rightfully rejects the attempt.

One question of course is where this pretty unusual "unusable"
memory block comes from on that system. Is this block visible the
same way when booting a native kernel, or is this being forced to
"unusable" by Xen?

Jan

David Vrabel

2013-Jul-08 12:06 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On 08/07/13 12:29, Jan Beulich wrote:>>>> On 08.07.13 at 12:31, Wei Liu <wei.liu2@citrix.com>
wrote:
> 
> $subject is probably the wrong way round, since ...
> 
>> (XEN)  0000000000000000 - 000000000008f000 (usable)
>> (XEN)  000000000008f000 - 00000000000a0000 (reserved)
>> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
>> (XEN)  0000000000100000 - 00000000ce69a000 (usable)
>> (XEN)  00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
>> (XEN)  00000000ce6f1000 - 00000000cf5fb000 (usable)
>> (XEN)  00000000cf5fb000 - 00000000cf608000 (reserved)
>> (XEN)  00000000cf608000 - 00000000cf6a5000 (usable)
>> (XEN)  00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
>> (XEN)  00000000cf6aa000 - 00000000cf6ab000 (usable)
>> (XEN)  00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
>> (XEN)  00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
>> (XEN)  00000000cf6ff000 - 00000000cf700000 (usable)
>> (XEN)  00000000cf700000 - 00000000d0000000 (reserved)
>> (XEN)  00000000fff00000 - 0000000100000000 (reserved)
>> (XEN)  0000000100000000 - 0000000228000000 (usable)
>> (XEN)  0000000228000000 - 000000022c000000 (unusable)
> 
> .. this and ...
> 
>> [    0.000000]  Xen: 0000000000000000 - 000000000008f000 (usable)
>> [    0.000000]  Xen: 000000000008f000 - 0000000000100000 (reserved)
>> [    0.000000]  Xen: 0000000000100000 - 00000000ce69a000 (usable)
>> [    0.000000]  Xen: 00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
>> [    0.000000]  Xen: 00000000ce6f1000 - 00000000cf5fb000 (usable)
>> [    0.000000]  Xen: 00000000cf5fb000 - 00000000cf608000 (reserved)
>> [    0.000000]  Xen: 00000000cf608000 - 00000000cf6a5000 (usable)
>> [    0.000000]  Xen: 00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
>> [    0.000000]  Xen: 00000000cf6aa000 - 00000000cf6ab000 (usable)
>> [    0.000000]  Xen: 00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
>> [    0.000000]  Xen: 00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
>> [    0.000000]  Xen: 00000000cf6ff000 - 00000000cf700000 (usable)
>> [    0.000000]  Xen: 00000000cf700000 - 00000000d0000000 (reserved)
>> [    0.000000]  Xen: 00000000fee00000 - 00000000fee01000 (reserved)
>> [    0.000000]  Xen: 00000000fff00000 - 0000000100000000 (reserved)
>> [    0.000000]  Xen: 0000000100000000 - 0000000228000000 (usable)
>> [    0.000000]  Xen: 0000000228000000 - 000000022c000000 (unusable)
> 
> ... this show that the kernel should be well aware that it
shouldn''t
> map (or use in any other way) this region.
This came up before when tboot (?) was incorrectly marking a region as
UNUSABLE instead of RESERVED.

Does the following (untested) patch help?

8<---------------------------------------
x86/xen: do not identity map UNUSABLE regions in the machine E820

If there are UNUSABLE regions in the machine memory map, dom0 will
attempt to map them 1:1 which is not permitted by Xen and the kernel
will crash.

There isn''t anything interesting in the UNUSABLE region that the dom0
kernel needs access to so we can avoid making the 1:1 mapping and
treat it as RAM.

Signed-off-by: David Vrabel <david.vrabel@citrix.com>
---
 arch/x86/xen/setup.c |   24 ++++++++++++++++++++++++
 1 files changed, 24 insertions(+), 0 deletions(-)

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index 94eac5c..73f621c 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -313,6 +313,17 @@ static void xen_align_and_add_e820_region(u64
start, u64 size, int type)
 	e820_add_region(start, end - start, type);
 }

+void xen_ignore_unusable(struct e820entry *list, size_t map_size)
+{
+	struct e820entry *entry;
+	unsigned int i;
+
+	for (i = 0, entry = list; i < map_size; i++, entry++) {
+		if (entry->type == E820_UNUSABLE)
+			entry->type = E820_RAM;
+	}
+}
+
 /**
  * machine_specific_memory_setup - Hook for machine specific memory setup.
  **/
@@ -353,6 +364,19 @@ char * __init xen_memory_setup(void)
 	}
 	BUG_ON(rc);

+	/*
+	 * Xen won''t allow a 1:1 mapping to be created to UNUSABLE
+	 * regions, so if we''re using the machine memory map convert
+	 * UNUSABLE to RAM.
+	 *
+	 * This might look odd but what we''re really doing here is
+	 * taking the psuedo-physical memory map and punching 1:1
+	 * holes through to interesting bits found in the machine
+	 * memory map.
+	 */
+	if (xen_initial_domain())
+		xen_ignore_unusable(map, memmap.nr_entries);
+
 	/* Make sure the Xen-supplied memory map is well-ordered. */
 	sanitize_e820_map(map, memmap.nr_entries, &memmap.nr_entries);

Wei Liu

2013-Jul-08 12:50 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On Mon, Jul 08, 2013 at 12:29:02PM +0100, Jan Beulich
wrote:> >>> On 08.07.13 at 12:31, Wei Liu <wei.liu2@citrix.com>
wrote:
> 
> $subject is probably the wrong way round, since ...
> 
Right... Xen is doing its job...
> > (XEN)  0000000000000000 - 000000000008f000 (usable)
> > (XEN)  000000000008f000 - 00000000000a0000 (reserved)
> > (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> > (XEN)  0000000000100000 - 00000000ce69a000 (usable)
> > (XEN)  00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
> > (XEN)  00000000ce6f1000 - 00000000cf5fb000 (usable)
> > (XEN)  00000000cf5fb000 - 00000000cf608000 (reserved)
> > (XEN)  00000000cf608000 - 00000000cf6a5000 (usable)
> > (XEN)  00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
> > (XEN)  00000000cf6aa000 - 00000000cf6ab000 (usable)
> > (XEN)  00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
> > (XEN)  00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
> > (XEN)  00000000cf6ff000 - 00000000cf700000 (usable)
> > (XEN)  00000000cf700000 - 00000000d0000000 (reserved)
> > (XEN)  00000000fff00000 - 0000000100000000 (reserved)
> > (XEN)  0000000100000000 - 0000000228000000 (usable)
> > (XEN)  0000000228000000 - 000000022c000000 (unusable)
> 
> .. this and ...
> 
> > [    0.000000]  Xen: 0000000000000000 - 000000000008f000 (usable)
> > [    0.000000]  Xen: 000000000008f000 - 0000000000100000 (reserved)
> > [    0.000000]  Xen: 0000000000100000 - 00000000ce69a000 (usable)
> > [    0.000000]  Xen: 00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
> > [    0.000000]  Xen: 00000000ce6f1000 - 00000000cf5fb000 (usable)
> > [    0.000000]  Xen: 00000000cf5fb000 - 00000000cf608000 (reserved)
> > [    0.000000]  Xen: 00000000cf608000 - 00000000cf6a5000 (usable)
> > [    0.000000]  Xen: 00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
> > [    0.000000]  Xen: 00000000cf6aa000 - 00000000cf6ab000 (usable)
> > [    0.000000]  Xen: 00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
> > [    0.000000]  Xen: 00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
> > [    0.000000]  Xen: 00000000cf6ff000 - 00000000cf700000 (usable)
> > [    0.000000]  Xen: 00000000cf700000 - 00000000d0000000 (reserved)
> > [    0.000000]  Xen: 00000000fee00000 - 00000000fee01000 (reserved)
> > [    0.000000]  Xen: 00000000fff00000 - 0000000100000000 (reserved)
> > [    0.000000]  Xen: 0000000100000000 - 0000000228000000 (usable)
> > [    0.000000]  Xen: 0000000228000000 - 000000022c000000 (unusable)
> 
> ... this show that the kernel should be well aware that it
shouldn''t
> map (or use in any other way) this region.
> 
> > [    0.000000] init_memory_mapping: 0000000000000000-00000000cf700000
> > [    0.000000] init_memory_mapping: 0000000100000000-0000000228000000
> > [    0.000000] init_memory_mapping: 0000000228000000-000000022c000000
> 
> Yet it does, ...
> 
> > (XEN) mm.c:783:d0 Non-privileged (0) attempt to map I/O space 00228000
> > (XEN) mm.c:1219:d0 Failure in alloc_l1_table: entry 0
> > (XEN) mm.c:2096:d0 Error while validating mfn 226221 (pfn 1e9760) for
type
> > 1000000000000000: caf=8000000000000003 taf=1000000000000001
> > (XEN) mm.c:2992:d0 Error while pinning mfn 226221
> 
> ... and Xen rightfully rejects the attempt.
> 
> One question of course is where this pretty unusual "unusable"
> memory block comes from on that system. Is this block visible the
> same way when booting a native kernel, or is this being forced to
> "unusable" by Xen?
> 
Vanilla 3.10 + Xen unstable:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000008efff] usable
[    0.000000] Xen: [mem 0x000000000008f000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x00000000ce699fff] usable
[    0.000000] Xen: [mem 0x00000000ce69a000-0x00000000ce6f0fff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000ce6f1000-0x00000000cf5fafff] usable
[    0.000000] Xen: [mem 0x00000000cf5fb000-0x00000000cf607fff] reserved
[    0.000000] Xen: [mem 0x00000000cf608000-0x00000000cf6a4fff] usable
[    0.000000] Xen: [mem 0x00000000cf6a5000-0x00000000cf6a9fff] ACPI data
[    0.000000] Xen: [mem 0x00000000cf6aa000-0x00000000cf6aafff] usable
[    0.000000] Xen: [mem 0x00000000cf6ab000-0x00000000cf6f1fff] ACPI NVS
[    0.000000] Xen: [mem 0x00000000cf6f2000-0x00000000cf6fefff] ACPI data
[    0.000000] Xen: [mem 0x00000000cf6ff000-0x00000000cf6fffff] usable
[    0.000000] Xen: [mem 0x00000000cf700000-0x00000000cfffffff] reserved
[    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] Xen: [mem 0x00000000fff00000-0x00000000ffffffff] reserved
[    0.000000] Xen: [mem 0x0000000100000000-0x0000000227ffffff] usable
[    0.000000] Xen: [mem 0x0000000228000000-0x000000022bffffff] unusable
[    0.000000] ERROR: earlyprintk= xenboot already used
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x228000 max_arch_pfn = 0x400000000
[    0.000000] e820: last_pfn = 0xcf700 max_arch_pfn = 0x400000000
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000] init_memory_mapping: [mem 0x219e00000-0x219ffffff]
[    0.000000] init_memory_mapping: [mem 0x218000000-0x219dfffff]
[    0.000000] init_memory_mapping: [mem 0x200000000-0x217ffffff]
[    0.000000] init_memory_mapping: [mem 0x00100000-0xce699fff]
[    0.000000] init_memory_mapping: [mem 0xce6f1000-0xcf5fafff]
[    0.000000] init_memory_mapping: [mem 0xcf608000-0xcf6a4fff]
[    0.000000] init_memory_mapping: [mem 0xcf6aa000-0xcf6aafff]
[    0.000000] init_memory_mapping: [mem 0xcf6ff000-0xcf6fffff]
[    0.000000] init_memory_mapping: [mem 0x100000000-0x1ffffffff]
[    0.000000] init_memory_mapping: [mem 0x21a000000-0x227ffffff]

We also have the similar unusable block, however the kernel doesn''t map
it.

3.2.0-4-amd64 (just found out that there''s actually backtrace in
dmesg):
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000008f000 (usable)
[    0.000000]  BIOS-e820: 000000000008f000 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000ce69a000 (usable)
[    0.000000]  BIOS-e820: 00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000ce6f1000 - 00000000cf5fb000 (usable)
[    0.000000]  BIOS-e820: 00000000cf5fb000 - 00000000cf608000 (reserved)
[    0.000000]  BIOS-e820: 00000000cf608000 - 00000000cf6a5000 (usable)
[    0.000000]  BIOS-e820: 00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000cf6aa000 - 00000000cf6ab000 (usable)
[    0.000000]  BIOS-e820: 00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000cf6ff000 - 00000000cf700000 (usable)
[    0.000000]  BIOS-e820: 00000000cf700000 - 00000000d0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 000000022c000000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] DMI:                  /DG965SS, BIOS
MQ96510J.86A.1669.2007.0406.0107 04/06/2007
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable)
==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x22c000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-FFFFF uncachable
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000 mask F80000000 write-back
[    0.000000]   1 base 080000000 mask FC0000000 write-back
[    0.000000]   2 base 0C0000000 mask FF0000000 write-back
[    0.000000]   3 base 0CF800000 mask FFF800000 uncachable
[    0.000000]   4 base 0CF700000 mask FFFF00000 uncachable
[    0.000000]   5 base 100000000 mask F00000000 write-back
[    0.000000]   6 base 200000000 mask FE0000000 write-back
[    0.000000]   7 base 220000000 mask FF8000000 write-back
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] e820 update range: 00000000cf700000 - 0000000100000000 (usable)
==> (reserved)
[    0.000000] e820 update range: 0000000228000000 - 000000022c000000 (usable)
==> (reserved)
[    0.000000] WARNING: BIOS bug: CPU MTRRs don''t cover all of memory,
losing 64MB of RAM.
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: at
/build/linux-s5x2oE/linux-3.2.46/arch/x86/kernel/cpu/mtrr/cleanup.c:971
mtrr_trim_uncached_memory+0x2a8/0x2cd()
[    0.000000] Hardware name:         
[    0.000000] Modules linked in:
[    0.000000] Pid: 0, comm: swapper Not tainted 3.2.0-4-amd64 #1 Debian
3.2.46-1
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff81046b75>] ? warn_slowpath_common+0x78/0x8c
[    0.000000]  [<ffffffff816b56e9>] ?
mtrr_trim_uncached_memory+0x2a8/0x2cd
[    0.000000]  [<ffffffff816afa0c>] ? setup_arch+0x52a/0xc27
[    0.000000]  [<ffffffff8134833d>] ? printk+0x43/0x48
[    0.000000]  [<ffffffff81070df3>] ? arch_local_irq_disable+0x7/0x8
[    0.000000]  [<ffffffff8134eb77>] ? _raw_spin_unlock_irqrestore+0xe/0xf
[    0.000000]  [<ffffffff816ab84d>] ? start_kernel+0xcf/0x3c3
[    0.000000]  [<ffffffff816ab140>] ? early_idt_handlers+0x140/0x140
[    0.000000]  [<ffffffff816ab3c4>] ? x86_64_start_kernel+0x104/0x111
[    0.000000] ---[ end trace fbfbe82c378e4a32 ]---
[    0.000000] update e820 for mtrr
[    0.000000] modified physical RAM map:
[    0.000000]  modified: 0000000000000000 - 0000000000010000 (reserved)
[    0.000000]  modified: 0000000000010000 - 000000000008f000 (usable)
[    0.000000]  modified: 000000000008f000 - 00000000000a0000 (reserved)
[    0.000000]  modified: 00000000000e0000 - 0000000000100000 (reserved)
[    0.000000]  modified: 0000000000100000 - 00000000ce69a000 (usable)
[    0.000000]  modified: 00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
[    0.000000]  modified: 00000000ce6f1000 - 00000000cf5fb000 (usable)
[    0.000000]  modified: 00000000cf5fb000 - 00000000cf608000 (reserved)
[    0.000000]  modified: 00000000cf608000 - 00000000cf6a5000 (usable)
[    0.000000]  modified: 00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
[    0.000000]  modified: 00000000cf6aa000 - 00000000cf6ab000 (usable)
[    0.000000]  modified: 00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
[    0.000000]  modified: 00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
[    0.000000]  modified: 00000000cf6ff000 - 00000000cf700000 (usable)
[    0.000000]  modified: 00000000cf700000 - 00000000d0000000 (reserved)
[    0.000000]  modified: 00000000fff00000 - 0000000100000000 (reserved)
[    0.000000]  modified: 0000000100000000 - 0000000228000000 (usable)
[    0.000000]  modified: 0000000228000000 - 000000022c000000 (reserved)
[    0.000000] last_pfn = 0x228000 max_arch_pfn = 0x400000000
[    0.000000] last_pfn = 0xcf700 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [ffff8800000fe200] fe200
[    0.000000] initial memory mapped : 0 - 20000000
[    0.000000] Base memory trampoline at [ffff88000008a000] 8a000 size 20480
[    0.000000] init_memory_mapping: 0000000000000000-00000000cf700000
[    0.000000]  0000000000 - 00cf600000 page 2M
[    0.000000]  00cf600000 - 00cf700000 page 4k
[    0.000000] kernel direct mapping tables up to cf700000 @ 1fffa000-20000000
[    0.000000] init_memory_mapping: 0000000100000000-0000000228000000
[    0.000000]  0100000000 - 0228000000 page 2M
[    0.000000] kernel direct mapping tables up to 228000000 @ cf69f000-cf6a5000



Wei.
> Jan

Wei Liu

2013-Jul-08 12:52 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On Mon, Jul 08, 2013 at 01:06:18PM +0100, David Vrabel
wrote:> On 08/07/13 12:29, Jan Beulich wrote:
> >>>> On 08.07.13 at 12:31, Wei Liu <wei.liu2@citrix.com>
wrote:
> > 
> > $subject is probably the wrong way round, since ...
> > 
> >> (XEN)  0000000000000000 - 000000000008f000 (usable)
> >> (XEN)  000000000008f000 - 00000000000a0000 (reserved)
> >> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> >> (XEN)  0000000000100000 - 00000000ce69a000 (usable)
> >> (XEN)  00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
> >> (XEN)  00000000ce6f1000 - 00000000cf5fb000 (usable)
> >> (XEN)  00000000cf5fb000 - 00000000cf608000 (reserved)
> >> (XEN)  00000000cf608000 - 00000000cf6a5000 (usable)
> >> (XEN)  00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
> >> (XEN)  00000000cf6aa000 - 00000000cf6ab000 (usable)
> >> (XEN)  00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
> >> (XEN)  00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
> >> (XEN)  00000000cf6ff000 - 00000000cf700000 (usable)
> >> (XEN)  00000000cf700000 - 00000000d0000000 (reserved)
> >> (XEN)  00000000fff00000 - 0000000100000000 (reserved)
> >> (XEN)  0000000100000000 - 0000000228000000 (usable)
> >> (XEN)  0000000228000000 - 000000022c000000 (unusable)
> > 
> > .. this and ...
> > 
> >> [    0.000000]  Xen: 0000000000000000 - 000000000008f000 (usable)
> >> [    0.000000]  Xen: 000000000008f000 - 0000000000100000
(reserved)
> >> [    0.000000]  Xen: 0000000000100000 - 00000000ce69a000 (usable)
> >> [    0.000000]  Xen: 00000000ce69a000 - 00000000ce6f1000 (ACPI
NVS)
> >> [    0.000000]  Xen: 00000000ce6f1000 - 00000000cf5fb000 (usable)
> >> [    0.000000]  Xen: 00000000cf5fb000 - 00000000cf608000
(reserved)
> >> [    0.000000]  Xen: 00000000cf608000 - 00000000cf6a5000 (usable)
> >> [    0.000000]  Xen: 00000000cf6a5000 - 00000000cf6aa000 (ACPI
data)
> >> [    0.000000]  Xen: 00000000cf6aa000 - 00000000cf6ab000 (usable)
> >> [    0.000000]  Xen: 00000000cf6ab000 - 00000000cf6f2000 (ACPI
NVS)
> >> [    0.000000]  Xen: 00000000cf6f2000 - 00000000cf6ff000 (ACPI
data)
> >> [    0.000000]  Xen: 00000000cf6ff000 - 00000000cf700000 (usable)
> >> [    0.000000]  Xen: 00000000cf700000 - 00000000d0000000
(reserved)
> >> [    0.000000]  Xen: 00000000fee00000 - 00000000fee01000
(reserved)
> >> [    0.000000]  Xen: 00000000fff00000 - 0000000100000000
(reserved)
> >> [    0.000000]  Xen: 0000000100000000 - 0000000228000000 (usable)
> >> [    0.000000]  Xen: 0000000228000000 - 000000022c000000
(unusable)
> > 
> > ... this show that the kernel should be well aware that it
shouldn''t
> > map (or use in any other way) this region.
> 
> This came up before when tboot (?) was incorrectly marking a region as
> UNUSABLE instead of RESERVED.
> 
> Does the following (untested) patch help?
> 
Just saw this. I will test it this afternoon when I have time.
> 8<---------------------------------------
> x86/xen: do not identity map UNUSABLE regions in the machine E820
> 
> If there are UNUSABLE regions in the machine memory map, dom0 will
> attempt to map them 1:1 which is not permitted by Xen and the kernel
> will crash.
> 
> There isn''t anything interesting in the UNUSABLE region that the
dom0
> kernel needs access to so we can avoid making the 1:1 mapping and
> treat it as RAM.
> 
> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
> ---
>  arch/x86/xen/setup.c |   24 ++++++++++++++++++++++++
>  1 files changed, 24 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index 94eac5c..73f621c 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -313,6 +313,17 @@ static void xen_align_and_add_e820_region(u64
> start, u64 size, int type)
>  	e820_add_region(start, end - start, type);
>  }
> 
> +void xen_ignore_unusable(struct e820entry *list, size_t map_size)
> +{
> +	struct e820entry *entry;
> +	unsigned int i;
> +
> +	for (i = 0, entry = list; i < map_size; i++, entry++) {
> +		if (entry->type == E820_UNUSABLE)
> +			entry->type = E820_RAM;
> +	}
> +}
> +
>  /**
>   * machine_specific_memory_setup - Hook for machine specific memory setup.
>   **/
> @@ -353,6 +364,19 @@ char * __init xen_memory_setup(void)
>  	}
>  	BUG_ON(rc);
> 
> +	/*
> +	 * Xen won''t allow a 1:1 mapping to be created to UNUSABLE
> +	 * regions, so if we''re using the machine memory map convert
> +	 * UNUSABLE to RAM.
> +	 *
> +	 * This might look odd but what we''re really doing here is
> +	 * taking the psuedo-physical memory map and punching 1:1
> +	 * holes through to interesting bits found in the machine
> +	 * memory map.
> +	 */
> +	if (xen_initial_domain())
> +		xen_ignore_unusable(map, memmap.nr_entries);
> +
>  	/* Make sure the Xen-supplied memory map is well-ordered. */
>  	sanitize_e820_map(map, memmap.nr_entries, &memmap.nr_entries);

Wei Liu

2013-Jul-08 13:02 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On Mon, Jul 08, 2013 at 01:06:18PM +0100, David Vrabel
wrote:> On 08/07/13 12:29, Jan Beulich wrote:
> >>>> On 08.07.13 at 12:31, Wei Liu <wei.liu2@citrix.com>
wrote:
> > 
> > $subject is probably the wrong way round, since ...
> > 
> >> (XEN)  0000000000000000 - 000000000008f000 (usable)
> >> (XEN)  000000000008f000 - 00000000000a0000 (reserved)
> >> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> >> (XEN)  0000000000100000 - 00000000ce69a000 (usable)
> >> (XEN)  00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
> >> (XEN)  00000000ce6f1000 - 00000000cf5fb000 (usable)
> >> (XEN)  00000000cf5fb000 - 00000000cf608000 (reserved)
> >> (XEN)  00000000cf608000 - 00000000cf6a5000 (usable)
> >> (XEN)  00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
> >> (XEN)  00000000cf6aa000 - 00000000cf6ab000 (usable)
> >> (XEN)  00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
> >> (XEN)  00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
> >> (XEN)  00000000cf6ff000 - 00000000cf700000 (usable)
> >> (XEN)  00000000cf700000 - 00000000d0000000 (reserved)
> >> (XEN)  00000000fff00000 - 0000000100000000 (reserved)
> >> (XEN)  0000000100000000 - 0000000228000000 (usable)
> >> (XEN)  0000000228000000 - 000000022c000000 (unusable)
> > 
> > .. this and ...
> > 
> >> [    0.000000]  Xen: 0000000000000000 - 000000000008f000 (usable)
> >> [    0.000000]  Xen: 000000000008f000 - 0000000000100000
(reserved)
> >> [    0.000000]  Xen: 0000000000100000 - 00000000ce69a000 (usable)
> >> [    0.000000]  Xen: 00000000ce69a000 - 00000000ce6f1000 (ACPI
NVS)
> >> [    0.000000]  Xen: 00000000ce6f1000 - 00000000cf5fb000 (usable)
> >> [    0.000000]  Xen: 00000000cf5fb000 - 00000000cf608000
(reserved)
> >> [    0.000000]  Xen: 00000000cf608000 - 00000000cf6a5000 (usable)
> >> [    0.000000]  Xen: 00000000cf6a5000 - 00000000cf6aa000 (ACPI
data)
> >> [    0.000000]  Xen: 00000000cf6aa000 - 00000000cf6ab000 (usable)
> >> [    0.000000]  Xen: 00000000cf6ab000 - 00000000cf6f2000 (ACPI
NVS)
> >> [    0.000000]  Xen: 00000000cf6f2000 - 00000000cf6ff000 (ACPI
data)
> >> [    0.000000]  Xen: 00000000cf6ff000 - 00000000cf700000 (usable)
> >> [    0.000000]  Xen: 00000000cf700000 - 00000000d0000000
(reserved)
> >> [    0.000000]  Xen: 00000000fee00000 - 00000000fee01000
(reserved)
> >> [    0.000000]  Xen: 00000000fff00000 - 0000000100000000
(reserved)
> >> [    0.000000]  Xen: 0000000100000000 - 0000000228000000 (usable)
> >> [    0.000000]  Xen: 0000000228000000 - 000000022c000000
(unusable)
> > 
> > ... this show that the kernel should be well aware that it
shouldn''t
> > map (or use in any other way) this region.
> 
> This came up before when tboot (?) was incorrectly marking a region as
> UNUSABLE instead of RESERVED.
> 
> Does the following (untested) patch help?
> 
Oh wait, this patch is for Linux kernel. It might take me some time to
get it apply to Debian Wheezy''s kernel (I''ve never done this
before).

But one thing I need to point out is that 3.10 doens''t have any problem
booting on Xen unstable. 3.10 doesn''t seem to have similar logic in
xen_memory_setup...
> 8<---------------------------------------
> x86/xen: do not identity map UNUSABLE regions in the machine E820
> 
> If there are UNUSABLE regions in the machine memory map, dom0 will
> attempt to map them 1:1 which is not permitted by Xen and the kernel
> will crash.
> 
> There isn''t anything interesting in the UNUSABLE region that the
dom0
> kernel needs access to so we can avoid making the 1:1 mapping and
> treat it as RAM.
> 
> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
> ---
>  arch/x86/xen/setup.c |   24 ++++++++++++++++++++++++
>  1 files changed, 24 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index 94eac5c..73f621c 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -313,6 +313,17 @@ static void xen_align_and_add_e820_region(u64
> start, u64 size, int type)
>  	e820_add_region(start, end - start, type);
>  }
> 
> +void xen_ignore_unusable(struct e820entry *list, size_t map_size)
> +{
> +	struct e820entry *entry;
> +	unsigned int i;
> +
> +	for (i = 0, entry = list; i < map_size; i++, entry++) {
> +		if (entry->type == E820_UNUSABLE)
> +			entry->type = E820_RAM;
> +	}
> +}
> +
>  /**
>   * machine_specific_memory_setup - Hook for machine specific memory setup.
>   **/
> @@ -353,6 +364,19 @@ char * __init xen_memory_setup(void)
>  	}
>  	BUG_ON(rc);
> 
> +	/*
> +	 * Xen won''t allow a 1:1 mapping to be created to UNUSABLE
> +	 * regions, so if we''re using the machine memory map convert
> +	 * UNUSABLE to RAM.
> +	 *
> +	 * This might look odd but what we''re really doing here is
> +	 * taking the psuedo-physical memory map and punching 1:1
> +	 * holes through to interesting bits found in the machine
> +	 * memory map.
> +	 */
> +	if (xen_initial_domain())
> +		xen_ignore_unusable(map, memmap.nr_entries);
> +
>  	/* Make sure the Xen-supplied memory map is well-ordered. */
>  	sanitize_e820_map(map, memmap.nr_entries, &memmap.nr_entries);

Jan Beulich

2013-Jul-08 13:16 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

>>> On 08.07.13 at 14:50, Wei Liu <wei.liu2@citrix.com> wrote:
> On Mon, Jul 08, 2013 at 12:29:02PM +0100, Jan Beulich wrote:
>> One question of course is where this pretty unusual
"unusable"
>> memory block comes from on that system. Is this block visible the
>> same way when booting a native kernel, or is this being forced to
>> "unusable" by Xen?
> 
> Vanilla 3.10 + Xen unstable:
> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000008efff] usable
> [    0.000000] Xen: [mem 0x000000000008f000-0x00000000000fffff] reserved
> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ce699fff] usable
> [    0.000000] Xen: [mem 0x00000000ce69a000-0x00000000ce6f0fff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000ce6f1000-0x00000000cf5fafff] usable
> [    0.000000] Xen: [mem 0x00000000cf5fb000-0x00000000cf607fff] reserved
> [    0.000000] Xen: [mem 0x00000000cf608000-0x00000000cf6a4fff] usable
> [    0.000000] Xen: [mem 0x00000000cf6a5000-0x00000000cf6a9fff] ACPI data
> [    0.000000] Xen: [mem 0x00000000cf6aa000-0x00000000cf6aafff] usable
> [    0.000000] Xen: [mem 0x00000000cf6ab000-0x00000000cf6f1fff] ACPI NVS
> [    0.000000] Xen: [mem 0x00000000cf6f2000-0x00000000cf6fefff] ACPI data
> [    0.000000] Xen: [mem 0x00000000cf6ff000-0x00000000cf6fffff] usable
> [    0.000000] Xen: [mem 0x00000000cf700000-0x00000000cfffffff] reserved
> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> [    0.000000] Xen: [mem 0x00000000fff00000-0x00000000ffffffff] reserved
> [    0.000000] Xen: [mem 0x0000000100000000-0x0000000227ffffff] usable
> [    0.000000] Xen: [mem 0x0000000228000000-0x000000022bffffff] unusable
> [    0.000000] ERROR: earlyprintk= xenboot already used
> [    0.000000] NX (Execute Disable) protection: active
> [    0.000000] SMBIOS 2.4 present.
> [    0.000000] No AGP bridge found
> [    0.000000] e820: last_pfn = 0x228000 max_arch_pfn = 0x400000000
> [    0.000000] e820: last_pfn = 0xcf700 max_arch_pfn = 0x400000000
> [    0.000000] Scanning 1 areas for low memory corruption
> [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
> [    0.000000] init_memory_mapping: [mem 0x219e00000-0x219ffffff]
> [    0.000000] init_memory_mapping: [mem 0x218000000-0x219dfffff]
> [    0.000000] init_memory_mapping: [mem 0x200000000-0x217ffffff]
> [    0.000000] init_memory_mapping: [mem 0x00100000-0xce699fff]
> [    0.000000] init_memory_mapping: [mem 0xce6f1000-0xcf5fafff]
> [    0.000000] init_memory_mapping: [mem 0xcf608000-0xcf6a4fff]
> [    0.000000] init_memory_mapping: [mem 0xcf6aa000-0xcf6aafff]
> [    0.000000] init_memory_mapping: [mem 0xcf6ff000-0xcf6fffff]
> [    0.000000] init_memory_mapping: [mem 0x100000000-0x1ffffffff]
> [    0.000000] init_memory_mapping: [mem 0x21a000000-0x227ffffff]
> 
> We also have the similar unusable block, however the kernel
doesn''t map it.
Right, iirc a fix for this was done not too long ago. Konrad may
recall further details...
> 3.2.0-4-amd64 (just found out that there''s actually backtrace in
dmesg):
> [    0.000000] BIOS-provided physical RAM map:
> [    0.000000]  BIOS-e820: 0000000000000000 - 000000000008f000 (usable)
> [    0.000000]  BIOS-e820: 000000000008f000 - 00000000000a0000 (reserved)
> [    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
> [    0.000000]  BIOS-e820: 0000000000100000 - 00000000ce69a000 (usable)
> [    0.000000]  BIOS-e820: 00000000ce69a000 - 00000000ce6f1000 (ACPI NVS)
> [    0.000000]  BIOS-e820: 00000000ce6f1000 - 00000000cf5fb000 (usable)
> [    0.000000]  BIOS-e820: 00000000cf5fb000 - 00000000cf608000 (reserved)
> [    0.000000]  BIOS-e820: 00000000cf608000 - 00000000cf6a5000 (usable)
> [    0.000000]  BIOS-e820: 00000000cf6a5000 - 00000000cf6aa000 (ACPI data)
> [    0.000000]  BIOS-e820: 00000000cf6aa000 - 00000000cf6ab000 (usable)
> [    0.000000]  BIOS-e820: 00000000cf6ab000 - 00000000cf6f2000 (ACPI NVS)
> [    0.000000]  BIOS-e820: 00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
> [    0.000000]  BIOS-e820: 00000000cf6ff000 - 00000000cf700000 (usable)
> [    0.000000]  BIOS-e820: 00000000cf700000 - 00000000d0000000 (reserved)
> [    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
> [    0.000000]  BIOS-e820: 0000000100000000 - 000000022c000000 (usable)
So no such block right off the BIOS. You''re not using Xen TXT code
by chance? Off the top of my head I don''t recall any other place
where multiple RAM pages might get turned into "unusable".

Jan

Wei Liu

2013-Jul-08 13:42 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On Mon, Jul 08, 2013 at 02:16:07PM +0100, Jan Beulich wrote:
[...]> 
> > 3.2.0-4-amd64 (just found out that there''s actually backtrace
in dmesg):
> > [    0.000000] BIOS-provided physical RAM map:
> > [    0.000000]  BIOS-e820: 0000000000000000 - 000000000008f000
(usable)
> > [    0.000000]  BIOS-e820: 000000000008f000 - 00000000000a0000
(reserved)
> > [    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000
(reserved)
> > [    0.000000]  BIOS-e820: 0000000000100000 - 00000000ce69a000
(usable)
> > [    0.000000]  BIOS-e820: 00000000ce69a000 - 00000000ce6f1000 (ACPI
NVS)
> > [    0.000000]  BIOS-e820: 00000000ce6f1000 - 00000000cf5fb000
(usable)
> > [    0.000000]  BIOS-e820: 00000000cf5fb000 - 00000000cf608000
(reserved)
> > [    0.000000]  BIOS-e820: 00000000cf608000 - 00000000cf6a5000
(usable)
> > [    0.000000]  BIOS-e820: 00000000cf6a5000 - 00000000cf6aa000 (ACPI
data)
> > [    0.000000]  BIOS-e820: 00000000cf6aa000 - 00000000cf6ab000
(usable)
> > [    0.000000]  BIOS-e820: 00000000cf6ab000 - 00000000cf6f2000 (ACPI
NVS)
> > [    0.000000]  BIOS-e820: 00000000cf6f2000 - 00000000cf6ff000 (ACPI
data)
> > [    0.000000]  BIOS-e820: 00000000cf6ff000 - 00000000cf700000
(usable)
> > [    0.000000]  BIOS-e820: 00000000cf700000 - 00000000d0000000
(reserved)
> > [    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000
(reserved)
> > [    0.000000]  BIOS-e820: 0000000100000000 - 000000022c000000
(usable)
> 
> So no such block right off the BIOS. You''re not using Xen TXT code
> by chance? Off the top of my head I don''t recall any other place
> where multiple RAM pages might get turned into "unusable".
> 
I don''t think so but I''m not sure. I don''t see any
TXT option in BIOS
setting, so I presume that mother board doesn''t have such thing --
it''s
pretty old.


Wei.
> Jan

Jan Beulich

2013-Jul-08 14:03 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

>>> On 08.07.13 at 15:42, Wei Liu <wei.liu2@citrix.com> wrote:
> On Mon, Jul 08, 2013 at 02:16:07PM +0100, Jan Beulich wrote:
> [...]
>> 
>> > 3.2.0-4-amd64 (just found out that there''s actually
backtrace in dmesg):
>> > [    0.000000] BIOS-provided physical RAM map:
>> > [    0.000000]  BIOS-e820: 0000000000000000 - 000000000008f000
(usable)
>> > [    0.000000]  BIOS-e820: 000000000008f000 - 00000000000a0000
(reserved)
>> > [    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000
(reserved)
>> > [    0.000000]  BIOS-e820: 0000000000100000 - 00000000ce69a000
(usable)
>> > [    0.000000]  BIOS-e820: 00000000ce69a000 - 00000000ce6f1000
(ACPI NVS)
>> > [    0.000000]  BIOS-e820: 00000000ce6f1000 - 00000000cf5fb000
(usable)
>> > [    0.000000]  BIOS-e820: 00000000cf5fb000 - 00000000cf608000
(reserved)
>> > [    0.000000]  BIOS-e820: 00000000cf608000 - 00000000cf6a5000
(usable)
>> > [    0.000000]  BIOS-e820: 00000000cf6a5000 - 00000000cf6aa000
(ACPI data)
>> > [    0.000000]  BIOS-e820: 00000000cf6aa000 - 00000000cf6ab000
(usable)
>> > [    0.000000]  BIOS-e820: 00000000cf6ab000 - 00000000cf6f2000
(ACPI NVS)
>> > [    0.000000]  BIOS-e820: 00000000cf6f2000 - 00000000cf6ff000
(ACPI data)
>> > [    0.000000]  BIOS-e820: 00000000cf6ff000 - 00000000cf700000
(usable)
>> > [    0.000000]  BIOS-e820: 00000000cf700000 - 00000000d0000000
(reserved)
>> > [    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000
(reserved)
>> > [    0.000000]  BIOS-e820: 0000000100000000 - 000000022c000000
(usable)
>> 
>> So no such block right off the BIOS. You''re not using Xen TXT
code
>> by chance? Off the top of my head I don''t recall any other
place
>> where multiple RAM pages might get turned into "unusable".
>> 
> 
> I don''t think so but I''m not sure. I don''t see
any TXT option in BIOS
> setting, so I presume that mother board doesn''t have such thing --
it''s
> pretty old.
Odd - the only other place where we do such conversion is when
clipping memory, which shouldn''t be the case here (unless you
have a strange "mem=" option on your command line).

And with TXT, you''d at least have one line starting "TBOOT: "
in your
hypervisor log.

Jan

Wei Liu

2013-Jul-08 14:11 UTC

head link

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On Mon, Jul 08, 2013 at 03:03:20PM +0100, Jan Beulich
wrote:> >>> On 08.07.13 at 15:42, Wei Liu <wei.liu2@citrix.com>
wrote:
> > On Mon, Jul 08, 2013 at 02:16:07PM +0100, Jan Beulich wrote:
> > [...]
> >> 
> >> > 3.2.0-4-amd64 (just found out that there''s actually
backtrace in dmesg):
> >> > [    0.000000] BIOS-provided physical RAM map:
> >> > [    0.000000]  BIOS-e820: 0000000000000000 -
000000000008f000 (usable)
> >> > [    0.000000]  BIOS-e820: 000000000008f000 -
00000000000a0000 (reserved)
> >> > [    0.000000]  BIOS-e820: 00000000000e0000 -
0000000000100000 (reserved)
> >> > [    0.000000]  BIOS-e820: 0000000000100000 -
00000000ce69a000 (usable)
> >> > [    0.000000]  BIOS-e820: 00000000ce69a000 -
00000000ce6f1000 (ACPI NVS)
> >> > [    0.000000]  BIOS-e820: 00000000ce6f1000 -
00000000cf5fb000 (usable)
> >> > [    0.000000]  BIOS-e820: 00000000cf5fb000 -
00000000cf608000 (reserved)
> >> > [    0.000000]  BIOS-e820: 00000000cf608000 -
00000000cf6a5000 (usable)
> >> > [    0.000000]  BIOS-e820: 00000000cf6a5000 -
00000000cf6aa000 (ACPI data)
> >> > [    0.000000]  BIOS-e820: 00000000cf6aa000 -
00000000cf6ab000 (usable)
> >> > [    0.000000]  BIOS-e820: 00000000cf6ab000 -
00000000cf6f2000 (ACPI NVS)
> >> > [    0.000000]  BIOS-e820: 00000000cf6f2000 -
00000000cf6ff000 (ACPI data)
> >> > [    0.000000]  BIOS-e820: 00000000cf6ff000 -
00000000cf700000 (usable)
> >> > [    0.000000]  BIOS-e820: 00000000cf700000 -
00000000d0000000 (reserved)
> >> > [    0.000000]  BIOS-e820: 00000000fff00000 -
0000000100000000 (reserved)
> >> > [    0.000000]  BIOS-e820: 0000000100000000 -
000000022c000000 (usable)
> >> 
> >> So no such block right off the BIOS. You''re not using Xen
TXT code
> >> by chance? Off the top of my head I don''t recall any
other place
> >> where multiple RAM pages might get turned into
"unusable".
> >> 
> > 
> > I don''t think so but I''m not sure. I don''t
see any TXT option in BIOS
> > setting, so I presume that mother board doesn''t have such
thing -- it''s
> > pretty old.
> 
> Odd - the only other place where we do such conversion is when
> clipping memory, which shouldn''t be the case here (unless you
> have a strange "mem=" option on your command line).
> 
No, I don''t have any memory options in Xen command line.
> And with TXT, you''d at least have one line starting "TBOOT:
" in your
> hypervisor log.
Just grep through the log, no tboot output found.


Wei.
> 
> Jan

Konrad Rzeszutek Wilk

2013-Jul-08 19:47 UTC

head link

Is: MTRR issue + Xen + memory clipping? Was:Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On Mon, Jul 08, 2013 at 02:16:07PM +0100, Jan Beulich
wrote:> >>> On 08.07.13 at 14:50, Wei Liu <wei.liu2@citrix.com>
wrote:
> > On Mon, Jul 08, 2013 at 12:29:02PM +0100, Jan Beulich wrote:
> >> One question of course is where this pretty unusual
"unusable"
> >> memory block comes from on that system. Is this block visible the
> >> same way when booting a native kernel, or is this being forced to
> >> "unusable" by Xen?
> > 
> > Vanilla 3.10 + Xen unstable:
> > [    0.000000] Xen: [mem 0x0000000000000000-0x000000000008efff] usable
> > [    0.000000] Xen: [mem 0x000000000008f000-0x00000000000fffff]
reserved
> > [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ce699fff] usable
> > [    0.000000] Xen: [mem 0x00000000ce69a000-0x00000000ce6f0fff] ACPI
NVS
> > [    0.000000] Xen: [mem 0x00000000ce6f1000-0x00000000cf5fafff] usable
> > [    0.000000] Xen: [mem 0x00000000cf5fb000-0x00000000cf607fff]
reserved
> > [    0.000000] Xen: [mem 0x00000000cf608000-0x00000000cf6a4fff] usable
> > [    0.000000] Xen: [mem 0x00000000cf6a5000-0x00000000cf6a9fff] ACPI
data
> > [    0.000000] Xen: [mem 0x00000000cf6aa000-0x00000000cf6aafff] usable
> > [    0.000000] Xen: [mem 0x00000000cf6ab000-0x00000000cf6f1fff] ACPI
NVS
> > [    0.000000] Xen: [mem 0x00000000cf6f2000-0x00000000cf6fefff] ACPI
data
> > [    0.000000] Xen: [mem 0x00000000cf6ff000-0x00000000cf6fffff] usable
> > [    0.000000] Xen: [mem 0x00000000cf700000-0x00000000cfffffff]
reserved
> > [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff]
reserved
> > [    0.000000] Xen: [mem 0x00000000fff00000-0x00000000ffffffff]
reserved
> > [    0.000000] Xen: [mem 0x0000000100000000-0x0000000227ffffff] usable
> > [    0.000000] Xen: [mem 0x0000000228000000-0x000000022bffffff]
unusable
> > [    0.000000] ERROR: earlyprintk= xenboot already used
> > [    0.000000] NX (Execute Disable) protection: active
> > [    0.000000] SMBIOS 2.4 present.
> > [    0.000000] No AGP bridge found
> > [    0.000000] e820: last_pfn = 0x228000 max_arch_pfn = 0x400000000
> > [    0.000000] e820: last_pfn = 0xcf700 max_arch_pfn = 0x400000000
> > [    0.000000] Scanning 1 areas for low memory corruption
> > [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
> > [    0.000000] init_memory_mapping: [mem 0x219e00000-0x219ffffff]
> > [    0.000000] init_memory_mapping: [mem 0x218000000-0x219dfffff]
> > [    0.000000] init_memory_mapping: [mem 0x200000000-0x217ffffff]
> > [    0.000000] init_memory_mapping: [mem 0x00100000-0xce699fff]
> > [    0.000000] init_memory_mapping: [mem 0xce6f1000-0xcf5fafff]
> > [    0.000000] init_memory_mapping: [mem 0xcf608000-0xcf6a4fff]
> > [    0.000000] init_memory_mapping: [mem 0xcf6aa000-0xcf6aafff]
> > [    0.000000] init_memory_mapping: [mem 0xcf6ff000-0xcf6fffff]
> > [    0.000000] init_memory_mapping: [mem 0x100000000-0x1ffffffff]
> > [    0.000000] init_memory_mapping: [mem 0x21a000000-0x227ffffff]
> > 
> > We also have the similar unusable block, however the kernel
doesn''t map it.
> 
> Right, iirc a fix for this was done not too long ago. Konrad may
> recall further details...
Stefan hit this I think. With the MTRR clipping blowing things. CC-ing him
here.> 
> > 3.2.0-4-amd64 (just found out that there''s actually backtrace
in dmesg):
> > [    0.000000] BIOS-provided physical RAM map:
> > [    0.000000]  BIOS-e820: 0000000000000000 - 000000000008f000
(usable)
> > [    0.000000]  BIOS-e820: 000000000008f000 - 00000000000a0000
(reserved)
> > [    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000
(reserved)
> > [    0.000000]  BIOS-e820: 0000000000100000 - 00000000ce69a000
(usable)
> > [    0.000000]  BIOS-e820: 00000000ce69a000 - 00000000ce6f1000 (ACPI
NVS)
> > [    0.000000]  BIOS-e820: 00000000ce6f1000 - 00000000cf5fb000
(usable)
> > [    0.000000]  BIOS-e820: 00000000cf5fb000 - 00000000cf608000
(reserved)
> > [    0.000000]  BIOS-e820: 00000000cf608000 - 00000000cf6a5000
(usable)
> > [    0.000000]  BIOS-e820: 00000000cf6a5000 - 00000000cf6aa000 (ACPI
data)
> > [    0.000000]  BIOS-e820: 00000000cf6aa000 - 00000000cf6ab000
(usable)
> > [    0.000000]  BIOS-e820: 00000000cf6ab000 - 00000000cf6f2000 (ACPI
NVS)
> > [    0.000000]  BIOS-e820: 00000000cf6f2000 - 00000000cf6ff000 (ACPI
data)
> > [    0.000000]  BIOS-e820: 00000000cf6ff000 - 00000000cf700000
(usable)
> > [    0.000000]  BIOS-e820: 00000000cf700000 - 00000000d0000000
(reserved)
> > [    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000
(reserved)
> > [    0.000000]  BIOS-e820: 0000000100000000 - 000000022c000000
(usable)
> 
> So no such block right off the BIOS. You''re not using Xen TXT code
> by chance? Off the top of my head I don''t recall any other place
> where multiple RAM pages might get turned into "unusable".
> 
> Jan
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Stefan Bader

2013-Jul-09 09:46 UTC

head link

Re: Is: MTRR issue + Xen + memory clipping? Was:Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On 08.07.2013 21:47, Konrad Rzeszutek Wilk wrote:> On Mon, Jul 08, 2013 at 02:16:07PM +0100, Jan Beulich wrote:
>>>>> On 08.07.13 at 14:50, Wei Liu <wei.liu2@citrix.com>
wrote:
>>> On Mon, Jul 08, 2013 at 12:29:02PM +0100, Jan Beulich wrote:
>>>> One question of course is where this pretty unusual
"unusable"
>>>> memory block comes from on that system. Is this block visible
the
>>>> same way when booting a native kernel, or is this being forced
to
>>>> "unusable" by Xen?
>>>
>>> Vanilla 3.10 + Xen unstable:
>>> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000008efff]
usable
>>> [    0.000000] Xen: [mem 0x000000000008f000-0x00000000000fffff]
reserved
>>> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ce699fff]
usable
>>> [    0.000000] Xen: [mem 0x00000000ce69a000-0x00000000ce6f0fff]
ACPI NVS
>>> [    0.000000] Xen: [mem 0x00000000ce6f1000-0x00000000cf5fafff]
usable
>>> [    0.000000] Xen: [mem 0x00000000cf5fb000-0x00000000cf607fff]
reserved
>>> [    0.000000] Xen: [mem 0x00000000cf608000-0x00000000cf6a4fff]
usable
>>> [    0.000000] Xen: [mem 0x00000000cf6a5000-0x00000000cf6a9fff]
ACPI data
>>> [    0.000000] Xen: [mem 0x00000000cf6aa000-0x00000000cf6aafff]
usable
>>> [    0.000000] Xen: [mem 0x00000000cf6ab000-0x00000000cf6f1fff]
ACPI NVS
>>> [    0.000000] Xen: [mem 0x00000000cf6f2000-0x00000000cf6fefff]
ACPI data
>>> [    0.000000] Xen: [mem 0x00000000cf6ff000-0x00000000cf6fffff]
usable
>>> [    0.000000] Xen: [mem 0x00000000cf700000-0x00000000cfffffff]
reserved
>>> [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000fee00fff]
reserved
>>> [    0.000000] Xen: [mem 0x00000000fff00000-0x00000000ffffffff]
reserved
>>> [    0.000000] Xen: [mem 0x0000000100000000-0x0000000227ffffff]
usable
>>> [    0.000000] Xen: [mem 0x0000000228000000-0x000000022bffffff]
unusable
>>> [    0.000000] ERROR: earlyprintk= xenboot already used
>>> [    0.000000] NX (Execute Disable) protection: active
>>> [    0.000000] SMBIOS 2.4 present.
>>> [    0.000000] No AGP bridge found
>>> [    0.000000] e820: last_pfn = 0x228000 max_arch_pfn = 0x400000000
>>> [    0.000000] e820: last_pfn = 0xcf700 max_arch_pfn = 0x400000000
>>> [    0.000000] Scanning 1 areas for low memory corruption
>>> [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
>>> [    0.000000] init_memory_mapping: [mem 0x219e00000-0x219ffffff]
>>> [    0.000000] init_memory_mapping: [mem 0x218000000-0x219dfffff]
>>> [    0.000000] init_memory_mapping: [mem 0x200000000-0x217ffffff]
>>> [    0.000000] init_memory_mapping: [mem 0x00100000-0xce699fff]
>>> [    0.000000] init_memory_mapping: [mem 0xce6f1000-0xcf5fafff]
>>> [    0.000000] init_memory_mapping: [mem 0xcf608000-0xcf6a4fff]
>>> [    0.000000] init_memory_mapping: [mem 0xcf6aa000-0xcf6aafff]
>>> [    0.000000] init_memory_mapping: [mem 0xcf6ff000-0xcf6fffff]
>>> [    0.000000] init_memory_mapping: [mem 0x100000000-0x1ffffffff]
>>> [    0.000000] init_memory_mapping: [mem 0x21a000000-0x227ffffff]
>>>
>>> We also have the similar unusable block, however the kernel
doesn''t map it.
>>
>> Right, iirc a fix for this was done not too long ago. Konrad may
>> recall further details...
> 
> Stefan hit this I think. With the MTRR clipping blowing things. CC-ing him
> here.
Not personally but we had a bug report about this. Unfortunately it was a bit
confusing as in the beginning it was not really obvious whether the problem was
or was not fixed in later kernels or whether the different installations used
different dom0_mem arguments.

Reading the old bug report (which never completed as it seemed the reporters
apparently lost interest) I think the problem was that the hypervisor detects
the MTRR problem and set the remainder of memory as unusable.
Then dom0 kernel (and if I parse that report right, it may have been fixed
between 3.2 and 3.3 but hard to say when all you get is them saying yes or no)
would somehow still try to put mappings in there.

The link below is that bug report. Not sure of how much help it is.


[1]
https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1111470>>
>>> 3.2.0-4-amd64 (just found out that there''s actually
backtrace in dmesg):
>>> [    0.000000] BIOS-provided physical RAM map:
>>> [    0.000000]  BIOS-e820: 0000000000000000 - 000000000008f000
(usable)
>>> [    0.000000]  BIOS-e820: 000000000008f000 - 00000000000a0000
(reserved)
>>> [    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000
(reserved)
>>> [    0.000000]  BIOS-e820: 0000000000100000 - 00000000ce69a000
(usable)
>>> [    0.000000]  BIOS-e820: 00000000ce69a000 - 00000000ce6f1000
(ACPI NVS)
>>> [    0.000000]  BIOS-e820: 00000000ce6f1000 - 00000000cf5fb000
(usable)
>>> [    0.000000]  BIOS-e820: 00000000cf5fb000 - 00000000cf608000
(reserved)
>>> [    0.000000]  BIOS-e820: 00000000cf608000 - 00000000cf6a5000
(usable)
>>> [    0.000000]  BIOS-e820: 00000000cf6a5000 - 00000000cf6aa000
(ACPI data)
>>> [    0.000000]  BIOS-e820: 00000000cf6aa000 - 00000000cf6ab000
(usable)
>>> [    0.000000]  BIOS-e820: 00000000cf6ab000 - 00000000cf6f2000
(ACPI NVS)
>>> [    0.000000]  BIOS-e820: 00000000cf6f2000 - 00000000cf6ff000
(ACPI data)
>>> [    0.000000]  BIOS-e820: 00000000cf6ff000 - 00000000cf700000
(usable)
>>> [    0.000000]  BIOS-e820: 00000000cf700000 - 00000000d0000000
(reserved)
>>> [    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000
(reserved)
>>> [    0.000000]  BIOS-e820: 0000000100000000 - 000000022c000000
(usable)
>>
>> So no such block right off the BIOS. You''re not using Xen TXT
code
>> by chance? Off the top of my head I don''t recall any other
place
>> where multiple RAM pages might get turned into "unusable".
>>
>> Jan
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Wei Liu

2013-Jul-09 13:37 UTC

head link

Re: Is: MTRR issue + Xen + memory clipping? Was:Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

On Tue, Jul 09, 2013 at 11:46:51AM +0200, Stefan Bader
wrote:> On 08.07.2013 21:47, Konrad Rzeszutek Wilk wrote:
> > On Mon, Jul 08, 2013 at 02:16:07PM +0100, Jan Beulich wrote:
> >>>>> On 08.07.13 at 14:50, Wei Liu
<wei.liu2@citrix.com> wrote:
> >>> On Mon, Jul 08, 2013 at 12:29:02PM +0100, Jan Beulich wrote:
> >>>> One question of course is where this pretty unusual
"unusable"
> >>>> memory block comes from on that system. Is this block
visible the
> >>>> same way when booting a native kernel, or is this being
forced to
> >>>> "unusable" by Xen?
> >>>
> >>> Vanilla 3.10 + Xen unstable:
> >>> [    0.000000] Xen: [mem
0x0000000000000000-0x000000000008efff] usable
> >>> [    0.000000] Xen: [mem
0x000000000008f000-0x00000000000fffff] reserved
> >>> [    0.000000] Xen: [mem
0x0000000000100000-0x00000000ce699fff] usable
> >>> [    0.000000] Xen: [mem
0x00000000ce69a000-0x00000000ce6f0fff] ACPI NVS
> >>> [    0.000000] Xen: [mem
0x00000000ce6f1000-0x00000000cf5fafff] usable
> >>> [    0.000000] Xen: [mem
0x00000000cf5fb000-0x00000000cf607fff] reserved
> >>> [    0.000000] Xen: [mem
0x00000000cf608000-0x00000000cf6a4fff] usable
> >>> [    0.000000] Xen: [mem
0x00000000cf6a5000-0x00000000cf6a9fff] ACPI data
> >>> [    0.000000] Xen: [mem
0x00000000cf6aa000-0x00000000cf6aafff] usable
> >>> [    0.000000] Xen: [mem
0x00000000cf6ab000-0x00000000cf6f1fff] ACPI NVS
> >>> [    0.000000] Xen: [mem
0x00000000cf6f2000-0x00000000cf6fefff] ACPI data
> >>> [    0.000000] Xen: [mem
0x00000000cf6ff000-0x00000000cf6fffff] usable
> >>> [    0.000000] Xen: [mem
0x00000000cf700000-0x00000000cfffffff] reserved
> >>> [    0.000000] Xen: [mem
0x00000000fee00000-0x00000000fee00fff] reserved
> >>> [    0.000000] Xen: [mem
0x00000000fff00000-0x00000000ffffffff] reserved
> >>> [    0.000000] Xen: [mem
0x0000000100000000-0x0000000227ffffff] usable
> >>> [    0.000000] Xen: [mem
0x0000000228000000-0x000000022bffffff] unusable
> >>> [    0.000000] ERROR: earlyprintk= xenboot already used
> >>> [    0.000000] NX (Execute Disable) protection: active
> >>> [    0.000000] SMBIOS 2.4 present.
> >>> [    0.000000] No AGP bridge found
> >>> [    0.000000] e820: last_pfn = 0x228000 max_arch_pfn =
0x400000000
> >>> [    0.000000] e820: last_pfn = 0xcf700 max_arch_pfn =
0x400000000
> >>> [    0.000000] Scanning 1 areas for low memory corruption
> >>> [    0.000000] init_memory_mapping: [mem
0x00000000-0x000fffff]
> >>> [    0.000000] init_memory_mapping: [mem
0x219e00000-0x219ffffff]
> >>> [    0.000000] init_memory_mapping: [mem
0x218000000-0x219dfffff]
> >>> [    0.000000] init_memory_mapping: [mem
0x200000000-0x217ffffff]
> >>> [    0.000000] init_memory_mapping: [mem
0x00100000-0xce699fff]
> >>> [    0.000000] init_memory_mapping: [mem
0xce6f1000-0xcf5fafff]
> >>> [    0.000000] init_memory_mapping: [mem
0xcf608000-0xcf6a4fff]
> >>> [    0.000000] init_memory_mapping: [mem
0xcf6aa000-0xcf6aafff]
> >>> [    0.000000] init_memory_mapping: [mem
0xcf6ff000-0xcf6fffff]
> >>> [    0.000000] init_memory_mapping: [mem
0x100000000-0x1ffffffff]
> >>> [    0.000000] init_memory_mapping: [mem
0x21a000000-0x227ffffff]
> >>>
> >>> We also have the similar unusable block, however the kernel
doesn''t map it.
> >>
> >> Right, iirc a fix for this was done not too long ago. Konrad may
> >> recall further details...
> > 
> > Stefan hit this I think. With the MTRR clipping blowing things. CC-ing
him
> > here.
> 
> Not personally but we had a bug report about this. Unfortunately it was a
bit
> confusing as in the beginning it was not really obvious whether the problem
was
> or was not fixed in later kernels or whether the different installations
used
> different dom0_mem arguments.
> 
> Reading the old bug report (which never completed as it seemed the
reporters
> apparently lost interest) I think the problem was that the hypervisor
detects
> the MTRR problem and set the remainder of memory as unusable.
> Then dom0 kernel (and if I parse that report right, it may have been fixed
> between 3.2 and 3.3 but hard to say when all you get is them saying yes or
no)
> would somehow still try to put mappings in there.
> 
> The link below is that bug report. Not sure of how much help it is.
> 
> 
> [1] https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1111470
Thanks Stefan.

At least I now know that a fix went in between 3.2 and 3.3, which is
much better range than 3.2..3.10.

I will try to bisect that if I have time, however it is low priority on
my list...


Wei.

> >>> 3.2.0-4-amd64 (just found out that there''s actually
backtrace in dmesg):
> >>> [    0.000000] BIOS-provided physical RAM map:
> >>> [    0.000000]  BIOS-e820: 0000000000000000 - 000000000008f000
(usable)
> >>> [    0.000000]  BIOS-e820: 000000000008f000 - 00000000000a0000
(reserved)
> >>> [    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000
(reserved)
> >>> [    0.000000]  BIOS-e820: 0000000000100000 - 00000000ce69a000
(usable)
> >>> [    0.000000]  BIOS-e820: 00000000ce69a000 - 00000000ce6f1000
(ACPI NVS)
> >>> [    0.000000]  BIOS-e820: 00000000ce6f1000 - 00000000cf5fb000
(usable)
> >>> [    0.000000]  BIOS-e820: 00000000cf5fb000 - 00000000cf608000
(reserved)
> >>> [    0.000000]  BIOS-e820: 00000000cf608000 - 00000000cf6a5000
(usable)
> >>> [    0.000000]  BIOS-e820: 00000000cf6a5000 - 00000000cf6aa000
(ACPI data)
> >>> [    0.000000]  BIOS-e820: 00000000cf6aa000 - 00000000cf6ab000
(usable)
> >>> [    0.000000]  BIOS-e820: 00000000cf6ab000 - 00000000cf6f2000
(ACPI NVS)
> >>> [    0.000000]  BIOS-e820: 00000000cf6f2000 - 00000000cf6ff000
(ACPI data)
> >>> [    0.000000]  BIOS-e820: 00000000cf6ff000 - 00000000cf700000
(usable)
> >>> [    0.000000]  BIOS-e820: 00000000cf700000 - 00000000d0000000
(reserved)
> >>> [    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000
(reserved)
> >>> [    0.000000]  BIOS-e820: 0000000100000000 - 000000022c000000
(usable)
> >>
> >> So no such block right off the BIOS. You''re not using Xen
TXT code
> >> by chance? Off the top of my head I don''t recall any
other place
> >> where multiple RAM pages might get turned into
"unusable".
> >>
> >> Jan
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> http://lists.xen.org/xen-devel
> 
>

Xen devel - Jul 2013 - Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Is: MTRR issue + Xen + memory clipping? Was:Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Is: MTRR issue + Xen + memory clipping? Was:Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box

Re: Is: MTRR issue + Xen + memory clipping? Was:Re: Xen not working with stock Debian Wheezy 3.2 kernel on a Core 2 Duo box