Hi:
Recently I am trying pvops dom0 kernel, 2.6.32.36, but confronted some
strange things.
In my grub, i give dom0 5630M memory. But when system startup, use free -m,
only see 2869M in total.
title Red Hat Enterprise Linux Server (2.6.32.36xen)
root (hd0,0)
kernel /xen-4.0.1.gz msi=1 iommu=off x2apic=off console=com1,vga
com1=115200,8n1 noreboot dom0_mem=5630M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=
0 cpufreq=none no-xsave
module /vmlinuz-2.6.32.36xen ro root=LABEL=/ hda=noprobe console=hvc0
module /initrd-2.6.32.36xen.img
Free -m output
root@xmao # free -m
total used free shared buffers cached
Mem: 2869 1068 1800 0 25 65
-/+ buffers/cache: 977 1892
Swap: 1027 0 1027
root@xmao# xm li
Name ID Mem VCPUs State Time(s)
Domain-0 0 4862 4 r----- 1272.6
root@xmao # xenstore-read /local/domain/0/memory/target
4979676
and dom0 auto balloon is disabled in /etc/xen/xend-config.
Well, when I use 2.6.32.10. Everything is OK
Free output in 2.6.32.10
root@xmao # free -m
total used free shared buffers cached
Mem: 4860 1550 3309 0 14 75
-/+ buffers/cache: 1460 3399
Swap: 1027 0 1027
Where is my memory going with 32.36?
Thanks.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
if xend is not started at the very beginning
root@xmao # free -m
total used free shared buffers cached
Mem: 3636 1020 2616 0 12 52
-/+ buffers/cache: 955 2681
Swap: 1027 0 1027
From: tinnycloud@hotmail.com
To: xen-devel@lists.xensource.com
Subject: Memory missing in 2.6.32.36
Date: Wed, 18 May 2011 22:58:55 +0800
Hi:
Recently I am trying pvops dom0 kernel, 2.6.32.36, but confronted some
strange things.
In my grub, i give dom0 5630M memory. But when system startup, use free -m,
only see 2869M in total.
title Red Hat Enterprise Linux Server (2.6.32.36xen)
root (hd0,0)
kernel /xen-4.0.1.gz msi=1 iommu=off x2apic=off console=com1,vga
com1=115200,8n1 noreboot dom0_mem=5630M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=
0 cpufreq=none no-xsave
module /vmlinuz-2.6.32.36xen ro root=LABEL=/ hda=noprobe console=hvc0
module /initrd-2.6.32.36xen.img
Free -m output
root@xmao # free -m
total used free shared buffers cached
Mem: 2869 1068 1800 0 25 65
-/+ buffers/cache: 977 1892
Swap: 1027 0 1027
root@xmao# xm li
Name ID Mem VCPUs State Time(s)
Domain-0 0 4862 4 r----- 1272.6
root@xmao # xenstore-read /local/domain/0/memory/target
4979676
and dom0 auto balloon is disabled in /etc/xen/xend-config.
Well, when I use 2.6.32.10. Everything is OK
Free output in 2.6.32.10
root@xmao # free -m
total used free shared buffers cached
Mem: 4860 1550 3309 0 14 75
-/+ buffers/cache: 1460 3399
Swap: 1027 0 1027
Where is my memory going with 32.36?
Thanks.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Boot up info below.>From line 4 and 70, see I give dom0 5630M memory,but from line 230: 229 Memory: 3717504k/52165668k available (4404k kernel code, 1057688k absent, 47389772k reserved, 2847k data, 732k init) which is printed in arch/x86/mm/init_64.c, nr_free_pages() only 3717504K, something wrong? static struct kcore_list kcore_vsyscall; void __init mem_init(void) { long codesize, reservedpages, datasize, initsize; unsigned long absent_pages; pci_iommu_alloc(); /* clear_bss() already clear the empty_zero_page */ reservedpages = 0; /* this will put all low memory onto the freelists */ #ifdef CONFIG_NUMA totalram_pages = numa_free_all_bootmem(); #else totalram_pages = free_all_bootmem(); #endif absent_pages = absent_pages_in_range(0, max_pfn); reservedpages = max_pfn - totalram_pages - absent_pages; after_bootmem = 1; codesize = (unsigned long) &_etext - (unsigned long) &_text; datasize = (unsigned long) &_edata - (unsigned long) &_etext; initsize = (unsigned long) &__init_end - (unsigned long) &__init_begin; /* Register memory areas for /proc/kcore */ kclist_add(&kcore_vsyscall, (void *)VSYSCALL_START, VSYSCALL_END - VSYSCALL_START, KCORE_OTHER); printk(KERN_INFO "Memory: %luk/%luk available (%ldk kernel code, " "%ldk absent, %ldk reserved, %ldk data, %ldk init)\n", nr_free_pages() << (PAGE_SHIFT-10), max_pfn << (PAGE_SHIFT-10), codesize >> 10, absent_pages << (PAGE_SHIFT-10), reservedpages << (PAGE_SHIFT-10), datasize >> 10, initsize >> 10); } 1 root (hd0,0) 2 Filesystem type is ext2fs, partition type 0x83 3 kernel /xen-4.0.1.gz msi=1 iommu=off x2apic=off console=com1,vga com1=115200,8n 4 1 noreboot dom0_mem=5630M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=0 cpufreq=none 5 6 \ \/ /___ _ __ | || | / _ \ / | * 7 \ // _ \ ''_ \ | || |_| | | || | * 8 / \ __/ | | | |__ _| |_| || | * 9 /_/\_\___|_| |_| |_|(_)___(_)_| * 10 (XEN) ACPI: RSDP 000FA000, 0024 (r2 ACPIAM) 11 (XEN) ACPI: XSDT BF790100, 005C (r1 080510 XSDT1021 20100805 MSFT 97) 12 (XEN) ACPI: FACP BF790290, 00F4 (r4 080510 FACP1021 20100805 MSFT 97) 13 (XEN) ACPI: DSDT BF7904F0, 4E48 (r2 CTSAV CTSAV031 31 INTL 20051117) 14 (XEN) ACPI: FACS BF79E000, 0040 15 (XEN) ACPI: APIC BF790390, 0118 (r2 080510 APIC1021 20100805 MSFT 97) 16 (XEN) ACPI: MCFG BF7904B0, 003C (r1 080510 OEMMCFG 20100805 MSFT 97) 17 (XEN) ACPI: OEMB BF79E040, 0082 (r1 080510 OEMB1021 20100805 MSFT 97) 18 (XEN) ACPI: SRAT BF79A4F0, 01D0 (r2 080510 OEMSRAT 1 INTL 1) 19 (XEN) ACPI: HPET BF79A6C0, 0038 (r1 080510 OEMHPET 20100805 MSFT 97) 20 (XEN) ACPI: SSDT BF7A1A10, 0363 (r1 DpgPmm CpuPm 12 INTL 20051117) 21 (XEN) System RAM: 49143MB (50322600kB) 22 (XEN) Domain heap initialised DMA width 32 bits 23 (XEN) Processor #0 6:12 APIC version 21 24 (XEN) Processor #2 6:12 APIC version 21 25 (XEN) Processor #18 6:12 APIC version 21 26 (XEN) Processor #20 6:12 APIC version 21 27 (XEN) Processor #32 6:12 APIC version 21 28 (XEN) Processor #34 6:12 APIC version 21 29 (XEN) Processor #50 6:12 APIC version 21 30 (XEN) Processor #52 6:12 APIC version 21 31 (XEN) Processor #1 6:12 APIC version 21 32 (XEN) Processor #3 6:12 APIC version 21 33 (XEN) Processor #19 6:12 APIC version 21 66 (XEN) *** LOADING DOMAIN 0 *** 67 (XEN) Xen kernel: 64-bit, lsb, compat32 68 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x19b7000 69 (XEN) PHYSICAL MEMORY ARRANGEMENT: 70 (XEN) Dom0 alloc.: 0000000434000000->0000000438000000 (1424896 pages to be allocated) 71 (XEN) VIRTUAL MEMORY ARRANGEMENT: 72 (XEN) Loaded kernel: ffffffff81000000->ffffffff819b7000 73 (XEN) Init. ramdisk: ffffffff819b7000->ffffffff81ec6e00 74 (XEN) Phys-Mach map: ffffffff81ec7000->ffffffff829c6000 75 (XEN) Start info: ffffffff829c6000->ffffffff829c64b4 76 (XEN) Page tables: ffffffff829c7000->ffffffff829e0000 77 (XEN) Boot stack: ffffffff829e0000->ffffffff829e1000 78 (XEN) TOTAL: ffffffff80000000->ffffffff82c00000 79 (XEN) ENTRY ADDRESS: ffffffff8172c200 80 (XEN) Dom0 has maximum 4 VCPUs 81 (XEN) Scrubbing Free RAM: ............................................................................................................................... ......................................................................................................................................................... .............. 82 ............................... 83 (XEN) Xen trace buffers: disabled 84 (XEN) Std. Loglevel: Errors and warnings 85 (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings) 86 (XEN) Xen is relinquishing VGA console. 87 (XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times to switch input to Xen) 88 (XEN) Freed 168kB init memory. 89 mapping kernel into physical memory 90 Xen: setup ISA identity maps 91 about to get started... 92 Initializing cgroup subsys cpuset 93 Initializing cgroup subsys cpu 94 Linux version 2.6.32.36xen (root@houyi-chunk2.dev.sd.aliyun.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Wed May 18 22:10:07 CST 2011 95 Command line: ro root=LABEL=/ hda=noprobe console=hvc0 96 KERNEL supported cpus: 97 Intel GenuineIntel 98 AMD AuthenticAMD 99 Centaur CentaurHauls 100 xen_release_chunk: looking at area pfn bf7e0-bf7ec: 12 pages freed 101 xen_release_chunk: looking at area pfn c0000-e0000: 131072 pages freed 102 xen_release_chunk: looking at area pfn f0000-fec00: 60416 pages freed 103 xen_release_chunk: looking at area pfn fec01-fec8a: 137 pages freed 104 xen_release_chunk: looking at area pfn fec8b-fee00: 373 pages freed 105 xen_release_chunk: looking at area pfn fee01-fff00: 4351 pages freed 106 released 196361 pages of unused memory 107 BIOS-provided physical RAM map: 108 Xen: 0000000000000000 - 000000000009a000 (usable) 109 Xen: 000000000009a400 - 0000000000100000 (reserved) 110 Xen: 0000000000100000 - 00000000bf790000 (usable) 111 Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data) 112 Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) 113 Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved) 114 Xen: 00000000bf7ec000 - 00000000c0000000 (reserved) 115 Xen: 00000000e0000000 - 00000000f0000000 (reserved) 116 Xen: 00000000fec00000 - 00000000fec01000 (reserved) 117 Xen: 00000000fec8a000 - 00000000fec8b000 (reserved) 118 Xen: 00000000fee00000 - 00000000fee01000 (reserved) 119 Xen: 00000000fff00000 - 0000000100000000 (reserved) 120 Xen: 0000000100000000 - 0000000c6ff09000 (usable) 121 DMI present. 122 AMI BIOS detected: BIOS may corrupt low RAM, working around it. 123 last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000 124 last_pfn = 0xbf790 max_arch_pfn = 0x400000000 125 init_memory_mapping: 0000000000000000-00000000bf790000 126 init_memory_mapping: 0000000100000000-0000000c6ff09000 127 RAMDISK: 019b7000 - 01ec6e00 128 ACPI: RSDP 00000000000fa000 00024 (v02 ACPIAM) 129 ACPI: XSDT 00000000bf790100 0005C (v01 080510 XSDT1021 20100805 MSFT 00000097) 130 ACPI: FACP 00000000bf790290 000F4 (v04 080510 FACP1021 20100805 MSFT 00000097) 131 ACPI: DSDT 00000000bf7904f0 04E48 (v02 CTSAV CTSAV031 00000031 INTL 20051117) 132 ACPI: FACS 00000000bf79e000 00040 133 ACPI: APIC 00000000bf790390 00118 (v02 080510 APIC1021 20100805 MSFT 00000097) 134 ACPI: MCFG 00000000bf7904b0 0003C (v01 080510 OEMMCFG 20100805 MSFT 00000097) 135 ACPI: OEMB 00000000bf79e040 00082 (v01 080510 OEMB1021 20100805 MSFT 00000097) 136 ACPI: SRAT 00000000bf79a4f0 001D0 (v02 080510 OEMSRAT 00000001 INTL 00000001) 137 ACPI: HPET 00000000bf79a6c0 00038 (v01 080510 OEMHPET 20100805 MSFT 00000097) 138 ACPI: SSDT 00000000bf7a1a10 00363 (v01 DpgPmm CpuPm 00000012 INTL 20051117) 139 (10 early reservations) ==> bootmem [0000000000 - 0c6ff09000] 140 #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] 141 #1 [00029c7000 - 00029e0000] XEN PAGETABLES ==> [00029c7000 - 00029e0000] 142 #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] 143 #3 [0001000000 - 000188c3b8] TEXT DATA BSS ==> [0001000000 - 000188c3b8] 144 #4 [00019b7000 - 0001ec6e00] RAMDISK ==> [00019b7000 - 0001ec6e00] 145 #5 [0001ec7000 - 00029c7000] XEN START INFO ==> [0001ec7000 - 00029c7000] 146 #6 [015fe00000 - 0c6ff09000] XEN EXTRA ==> [015fe00000 - 0c6ff09000] 147 #7 [000188d000 - 00018a31c0] BRK ==> [000188d000 - 00018a31c0] 148 #8 [0000100000 - 00006e4000] PGTABLE ==> [0000100000 - 00006e4000] 149 #9 [00029e0000 - 000858e000] PGTABLE ==> [00029e0000 - 000858e000] 150 Zone PFN ranges: 151 DMA 0x00000010 -> 0x00001000 152 DMA32 0x00001000 -> 0x00100000 153 Normal 0x00100000 -> 0x00c6ff09 154 Movable zone start PFN for each node 155 early_node_map[3] active PFN ranges 156 0: 0x00000010 -> 0x0000009a 157 0: 0x00000100 -> 0x000bf790 158 0: 0x00100000 -> 0x00c6ff09 159 ACPI: PM-Timer IO Port: 0x808 160 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) 161 ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) 162 ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled) 163 ACPI: LAPIC (acpi_id[0x04] lapic_id[0x14] enabled) 164 ACPI: LAPIC (acpi_id[0x05] lapic_id[0x20] enabled) 165 ACPI: LAPIC (acpi_id[0x06] lapic_id[0x22] enabled) 166 ACPI: LAPIC (acpi_id[0x07] lapic_id[0x32] enabled) 167 ACPI: LAPIC (acpi_id[0x08] lapic_id[0x34] enabled) 168 ACPI: LAPIC (acpi_id[0x09] lapic_id[0x01] enabled) 169 ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x03] enabled) 170 ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] enabled) 171 ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x15] enabled) 172 ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x21] enabled) 173 ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x23] enabled) 174 ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x33] enabled) 175 ACPI: LAPIC (acpi_id[0x10] lapic_id[0x35] enabled) 176 ACPI: LAPIC (acpi_id[0x11] lapic_id[0x90] disabled) 177 ACPI: LAPIC (acpi_id[0x12] lapic_id[0x91] disabled) 178 ACPI: LAPIC (acpi_id[0x13] lapic_id[0x92] disabled) 179 ACPI: LAPIC (acpi_id[0x14] lapic_id[0x93] disabled) 180 ACPI: LAPIC (acpi_id[0x15] lapic_id[0x94] disabled) 181 ACPI: LAPIC (acpi_id[0x16] lapic_id[0x95] disabled) 182 ACPI: LAPIC (acpi_id[0x17] lapic_id[0x96] disabled) 183 ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled) 184 ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0]) 185 IOAPIC[0]: apic_id 6, version 0, address 0xfec00000, GSI 0-0 186 ACPI: IOAPIC (id[0x07] address[0xfec8a000] gsi_base[24]) 187 IOAPIC[1]: apic_id 7, version 0, address 0xfec8a000, GSI 24-24 188 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 189 ERROR: Unable to locate IOAPIC for GSI 2 190 ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) 191 ERROR: Unable to locate IOAPIC for GSI 9 192 Using ACPI (MADT) for SMP configuration information 193 ACPI: HPET id: 0x8086a301 base: 0xfed00000 194 SMP: Allowing 4 CPUs, 0 hotplug CPUs 195 PM: Registered nosave memory: 000000000009a000 - 000000000009b000 196 PM: Registered nosave memory: 000000000009b000 - 0000000000100000 197 PM: Registered nosave memory: 00000000bf790000 - 00000000bf79e000 198 PM: Registered nosave memory: 00000000bf79e000 - 00000000bf7d0000 199 PM: Registered nosave memory: 00000000bf7d0000 - 00000000bf7e0000 200 PM: Registered nosave memory: 00000000bf7e0000 - 00000000bf7ec000 201 PM: Registered nosave memory: 00000000bf7ec000 - 00000000c0000000 202 PM: Registered nosave memory: 00000000c0000000 - 00000000e0000000 203 PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000 204 PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000 205 PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000 206 PM: Registered nosave memory: 00000000fec01000 - 00000000fec8a000 207 PM: Registered nosave memory: 00000000fec8a000 - 00000000fec8b000 208 PM: Registered nosave memory: 00000000fec8b000 - 00000000fee00000 209 PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000 210 PM: Registered nosave memory: 00000000fee01000 - 00000000fff00000 211 PM: Registered nosave memory: 00000000fff00000 - 0000000100000000 212 Allocating PCI resources starting at c0000000 (gap: c0000000:20000000) 213 Booting paravirtualized kernel on Xen 214 Xen version: 4.0.1 (preserve-AD) (dom0) 215 NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:4 nr_node_ids:1 216 PERCPU: Embedded 29 pages/cpu @ffff88002803b000 s90072 r8192 d20520 u118784 217 pcpu-alloc: s90072 r8192 d20520 u118784 alloc=29*4096 218 pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 219 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 12597183 220 Kernel command line: ro root=LABEL=/ hda=noprobe console=hvc0 221 PID hash table entries: 4096 (order: 3, 32768 bytes) 222 Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) 223 Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) 224 Initializing CPU#0 225 DMA: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000 226 DMA: software IO TLB at phys 0x20000000 - 0x24000000 227 xen_swiotlb_fixup: buf=ffff880020000000 size=67108864 228 xen_swiotlb_fixup: buf=ffff880024060000 size=32768 229 Memory: 3717504k/52165668k available (4404k kernel code, 1057688k absent, 47389772k reserved, 2847k data, 732k init) 230 SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 231 Hierarchical RCU implementation. 232 NR_IRQS:4352 nr_irqs:1280 233 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 234 ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) 235 xen: sci override: source_irq=9 global_irq=9 trigger=c polarity=1 236 xen_allocate_pirq: returning irq 9 for gsi 9 237 xen: acpi sci 9 238 Console: colour VGA+ 80x25 239 console [hvc0] enabled 240 allocated 511180800 bytes of page_cgroup 241 please try ''cgroup_disable=memory'' option if you don''t want memory cgroups 242 installing Xen timer for CPU 0 From: tinnycloud@hotmail.com To: xen-devel@lists.xensource.com Subject: RE: Memory missing in 2.6.32.36 Date: Thu, 19 May 2011 11:58:01 +0800 if xend is not started at the very beginning root@xmao # free -m total used free shared buffers cached Mem: 3636 1020 2616 0 12 52 -/+ buffers/cache: 955 2681 Swap: 1027 0 1027 From: tinnycloud@hotmail.com To: xen-devel@lists.xensource.com Subject: Memory missing in 2.6.32.36 Date: Wed, 18 May 2011 22:58:55 +0800 Hi: Recently I am trying pvops dom0 kernel, 2.6.32.36, but confronted some strange things. In my grub, i give dom0 5630M memory. But when system startup, use free -m, only see 2869M in total. title Red Hat Enterprise Linux Server (2.6.32.36xen) root (hd0,0) kernel /xen-4.0.1.gz msi=1 iommu=off x2apic=off console=com1,vga com1=115200,8n1 noreboot dom0_mem=5630M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle= 0 cpufreq=none no-xsave module /vmlinuz-2.6.32.36xen ro root=LABEL=/ hda=noprobe console=hvc0 module /initrd-2.6.32.36xen.img Free -m output root@xmao # free -m total used free shared buffers cached Mem: 2869 1068 1800 0 25 65 -/+ buffers/cache: 977 1892 Swap: 1027 0 1027 root@xmao# xm li Name ID Mem VCPUs State Time(s) Domain-0 0 4862 4 r----- 1272.6 root@xmao # xenstore-read /local/domain/0/memory/target 4979676 and dom0 auto balloon is disabled in /etc/xen/xend-config. Well, when I use 2.6.32.10. Everything is OK Free output in 2.6.32.10 root@xmao # free -m total used free shared buffers cached Mem: 4860 1550 3309 0 14 75 -/+ buffers/cache: 1460 3399 Swap: 1027 0 1027 Where is my memory going with 32.36? Thanks. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi:
I was able to do some debug. And I''d like to sum this issue
again. It looks like has relation with xen_memory_setup(). arch/x86/xen/setup.c
1) first in grub config, give dom0 5630M memory
kernel /xen-4.0.1.gz msi=1 iommu=off x2apic=off console=com1,vga
com1=115200,8n1 noreboot dom0_mem=5630M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=
0 cpufreq=none no-xsave
2) use 2.6.32.36 kernel, free -m only see 2869 memory
root@xmao# xm info | grep memory
total_memory : 49143
free_memory : 44512
node_to_memory : node0:24424
root@xmao # free -m
total used free shared buffers cached
Mem: 2869 1068 1800 0 25 65
-/+ buffers/cache: 977 1892
Swap: 1027 0 1027
root@xmao# xm li
Name ID Mem VCPUs State Time(s)
Domain-0 0 4862 4 r----- 1272.6
3) use 2.6.32.10 kernel, I got 4860, which means 4860 - 2869 memory is lost!
root@xmao # free -m
total used free shared buffers cached
Mem: 4860 1550 3309 0 14 75
-/+ buffers/cache: 1460 3399
Swap: 1027 0 1027
4) Compare the boot log between kernel 10 and 36, I noticed something
different, they have different last_pfn.
===below is 36=== Xen: 0000000000000000 - 000000000009a000 (usable)
Xen: 000000000009a400 - 0000000000100000 (reserved)
Xen: 0000000000100000 - 00000000bf790000 (usable)
Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data)
Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
Xen: 00000000bf7ec000 - 00000000c0000000 (reserved)
Xen: 00000000e0000000 - 00000000f0000000 (reserved)
Xen: 00000000fec00000 - 00000000fec01000 (reserved)
Xen: 00000000fec8a000 - 00000000fec8b000 (reserved)
Xen: 00000000fee00000 - 00000000fee01000 (reserved)
Xen: 00000000fff00000 - 0000000100000000 (reserved)
Xen: 0000000100000000 - 0000000c6ff09000 (usable)
DMI present.
AMI BIOS detected: BIOS may corrupt low RAM, working around it.
last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000
last_pfn = 0xbf790 max_arch_pfn = 0x400000000
=========below is 10
Xen: 0000000000000000 - 000000000009a400 (usable)
Xen: 000000000009a400 - 0000000000100000 (reserved)
Xen: 0000000000100000 - 00000000bf790000 (usable)
Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data)
Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
Xen: 00000000bf7ec000 - 00000000c0000000 (reserved)
Xen: 00000000e0000000 - 00000000f0000000 (reserved)
Xen: 00000000fec00000 - 00000000fec01000 (reserved)
Xen: 00000000fec8a000 - 00000000fec8b000 (reserved)
Xen: 00000000fee00000 - 00000000fee01000 (reserved)
Xen: 00000000fff00000 - 0000000100000000 (reserved)
Xen: 0000000100000000 - 000000015fe00000 (usable)
DMI present.
AMI BIOS detected: BIOS may corrupt low RAM, working around it.
last_pfn = 0x15fe00 max_arch_pfn = 0x400000000
last_pfn = 0xbf790 max_arch_pfn = 0x400000000
5) this makes me take a look char * __init xen_memory_setup(void) in
arch/x86/xen/setup.c
compare with the source code with 2.6.31, I noticed that code from line 242 is
newly added, and I comment line 261.
And the result is some of the memory comes back. I got 3662. But some others
still lost.
root@xmao# free -m
total used free shared buffers cached
Mem: 3662 406 3255 0 14 105
-/+ buffers/cache: 286 3375
Swap: 1027 0 1027
root@xmao # xm info | grep memory
total_memory : 49143
free_memory : 44512
node_to_memory : node0:24424
and BIOS got info same as kernel 10
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 000000000009a000 (usable)
Xen: 000000000009a400 - 0000000000100000 (reserved)
Xen: 0000000000100000 - 00000000bf790000 (usable)
Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data)
Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
Xen: 00000000bf7ec000 - 00000000c0000000 (reserved)
Xen: 00000000e0000000 - 00000000f0000000 (reserved)
Xen: 00000000fec00000 - 00000000fec01000 (reserved)
Xen: 00000000fec8a000 - 00000000fec8b000 (reserved)
Xen: 00000000fee00000 - 00000000fee01000 (reserved)
Xen: 00000000fff00000 - 0000000100000000 (reserved)
Xen: 0000000100000000 - 000000015fe00000 (usable)
DMI present.
AMI BIOS detected: BIOS may corrupt low RAM, working around it.
last_pfn = 0x15fe00 max_arch_pfn = 0x400000000
last_pfn = 0xbf790 max_arch_pfn = 0x400000000
=======code below=======
144 char * __init xen_memory_setup(void)
145 {
146 <+++static struct e820entry map[E820MAX] __initdata;
147
148 <+++unsigned long max_pfn = xen_start_info->nr_pages;
149 <+++unsigned long long mem_end;
150 <+++int rc;
151 <+++struct xen_memory_map memmap;
152 <+++unsigned long extra_pages = 0;
153 <+++unsigned long extra_limit;
154 <+++int op;
155 <+++int i;
156
157 <+++max_pfn = min(MAX_DOMAIN_PAGES, max_pfn);
158 <+++mem_end = PFN_PHYS(max_pfn);
159
160 <+++memmap.nr_entries = E820MAX;
161 <+++set_xen_guest_handle(memmap.buffer, map);
162
163 <+++op = xen_initial_domain() ?
164 <+++<+++XENMEM_machine_memory_map :
165 <+++<+++XENMEM_memory_map;
166 <+++rc = HYPERVISOR_memory_op(op, &memmap);
167 <+++if (rc == -ENOSYS) {
168 <+++<+++BUG_ON(xen_initial_domain());
169 <+++<+++memmap.nr_entries = 1;
170 <+++<+++map[0].addr = 0ULL;
171 <+++<+++map[0].size = mem_end;
172 <+++<+++/* 8MB slack (to balance backend allocations). */
173 <+++<+++map[0].size += 8ULL << 20;
174 <+++<+++map[0].type = E820_RAM;
175 <+++<+++rc = 0;
176 <+++}
177 <+++BUG_ON(rc);
178
179 <+++e820.nr_map = 0;
242 <+++/*
243 <+++ * Clamp the amount of extra memory to a EXTRA_MEM_RATIO
244 <+++ * factor the base size. On non-highmem systems, the base
245 <+++ * size is the full initial memory allocation; on highmem it
246 <+++ * is limited to the max size of lowmem, so that it doesn''t
247 <+++ * get completely filled.
248 <+++ *
249 <+++ * In principle there could be a problem in lowmem systems if
250 <+++ * the initial memory is also very large with respect to
251 <+++ * lowmem, but we won''t try to deal with that here.
252 <+++ */
253 <+++extra_limit = min(EXTRA_MEM_RATIO * min(max_pfn, PFN_DOWN(MAXMEM)),
254 <+++<+++<+++ max_pfn + extra_pages);
255
256 <+++if (extra_limit >= max_pfn)
257 <+++<+++extra_pages = extra_limit - max_pfn;
258 <+++else
259 <+++<+++extra_pages = 0;
260
261 <+++//xen_add_extra_mem(extra_pages);
262
263 <+++return "Xen";
From: tinnycloud@hotmail.com
To: xen-devel@lists.xensource.com
Subject: RE: Memory missing in 2.6.32.36
Date: Thu, 19 May 2011 13:30:03 +0800
Boot up info below. >From line 4 and 70, see I give dom0 5630M memory,
but from line 230:
229 Memory: 3717504k/52165668k available (4404k kernel code, 1057688k absent,
47389772k reserved, 2847k data, 732k init)
which is printed in arch/x86/mm/init_64.c, nr_free_pages() only 3717504K,
something wrong?
static struct kcore_list kcore_vsyscall;
void __init mem_init(void)
{
long codesize, reservedpages, datasize, initsize;
unsigned long absent_pages;
pci_iommu_alloc();
/* clear_bss() already clear the empty_zero_page */
reservedpages = 0;
/* this will put all low memory onto the freelists */
#ifdef CONFIG_NUMA
totalram_pages = numa_free_all_bootmem();
#else
totalram_pages = free_all_bootmem();
#endif
absent_pages = absent_pages_in_range(0, max_pfn);
reservedpages = max_pfn - totalram_pages - absent_pages;
after_bootmem = 1;
codesize = (unsigned long) &_etext - (unsigned long) &_text;
datasize = (unsigned long) &_edata - (unsigned long) &_etext;
initsize = (unsigned long) &__init_end - (unsigned long)
&__init_begin;
/* Register memory areas for /proc/kcore */
kclist_add(&kcore_vsyscall, (void *)VSYSCALL_START,
VSYSCALL_END - VSYSCALL_START, KCORE_OTHER);
printk(KERN_INFO "Memory: %luk/%luk available (%ldk kernel code, "
"%ldk absent, %ldk reserved, %ldk data, %ldk init)\n",
nr_free_pages() << (PAGE_SHIFT-10),
max_pfn << (PAGE_SHIFT-10),
codesize >> 10,
absent_pages << (PAGE_SHIFT-10),
reservedpages << (PAGE_SHIFT-10),
datasize >> 10,
initsize >> 10);
}
1 root (hd0,0)
2 Filesystem type is ext2fs, partition type 0x83
3 kernel /xen-4.0.1.gz msi=1 iommu=off x2apic=off console=com1,vga
com1=115200,8n
4 1 noreboot dom0_mem=5630M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=0
cpufreq=none
5
6 \ \/ /___ _ __ | || | / _ \ / | *
7 \ // _ \ ''_ \ | || |_| | | || |
*
8 / \ __/ | | | |__ _| |_| || | *
9 /_/\_\___|_| |_| |_|(_)___(_)_| *
10 (XEN) ACPI: RSDP 000FA000, 0024 (r2 ACPIAM)
11 (XEN) ACPI: XSDT BF790100, 005C (r1 080510 XSDT1021 20100805 MSFT 97)
12 (XEN) ACPI: FACP BF790290, 00F4 (r4 080510 FACP1021 20100805 MSFT 97)
13 (XEN) ACPI: DSDT BF7904F0, 4E48 (r2 CTSAV CTSAV031 31 INTL 20051117)
14 (XEN) ACPI: FACS BF79E000, 0040
15 (XEN) ACPI: APIC BF790390, 0118 (r2 080510 APIC1021 20100805 MSFT 97)
16 (XEN) ACPI: MCFG BF7904B0, 003C (r1 080510 OEMMCFG 20100805 MSFT 97)
17 (XEN) ACPI: OEMB BF79E040, 0082 (r1 080510 OEMB1021 20100805 MSFT 97)
18 (XEN) ACPI: SRAT BF79A4F0, 01D0 (r2 080510 OEMSRAT 1 INTL 1)
19 (XEN) ACPI: HPET BF79A6C0, 0038 (r1 080510 OEMHPET 20100805 MSFT 97)
20 (XEN) ACPI: SSDT BF7A1A10, 0363 (r1 DpgPmm CpuPm 12 INTL 20051117)
21 (XEN) System RAM: 49143MB (50322600kB)
22 (XEN) Domain heap initialised DMA width 32 bits
23 (XEN) Processor #0 6:12 APIC version 21
24 (XEN) Processor #2 6:12 APIC version 21
25 (XEN) Processor #18 6:12 APIC version 21
26 (XEN) Processor #20 6:12 APIC version 21
27 (XEN) Processor #32 6:12 APIC version 21
28 (XEN) Processor #34 6:12 APIC version 21
29 (XEN) Processor #50 6:12 APIC version 21
30 (XEN) Processor #52 6:12 APIC version 21
31 (XEN) Processor #1 6:12 APIC version 21
32 (XEN) Processor #3 6:12 APIC version 21
33 (XEN) Processor #19 6:12 APIC version 21
66 (XEN) *** LOADING DOMAIN 0 ***
67 (XEN) Xen kernel: 64-bit, lsb, compat32
68 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x19b7000
69 (XEN) PHYSICAL MEMORY ARRANGEMENT:
70 (XEN) Dom0 alloc.: 0000000434000000->0000000438000000 (1424896 pages
to be allocated)
71 (XEN) VIRTUAL MEMORY ARRANGEMENT:
72 (XEN) Loaded kernel: ffffffff81000000->ffffffff819b7000
73 (XEN) Init. ramdisk: ffffffff819b7000->ffffffff81ec6e00
74 (XEN) Phys-Mach map: ffffffff81ec7000->ffffffff829c6000
75 (XEN) Start info: ffffffff829c6000->ffffffff829c64b4
76 (XEN) Page tables: ffffffff829c7000->ffffffff829e0000
77 (XEN) Boot stack: ffffffff829e0000->ffffffff829e1000
78 (XEN) TOTAL: ffffffff80000000->ffffffff82c00000
79 (XEN) ENTRY ADDRESS: ffffffff8172c200
80 (XEN) Dom0 has maximum 4 VCPUs
81 (XEN) Scrubbing Free RAM:
...............................................................................................................................
.........................................................................................................................................................
..............
82 ...............................
83 (XEN) Xen trace buffers: disabled
84 (XEN) Std. Loglevel: Errors and warnings
85 (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
86 (XEN) Xen is relinquishing VGA console.
87 (XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three
times to switch input to Xen)
88 (XEN) Freed 168kB init memory.
89 mapping kernel into physical memory
90 Xen: setup ISA identity maps
91 about to get started...
92 Initializing cgroup subsys cpuset
93 Initializing cgroup subsys cpu
94 Linux version 2.6.32.36xen (root@houyi-chunk2.dev.sd.aliyun.com) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Wed May 18 22:10:07 CST 2011
95 Command line: ro root=LABEL=/ hda=noprobe console=hvc0
96 KERNEL supported cpus:
97 Intel GenuineIntel
98 AMD AuthenticAMD
99 Centaur CentaurHauls
100 xen_release_chunk: looking at area pfn bf7e0-bf7ec: 12 pages freed
101 xen_release_chunk: looking at area pfn c0000-e0000: 131072 pages freed
102 xen_release_chunk: looking at area pfn f0000-fec00: 60416 pages freed
103 xen_release_chunk: looking at area pfn fec01-fec8a: 137 pages freed
104 xen_release_chunk: looking at area pfn fec8b-fee00: 373 pages freed
105 xen_release_chunk: looking at area pfn fee01-fff00: 4351 pages freed
106 released 196361 pages of unused memory
107 BIOS-provided physical RAM map:
108 Xen: 0000000000000000 - 000000000009a000 (usable)
109 Xen: 000000000009a400 - 0000000000100000 (reserved)
110 Xen: 0000000000100000 - 00000000bf790000 (usable)
111 Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data)
112 Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
113 Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
114 Xen: 00000000bf7ec000 - 00000000c0000000 (reserved)
115 Xen: 00000000e0000000 - 00000000f0000000 (reserved)
116 Xen: 00000000fec00000 - 00000000fec01000 (reserved)
117 Xen: 00000000fec8a000 - 00000000fec8b000 (reserved)
118 Xen: 00000000fee00000 - 00000000fee01000 (reserved)
119 Xen: 00000000fff00000 - 0000000100000000 (reserved)
120 Xen: 0000000100000000 - 0000000c6ff09000 (usable)
121 DMI present.
122 AMI BIOS detected: BIOS may corrupt low RAM, working around it.
123 last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000
124 last_pfn = 0xbf790 max_arch_pfn = 0x400000000
125 init_memory_mapping: 0000000000000000-00000000bf790000
126 init_memory_mapping: 0000000100000000-0000000c6ff09000
127 RAMDISK: 019b7000 - 01ec6e00
128 ACPI: RSDP 00000000000fa000 00024 (v02 ACPIAM)
129 ACPI: XSDT 00000000bf790100 0005C (v01 080510 XSDT1021 20100805 MSFT
00000097)
130 ACPI: FACP 00000000bf790290 000F4 (v04 080510 FACP1021 20100805 MSFT
00000097)
131 ACPI: DSDT 00000000bf7904f0 04E48 (v02 CTSAV CTSAV031 00000031 INTL
20051117)
132 ACPI: FACS 00000000bf79e000 00040
133 ACPI: APIC 00000000bf790390 00118 (v02 080510 APIC1021 20100805 MSFT
00000097)
134 ACPI: MCFG 00000000bf7904b0 0003C (v01 080510 OEMMCFG 20100805 MSFT
00000097)
135 ACPI: OEMB 00000000bf79e040 00082 (v01 080510 OEMB1021 20100805 MSFT
00000097)
136 ACPI: SRAT 00000000bf79a4f0 001D0 (v02 080510 OEMSRAT 00000001 INTL
00000001)
137 ACPI: HPET 00000000bf79a6c0 00038 (v01 080510 OEMHPET 20100805 MSFT
00000097)
138 ACPI: SSDT 00000000bf7a1a10 00363 (v01 DpgPmm CpuPm 00000012 INTL
20051117)
139 (10 early reservations) ==> bootmem [0000000000 - 0c6ff09000]
140 #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 -
0000001000]
141 #1 [00029c7000 - 00029e0000] XEN PAGETABLES ==> [00029c7000 -
00029e0000]
142 #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 -
0000008000]
143 #3 [0001000000 - 000188c3b8] TEXT DATA BSS ==> [0001000000 -
000188c3b8]
144 #4 [00019b7000 - 0001ec6e00] RAMDISK ==> [00019b7000 -
0001ec6e00]
145 #5 [0001ec7000 - 00029c7000] XEN START INFO ==> [0001ec7000 -
00029c7000]
146 #6 [015fe00000 - 0c6ff09000] XEN EXTRA ==> [015fe00000 -
0c6ff09000]
147 #7 [000188d000 - 00018a31c0] BRK ==> [000188d000 -
00018a31c0]
148 #8 [0000100000 - 00006e4000] PGTABLE ==> [0000100000 -
00006e4000]
149 #9 [00029e0000 - 000858e000] PGTABLE ==> [00029e0000 -
000858e000]
150 Zone PFN ranges:
151 DMA 0x00000010 -> 0x00001000
152 DMA32 0x00001000 -> 0x00100000
153 Normal 0x00100000 -> 0x00c6ff09
154 Movable zone start PFN for each node
155 early_node_map[3] active PFN ranges
156 0: 0x00000010 -> 0x0000009a
157 0: 0x00000100 -> 0x000bf790
158 0: 0x00100000 -> 0x00c6ff09
159 ACPI: PM-Timer IO Port: 0x808
160 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
161 ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
162 ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled)
163 ACPI: LAPIC (acpi_id[0x04] lapic_id[0x14] enabled)
164 ACPI: LAPIC (acpi_id[0x05] lapic_id[0x20] enabled)
165 ACPI: LAPIC (acpi_id[0x06] lapic_id[0x22] enabled)
166 ACPI: LAPIC (acpi_id[0x07] lapic_id[0x32] enabled)
167 ACPI: LAPIC (acpi_id[0x08] lapic_id[0x34] enabled)
168 ACPI: LAPIC (acpi_id[0x09] lapic_id[0x01] enabled)
169 ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x03] enabled)
170 ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] enabled)
171 ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x15] enabled)
172 ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x21] enabled)
173 ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x23] enabled)
174 ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x33] enabled)
175 ACPI: LAPIC (acpi_id[0x10] lapic_id[0x35] enabled)
176 ACPI: LAPIC (acpi_id[0x11] lapic_id[0x90] disabled)
177 ACPI: LAPIC (acpi_id[0x12] lapic_id[0x91] disabled)
178 ACPI: LAPIC (acpi_id[0x13] lapic_id[0x92] disabled)
179 ACPI: LAPIC (acpi_id[0x14] lapic_id[0x93] disabled)
180 ACPI: LAPIC (acpi_id[0x15] lapic_id[0x94] disabled)
181 ACPI: LAPIC (acpi_id[0x16] lapic_id[0x95] disabled)
182 ACPI: LAPIC (acpi_id[0x17] lapic_id[0x96] disabled)
183 ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled)
184 ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
185 IOAPIC[0]: apic_id 6, version 0, address 0xfec00000, GSI 0-0
186 ACPI: IOAPIC (id[0x07] address[0xfec8a000] gsi_base[24])
187 IOAPIC[1]: apic_id 7, version 0, address 0xfec8a000, GSI 24-24
188 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
189 ERROR: Unable to locate IOAPIC for GSI 2
190 ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
191 ERROR: Unable to locate IOAPIC for GSI 9
192 Using ACPI (MADT) for SMP configuration information
193 ACPI: HPET id: 0x8086a301 base: 0xfed00000
194 SMP: Allowing 4 CPUs, 0 hotplug CPUs
195 PM: Registered nosave memory: 000000000009a000 - 000000000009b000
196 PM: Registered nosave memory: 000000000009b000 - 0000000000100000
197 PM: Registered nosave memory: 00000000bf790000 - 00000000bf79e000
198 PM: Registered nosave memory: 00000000bf79e000 - 00000000bf7d0000
199 PM: Registered nosave memory: 00000000bf7d0000 - 00000000bf7e0000
200 PM: Registered nosave memory: 00000000bf7e0000 - 00000000bf7ec000
201 PM: Registered nosave memory: 00000000bf7ec000 - 00000000c0000000
202 PM: Registered nosave memory: 00000000c0000000 - 00000000e0000000
203 PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000
204 PM: Registered nosave memory: 00000000f0000000 - 00000000fec00000
205 PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
206 PM: Registered nosave memory: 00000000fec01000 - 00000000fec8a000
207 PM: Registered nosave memory: 00000000fec8a000 - 00000000fec8b000
208 PM: Registered nosave memory: 00000000fec8b000 - 00000000fee00000
209 PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
210 PM: Registered nosave memory: 00000000fee01000 - 00000000fff00000
211 PM: Registered nosave memory: 00000000fff00000 - 0000000100000000
212 Allocating PCI resources starting at c0000000 (gap: c0000000:20000000)
213 Booting paravirtualized kernel on Xen
214 Xen version: 4.0.1 (preserve-AD) (dom0)
215 NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:4 nr_node_ids:1
216 PERCPU: Embedded 29 pages/cpu @ffff88002803b000 s90072 r8192 d20520 u118784
217 pcpu-alloc: s90072 r8192 d20520 u118784 alloc=29*4096
218 pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
219 Built 1 zonelists in Zone order, mobility grouping on. Total pages:
12597183
220 Kernel command line: ro root=LABEL=/ hda=noprobe console=hvc0
221 PID hash table entries: 4096 (order: 3, 32768 bytes)
222 Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
223 Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
224 Initializing CPU#0
225 DMA: Placing 64MB software IO TLB between ffff880020000000 -
ffff880024000000
226 DMA: software IO TLB at phys 0x20000000 - 0x24000000
227 xen_swiotlb_fixup: buf=ffff880020000000 size=67108864
228 xen_swiotlb_fixup: buf=ffff880024060000 size=32768
229 Memory: 3717504k/52165668k available (4404k kernel code, 1057688k absent,
47389772k reserved, 2847k data, 732k init)
230 SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
231 Hierarchical RCU implementation.
232 NR_IRQS:4352 nr_irqs:1280
233 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
234 ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
235 xen: sci override: source_irq=9 global_irq=9 trigger=c polarity=1
236 xen_allocate_pirq: returning irq 9 for gsi 9
237 xen: acpi sci 9
238 Console: colour VGA+ 80x25
239 console [hvc0] enabled
240 allocated 511180800 bytes of page_cgroup
241 please try ''cgroup_disable=memory'' option if you
don''t want memory cgroups
242 installing Xen timer for CPU 0
From: tinnycloud@hotmail.com
To: xen-devel@lists.xensource.com
Subject: RE: Memory missing in 2.6.32.36
Date: Thu, 19 May 2011 11:58:01 +0800
if xend is not started at the very beginning
root@xmao # free -m
total used free shared buffers cached
Mem: 3636 1020 2616 0 12 52
-/+ buffers/cache: 955 2681
Swap: 1027 0 1027
From: tinnycloud@hotmail.com
To: xen-devel@lists.xensource.com
Subject: Memory missing in 2.6.32.36
Date: Wed, 18 May 2011 22:58:55 +0800
Hi:
Recently I am trying pvops dom0 kernel, 2.6.32.36, but confronted some
strange things.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-May-19 14:42 UTC
Re: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable.
> 4) Compare the boot log between kernel 10 and 36, I noticed something different, they have different last_pfn. > > ===below is 36===> Xen: 0000000000000000 - 000000000009a000 (usable) > Xen: 000000000009a400 - 0000000000100000 (reserved) > Xen: 0000000000100000 - 00000000bf790000 (usable) > Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data) > Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) > Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved) > Xen: 00000000bf7ec000 - 00000000c0000000 (reserved) > Xen: 00000000e0000000 - 00000000f0000000 (reserved) > Xen: 00000000fec00000 - 00000000fec01000 (reserved) > Xen: 00000000fec8a000 - 00000000fec8b000 (reserved) > Xen: 00000000fee00000 - 00000000fee01000 (reserved) > Xen: 00000000fff00000 - 0000000100000000 (reserved) > Xen: 0000000100000000 - 0000000c6ff09000 (usable)Ok, C6FF09-100000 is 46GB> DMI present. > AMI BIOS detected: BIOS may corrupt low RAM, working around it. > last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000 > last_pfn = 0xbf790 max_arch_pfn = 0x400000000 > > =========below is 10 > > Xen: 0000000000000000 - 000000000009a400 (usable) > Xen: 000000000009a400 - 0000000000100000 (reserved) > Xen: 0000000000100000 - 00000000bf790000 (usable) > Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data) > Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) > Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved) > Xen: 00000000bf7ec000 - 00000000c0000000 (reserved) > Xen: 00000000e0000000 - 00000000f0000000 (reserved) > Xen: 00000000fec00000 - 00000000fec01000 (reserved) > Xen: 00000000fec8a000 - 00000000fec8b000 (reserved) > Xen: 00000000fee00000 - 00000000fee01000 (reserved) > Xen: 00000000fff00000 - 0000000100000000 (reserved) > Xen: 0000000100000000 - 000000015fe00000 (usable)And, 15FE00-100000 = 392704*4K/1K/1K = 1534MB = 1.5G> DMI present. > AMI BIOS detected: BIOS may corrupt low RAM, working around it. > last_pfn = 0x15fe00 max_arch_pfn = 0x400000000 > last_pfn = 0xbf790 max_arch_pfn = 0x400000000 >So the problem looks as the .36 is creating a pagetable for the memory above the 5G you provided - which sounds correct as you gave it 5G and the expectation that it can balloon up to 50G. What happens if you provide dom0_mem=max:5G to clamp on the size? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, May 19, 2011 at 09:17:11PM +0800, MaoXiaoyun wrote:> > Hi: > > I was able to do some debug. And I''d like to sum this issue again. It looks like has relation with xen_memory_setup(). arch/x86/xen/setup.c > > 1) first in grub config, give dom0 5630M memory > kernel /xen-4.0.1.gz msi=1 iommu=off x2apic=off console=com1,vga com1=115200,8n1 noreboot dom0_mem=5630M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle= 0 cpufreq=none no-xsaveHm, no-xsave? I think you mean xsave=0 Also the cpuidle looks wrong. It should be cpuidle=0 (no space). Lastly, why ''msi=1''? It is turned on by defualt. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2011-May-19 15:31 UTC
RE: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable.
> Date: Thu, 19 May 2011 10:42:42 -0400 > From: konrad.wilk@oracle.com > To: tinnycloud@hotmail.com > CC: xen-devel@lists.xensource.com; jeremy@goop.org > Subject: Re: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable. > > > 4) Compare the boot log between kernel 10 and 36, I noticed something different, they have different last_pfn. > > > > ===below is 36===> > Xen: 0000000000000000 - 000000000009a000 (usable) > > Xen: 000000000009a400 - 0000000000100000 (reserved) > > Xen: 0000000000100000 - 00000000bf790000 (usable) > > Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data) > > Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) > > Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved) > > Xen: 00000000bf7ec000 - 00000000c0000000 (reserved) > > Xen: 00000000e0000000 - 00000000f0000000 (reserved) > > Xen: 00000000fec00000 - 00000000fec01000 (reserved) > > Xen: 00000000fec8a000 - 00000000fec8b000 (reserved) > > Xen: 00000000fee00000 - 00000000fee01000 (reserved) > > Xen: 00000000fff00000 - 0000000100000000 (reserved) > > Xen: 0000000100000000 - 0000000c6ff09000 (usable) > > Ok, C6FF09-100000 is 46GB > > > DMI present. > > AMI BIOS detected: BIOS may corrupt low RAM, working around it. > > last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000 > > last_pfn = 0xbf790 max_arch_pfn = 0x400000000 > > > > =========below is 10 > > > > Xen: 0000000000000000 - 000000000009a400 (usable) > > Xen: 000000000009a400 - 0000000000100000 (reserved) > > Xen: 0000000000100000 - 00000000bf790000 (usable) > > Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data) > > Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) > > Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved) > > Xen: 00000000bf7ec000 - 00000000c0000000 (reserved) > > Xen: 00000000e0000000 - 00000000f0000000 (reserved) > > Xen: 00000000fec00000 - 00000000fec01000 (reserved) > > Xen: 00000000fec8a000 - 00000000fec8b000 (reserved) > > Xen: 00000000fee00000 - 00000000fee01000 (reserved) > > Xen: 00000000fff00000 - 0000000100000000 (reserved) > > Xen: 0000000100000000 - 000000015fe00000 (usable) > > And, 15FE00-100000 = 392704*4K/1K/1K = 1534MB = 1.5G > > > DMI present. > > AMI BIOS detected: BIOS may corrupt low RAM, working around it. > > last_pfn = 0x15fe00 max_arch_pfn = 0x400000000 > > last_pfn = 0xbf790 max_arch_pfn = 0x400000000 > > > > So the problem looks as the .36 is creating a pagetable for the > memory above the 5G you provided - which sounds correct as > you gave it 5G and the expectation that it can balloon up to 50G. > > What happens if you provide dom0_mem=max:5G to clamp on the size? >Seems nothing different(what is clamp means? ^_^ ) released 196361 pages of unused memory BIOS-provided physical RAM map: Xen: 0000000000000000 - 000000000009a000 (usable) Xen: 000000000009a400 - 0000000000100000 (reserved) Xen: 0000000000100000 - 00000000bf790000 (usable) Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data) Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved) Xen: 00000000bf7ec000 - 00000000c0000000 (reserved) Xen: 00000000e0000000 - 00000000f0000000 (reserved) Xen: 00000000fec00000 - 00000000fec01000 (reserved) Xen: 00000000fec8a000 - 00000000fec8b000 (reserved) Xen: 00000000fee00000 - 00000000fee01000 (reserved) Xen: 00000000fff00000 - 0000000100000000 (reserved) Xen: 0000000100000000 - 0000000c6ff09000 (usable) DMI present. AMI BIOS detected: BIOS may corrupt low RAM, working around it. last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000 last_pfn = 0xbf790 max_arch_pfn = 0x400000000 init_memory_mapping: 0000000000000000-00000000bf790000 init_memory_mapping: 0000000100000000-0000000c6ff09000 RAMDISK: 019b7000 - 01ec6e00 root@xmao # xm info | grep ''memory\|command'' total_memory : 49143 free_memory : 45022 node_to_memory : node0:24424 xen_commandline : msi=1 iommu=off x2apic=off console=com1,vga com1=115200,8n1 noreboot dom0_mem=max:5G dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=0 cpufreq=none xsave=0 root@xmao # free -m total used free shared buffers cached Mem: 2360 1136 1224 0 15 122 -/+ buffers/cache: 997 1363 Swap: 1027 0 1027 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Date: Thu, 19 May 2011 10:44:57 -0400 > From: konrad.wilk@oracle.com > To: tinnycloud@hotmail.com > CC: xen-devel@lists.xensource.com; jeremy@goop.org > Subject: Re: 2Gb Memory lost in 2.6.32.36 > > On Thu, May 19, 2011 at 09:17:11PM +0800, MaoXiaoyun wrote: > > > > Hi: > > > > I was able to do some debug. And I''d like to sum this issue again. It looks like has relation with xen_memory_setup(). arch/x86/xen/setup.c > > > > 1) first in grub config, give dom0 5630M memory > > kernel /xen-4.0.1.gz msi=1 iommu=off x2apic=off console=com1,vga com1=115200,8n1 noreboot dom0_mem=5630M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle= 0 cpufreq=none no-xsave > > Hm, no-xsave? I think you mean xsave=0 > > Also the cpuidle looks wrong. It should be cpuidle=0 (no space). > Lastly, why ''msi=1''? It is turned on by defualt.Thanks for correction. I''ll apply the change. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-May-19 16:16 UTC
Re: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
> > What happens if you provide dom0_mem=max:5G to clamp on the size? > > > > Seems nothing different(what is clamp means? ^_^ )http://en.wikipedia.org/wiki/Clamp_(tool) .. snip..> Xen: 0000000100000000 - 0000000c6ff09000 (usable) > DMI present. > AMI BIOS detected: BIOS may corrupt low RAM, working around it. > last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000.. snip..> xen_commandline : msi=1 iommu=off x2apic=off console=com1,vga com1=115200,8n1 noreboot dom0_mem=max:5G dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=0 cpufreq=none xsave=0OK, I would have expected your last entry in the E820 to be truncated to 15fe00000, but it is not with the max:5G parameter. Looks like you found a bug - but I am not sure if it is the Xen hypervisor or the E820 parsing code in Linux. Fortunatly for you, I just got my hands on 32G machine so I can work on this. Can you remind me in a week if I you do not see any patches from me? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2011-May-25 09:08 UTC
RE: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
Hi Konrad:
I think I''ve located where the lost memory going. It is in bootmem
when called sparse_init();
I enabled bootmem_debug=1 in kernel start args, and log every __reserve and
__free memory with ''count'' taged.
Like ___free & __reserve code(end of mail) :
Attached is the output log.
1) We give dom0 5630M, that is 5630 * 1024 /4 = 1441280 pages
2) Line 204-237 is the stage of early reservations.
Add up memory in *early reservations* except adderss beyond 15ffe0(line 228) :
6329(line 225) + 23 + 1508 + 23470 = 31330 pages ( 123M)
3) I will caculated the mem reserved after line 237.
In line 237 , tot_res_pages is 11631467 and line 1162, its values is 11881636,
so 11881636 - 11631467 = 250169 pages are reserved.
In line 203, tot_free_pages is 12776995 and line 1158, its values is 12810787,
so 12810787 - 12776995 = 33792 pages are freed
So 250169 - 33792 = 216377 pages (216377 *4 / 1024 = 845M) are still in
bootmem, I''ve been located it is in sparse_init.
4) Line 162, 196361 ( 768M ) pages of unused memory are been relesed to Xen.
5) Line 1169 , Memory is 3717504k = 3630M , 3630 + 768 + 845 + 123 = 5366.
5630 - 5366 = 264 M (Did I miscaulated? )
Questions
1) Normally, sparse_init will only reserver about 173M in previous kernel, and
in .36 is it meant to reserve so much?
2) where is 264M memory I last caculated, did I make mistake?
Many thanks.
============================code to log========mm/bootmem.c===
253 static void __init __free(bootmem_data_t *bdata,
254 <+++<+++<+++unsigned long sidx, unsigned long eidx)
255 {
256 <+++unsigned long idx;
257 static int count;
258 static int tot_free_pages;
259
260 tot_free_pages += (eidx - sidx);
261
262 <+++bdebug("count %d nid=%td start=%lx end=%lx pages %d
tot_free_pages %d\n", ++count, bdata - bootmem_node_data,
263 <+++<+++sidx + bdata->node_min_pfn,
264 <+++<+++eidx + bdata->node_min_pfn,
265 eidx - sidx,
266 tot_free_pages
267 );
268
277 static int __init __reserve(bootmem_data_t *bdata, unsigned long sidx,
278 unsigned long eidx, int flags)
279 {
280 unsigned long idx;
281 int exclusive = flags & BOOTMEM_EXCLUSIVE;
282 static int count;
283 static int tot_res_pages;
284
285 tot_res_pages += (eidx - sidx);
286 <+++bdebug("count %d nid=%td start=%lx end=%lx flags=%x pages=%d,
tot_res_pages %d\n",
287 ++count,
288 <+++<+++bdata - bootmem_node_data,
289 <+++<+++sidx + bdata->node_min_pfn,
290 <+++<+++eidx + bdata->node_min_pfn,
291 <+++<+++flags,
292 eidx - sidx,
293 tot_res_pages
294 );
> Date: Thu, 19 May 2011 12:16:56 -0400
> From: konrad.wilk@oracle.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; jeremy@goop.org
> Subject: Re: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory,
maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
>
> > > What happens if you provide dom0_mem=max:5G to clamp on the size?
> > >
> >
> > Seems nothing different(what is clamp means? ^_^ )
>
> http://en.wikipedia.org/wiki/Clamp_(tool)
>
> .. snip..
> > Xen: 0000000100000000 - 0000000c6ff09000 (usable)
> > DMI present.
> > AMI BIOS detected: BIOS may corrupt low RAM, working around it.
> > last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000
> .. snip..
> > xen_commandline : msi=1 iommu=off x2apic=off console=com1,vga
com1=115200,8n1 noreboot dom0_mem=max:5G dom0_max_vcpus=4 dom0_vcpus_pin
cpuidle=0 cpufreq=none xsave=0
>
> OK, I would have expected your last entry in the E820 to be truncated
> to 15fe00000, but it is not with the max:5G parameter. Looks like
> you found a bug - but I am not sure if it is the Xen hypervisor or
> the E820 parsing code in Linux. Fortunatly for you, I just got my hands on
32G
> machine so I can work on this. Can you remind me in a week if I you do
> not see any patches from me?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
MaoXiaoyun
2011-May-31 08:33 UTC
RE: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
Hi Konrad;
I realized that in 32.36 kernel dom0 can balloon up to use whole memory
address, in my case it is 48G. But it consumes 845M memory
for sparse memory to manage them, Well, my dom0 doesn''t need so big
flexable, and I learnt that use "mem=" option in grub can limit
dom0 memory, I think it works for me now.
Any comments?
title Red Hat Enterprise Linux Server (2.6.32.36xen)
root (hd0,0)
kernel /xen-4.0.1.gz iommu=off x2apic=off console=com1,vga
com1=115200,8n1 noreboot dom0_mem=6144M dom0_max_vcpus=4 dom0_vcpus_pin
cpuidle=0 cpufreq=none
module /vmlinuz-2.6.32.36xen ro root=LABEL=/ hda=noprobe console=hvc0
mem=8192M
module /initrd-2.6.32.36xen.img
After dom0 start,
root@xmao# free -m
total used free shared buffers cached
Mem: 4135 508 3626 0 21 166
-/+ buffers/cache: 320 3814
Swap: 1027 0 1027
======================xen_release_chunk: looking at area pfn bf7e0-bf7ec: 12
pages freed
xen_release_chunk: looking at area pfn c0000-e0000: 131072 pages freed
xen_release_chunk: looking at area pfn f0000-fec00: 60416 pages freed
xen_release_chunk: looking at area pfn fec01-fec8a: 137 pages freed
xen_release_chunk: looking at area pfn fec8b-fee00: 373 pages freed
xen_release_chunk: looking at area pfn fee01-fff00: 4351 pages freed
released 196361 pages of unused memory
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 000000000009a000 (usable)
Xen: 000000000009a400 - 0000000000100000 (reserved)
Xen: 0000000000100000 - 00000000bf790000 (usable)
Xen: 00000000bf790000 - 00000000bf79e000 (ACPI data)
Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
Xen: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
Xen: 00000000bf7ec000 - 00000000c0000000 (reserved)
Xen: 00000000e0000000 - 00000000f0000000 (reserved)
Xen: 00000000fec00000 - 00000000fec01000 (reserved)
Xen: 00000000fec8a000 - 00000000fec8b000 (reserved)
Xen: 00000000fee00000 - 00000000fee01000 (reserved)
Xen: 00000000fff00000 - 0000000100000000 (reserved)
Xen: 0000000100000000 - 0000000c6ff09000 (usable)
user-defined physical RAM map:
user: 0000000000000000 - 000000000009a000 (usable)
user: 000000000009a400 - 0000000000100000 (reserved)
user: 0000000000100000 - 00000000bf790000 (usable)
user: 00000000bf790000 - 00000000bf79e000 (ACPI data)
user: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
user: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
user: 00000000bf7ec000 - 00000000c0000000 (reserved)
user: 00000000e0000000 - 00000000f0000000 (reserved)
user: 00000000fec00000 - 00000000fec01000 (reserved)
user: 00000000fec8a000 - 00000000fec8b000 (reserved)
user: 00000000fee00000 - 00000000fee01000 (reserved)
user: 00000000fff00000 - 0000000100000000 (reserved)
user: 0000000100000000 - 0000000200000000 (usable)
From: tinnycloud@hotmail.com
To: konrad.wilk@oracle.com
CC: xen-devel@lists.xensource.com; jeremy@goop.org
Subject: RE: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory,
maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
Date: Wed, 25 May 2011 17:08:49 +0800
Hi Konrad:
I think I''ve located where the lost memory going. It is in bootmem
when called sparse_init();
I enabled bootmem_debug=1 in kernel start args, and log every __reserve and
__free memory with ''count'' taged.
Like ___free & __reserve code(end of mail) :
Attached is the output log.
1) We give dom0 5630M, that is 5630 * 1024 /4 = 1441280 pages
2) Line 204-237 is the stage of early reservations.
Add up memory in *early reservations* except adderss beyond 15ffe0(line 228) :
6329(line 225) + 23 + 1508 + 23470 = 31330 pages ( 123M)
3) I will caculated the mem reserved after line 237.
In line 237 , tot_res_pages is 11631467 and line 1162, its values is 11881636,
so 11881636 - 11631467 = 250169 pages are reserved.
In line 203, tot_free_pages is 12776995 and line 1158, its values is 12810787,
so 12810787 - 12776995 = 33792 pages are freed
So 250169 - 33792 = 216377 pages (216377 *4 / 1024 = 845M) are still in
bootmem, I''ve been located it is in sparse_init.
4) Line 162, 196361 ( 768M ) pages of unused memory are been relesed to Xen.
5) Line 1169 , Memory is 3717504k = 3630M , 3630 + 768 + 845 + 123 = 5366.
5630 - 5366 = 264 M (Did I miscaulated? )
Questions
1) Normally, sparse_init will only reserver about 173M in previous kernel, and
in .36 is it meant to reserve so much?
2) where is 264M memory I last caculated, did I make mistake?
Many thanks.
============================code to log========mm/bootmem.c===
253 static void __init __free(bootmem_data_t *bdata,
254 <+++<+++<+++unsigned long sidx, unsigned long eidx)
255 {
256 <+++unsigned long idx;
257 static int count;
258 static int tot_free_pages;
259
260 tot_free_pages += (eidx - sidx);
261
262 <+++bdebug("count %d nid=%td start=%lx end=%lx pages %d
tot_free_pages %d\n", ++count, bdata - bootmem_node_data,
263 <+++<+++sidx + bdata->node_min_pfn,
264 <+++<+++eidx + bdata->node_min_pfn,
265 eidx - sidx,
266 tot_free_pages
267 );
268
277 static int __init __reserve(bootmem_data_t *bdata, unsigned long sidx,
278 unsigned long eidx, int flags)
279 {
280 unsigned long idx;
281 int exclusive = flags & BOOTMEM_EXCLUSIVE;
282 static int count;
283 static int tot_res_pages;
284
285 tot_res_pages += (eidx - sidx);
286 <+++bdebug("count %d nid=%td start=%lx end=%lx flags=%x pages=%d,
tot_res_pages %d\n",
287 ++count,
288 <+++<+++bdata - bootmem_node_data,
289 <+++<+++sidx + bdata->node_min_pfn,
290 <+++<+++eidx + bdata->node_min_pfn,
291 <+++<+++flags,
292 eidx - sidx,
293 tot_res_pages
294 );
> Date: Thu, 19 May 2011 12:16:56 -0400
> From: konrad.wilk@oracle.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; jeremy@goop.org
> Subject: Re: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory,
maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
>
> > > What happens if you provide dom0_mem=max:5G to clamp on the size?
> > >
> >
> > Seems nothing different(what is clamp means? ^_^ )
>
> http://en.wikipedia.org/wiki/Clamp_(tool)
>
> .. snip..
> > Xen: 0000000100000000 - 0000000c6ff09000 (usable)
> > DMI present.
> > AMI BIOS detected: BIOS may corrupt low RAM, working around it.
> > last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000
> .. snip..
> > xen_commandline : msi=1 iommu=off x2apic=off console=com1,vga
com1=115200,8n1 noreboot dom0_mem=max:5G dom0_max_vcpus=4 dom0_vcpus_pin
cpuidle=0 cpufreq=none xsave=0
>
> OK, I would have expected your last entry in the E820 to be truncated
> to 15fe00000, but it is not with the max:5G parameter. Looks like
> you found a bug - but I am not sure if it is the Xen hypervisor or
> the E820 parsing code in Linux. Fortunatly for you, I just got my hands on
32G
> machine so I can work on this. Can you remind me in a week if I you do
> not see any patches from me?
--_4538a01d-59d6-427b-8f51-cec729dbb141_
Content-Type: text/html; charset="gb2312"
Content-Transfer-Encoding: 8bit
<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:微软雅黑
}
--></style>
</head>
<body class=''hmmessage''>
Hi Konrad;<BR>
<BR>
I realized that in 32.36
kernel dom0 can balloon up to use whole memory address, in my case it is 48G.
But it consumes 845M memory <BR>
for sparse memory to manage them, Well, my dom0
doesn''t need so big flexable,
and I learnt that use "mem="
option in grub can limit <BR>
dom0 memory, I think it works for me now. <BR>
<BR>
Any comments?<BR>
<BR>
title Red Hat Enterprise Linux Server
(2.6.32.36xen)<BR>
root
(hd0,0)<BR>
kernel /xen-4.0.1.gz iommu=off x2apic=off console=com1,vga
com1=115200,8n1 noreboot dom0_mem=6144M dom0_max_vcpus=4 dom0_vcpus_pin
cpuidle=0 cpufreq=none <BR>
module
/vmlinuz-2.6.32.36xen ro root=LABEL=/ hda=noprobe console=hvc0
mem=8192M<BR>
module /initrd-2.6.32.36xen.img<BR>
<BR>
After dom0 start,<BR>
<A href="mailto:root@xmao">root@xmao</A># free
-m<BR>
total
used
free
shared
buffers
cached<BR>Mem:
4135
508
3626
0
21
166<BR>-/+
buffers/cache:
320
3814<BR>Swap:
1027
0 1027<BR>
<BR>
=======================<BR>
xen_release_chunk: looking at area pfn bf7e0-bf7ec: 12 pages
freed<BR>xen_release_chunk: looking at area pfn c0000-e0000: 131072 pages
freed<BR>xen_release_chunk: looking at area pfn f0000-fec00: 60416 pages
freed<BR>xen_release_chunk: looking at area pfn fec01-fec8a: 137 pages
freed<BR>xen_release_chunk: looking at area pfn fec8b-fee00: 373 pages
freed<BR>xen_release_chunk: looking at area pfn fee01-fff00: 4351 pages
freed<BR>released 196361 pages of unused memory<BR>BIOS-provided
physical RAM map:<BR> Xen: 0000000000000000 - 000000000009a000
(usable)<BR> Xen: 000000000009a400 - 0000000000100000
(reserved)<BR> Xen: 0000000000100000 - 00000000bf790000
(usable)<BR> Xen: 00000000bf790000 - 00000000bf79e000 (ACPI
data)<BR> Xen: 00000000bf79e000 - 00000000bf7d0000 (ACPI
NVS)<BR> Xen: 00000000bf7d0000 - 00000000bf7e0000
(reserved)<BR> Xen: 00000000bf7ec000 - 00000000c0000000
(reserved)<BR> Xen: 00000000e0000000 - 00000000f0000000
(reserved)<BR> Xen: 00000000fec00000 - 00000000fec01000
(reserved)<BR> Xen: 00000000fec8a000 - 00000000fec8b000
(reserved)<BR> Xen: 00000000fee00000 - 00000000fee01000
(reserved)<BR> Xen: 00000000fff00000 - 0000000100000000
(reserved)<BR> Xen: 0000000100000000 - 0000000c6ff09000
(usable)<BR>user-defined physical RAM map:<BR> user:
0000000000000000 - 000000000009a000 (usable)<BR> user:
000000000009a400 - 0000000000100000 (reserved)<BR> user:
0000000000100000 - 00000000bf790000 (usable)<BR> user:
00000000bf790000 - 00000000bf79e000 (ACPI data)<BR> user:
00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)<BR> user:
00000000bf7d0000 - 00000000bf7e0000 (reserved)<BR> user:
00000000bf7ec000 - 00000000c0000000 (reserved)<BR> user:
00000000e0000000 - 00000000f0000000 (reserved)<BR> user:
00000000fec00000 - 00000000fec01000 (reserved)<BR> user:
00000000fec8a000 - 00000000fec8b000 (reserved)<BR> user:
00000000fee00000 - 00000000fee01000 (reserved)<BR> user:
00000000fff00000 - 0000000100000000 (reserved)<BR> user:
0000000100000000 - 0000000200000000 (usable)<BR>
<BR> <BR>
<HR id=stopSpelling>
From: tinnycloud@hotmail.com<BR>To: konrad.wilk@oracle.com<BR>CC:
xen-devel@lists.xensource.com; jeremy@goop.org<BR>Subject: RE: [Xen-devel]
RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable,
clamping (max:5G) is not helping.<BR>Date: Wed, 25 May 2011 17:08:49
+0800<BR><BR>
<META name=Generator content="Microsoft SafeHTML">
<STYLE>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:微软雅黑;}
</STYLE>
Hi Konrad:<BR> <BR>
I think I''ve located where the lost
memory going. It is in bootmem when called sparse_init();
<BR> <BR> I enabled
bootmem_debug=1 in kernel start args, and log every __reserve and __free memory
with ''count''
taged.<BR> <BR> Like ___free
& __reserve code(end of
mail) :<BR> <BR>
Attached is the output log. <BR>
<BR>1) We give dom0 5630M, that is 5630 * 1024 /4 = 1441280
pages<BR> <BR>2) Line 204-237 is the stage of early
reservations. <BR>Add up memory in *early reservations* except
adderss beyond 15ffe0(line 228) : <BR>
6329(line 225) + 23 + 1508 + 23470 = 31330 pages (
123M)<BR> <BR>3) I will caculated the mem reserved after
line 237.<BR> <BR>In line
237 , tot_res_pages is 11631467 and line 1162, its
values is 11881636, so 11881636 - 11631467 = 250169 pages are
reserved.<BR>In line 203, tot_free_pages is 12776995 and line 1158, its
values is 12810787, so 12810787 - 12776995 = 33792 pages are
freed<BR>So 250169 - 33792 = 216377 pages (216377 *4 /
1024 = 845M) are still in bootmem, I''ve been located
it is in sparse_init.<BR> <BR>4) Line 162, 196361
( 768M ) pages of unused memory are been relesed to
Xen.<BR> <BR>5) Line 1169 ,
Memory is 3717504k = 3630M , 3630 + 768 +
845 + 123 =
5366. 5630
- 5366 = 264 M (Did I miscaulated? )<BR>
<BR>Questions<BR> <BR>1) Normally,
sparse_init will only reserver about 173M in previous kernel, and in .36 is it
meant to reserve so much?<BR>2) where is 264M memory I last caculated, did
I make mistake?<BR> <BR>Many thanks.
<BR> <BR> <BR> <BR> <BR> <BR> <BR> <BR> <BR>============================code
to log========mm/bootmem.c====<BR> <BR>253 static void
__init __free(bootmem_data_t *bdata,<BR>254
<+++<+++<+++unsigned long sidx, unsigned long
eidx)<BR>255 {<BR>256 <+++unsigned long
idx;<BR>257 static int
count;<BR>258 static int
tot_free_pages;<BR>259
<BR>260 tot_free_pages += (eidx -
sidx);<BR>261 <BR>262 <+++bdebug("count %d nid=%td
start=%lx end=%lx pages %d tot_free_pages %d\n", ++count, bdata -
bootmem_node_data,<BR>263 <+++<+++sidx +
bdata->node_min_pfn,<BR>264 <+++<+++eidx +
bdata->node_min_pfn,<BR>265
eidx -
sidx,<BR>266
tot_free_pages<BR>267
);<BR>268 <BR> <BR>277 static int __init
__reserve(bootmem_data_t *bdata, unsigned long sidx,<BR>278
unsigned long eidx, int flags)<BR>279
{<BR>280 unsigned long
idx;<BR>281 int exclusive = flags
&
BOOTMEM_EXCLUSIVE;<BR>282 static
int count;<BR>283 static int
tot_res_pages;<BR>284
<BR>285 tot_res_pages += (eidx -
sidx);<BR>286 <+++bdebug("count %d nid=%td start=%lx end=%lx
flags=%x pages=%d, tot_res_pages
%d\n",
<BR>287
++count,<BR>288 <+++<+++bdata -
bootmem_node_data,<BR>289 <+++<+++sidx +
bdata->node_min_pfn,<BR>290 <+++<+++eidx +
bdata->node_min_pfn,<BR>291
<+++<+++flags,<BR>292
eidx -
sidx,<BR>293
tot_res_pages<BR>294
);<BR>
<BR> <BR><BR> <BR>> Date: Thu,
19 May 2011 12:16:56 -0400<BR>> From:
konrad.wilk@oracle.com<BR>> To:
tinnycloud@hotmail.com<BR>> CC: xen-devel@lists.xensource.com;
jeremy@goop.org<BR>> Subject: Re: [Xen-devel] RE:2Gb Memory lost in
2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable, clamping (max:5G) is
not helping.<BR>> <BR>> > > What happens
if you provide dom0_mem=max:5G to clamp on the size?<BR>> >
> <BR>> > <BR>> > Seems nothing
different(what is clamp means? ^_^ )<BR>> <BR>>
http://en.wikipedia.org/wiki/Clamp_(tool)<BR>> <BR>>
.. snip..<BR>> > Xen: 0000000100000000 - 0000000c6ff09000
(usable)<BR>> > DMI present.<BR>> > AMI
BIOS detected: BIOS may corrupt low RAM, working around it.<BR>>
> last_pfn = 0xc6ff09 max_arch_pfn = 0x400000000<BR>> ..
snip..<BR>> > xen_commandline : msi=1 iommu=off x2apic=off
console=com1,vga com1=115200,8n1 noreboot dom0_mem=max:5G dom0_max_vcpus=4
dom0_vcpus_pin cpuidle=0 cpufreq=none xsave=0<BR>>
<BR>> OK, I would have expected your last entry in the E820 to be
truncated<BR>> to 15fe00000, but it is not with the max:5G
parameter. Looks like<BR>> you found a bug - but I am not sure if
it is the Xen hypervisor or<BR>> the E820 parsing code in Linux.
Fortunatly for you, I just got my hands on 32G<BR>> machine so I
can work on this. Can you remind me in a week if I you do<BR>> not
see any patches from me?<BR> </body>
</html>
--_4538a01d-59d6-427b-8f51-cec729dbb141_--
--===============0633053430=Content-Type: text/plain;
charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
--===============0633053430==--
Konrad Rzeszutek Wilk
2011-May-31 14:44 UTC
Re: [Xen-devel] RE:2Gb Memory lost in 2.6.32.36 - 50G memory, maxmem=50G, 2G used for pagetable, clamping (max:5G) is not helping.
On Tue, May 31, 2011 at 04:33:52PM +0800, MaoXiaoyun wrote:> > Hi Konrad; > > I realized that in 32.36 kernel dom0 can balloon up to use whole memory address, in my case it is 48G. But it consumes 845M memory > for sparse memory to manage them, Well, my dom0 doesn''t need so big flexable, and I learnt that use "mem=" option in grub can limit > dom0 memory, I think it works for me now.HA! That does it too.> > Any comments?> > title Red Hat Enterprise Linux Server (2.6.32.36xen) > root (hd0,0) > kernel /xen-4.0.1.gz iommu=off x2apic=off console=com1,vga com1=115200,8n1 noreboot dom0_mem=6144M dom0_max_vcpus=4 dom0_vcpus_pin cpuidle=0 cpufreq=none > module /vmlinuz-2.6.32.36xen ro root=LABEL=/ hda=noprobe console=hvc0 mem=8192M > module /initrd-2.6.32.36xen.imgThat is a good work-around. Still need to fix the bug in the E820 setup code to take dom0_mem=max: parameter under advisment. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel