Hello, I have an odd problem on a dual processor, dual core Opteron system. Obviosously it is x86_64 so should have no problem large amounts of RAM. The system has 4 GB installed (2GB on each processor). If I boot the system with a fresh install of Debian Etch it sees all the memory fine. dmesg reports: Memory: 4107008k/5242880k available (1929k kernel code, 86836k reserved, 864k data, 176k init) However if I boot a xen-3.0.3 kernel, it only sees about half of this: The xen dmesg reports: Memory: 183852k/270336k available (3531k kernel code, 77948k reserved, 1250k data, 208k init) However ''xm info'' shows: total_memory : 3071 This is running a source install of Xen 3.0.3 with kernel 2.6.16.29 I have checked the kernel configuration that Etch uses, compared to my Xen kernel, and see that Etch is using "Discontiguos Memory" model, where as the only option in the xen compile is "Flat Memory" model. Also Etch is using NUMA, which doesn''t seem to be an option during the xen kernel compile. Surely, xen can access the full memory of this system? Can anyone recommend any things to try? My kernel config for "processor type and features" is below. Etch kernel uses: CONFIG_ARCH_DISCONTIGMEM_ENABLE=y CONFIG_ARCH_DISCONTIGMEM_DEFAULT=y CONFIG_ARCH_SPARSEMEM_ENABLE=y CONFIG_SELECT_MEMORY_MODEL=y # CONFIG_FLATMEM_MANUAL is not set CONFIG_DISCONTIGMEM_MANUAL=y # CONFIG_SPARSEMEM_MANUAL is not set CONFIG_DISCONTIGMEM=y CONFIG_FLAT_NODE_MEM_MAP=y Whereas my Xen kernel config uses: # # Processor type and features # CONFIG_X86_PC=y # CONFIG_X86_VSMP is not set CONFIG_MK8=y # CONFIG_MPSC is not set # CONFIG_GENERIC_CPU is not set CONFIG_X86_64_XEN=y CONFIG_X86_NO_TSS=y CONFIG_X86_NO_IDT=y CONFIG_X86_L1_CACHE_BYTES=64 CONFIG_X86_L1_CACHE_SHIFT=6 CONFIG_X86_GOOD_APIC=y # CONFIG_MICROCODE is not set # CONFIG_X86_MSR is not set # CONFIG_X86_CPUID is not set CONFIG_X86_IO_APIC=y CONFIG_X86_XEN_GENAPIC=y CONFIG_X86_LOCAL_APIC=y CONFIG_MTRR=y CONFIG_SMP=y CONFIG_PREEMPT_NONE=y # CONFIG_PREEMPT_VOLUNTARY is not set # CONFIG_PREEMPT is not set # CONFIG_PREEMPT_BKL is not set CONFIG_ARCH_FLATMEM_ENABLE=y CONFIG_SELECT_MEMORY_MODEL=y CONFIG_FLATMEM_MANUAL=y # CONFIG_DISCONTIGMEM_MANUAL is not set # CONFIG_SPARSEMEM_MANUAL is not set CONFIG_FLATMEM=y CONFIG_FLAT_NODE_MEM_MAP=y # CONFIG_SPARSEMEM_STATIC is not set CONFIG_SPLIT_PTLOCK_CPUS=4096 CONFIG_NR_CPUS=32 # CONFIG_HOTPLUG_CPU is not set CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y CONFIG_SWIOTLB=y # CONFIG_CRASH_DUMP is not set CONFIG_PHYSICAL_START=0x100000 # CONFIG_SECCOMP is not set # CONFIG_HZ_100 is not set CONFIG_HZ_250=y # CONFIG_HZ_1000 is not set CONFIG_HZ=250 CONFIG_GENERIC_HARDIRQS=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_ISA_DMA_API=y CONFIG_GENERIC_PENDING_IRQ=y -- John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Apr 28, 2007 at 02:13:07PM +0100, John Hannfield wrote: Hello,> I have an odd problem on a dual processor, dual core Opteron system. > Obviosously it is x86_64 so should have no problem large amounts of > RAM. The system has 4 GB installed (2GB on each processor). > > If I boot the system with a fresh install of Debian Etch it sees all the > memory > fine. dmesg reports: > > Memory: 4107008k/5242880k available (1929k kernel code, 86836k > reserved, 864k data, 176k init) > > However if I boot a xen-3.0.3 kernel, it only sees about half of this: > The xen dmesg reports: > > Memory: 183852k/270336k available (3531k kernel code, 77948k reserved, > 1250k data, 208k init)Can you check if this problem is related to debian bug #419994? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419994 Xen system gets memory map directly from grub loader, differently to plain linux kernel. On some systems grub have trouble calling BIOS function 0x810 int 0x15 and falls back to function 0x801, which cannot report full memory in presence of PCI memory hole. You can easily check that, just try command displaymem in grub command line. If it shows full memory map then I am wrong and your problem is unrelated to mine. Regards, Kupson -- Great software without the knowledge to run it is pretty useless. (Linux Gazette #1) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Kupson,> Can you check if this problem is related to debian bug #419994? > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419994OK. Interesting.> You can easily check that, just try command displaymem in grub command > line. If it shows full memory map then I am wrong and your problem is > unrelated to mine.This is what it shows: grub> displaymem displaymem EISA Memory BIOS Interface is present Address Map BIOS Interface is present Lower memory: 640K, Upper memory (to first chipset hole): 3072K [Address Range Descriptor entries immediately follow (values are 64-bit)] Usable RAM: Base Address: 0x0 X 4GB + 0x0, Length: 0x0 X 4GB + 0xa0000 bytes Reserved: Base Address: 0x0 X 4GB + 0xa0000, Length: 0x0 X 4GB + 0x60000 bytes Usable RAM: Base Address: 0x0 X 4GB + 0x100000, Length: 0x0 X 4GB + 0x300000 bytes-- It seems to say 4GB there. Is that OK? Or does it stop at 3072K (first chipset hole) ? John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Apr 28, 2007 at 03:35:43PM +0100, John Hannfield wrote: Hello,> >You can easily check that, just try command displaymem in grub command > >line. If it shows full memory map then I am wrong and your problem is > >unrelated to mine. > > This is what it shows: > > grub> displaymem > displaymem > EISA Memory BIOS Interface is present > Address Map BIOS Interface is present > Lower memory: 640K, Upper memory (to first chipset hole): 3072K > [Address Range Descriptor entries immediately follow (values are 64-bit)] > Usable RAM: Base Address: 0x0 X 4GB + 0x0, > Length: 0x0 X 4GB + 0xa0000 bytes > Reserved: Base Address: 0x0 X 4GB + 0xa0000, > Length: 0x0 X 4GB + 0x60000 bytes > Usable RAM: Base Address: 0x0 X 4GB + 0x100000, > Length: 0x0 X 4GB + 0x300000 bytes--There is memory map, so it''s not debian bug #419994.> It seems to say 4GB there. Is that OK?Hmm, there is 640KB memory, then PC legacy hole up to 1MB. Next to it 3GB usable memory region, then nothing? Did you put whole displaymem output in email? If so, there is something wrong -- missing memory above 4GB address. (I think that bios reserve 1GB space at ~3GB address for PCI devices.)> Or does it stop at 3072K (first chipset hole) ?Yes, there is no memory detected (according to presented memory map) above chipset/PCI hole. Please send memory map from linux kernel that correctly detects all memory. You can use: $ dmesg | grep e820 or locate it in kernel boot logs. Kupson -- Great software without the knowledge to run it is pretty useless. (Linux Gazette #1) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Rafal. That is odd. So it only report about 4 MB or memory?> Please send memory map from linux kernel that correctly detects all > memory.OK, for a vanilla linux boot (Debian Etch AMD64), I get this e820 in the dmesg. BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 00000000bfff0000 (usable) BIOS-e820: 00000000bfff0000 - 00000000bfffe000 (ACPI data) BIOS-e820: 00000000bfffe000 - 00000000c0000000 (ACPI NVS) BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved) BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved) BIOS-e820: 0000000100000000 - 0000000140000000 (usable) DMI present. and the Vanilla linux boot can access all 4GB of memory. However if I boot xen 3.0.3 my ''xm dmesg'' is this: (XEN) Command line: /boot/xen-3.0.3-0.gz dom0_mem=262144 com1=115200,8n1 console=vga,com1 (XEN) Physical RAM map: (XEN) 0000000000000000 - 000000000009dc00 (usable) (XEN) 0000000000100000 - 00000000bfff0000 (usable) (XEN) System RAM: 3071MB (3145268kB) (XEN) Xen heap: 14MB (14384kB) (XEN) found SMP MP-table at 000ff780 (XEN) DMI present. (XEN) Using APIC driver default Any ideas what the problem is? FYI - In BIOS I have "Memory Hole ReMapping" set to "enabled", and "MTRR Mapping" set to "Discreet". Thanks -- John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, Apr 28, 2007 at 05:07:36PM +0100, John Hannfield wrote:> Hi Rafal. > > That is odd. So it only report about 4 MB or memory?Grub detects about 3G total usable memory.> >Please send memory map from linux kernel that correctly detects all > >memory. > > OK, for a vanilla linux boot (Debian Etch AMD64), I get this e820 in the > dmesg. > > BIOS-provided physical RAM map: > BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) > BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) > BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) > BIOS-e820: 0000000000100000 - 00000000bfff0000 (usable)grub memory map ends here> BIOS-e820: 00000000bfff0000 - 00000000bfffe000 (ACPI data) > BIOS-e820: 00000000bfffe000 - 00000000c0000000 (ACPI NVS) > BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) > BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) > BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved) > BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved)some reserved regions, acpi and pci memory hole> BIOS-e820: 0000000100000000 - 0000000140000000 (usable)Look, there is! 1GB of RAM starting at address 4GB.> Any ideas what the problem is?There is difference between grub and native linux memory map. Strange, both of them gets it from BIOS e820 function. Maybe after all, our problems are similar. We both have systems where grub detects memory map wrongly. In my case -- entirely wrong, no single region in memory map. In yours some regions are missing. Can you try patched version od grub? Try first patch from message http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419994;msg=15 or this: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419994;msg=5 On my system both of them make grub works correctly [1]. (And full 4GB memory in Xen) Kupson [1] Sadly, I don''t know why this patches helps. -- Great software without the knowledge to run it is pretty useless. (Linux Gazette #1) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Can you try patched version od grub? Try first patch from message > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419994;msg=15 > or this: > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419994;msg=5 >OK. Yes, I will try upgrading and patching grub. I am on a earlier etch version of grub - version 0.97-23 I see 0.97-27 is the release with Etch stable. I will upgrade to that and if I still have problems, apply those patches you mention. I will not be able to do this until Tuesday though. Thanks for your help Rafal. I will let you know how it goes. -- John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Rafal,
Well I tried the grub 0.97 patches. I recompiled grub with both patches, then
replaced the etch grub withthe new .deb. Then re-installed grub stages on the
disk with
grub> root (hd0,0)
grub> setup (hd0)
Now if I reboot the machine at at the live grub prompt, I do a displaymem,
it says:
EISA Memory BIOS Interface is present
Address Map BIOS Interface is present
Lower memory: 631K, Upper memory (to first chipset hole): 3144640K
There are no address ranges given.
So it looks like it only sees 3 GB.
If I run grub from the booted system, on the command line, it gives the ranges,
and also the debug info from the patched version of grub.
# grub
Probing devices to guess BIOS drives. This may take a long time.
SMAP: addr:f7b4fc00 cont:1
SMAP: addr:f7b4fc18 cont:2
SMAP: addr:f7b4fc30 cont:0
mbi.mmap_length: 48
GNU GRUB version 0.97 (640K lower / 3072K upper memory)
[ Minimal BASH-like line editing is supported. For
the first word, TAB lists possible command
completions. Anywhere else TAB lists the possible
completions of a device/filename. ]
grub> displaymem
displaymem
EISA Memory BIOS Interface is present
Address Map BIOS Interface is present
Lower memory: 640K, Upper memory (to first chipset hole): 3072K
[Address Range Descriptor entries immediately follow (values are 64-bit)]
Usable RAM: Base Address: 0x0 X 4GB + 0x0,
Length: 0x0 X 4GB + 0xa0000 bytes
Reserved: Base Address: 0x0 X 4GB + 0xa0000,
Length: 0x0 X 4GB + 0x60000 bytes
Usable RAM: Base Address: 0x0 X 4GB + 0x100000,
Length: 0x0 X 4GB + 0x300000 bytes
grub>
any ideas on where to go from here?
Is it normal for it to only show
Upper memory (to first chipset hole): 3144640K
at the grub boot menu?
Thanks
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
John,> any ideas on where to go from here? > Is it normal for it to only show > Upper memory (to first chipset hole): 3144640K > at the grub boot menu?What revision of processor do you have, and how large is your graphics memory? If you have a graphics card that eats a fair bit of memory (256 or 512MB) on a processor that doesn''t support "memory hoisting"[1] (i.e. remapping of "behind PCI memory" to a higher address), then I wouldn''t be surprised if you don''t see more than 3.x GB - the reason being that the rest of the memory is "behind" the PCI bus devices. Processors from Rev E has a more "flexible" memory hoisting functionality, where there is a proper "set of memory hole registers". Also, if you have a newer processor you may need a new BIOS to support this, and you''ll possibly also need to enable "memory hoisting" (or memory hole or some such) in the BIOS setup. [1] All processors do have a form of memory hoisting using the "chipselect" method, however, this is only possible to use under certain conditions, and it''s limited to moving a whole chipselect-bank at a time [which in your case may be 1-2GB, and the BIOS may deem it better to leave a few hundred MB below 32-bit than to move a whole heap of 1 or 2GB above the 4GB boundary - particularly since a majority of users still prefer to use 32-bit OS''s that can''t get past 4GB!]. -- Mats> > Thanks > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Mats> What revision of processor do you have, and how large is your graphics > memory?The system has 2 dual core Opteron 2210''s in it. cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 65 model name : Dual-Core AMD Opteron(tm) Processor 2210 stepping : 2 cpu MHz : 1800.283 cache size : 1024 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy bogomips : 3604.24 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp tm stc> If you have a graphics card that eats a fair bit of memory (256 or > 512MB) on a processor that doesn''t support "memory hoisting"[1] (i.e.It has an onboard graphics card, but I doubt it is high MB, as it is a dedicated server. A PenguinComputing Altus 1600> remapping of "behind PCI memory" to a higher address), then I wouldn''t > be surprised if you don''t see more than 3.x GB - the reason being that > the rest of the memory is "behind" the PCI bus devices.> Also, if you have a newer processor you may need a new BIOS to support > this, and you''ll possibly also need to enable "memory hoisting" (or > memory hole or some such) in the BIOS setup.My BIOS has options for MemoryHole ReMapping this is "Enabled" MTRR Mapping this is "Discrete" IOMMU Mode set to "256 MB" CS Sparing Enable this is "Disabled" The thing is it boots and shows all 4GB with a vanilla debian etch bootup. It only has problems when booting in to Xen. -- John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: John Hannfield [mailto:hal9020@gmail.com] > Sent: 01 May 2007 11:53 > To: Petersson, Mats > Cc: xen-users@lists.xensource.com > Subject: Re: [Xen-users] X86_64 and 4GB RAM > > Hi Mats > > > What revision of processor do you have, and how large is > your graphics > > memory? > > The system has 2 dual core Opteron 2210''s in it.Ah, yes that is a Rev F, so it definitely allows memory hoisting.> cat /proc/cpuinfo > > processor : 0 > vendor_id : AuthenticAMD > cpu family : 15 > model : 65[snip]> > > > If you have a graphics card that eats a fair bit of memory (256 or > > 512MB) on a processor that doesn''t support "memory > hoisting"[1] (i.e. > > It has an onboard graphics card, but I doubt it is high MB, as it is a > dedicated server. A PenguinComputing Altus 1600Yeah, onboard graphics usuall use around 16-32MB, so not a whole lot.> > > remapping of "behind PCI memory" to a higher address), then > I wouldn''t > > be surprised if you don''t see more than 3.x GB - the reason > being that > > the rest of the memory is "behind" the PCI bus devices. > > > Also, if you have a newer processor you may need a new BIOS > to support > > this, and you''ll possibly also need to enable "memory hoisting" (or > > memory hole or some such) in the BIOS setup. > > My BIOS has options for > > MemoryHole ReMapping this is "Enabled"That''s the one we need...> MTRR Mapping this is "Discrete" > IOMMU Mode set to "256 MB" > CS Sparing Enable this is "Disabled" > > The thing is it boots and shows all 4GB with a vanilla debian > etch bootup. > It only has problems when booting in to Xen.Hmm. Something is certainly going wrong here... -- Mats> > -- > > John > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, May 01, 2007 at 11:18:49AM +0100, John Hannfield wrote: Hello,> Well I tried the grub 0.97 patches. I recompiled grub with both patches, > then replaced the etch grub withthe new .deb. Then re-installed grub > stages on the disk with > grub> root (hd0,0) > grub> setup (hd0)That patches only touches stage2 file, reinstalling whole grub (setup command) wasn''t necessary.> Now if I reboot the machine at at the live grub prompt, I do a displaymem, > it says: > > EISA Memory BIOS Interface is present > Address Map BIOS Interface is present > Lower memory: 631K, Upper memory (to first chipset hole): 3144640K > > There are no address ranges given. > So it looks like it only sees 3 GB.It''s exactly like (non-patched) grub 0.97 on my system. Please do one more try. Get stage2 file from http://kupson.net/grub/stage2 and put in /boot/grub/stage2. Then run displaymem command in grub.> If I run grub from the booted system, on the command line, it gives > the ranges, and also the debug info from the patched version of grub.That''s fake memory map, not the BIOS one. It''s not related to problem.> any ideas on where to go from here?You can try another multiboot compatible boot loader, grub2[1] maybe. Is there any newer BIOS version for your motherboard?> Is it normal for it to only show Upper memory (to first chipset hole): > 3144640K at the grub boot menu?Yes, that line is normal, but there should be memory region above address 4G (in displaymem), just like the one in plain linux kernel. Kupson [1]. I find it difficult to setup on my system. -- Great software without the knowledge to run it is pretty useless. (Linux Gazette #1) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Rafal, Mats I managed to get it sorted out. See the thread on the xen-devel list. Kier Fraser, one of the xensource people, was able to provide me with a patch which hard coded the correct BIOS memory map (from the vanilla linux install) in to Xen, so it doesn''t need to rely on the (incomplete) info from Grub. This is fine, as the memory configuration will not change. If I get time, I will troubleshoot this in grub on a similar machine, as it must be a grub stage2 problem with my BIOS. Thanks for your help, -- John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Hannfield wrote:> Kier Fraser, one of the xensource people, was able to provide me with a > patch > which hard coded the correct BIOS memory map (from the vanilla linux > install)Hi John, I''m having the same problem with the same machine, Penguin Altus 1600s with 8GB of memory. I couldn''t find the patches from Kier on Xen-devel, can you repost? thanks -joe _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users