Xen users - Jun 2007 - amd64 system only seeing 3.5gb instead of 4gb of memory

I''ve just had a server upgraded to 4gb of memory... BIOS reports 4gb of
memory but xen only sees 3.5gb (3583mb)

I was under the impression that an amd64 server should be able to see
well over 4gb of memory. I believe that PCI space and other stuff gets
mapped to the very top of memory... is there some setting somewhere that
I might need to set to get that .5gb mapped to somewhere visible?

Thanks

James

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Jun 15, 2007 at 03:38:24PM +1000, James Harper wrote:
Hello,
> I''ve just had a server upgraded to 4gb of memory... BIOS reports
4gb of
> memory but xen only sees 3.5gb (3583mb)
What about plain linux kernel? How much it see?
> I was under the impression that an amd64 server should be able to see
> well over 4gb of memory. I believe that PCI space and other stuff gets
> mapped to the very top of memory... is there some setting somewhere that
> I might need to set to get that .5gb mapped to somewhere visible?
Only BIOS can do that. Search for "PCI memory hole" or some memory
remapping option (maybe with values: continuous/discontig).

By "amd64" you mean AMD processor[1]? Some older Inter chipsets cannot
remap RAM above 4G.

If problem persists please post e820 memory map from xm dmesg (that
lines after "(XEN) Physical RAM map:") and from plain linux kernel.

Kupson
[1] Debian uses amd64 as arch name, instead of x86-64.
-- 
Great software without the knowledge to run it is pretty useless.
(Linux Gazette #1)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Jun 15, 2007 at 03:38:24PM +1000, James Harper
wrote:
> I''ve just had a server upgraded to 4gb of memory... BIOS reports
4gb of
> memory but xen only sees 3.5gb (3583mb)
I had a similar problem, that was resolved with a BIOS upgrade.

-- 
lfr
0/0


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Have you tried boot without XEN  ? Can you see 4 GB ?.



> On Fri, Jun 15, 2007 at 03:38:24PM +1000, James Harper wrote:
>> I''ve just had a server upgraded to 4gb of memory... BIOS
reports 4gb of
>> memory but xen only sees 3.5gb (3583mb)
>
> I had a similar problem, that was resolved with a BIOS upgrade.
>
> --
> lfr
> 0/0
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

-- 
Juan Antonio Vera
Responsable Departament Informàtica
Ajuntament Sant Adrià de Besòs


Telf: 93 381 20 04 (ext 367)
Fax:  93 381 70 56
Email: javera@sant-adria.net
http://www.sant-adria.net


-- 
Aquest missatge ha estat verificat amb antivirus i
antispam. Es considera que esta net.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> James Harper
> Sent: 15 June 2007 06:38
> To: xen-users@lists.xensource.com
> Subject: [Xen-users] amd64 system only seeing 3.5gb instead 
> of 4gb of memory
> 
> I''ve just had a server upgraded to 4gb of memory... BIOS 
> reports 4gb of
> memory but xen only sees 3.5gb (3583mb)
> 
> I was under the impression that an amd64 server should be able to see
> well over 4gb of memory. I believe that PCI space and other stuff gets
> mapped to the very top of memory... is there some setting 
> somewhere that
> I might need to set to get that .5gb mapped to somewhere visible?
Further to what has already been said (all of which is sensible and good
suggestions): 
1. Assuming you have an AMD processor, you need a procesor that is Rev E
or later. 
2. You need to have PAE or 64-bit mode to see any memory ABOVE 4GB (so
even if it''s remapped, non-PAE will not be able to "get
there" because
it''s strictly limited to 32-bit memory addresses, which is 4GB in total
- this is why the PCI memory is located around the end of 4GB). 
3. Your BIOS must enable "memory hoisting", which is the techie name
for
"move memory from under PCI space to someplace else".

--
Mats
> 
> Thanks
> 
> James
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 
> 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> Further to what has already been said (all of which is sensible and
good
> suggestions):
> 1. Assuming you have an AMD processor, you need a procesor that is Rev
E
> or later.
First half of cat /proc/cpuinfo:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 67
model name      : Dual-Core AMD Opteron(tm) Processor 1210
stepping        : 3
cpu MHz         : 1800.018
cache size      : 1024 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext
3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy
bogomips        : 4501.72
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

40 bits of physical addressing should be enough shouldn''t it? Or are
you
talking about revision E being required for the remapping?
> 2. You need to have PAE or 64-bit mode to see any memory ABOVE 4GB (so
> even if it''s remapped, non-PAE will not be able to "get
there" because
> it''s strictly limited to 32-bit memory addresses, which is 4GB in
total
> - this is why the PCI memory is located around the end of 4GB).
xm info:

host                   : dev
release                : 2.6.18-4-xen-amd64
version                : #1 SMP Fri May 4 02:40:51 UTC 2007
machine                : x86_64
nr_cpus                : 2

definitely running in 64 bit mode yes?
> 3. Your BIOS must enable "memory hoisting", which is the techie
name
for
> "move memory from under PCI space to someplace else".
Now this I haven''t checked... I''ll have a look this weekend.

Subsequent reading showed at least one other person had such a problem,
and it was probably related to grub passing a ''basic'' memory
map instead
of the modern type (I even followed up to that post originally... what a
short memory I have these days!!!). I can probably confirm this from xm
dmesg:

(XEN) Command line: /xen-3.1.0-1-amd64.gz
(XEN)  0000000000000000 - 000000000009f000 (usable)
(XEN)  0000000000100000 - 00000000dfff0000 (usable)
(XEN) System RAM: 3583MB (3669564kB)
(XEN) Xen heap: 13MB (14176kB)
(XEN) Domain heap initialised: DMA width 32 bits

D''oh!

I''ll try and get into the office this weekend and have a better look.
There are no BIOS upgrades at this point.

Thanks

James

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: James Harper [mailto:james.harper@bendigoit.com.au] 
> Sent: 15 June 2007 11:16
> To: Petersson, Mats; xen-users@lists.xensource.com
> Subject: RE: [Xen-users] amd64 system only seeing 3.5gb 
> instead of 4gb of memory
> 
> > 
> > Further to what has already been said (all of which is sensible and
> good
> > suggestions):
> > 1. Assuming you have an AMD processor, you need a procesor 
> that is Rev
> E
> > or later.
> 
> First half of cat /proc/cpuinfo:
> 
> processor       : 0
> vendor_id       : AuthenticAMD
> cpu family      : 15
> model           : 67
> model name      : Dual-Core AMD Opteron(tm) Processor 1210
> stepping        : 3
> cpu MHz         : 1800.018
> cache size      : 1024 KB
> physical id     : 0
> siblings        : 1
> core id         : 0
> cpu cores       : 1
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 1
> wp              : yes
> flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36
> clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext
> 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy
> bogomips        : 4501.72
> TLB size        : 1024 4K pages
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 40 bits physical, 48 bits virtual
> power management: ts fid vid ttp tm stc
> 
> 40 bits of physical addressing should be enough shouldn''t it? 
> Or are you
> talking about revision E being required for the remapping?
Yes, (flexible) memory hoisting is only available on Rev E or later. Not
an issue as your machine is a Rev F processor (Opteron 1xxx, 2xxx and
8xxx are Rev F). 
> 
> > 2. You need to have PAE or 64-bit mode to see any memory 
> ABOVE 4GB (so
> > even if it''s remapped, non-PAE will not be able to "get 
> there" because
> > it''s strictly limited to 32-bit memory addresses, which is
4GB in
> total
> > - this is why the PCI memory is located around the end of 4GB).
> 
> xm info:
> 
> host                   : dev
> release                : 2.6.18-4-xen-amd64
> version                : #1 SMP Fri May 4 02:40:51 UTC 2007
> machine                : x86_64
> nr_cpus                : 2
> 
> definitely running in 64 bit mode yes?
Looks so. 
> 
> > 3. Your BIOS must enable "memory hoisting", which is the
techie name
> for
> > "move memory from under PCI space to someplace else".
> 
> Now this I haven''t checked... I''ll have a look this
weekend.
> 
> Subsequent reading showed at least one other person had such 
> a problem,
> and it was probably related to grub passing a ''basic''
memory
> map instead
> of the modern type (I even followed up to that post 
> originally... what a
> short memory I have these days!!!). I can probably confirm 
> this from xm
> dmesg:
> 
> (XEN) Command line: /xen-3.1.0-1-amd64.gz
> (XEN)  0000000000000000 - 000000000009f000 (usable)
> (XEN)  0000000000100000 - 00000000dfff0000 (usable)
> (XEN) System RAM: 3583MB (3669564kB)
> (XEN) Xen heap: 13MB (14176kB)
> (XEN) Domain heap initialised: DMA width 32 bits
So that tells us that Xen is doing exactly what the BIOS/Grub says - and
of course, GRUB does things differently for Xen and a normal Linux
distro, simply because it''s using a different model of booting. 

--
Mats
> 
> D''oh!
> 
> I''ll try and get into the office this weekend and have a better
look.
> There are no BIOS upgrades at this point.
> 
> Thanks
> 
> James
> 
> 
> 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> >
> > (XEN) Command line: /xen-3.1.0-1-amd64.gz
> > (XEN)  0000000000000000 - 000000000009f000 (usable)
> > (XEN)  0000000000100000 - 00000000dfff0000 (usable)
> > (XEN) System RAM: 3583MB (3669564kB)
> > (XEN) Xen heap: 13MB (14176kB)
> > (XEN) Domain heap initialised: DMA width 32 bits
> 
> So that tells us that Xen is doing exactly what the BIOS/Grub says -
and
> of course, GRUB does things differently for Xen and a normal Linux
> distro, simply because it''s using a different model of booting.
I applied the patch mentioned here
"http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419994" and
rebooted,
and now xm dmesg says:

"
(XEN) Command line: /xen-3.1.0-1-amd64.gz
(XEN)  0000000000000000 - 000000000009f000 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000dfff0000 (usable)
(XEN)  00000000dfff0000 - 00000000dfffe000 (ACPI data)
(XEN)  00000000dfffe000 - 00000000e0000000 (ACPI NVS)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fee00000 - 00000000fef00000 (reserved)
(XEN)  00000000ff780000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000120000000 (usable)
(XEN) System RAM: 4095MB (4193852kB)
"

Which is a much healthier looking memory map, and I''m now seeing the
right amount of memory!

The fix makes me think that the bug is a bit heisenbug-ish... but it
works for me.

James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: James Harper [mailto:james.harper@bendigoit.com.au] 
> Sent: 17 June 2007 13:01
> To: Petersson, Mats; xen-users@lists.xensource.com
> Subject: RE: [Xen-users] FIXED!!! amd64 system only seeing 
> 3.5gb instead of 4gb of memory
> 
> > >
> > > (XEN) Command line: /xen-3.1.0-1-amd64.gz
> > > (XEN)  0000000000000000 - 000000000009f000 (usable)
> > > (XEN)  0000000000100000 - 00000000dfff0000 (usable)
> > > (XEN) System RAM: 3583MB (3669564kB)
> > > (XEN) Xen heap: 13MB (14176kB)
> > > (XEN) Domain heap initialised: DMA width 32 bits
> > 
> > So that tells us that Xen is doing exactly what the BIOS/Grub says -
> and
> > of course, GRUB does things differently for Xen and a normal Linux
> > distro, simply because it''s using a different model of
booting.
> 
> I applied the patch mentioned here
> "http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419994" and 
> rebooted,
> and now xm dmesg says:
> 
> "
> (XEN) Command line: /xen-3.1.0-1-amd64.gz
> (XEN)  0000000000000000 - 000000000009f000 (usable)
> (XEN)  000000000009fc00 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 00000000dfff0000 (usable)
> (XEN)  00000000dfff0000 - 00000000dfffe000 (ACPI data)
> (XEN)  00000000dfffe000 - 00000000e0000000 (ACPI NVS)
> (XEN)  00000000fec00000 - 00000000fec01000 (reserved)
> (XEN)  00000000fee00000 - 00000000fef00000 (reserved)
> (XEN)  00000000ff780000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 0000000120000000 (usable)
> (XEN) System RAM: 4095MB (4193852kB)
> "
> 
> Which is a much healthier looking memory map, and I''m now seeing
the
> right amount of memory!
> 
> The fix makes me think that the bug is a bit heisenbug-ish... but it
> works for me.
Yes, it looks like a "compiler issue" that is fixed by forcing the
compiler to not optimize the variable by using volatile - not the ideal
solution, but I guess fixing the compiler may be even more difficult.
[Not knowing what the code looks like - but I suppose the "cont"
variable isn''t updated by an interrupt or external hardware!]

--
Mats
> 
> James
> 
> 
> 
> 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Jun 18, 2007 at 11:35:56AM +0200, Petersson, Mats wrote:
Hello,
> > The fix makes me think that the bug is a bit heisenbug-ish... but it
> > works for me.
> 
> Yes, it looks like a "compiler issue" that is fixed by forcing
the
> compiler to not optimize the variable by using volatile - not the ideal
> solution, but I guess fixing the compiler may be even more difficult.
> [Not knowing what the code looks like - but I suppose the "cont"
> variable isn''t updated by an interrupt or external hardware!]
I thought the same, but one grub stage2 file behaves differently on
two motherboards. It fails to detect[1] e820 memory map on my computer,
but copied to friend''s one works perfectly. I think if it''s
compiler bug[2] that copy shouldn''t work on both computers.

Regerds,
Kupson
[1] displaymem command doesn''t work
[2] optimize loop or function call to nothing or something similar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> Rafal Kupka
> Sent: 18 June 2007 12:34
> To: xen-users@lists.xensource.com
> Subject: Re: [Xen-users] FIXED!!! amd64 system only seeing 
> 3.5gb instead of 4gb of memory
> 
> On Mon, Jun 18, 2007 at 11:35:56AM +0200, Petersson, Mats wrote:
> Hello,
> 
> > > The fix makes me think that the bug is a bit 
> heisenbug-ish... but it
> > > works for me.
> > 
> > Yes, it looks like a "compiler issue" that is fixed by
forcing the
> > compiler to not optimize the variable by using volatile - 
> not the ideal
> > solution, but I guess fixing the compiler may be even more 
> difficult.
> > [Not knowing what the code looks like - but I suppose the
"cont"
> > variable isn''t updated by an interrupt or external hardware!]
> 
> I thought the same, but one grub stage2 file behaves differently on
> two motherboards. It fails to detect[1] e820 memory map on my 
> computer,
> but copied to friend''s one works perfectly. I think if
it''s
> compiler bug[2] that copy shouldn''t work on both computers.
Well, the link in the original post says that the "while-loop is completely
removed", which I took to mean that the author of the bugfix had actually
examined the code generated by the compiler.

Obviously, copying files from one machine to another shouldn''t change
the behaviour of the code if "copy" works right (if in doubt, do
"md5sum some-file" to check that the content of the file
hasn''t changed).

Unfortunately, just because one bug is fixed, doesn''t mean that others
aren''t still in there.

--
Mats
> 
> Regerds,
> Kupson
> [1] displaymem command doesn''t work
> [2] optimize loop or function call to nothing or something similar
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 
> 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> On Mon, Jun 18, 2007 at 11:35:56AM +0200, Petersson, Mats wrote:
> Hello,
> 
> > > The fix makes me think that the bug is a bit heisenbug-ish... but
it
> > > works for me.
> >
> > Yes, it looks like a "compiler issue" that is fixed by
forcing the
> > compiler to not optimize the variable by using volatile - not the
ideal
> > solution, but I guess fixing the compiler may be even more difficult.
> > [Not knowing what the code looks like - but I suppose the
"cont"
> > variable isn''t updated by an interrupt or external hardware!]
> 
> I thought the same, but one grub stage2 file behaves differently on
> two motherboards. It fails to detect[1] e820 memory map on my computer,
> but copied to friend''s one works perfectly. I think if
it''s
> compiler bug[2] that copy shouldn''t work on both computers.
What are the differences between the two motherboards?

How positive are you that it was actually installed correctly on your computer?
Is the updated stage2 file definitely in /boot/grub?

Does Linux itself definitely detect the e820 memory map correctly in both
computers? Are you able to post the dmesg memory map under plain linux to the
list?

James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Jun 19, 2007 at 09:14:21AM +1000, James Harper wrote:
Hello,
> > I thought the same, but one grub stage2 file behaves differently on
> > two motherboards. It fails to detect[1] e820 memory map on my
> > computer, but copied to friend''s one works perfectly. I think
if
> > it''s compiler bug[2] that copy shouldn''t work on
both computers.
> 
> What are the differences between the two motherboards?
affected: MSI K9N SLI Platinum, MSI K9N SLI-2F[1].
(nvidia MCP55 chipset, socket AM2)

not affected: Asus P5AD2E (chipset Intel 925XE),
old intel 440BX based Abit (don''t know exact model)

So, they differ completely.
> How positive are you that it was actually installed correctly on your
> computer? Is the updated stage2 file definitely in /boot/grub?
I''m 100% sure.
> Does Linux itself definitely detect the e820 memory map correctly in
> both computers? Are you able to post the dmesg memory map under plain
> linux to the list?
Linux always detects memory correctly, only Grub/Xen has problems.

BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000bffd0000 (usable)
 BIOS-e820: 00000000bffd0000 - 00000000bffde000 (ACPI data)
 BIOS-e820: 00000000bffde000 - 00000000c0000000 (ACPI NVS)
 BIOS-e820: 00000000d0000000 - 00000000e0000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved)
 BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000140000000 (usable)

Kupson
[1] just model with less peripherals onboard
-- 
Great software without the knowledge to run it is pretty useless.
(Linux Gazette #1)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users