Xen devel - Jul 2013 - Re: [libvirt] [PATCH 1/4] libxl: implement NUMA capabilities reporting

[Moving the conversation on @xen-devel and adding Jan, as that seems
more appropriate]

[Jan, this came up as I''m implementing some NUMA bits in libvirt but,
as
you see, the core of Jim''s question is purely about Xen]

On lun, 2013-07-01 at 16:47 -0600, Jim Fehlig wrote:
> On my non-NUMA test machine I have the cell memory reported as
> 
>            <memory
unit=''KiB''>9175040</memory>
> 
Which is 8960, if divided by 1024, so at least it''s consistent.

However...
> The machine has 8G of memory, running xen 4.3 rc6, with dom0_mem=1024M. 
''xl
> info --numa'' says
> 
> total_memory           : 8190
> ...
> numa_info              :
> node:    memsize    memfree    distances
>     0:      8960       7116      10
> 
> Why is the node memsize > total_memory?
Mmm... Interesting question. I really never paid attention to this...
Jan (or anyone else), is that something known and/or expected?

I went checking this down in Xen, and here''s what I found.

total_memory is: info.total_pages/((1 << 20) / vinfo->pagesize)

where ''info'' is what libxl_get_physinfo() provides. On its
turn,
libxl_get_physinfo() is xc_physinfo(), which is XEN_SYSCTL_physinfo,
which uses total_pages, which is assigned the number of pages, down in
__start_xen(), as it results from parsing the E820 map (looking for RAM
blocks).

OTOH, memsize comes from libxl_get_numainfo(), which is xc_numainfo(),
which is XEN_SYSCTL_numainfo, which puts in memsize what
node_spanned_pages(<node_id>) says.

That seems to come, on a NUMA box, from the parsing of SRAT, and on a
non-NUMA box, from just (start_pfn-end_pfn) (in pages, of course).

Anyway, on my NUMA box, I see something similar to what Jim sees on a
non-NUMA one:

# xl info -n
...
total_memory           : 12285
...
numa_info              :
node:    memsize    memfree    distances
   0:      6144         23      10,20
   1:      6720        104      20,10

Where 6144+6720=12864 > 12285

Looking at what Xen says during boot, I see this (the [*], [+], [=] and
[|] are mine):

(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 0000000000096000 (usable)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)   [*]
(XEN)  0000000000100000 - 00000000dbdf9c00 (usable)
(XEN)  00000000dbdf9c00 - 00000000dbe4bc00 (ACPI NVS)   [+]
(XEN)  00000000dbe4bc00 - 00000000dbe4dc00 (ACPI data)  [=]
(XEN)  00000000dbe4dc00 - 00000000dc000000 (reserved)   [|]
(XEN)  00000000f8000000 - 00000000fd000000 (reserved)
(XEN)  00000000fe000000 - 00000000fed00400 (reserved)
(XEN)  00000000fee00000 - 00000000fef00000 (reserved)
(XEN)  00000000ffb00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000324000000 (usable)
...
(XEN) System RAM: 12285MB (12580412kB)

And my math says that 12285MB is the sum of the areas marked as
(usable), i.e., I guess, what during parsing is recognised as
E820_RAM... which makes total sense.

A bit below that I have this:

(XEN) SRAT: Node 1 PXM 1 0-dc000000
(XEN) SRAT: Node 1 PXM 1 100000000-1a4000000
(XEN) SRAT: Node 0 PXM 0 1a4000000-324000000

Which, after the needed calculations, gives exactly the same results
than memsize-s in `xl info -n''.

Now, if I add up the [*], [+], [=] and [|] regions above, and then
subtract that from node 1''s PXMs, I see that node 1 has only ~6141MB of
usable RAM, instead of 6720MB.

And in fact, 6720-6141=579, just as much as 12864-12285=579.

So, if I haven''t messed up with the calculations, it looks like that
Xen, when reporting to the upper layers the amount of memory it has
available, does filter out the non-RAM regions, if this happens via
XEN_SYSCTL_physinfo (i.e., by parsing E820), while it does not do that,
if this happens via XEN_SYSCTL_numainfo (i.e., by parsing ACPI SRAT).

What I''m not sure about is whether or not that was something
known/intended and whether or not it is something we should be concerned
about.

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Dario Faggioli <dario.faggioli@citrix.com> wrote:
>[Moving the conversation on @xen-devel and adding Jan, as that seems
>more appropriate]
>
>[Jan, this came up as I''m implementing some NUMA bits in libvirt
but,
>as
>you see, the core of Jim''s question is purely about Xen]
>
>On lun, 2013-07-01 at 16:47 -0600, Jim Fehlig wrote:
>> On my non-NUMA test machine I have the cell memory reported as
>> 
>>            <memory
unit=''KiB''>9175040</memory>
>> 
>Which is 8960, if divided by 1024, so at least it''s consistent.
>
>However...
>
>> The machine has 8G of memory, running xen 4.3 rc6, with
>dom0_mem=1024M.  ''xl 
>> info --numa'' says
>> 
>> total_memory           : 8190
>> ...
>> numa_info              :
>> node:    memsize    memfree    distances
>>     0:      8960       7116      10
>> 
>> Why is the node memsize > total_memory?
>
>Mmm... Interesting question. I really never paid attention to this...
>Jan (or anyone else), is that something known and/or expected?
>
Is the difference not the amount that Xen hypervisor consumes?  It just has not
been subtracted from the NUMA values.
-- 
Sent from my Android phone. Please excuse my brevity.

On gio, 2013-07-04 at 13:21 -0400, Konrad Rzeszutek Wilk
wrote:
> Dario Faggioli <dario.faggioli@citrix.com> wrote:
>
> >On lun, 2013-07-01 at 16:47 -0600, Jim Fehlig wrote:
> >> info --numa'' says
> >> 
> >> total_memory           : 8190
> >> ...
> >> numa_info              :
> >> node:    memsize    memfree    distances
> >>     0:      8960       7116      10
> >> 
> >> Why is the node memsize > total_memory?
> >
> >Mmm... Interesting question. I really never paid attention to this...
> >Jan (or anyone else), is that something known and/or expected?
> >
> Is the difference not the amount that Xen hypervisor consumes?  It just has
not been subtracted from the NUMA values.
>
I thought about that too, and I''m not that has nothing to do with it.

However, in the rest of my message, I reported the results of some
investigation I did, and the relevant (and hopefully correct enough)
calculations... Have you seen that? What do you think about it?

Moreover, wherever the difference is coming from, what I''m asking is,
should we stick with this situation or take some kind of action?

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel