Xen devel - Feb 2012 - Re: NUMA-aware VM placement in Xen

On 24/02/12 10:12, Dario Faggioli wrote:
> Hi guys,
>
> As some of you know I''m working on putting some kind of NUMA-aware
> placement of the various VMs within Xen. This means I''m
investigating
> deeply how memory allocation works, which isn''t an easy task for
me (I
> started completely from scratch), so forgive me if I say something
> wrong! :-P
>
> The status is I''m dealing for a while with a "design
issue" I''d be very
> glad to discuss with someone, as I''m not sure which path to go
for...
>
> To keep it short, what I need is a place --ideally during VM creation--
> where I can check how much memory a VM wants against how much memory is
> available in the various NUMA-nodes, and use this as the basis of my
> decision. The question is, where is this place?
>
> I traced memory related calls (e.g., for HVMs) from libxl__build_hvm to
> xc_hvm_build_target_mem to setup_guest to xc_domain_populate_physmap and
> alloc_domheap_pages. The last twos have been my target for a while, but
> I''m not so sure they would be the right choice, mainly because
both of
> them are called for allocating _only_part_ of the VM''s memory,
i.e.,
> some extents of it at each call (am I right?).
>
> Basically, given alloc_domheap_pages uses d->node_affinity for deciding
> from which node(s) to actually take memory from, I was planning to
> either use the same mask or build a new one with similar purposes, the
> problem being _where_ to populate it with the proper nodes.
> I''m now looking at
xc_domain_setmaxmem-->do_domctl(XEN_DOMCTL_max_mem),
> although I think it''s too early, and I''d end up guessing
wrt a lot of
> aspects... But considering xm/xend was doing the same even earlier (at
> least I think)...
So the first question is, where should the decision about NUMA placement 
be made, and the second is how that level should implement it.

Doing it at the libxc level I think is not right.  It seems to me we 
have two options:
* Have libxl do the NUMA placement on behalf of the toolstack.  In that 
case, the libxl_domain_create_new function should look at the available 
memory, the NUMA layout, &c, and then set d->node_affinity before 
calling xc_hvm_build.
* Have the toolstack do it.  In this case, you''d be modifying xl to set
d->node_affinity before calling libxl''s domain creation function.

Do those options work?  Let me know if I''ve misunderstood anything.

Any thoughts one way or the other from anyone?

I''d be tempted to have it be optional -- you can set
"numa=auto" and the
domain creation function will do the simple thing; or you can set 
"numa=manual" and have the toolstack / config file set the nodes 
manually.  That would translate pretty well to config files as well -- 
more "set the knobs" administrators could set the numa layout in the 
config file manually if they wanted.

Thoughts?

  -George
>
> Sorry fro writing so much... Any help/ideas you feel comfortable with
> sharing would be very appreciated! :-)
>
> Thanks a lot and regards,
> Dario
>

On Fri, Feb 24, 2012 at 10:50 AM, Dario Faggioli <raistlin@linux.it>
wrote:
>> It seems to me we
>> have two options:
>> * Have libxl do the NUMA placement on behalf of the toolstack.  In that
>> case, the libxl_domain_create_new function should look at the available
>> memory, the NUMA layout, &c, and then set d->node_affinity
before
>> calling xc_hvm_build.
>>
> This can be done. If I got it correctly it is more or less what xm/xend
> already does.
>
>> * Have the toolstack do it.  In this case, you''d be modifying
xl to set
>> d->node_affinity before calling libxl''s domain creation
function.
>>
> I''m not sure I''m getting this right... It seems very
similar to the one
> above.
From Xen''s perspective, yes.  But from the libxl perspective, no.
libxl is meant to be the interface we give to other external
toolstacks, so the interface there is important.
>> Do those options work?  Let me know if I''ve misunderstood
anything.
>>
> I think they can be implemented. "work", it depends on how we
define
> "work". :-D
>
> That''s why I was struggling for putting this in the hypervisor and
not
> in the toolstack because I really think it should live there if
> possible. For example it would be nice for the decision to be protected
> by the proper locking. I mean, what''s the point in checking the
amount
> of free memory in a node somewhere in (lib)xl, if when the actual
> allocation will happen (in Xen) that might be a completely different
> value (due to concurrent domain creation, destruction, etc.)?
At the moment, pages for a VM are not allocated in one big chunk
anyway -- xc_hvm_build.c:setup_guest() calls
xc_domain_populate_physmap() in a loop.  So Xen is in less of a
position to avoid the TOCTTOU race than the toolstack is.  The
toolstack in theory, at least, can refrain from starting a second VM
until the first is completely allocated.

I don''t think there''s any reason to do it in Xen --
it''s not
time-critical, it doesn''t require any information that the toolstack
and/or domain builder wouldn''t have available to it.
> I think the config file, supporting cpupools and vcpu-pinning, already
> offer almost all the facilities for manually deploying a VM reflecting a
> specific NUMA-layout. What I was thinking adding was the
"numa=auto" or
> whatever switch, so that if one does not (want to) specify cpupools or
> pinning, VM still gets NUMA-sensible placement.
Do we have a way to specify the NUMA layout?  That should be a
separate config option than vcpu pinning.  But yes, I think adding
"numa=auto" (on by default) is the big feature we need; and I think
that''s probably best implemented in either the toolstack or libxl.

 -George