thr3ads.net - Xen devel - [Xen-devel] Question about memory allocation on NUMA node. [Oct 2008]

If this information is useful, please help other people find it:
Share via:

Yuji Shimada

2008-Oct-17 10:28 UTC

[Xen-devel] Question about memory allocation on NUMA node.

I have a question about memory allocation on NUMA node for Xen
Hypervisor.

I think that the memory relating guest domain should be allocated from
the NUMA node on which the guest run.
Because the latency of the same NUMA node is better than that of a
different one.

According to this idea, most of the codes are good in xen-unstable.
But some memory relating guest domain are allocated from the NUMA node
on which CPU #0 run.

For example,
    - xen/arch/x86/domain.c
        setup_compat_l4(struct vcpu *v)
          struct page_info *pg = alloc_domheap_page(NULL, 0);

I think this memory should be allocated from the NUMA node on which the
guest run.

For example, 
    - xen/arch/x86/domain.c at
        setup_compat_l4(struct vcpu *v)
          struct page_info *pg = alloc_domheap_page(NULL,
                                 MEMF_node(domain_to_node(v->domain)));

As a result, machine performance becomes better.

What do you think about this idea? 
I''d like some comments.

If the developers agree with me, I would like to list them and submit
patch.

Thanks,

--
Yuji Shimada

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Andre Przywara

2008-Oct-17 12:10 UTC

head link

Re: [Xen-devel] Question about memory allocation on NUMA node.

Yuji,
> I have a question about memory allocation on NUMA node for Xen
> Hypervisor.
> 
> I think that the memory relating guest domain should be allocated from
> the NUMA node on which the guest run.
> Because the latency of the same NUMA node is better than that of a
> different one.
> 
> According to this idea, most of the codes are good in xen-unstable.
> But some memory relating guest domain are allocated from the NUMA node
> on which CPU #0 run.No, it will default to the node on which the domain runs, see
below.> 
> For example,
>     - xen/arch/x86/domain.c
>         setup_compat_l4(struct vcpu *v)
>           struct page_info *pg = alloc_domheap_page(NULL, 0);0 for memflags means default behaviour, if you look at the
implementation of alloc_domheap_pages
(http://lxr.xensource.com/lxr/source/xen/common/page_alloc.c?a=x86_64#L774)
a value of zero will evalutate to node=NUMA_NO_NODE, which in turn
will be replaced by ''node = domain_to_node(d);'', which is what
you
wanted to insert below. So I see no problem here.

Regards,
Andre.
> 
> I think this memory should be allocated from the NUMA node on which
> the guest run.
> 
> For example, 
>     - xen/arch/x86/domain.c at
>         setup_compat_l4(struct vcpu *v)
>           struct page_info *pg = alloc_domheap_page(NULL,
>                                  MEMF_node(domain_to_node(v->domain)));
> 
> As a result, machine performance becomes better.
> 
> What do you think about this idea? 
> I''d like some comments.
> 
> If the developers agree with me, I would like to list them and submit
> patch.
> --
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 277-84917
----to satisfy European Law for business letters:
AMD Saxony Limited Liability Company & Co. KG,
Wilschdorfer Landstr. 101, 01109 Dresden, Germany
Register Court Dresden: HRA 4896, General Partner authorized
to represent: AMD Saxony LLC (Wilmington, Delaware, US)
General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yuji Shimada

2008-Oct-20 07:42 UTC

head link

Re: [Xen-devel] Question about memory allocation on NUMA node.

Thanks for your reply.

On Fri, 17 Oct 2008 14:10:49 +0200
Andre Przywara <andre.przywara@amd.com> wrote:
> Yuji,
> 
> > I think that the memory relating guest domain should be allocated from
> > the NUMA node on which the guest run.
> > Because the latency of the same NUMA node is better than that of a
> > different one.
> > 
> > According to this idea, most of the codes are good in xen-unstable.
> > But some memory relating guest domain are allocated from the NUMA node
> > on which CPU #0 run.
> No, it will default to the node on which the domain runs, see below.
> > 
> > For example,
> >     - xen/arch/x86/domain.c
> >         setup_compat_l4(struct vcpu *v)
> >           struct page_info *pg = alloc_domheap_page(NULL, 0);
> 0 for memflags means default behaviour, if you look at the
> implementation of alloc_domheap_pages
> (http://lxr.xensource.com/lxr/source/xen/common/page_alloc.c?a=x86_64#L774)
> a value of zero will evalutate to node=NUMA_NO_NODE, which in turn
> will be replaced by ''node = domain_to_node(d);'', which is
what you
> wanted to insert below. So I see no problem here.
If first domain parameter is ''NULL'' and second memflag
parameter is
''0'' in alloc_domheap_page, the ''node'' value
is NOT replaced by
''domain_to_node(d)''.
So the ''node'' value is ''NUMA_NO_NODE''.

Please see below.

xen/common/page_alloc.c
774 struct page_info *alloc_domheap_pages(
775     struct domain *d, unsigned int order, unsigned int memflags)
776 {
777     struct page_info *pg = NULL;
778     unsigned int bits = memflags >> _MEMF_bits, zone_hi = NR_ZONES -
1;
779     unsigned int node = (uint8_t)((memflags >> _MEMF_node) - 1);
780 
781     ASSERT(!in_irq());
782 
783     if ( (node == NUMA_NO_NODE) && (d != NULL) )
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
784         node = domain_to_node(d);
…
810 }


So it is better to use NUMA node number of memflag parameter to allocate
memory from guest''s NUMA node.

For example,
    ''*pg = alloc_domheap_page(NULL, 0);''
    is better to be replaced by the following examples.

    ''*pg = alloc_domheap_page(NULL,
MEMF_node(domain_to_node(d)));''

What do you think?

Thanks,

--
Yuji Shimada
> 
> Regards,
> Andre.
> 
> > 
> > I think this memory should be allocated from the NUMA node on which
> > the guest run.
> > 
> > For example, 
> >     - xen/arch/x86/domain.c at
> >         setup_compat_l4(struct vcpu *v)
> >           struct page_info *pg = alloc_domheap_page(NULL,
> >                                 
MEMF_node(domain_to_node(v->domain)));
> > 
> > As a result, machine performance becomes better.
> > 
> > What do you think about this idea? 
> > I''d like some comments.
> > 
> > If the developers agree with me, I would like to list them and submit
> > patch.
> > 
> --
> Andre Przywara
> AMD-Operating System Research Center (OSRC), Dresden, Germany
> Tel: +49 351 277-84917
> ----to satisfy European Law for business letters:
> AMD Saxony Limited Liability Company & Co. KG,
> Wilschdorfer Landstr. 101, 01109 Dresden, Germany
> Register Court Dresden: HRA 4896, General Partner authorized
> to represent: AMD Saxony LLC (Wilmington, Delaware, US)
> General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Andre Przywara

2008-Oct-29 09:59 UTC

head link

Re: [Xen-devel] [PATCH] Question about memory allocation on NUMA node.

Yuji,

sorry for the late reply.>>> I think that the memory relating guest domain should be allocated
>>> from the NUMA node on which the guest run.
>>> Because the latency of the same NUMA node is better than that of a
>>> different one.
>>>
>>> According to this idea, most of the codes are good in xen-unstable.
>>> But some memory relating guest domain are allocated from the NUMA
>>> node on which CPU #0 run.
>> No, it will default to the node on which the domain runs, see below.
>>> For example,
>>>     - xen/arch/x86/domain.c
>>>         setup_compat_l4(struct vcpu *v)
>>>           struct page_info *pg = alloc_domheap_page(NULL, 0);
>> 0 for memflags means default behaviour, if you look at the
>> implementation of alloc_domheap_pages
>>
(http://lxr.xensource.com/lxr/source/xen/common/page_alloc.c?a=x86_64#L774)
>> a value of zero will evalutate to node=NUMA_NO_NODE, which in turn
>> will be replaced by ''node = domain_to_node(d);'',
which is what you
>> wanted to insert below. So I see no problem here.
> 
> If first domain parameter is ''NULL'' and second memflag
parameter is
> ''0'' in alloc_domheap_page, the ''node''
value is NOT replaced by
> ''domain_to_node(d)''.
> So the ''node'' value is ''NUMA_NO_NODE''.
Oh, you are right, I missed that.> 
> Please see below.
> 
> xen/common/page_alloc.c
> 774 struct page_info *alloc_domheap_pages(
> 775     struct domain *d, unsigned int order, unsigned int memflags)
> 776 {
> 777     struct page_info *pg = NULL;
> 778     unsigned int bits = memflags >> _MEMF_bits, zone_hi >
NR_ZONES - 1; 779     unsigned int node = (uint8_t)((memflags >>
> _MEMF_node) - 1); 780 
> 781     ASSERT(!in_irq());
> 782 
> 783     if ( (node == NUMA_NO_NODE) && (d != NULL) )
>         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 784         node = domain_to_node(d);
> …
> 810 }
> 
> 
> So it is better to use NUMA node number of memflag parameter to
> allocate memory from guest''s NUMA node.
> 
> For example,
>     ''*pg = alloc_domheap_page(NULL, 0);''
>     is better to be replaced by the following examples.
> 
>     ''*pg = alloc_domheap_page(NULL,
MEMF_node(domain_to_node(d)));''
> 
> What do you think?I would revert to giving a valid domain pointer as the first parameter
instead of duplicating the code in alloc_domheap_pages().
Patch attached.

Signed-off-by: Andre Przywara <andre.przywara@amd.com>

Regards,
Andre.
> 
> Thanks,
> 
> --
> Yuji Shimada
> 
>> Regards,
>> Andre.
>>
>>> I think this memory should be allocated from the NUMA node on which
>>> the guest run.
>>>
>>> For example, 
>>>     - xen/arch/x86/domain.c at
>>>         setup_compat_l4(struct vcpu *v)
>>>           struct page_info *pg = alloc_domheap_page(NULL,
>>>                                 
MEMF_node(domain_to_node(v->domain)));
>>>
>>> As a result, machine performance becomes better.
>>>
>>> What do you think about this idea? 
>>> I''d like some comments.
>>>
>>> If the developers agree with me, I would like to list them and
>>> submit patch.
>>>
>> --
>> Andre Przywara
>> AMD-Operating System Research Center (OSRC), Dresden, Germany
>> Tel: +49 351 277-84917
>> ----to satisfy European Law for business letters:
>> AMD Saxony Limited Liability Company & Co. KG,
>> Wilschdorfer Landstr. 101, 01109 Dresden, Germany
>> Register Court Dresden: HRA 4896, General Partner authorized
>> to represent: AMD Saxony LLC (Wilmington, Delaware, US)
>> General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy
> 
--
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 277-84917
----to satisfy European Law for business letters:
AMD Saxony Limited Liability Company & Co. KG,
Wilschdorfer Landstr. 101, 01109 Dresden, Germany
Register Court Dresden: HRA 4896, General Partner authorized
to represent: AMD Saxony LLC (Wilmington, Delaware, US)
General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2008-Oct-29 10:17 UTC

head link

Re: [Xen-devel] [PATCH] Question about memory allocation on NUMA node.

On 29/10/08 09:59, "Andre Przywara" <andre.przywara@amd.com>
wrote:
>> What do you think?
> I would revert to giving a valid domain pointer as the first parameter
> instead of duplicating the code in alloc_domheap_pages().
> Patch attached.
This is not correct, as the page must not be mappable and modifiable by the
guest. Yuji''s suggestion of specifying MEMF_node() is actually correct,
so
that''s what I''ll check in.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Oct 2008 - Question about memory allocation on NUMA node.

[Xen-devel] Question about memory allocation on NUMA node.

Re: [Xen-devel] Question about memory allocation on NUMA node.

Re: [Xen-devel] Question about memory allocation on NUMA node.

Re: [Xen-devel] [PATCH] Question about memory allocation on NUMA node.

Re: [Xen-devel] [PATCH] Question about memory allocation on NUMA node.