Xen devel - Nov 2009 - [PATCH]pcpu tuples [was Re: Xen 3.4.1 NUMA support]

Attached is a patch to construct pcpu tuples of the form
(node.socket.core.thread), and (currently)used by xm vcpu-pin utility.

-dulloor

On Fri, Nov 13, 2009 at 11:02 AM, Keir Fraser <keir.fraser@eu.citrix.com>
wrote:
> On 13/11/2009 15:40, "Ian Pratt" <Ian.Pratt@eu.citrix.com>
wrote:
>
>>> Even better would be to have pCPUs addressable and listable
explicitly as
>>> dotted tuples. That can be implemented entirely within the
toolstack, and
>>> could even allow wildcarding of tuple components to efficiently
express
>>> cpumasks.
>>
>> Yes, I''d certainly like to see the toolstack support dotted
tuple notation.
>>
>> However, I just don''t trust the toolstack to get this right
unless xen has
>> already set it up nicely for it with a sensible enumeration and defined
>> sockets-per-node, cores-per-socket and threads-per-core parameters. Xen
should
>> provide a clean interface to the toolstack in this respect.
>
> Xen provides a topology-interrogation hypercall which should suffice for
> tools to build up a {node,socket,core,thread}<->cpuid mapping table.
>
>  -- Keir
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I think this is good. However, the socket and node ids can be fairly
arbitrary small numbers -- we need a way for the admin to find out the
topology and ''addresses'' of physical cpus via xm. Perhaps a
new ''xm
cpu-list'' command to basically dump the pcpu_tuple information in
ascending
order of node, then socket, then core, then thread, with one row per cpu:
 node socket core thread xen-cpu-id

More info could be added beyond these five pieces of information, as we
later see fit.

An alternative would be to rename the socket/node identifiers in
pyxc_physinfo, or even in Xen itself, to achieve contiguity. However I think
a cpu-list command would still be useful, and it''s easy to implement.

 -- Keir

On 17/11/2009 06:56, "Dulloor" <dulloor@gmail.com> wrote:
> Attached is a patch to construct pcpu tuples of the form
> (node.socket.core.thread), and (currently)used by xm vcpu-pin utility.
> 
> -dulloor
> 
> On Fri, Nov 13, 2009 at 11:02 AM, Keir Fraser
<keir.fraser@eu.citrix.com>
> wrote:
>> On 13/11/2009 15:40, "Ian Pratt"
<Ian.Pratt@eu.citrix.com> wrote:
>> 
>>>> Even better would be to have pCPUs addressable and listable
explicitly as
>>>> dotted tuples. That can be implemented entirely within the
toolstack, and
>>>> could even allow wildcarding of tuple components to efficiently
express
>>>> cpumasks.
>>> 
>>> Yes, I''d certainly like to see the toolstack support
dotted tuple notation.
>>> 
>>> However, I just don''t trust the toolstack to get this
right unless xen has
>>> already set it up nicely for it with a sensible enumeration and
defined
>>> sockets-per-node, cores-per-socket and threads-per-core parameters.
Xen
>>> should
>>> provide a clean interface to the toolstack in this respect.
>> 
>> Xen provides a topology-interrogation hypercall which should suffice
for
>> tools to build up a {node,socket,core,thread}<->cpuid mapping
table.
>> 
>>  -- Keir
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dulloor wrote:
> Attached is a patch to construct pcpu tuples of the form
> (node.socket.core.thread), and (currently)used by xm vcpu-pin utility.
Without having looked further at the patch: There will be problems with 
that notation. The assumption that one node consist of at least one 
socket is no longer true with AMD''s upcoming Magny-Cours processors, 
which features _two_ nodes in one socket.
The socket information as it is of interest for licensing purposes and 
for the voltage domains. I suppose that power aware scheduling is out of 
scope for the current scheduler, so we could ignore the socket 
information here at all.
Shared cache would be an interesting information to consider for 
scheduling purposes, but again here the socket information is 
misleading, as each node of the Magny-Cours processor has it''s own L3 
cache, there is no cache shared across the two nodes in one package.
Xen already detects the NUMA topology on the new system correctly:
nr_cpus                : 48
nr_nodes               : 8
cores_per_socket       : 12
threads_per_core       : 1
I don''t know details about the usual IA64 topology, though.

I see currently these possible topologies for x86-64 systems:
Core2-based: 1 (fake) node, n sockets
AMD64/Nehalem: n nodes, 1 socket/node
AMD G34: n nodes, 2 or 1 nodes/socket(!)
That looks like that it will not be easy to combine all of those. One 
possibility would be to join nodes and sockets into one entity (use 
sockets on older systems (L2 cache domains) and nodes on AMD/newer Intel 
systems (memory controller domains)). But I don''t have a handy name for
that beast (left alone nockets ;-)


Although it can be quite useful to have such a notation, I am not sure 
whether it will really help. Eventually you want to go away from manual 
assignment (and be it at domain''s runtime via "xm vcpu-pin").

Looking forward to any comments.

Regards,
Andre.
> 
> -dulloor
> 
> On Fri, Nov 13, 2009 at 11:02 AM, Keir Fraser
<keir.fraser@eu.citrix.com> wrote:
>> On 13/11/2009 15:40, "Ian Pratt"
<Ian.Pratt@eu.citrix.com> wrote:
>>
>>>> Even better would be to have pCPUs addressable and listable
explicitly as
>>>> dotted tuples. That can be implemented entirely within the
toolstack, and
>>>> could even allow wildcarding of tuple components to efficiently
express
>>>> cpumasks.
>>> Yes, I''d certainly like to see the toolstack support
dotted tuple notation.
>>>
>>> However, I just don''t trust the toolstack to get this
right unless xen has
>>> already set it up nicely for it with a sensible enumeration and
defined
>>> sockets-per-node, cores-per-socket and threads-per-core parameters.
Xen should
>>> provide a clean interface to the toolstack in this respect.
>> Xen provides a topology-interrogation hypercall which should suffice
for
>> tools to build up a {node,socket,core,thread}<->cpuid mapping
table.
>>
>>  -- Keir
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>

-- 
Andre Przywara
AMD-OSRC (Dresden)
Tel: x29712


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Andre Przywara <andre.przywara@amd.com> 19.11.09 10:05
>>>
>Without having looked further at the patch: There will be problems with 
>that notation. The assumption that one node consist of at least one 
>socket is no longer true with AMD''s upcoming Magny-Cours
processors,
>which features _two_ nodes in one socket.
Hmm, the term "node" doesn''t seem right here: How would you
call the
aggregation of several sockets to a unit, several of which could form a
bigger system (which is what so far is being called "node" afaict)?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> From: Jan Beulich [mailto:JBeulich@novell.com]
> Sent: Thursday, November 19, 2009 2:33 AM
> To: Andre Przywara
> Cc: George Dunlap; Ian Pratt; Keir Fraser; Dulloor; Papagiannis
> Anastasios; xen-devel@lists.xensource.com; Dan Magenheimer
> Subject: Re: [Xen-devel][PATCH]pcpu tuples [was Re: [Xen-devel] Xen
> 3.4.1 NUMA support]
> 
> 
> >>> Andre Przywara <andre.przywara@amd.com> 19.11.09 10:05
>>>
> >Without having looked further at the patch: There will be 
> problems with 
> >that notation. The assumption that one node consist of at least one 
> >socket is no longer true with AMD''s upcoming Magny-Cours
processors,
> >which features _two_ nodes in one socket.
> 
> Hmm, the term "node" doesn''t seem right here: How would
you call the
> aggregation of several sockets to a unit, several of which 
> could form a
> bigger system (which is what so far is being called "node"
afaict)?
Actually, I think it is the term "socket" that is now irrelevant.
While it is interesting from the point-of-view of a computer
vendor and for software licensing, it is irrelevant for a
scheduler.  What is important I think is:

"buddies"
- cpus within this unit MUST share a memory controller
- cpus within this unit MUST share at least one level of cache

"grouping"
- cpus within the same "grouping" MUST share a memory controller
- cpus in different "groupings" MUST NOT share a cache

"node"
- cpus in different nodes MUST NOT share a memory controller

Is ACPI able to describe these differences today?  If not,
has any of this been discussed on lkml or in acpi mailing lists?
How is Linux (and KVM and VMware and Hyper-V) dealing with
this new level of topology?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, dulloor/keir, for the changes to XEN_SYSCTL_physinfo, I''m not sure
part of my work will be helpful.

I sent out a patch to present pcpu information in dom0''s
/sys/devices/system/xen_pcpu/pcpuX directory. Currently I simply present the
apic_id/acpi_id in this directory for cpu hotplug.
But how about using it to present the whole topology information, for example,
we can add initial_apicid/core_id etc to the xen_pcpu/pcpuX directory.

Furthermore, if we can create a directory like
/sys/devices/system/xen_pcpu/pcpuX/topology, make the arrangement below this
directory same as native linux''s cpu directory in sysfs, I think it
will benifit more. For example, we can simply change current linux tools to show
the topo information (just /sys/device/system/cpu/ to 
/sys/devices/system/xen_pcpu/).

Of course, I understand maybe most management tools for virtualization will be
different with native tools, but this similar arrangement will be helpful still.

Any idea?

--jyh

xen-devel-bounces@lists.xensource.com wrote:
> [Xen-devel] Xen 3.4.1 NUMA support]
> 
> I think this is good. However, the socket and node ids can be fairly
> arbitrary small numbers -- we need a way for the admin to find out the
> topology and ''addresses'' of physical cpus via xm. Perhaps
a new ''xm
> cpu-list'' command to basically dump the pcpu_tuple information
> in ascending
> order of node, then socket, then core, then thread, with one
> row per cpu:
> node socket core thread xen-cpu-id
> 
> More info could be added beyond these five pieces of information, as
> we later see fit. 
> 
> An alternative would be to rename the socket/node identifiers in
> pyxc_physinfo, or even in Xen itself, to achieve contiguity.
> However I think
> a cpu-list command would still be useful, and it''s easy to
implement.
> 
> -- Keir
> 
> On 17/11/2009 06:56, "Dulloor" <dulloor@gmail.com> wrote:
> 
>> Attached is a patch to construct pcpu tuples of the form
>> (node.socket.core.thread), and (currently)used by xm vcpu-pin
>> utility. 
>> 
>> -dulloor
>> 
>> On Fri, Nov 13, 2009 at 11:02 AM, Keir Fraser
>> <keir.fraser@eu.citrix.com> wrote:
>>> On 13/11/2009 15:40, "Ian Pratt"
<Ian.Pratt@eu.citrix.com> wrote:
>>> 
>>>>> Even better would be to have pCPUs addressable and listable
>>>>> explicitly as dotted tuples. That can be implemented
entirely
>>>>> within the toolstack, and could even allow wildcarding of
tuple
>>>>> components to efficiently express cpumasks.
>>>> 
>>>> Yes, I''d certainly like to see the toolstack support
dotted tuple
>>>> notation. 
>>>> 
>>>> However, I just don''t trust the toolstack to get this
right unless
>>>> xen has already set it up nicely for it with a sensible
>>>> enumeration and defined sockets-per-node, cores-per-socket and
>>>> threads-per-core parameters. Xen should provide a clean
interface
>>>> to the toolstack in this respect. 
>>> 
>>> Xen provides a topology-interrogation hypercall which should
>>> suffice for tools to build up a
{node,socket,core,thread}<->cpuid
>>> mapping table. 
>>> 
>>>  -- Keir
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel