Hi! Changeset 21118 breaks NUMA on AMD Magny-Coure due to introduction of ''sockets_per_node''. On AMD Magny-Coure we have two nodes on one socket, hence the existence of that field member introduces a breake on design level. Please revert changeset 21118 or rework the patch to get rid of this field member. Christoph -- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen Geschaeftsfuehrer: Andrew Bowd, Thomas M. McCoy, Giuliano Meroni Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
We''re at an intermediate state in 4.1 NUMA implementation I would say. That changeset can be revised as necessary in future patches, but I think it is broadly correct in that it''s adding the right kind of extra things to the control interfaces. Since we have 6-9 months before 4.1 is released we don''t need panic about regressions just yet. -- Keir On 12/04/2010 11:07, "Christoph Egger" <Christoph.Egger@amd.com> wrote:> > Hi! > > Changeset 21118 breaks NUMA on AMD Magny-Coure due to introduction of > ''sockets_per_node''. On AMD Magny-Coure we have two nodes on one socket, hence > the existence of that field member introduces a breake on design level. > > Please revert changeset 21118 or rework the patch to get rid of this field > member. > > Christoph >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
It is worth discussing whether it makes sense to have the concept of sockets in the NUMA interfaces. Would threads/cores/nodes suffice? Not sure sockets give us any more than a hint about possible caching hierarchy (possible socket-wide L3) and communication latency. That may be too vague to be any use even where the concept of sockets-per-node is applicable. -- Keir On 12/04/2010 11:41, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:> We''re at an intermediate state in 4.1 NUMA implementation I would say. That > changeset can be revised as necessary in future patches, but I think it is > broadly correct in that it''s adding the right kind of extra things to the > control interfaces. Since we have 6-9 months before 4.1 is released we don''t > need panic about regressions just yet. > > -- Keir > > On 12/04/2010 11:07, "Christoph Egger" <Christoph.Egger@amd.com> wrote: > >> >> Hi! >> >> Changeset 21118 breaks NUMA on AMD Magny-Coure due to introduction of >> ''sockets_per_node''. On AMD Magny-Coure we have two nodes on one socket, hence >> the existence of that field member introduces a breake on design level. >> >> Please revert changeset 21118 or rework the patch to get rid of this field >> member. >> >> Christoph >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
A large number of software licensing beancounters in the industry are going to be busy on this one ;-) Christophe, can you comment on how Linux and/or other bare metal operating systems will be reporting the Magny Cours hierarchy? I''d hope that Xen could follow their lead on this rather than forge new ground, which may result in an incompatible implementation. Dan> -----Original Message----- > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > Sent: Monday, April 12, 2010 5:01 AM > To: Christoph Egger; Nitin A Kamble > Cc: Andre Przywara; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Re: c/s 21118: Magny-Coure breakage > > It is worth discussing whether it makes sense to have the concept of > sockets > in the NUMA interfaces. Would threads/cores/nodes suffice? Not sure > sockets > give us any more than a hint about possible caching hierarchy (possible > socket-wide L3) and communication latency. That may be too vague to be > any > use even where the concept of sockets-per-node is applicable. > > -- Keir > > On 12/04/2010 11:41, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote: > > > We''re at an intermediate state in 4.1 NUMA implementation I would > say. That > > changeset can be revised as necessary in future patches, but I think > it is > > broadly correct in that it''s adding the right kind of extra things to > the > > control interfaces. Since we have 6-9 months before 4.1 is released > we don''t > > need panic about regressions just yet. > > > > -- Keir > > > > On 12/04/2010 11:07, "Christoph Egger" <Christoph.Egger@amd.com> > wrote: > > > >> > >> Hi! > >> > >> Changeset 21118 breaks NUMA on AMD Magny-Coure due to introduction > of > >> ''sockets_per_node''. On AMD Magny-Coure we have two nodes on one > socket, hence > >> the existence of that field member introduces a breake on design > level. > >> > >> Please revert changeset 21118 or rework the patch to get rid of this > field > >> member. > >> > >> Christoph > >> > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer wrote:> A large number of software licensing beancounters in the industry > are going to be busy on this one ;-)Yes, but do they use the Xen hypervisor interface for that? And do they look at the actual socket number? As far as I remember some companies ;-) are counting MCMs twice, so actually counting the nodes?> Christophe, can you comment on how Linux and/or other bare metal > operating systems will be reporting the Magny Cours hierarchy?As long as there is no such notion as ''sockets per node'' (which isn''t on Linux), everything is fine. It will just report 8 nodes with 6 cores each (in case of a 4 socket M-C). Each core has a associated physical package id, which is just the same for 12 cores or 2 nodes. I have a machine here up and running, so just tell me what sysfs path you are interested in and I can send you the output.> I''d hope that Xen could follow their lead on this rather > than forge new ground, which may result in an incompatible > implementation.Until c/s 21118 Xen was just fine, reporting 8 nodes, 4 sockets and 48 cores. It is just this socket per node (which got removed some years ago), which is doing harm. (BTW: I already complained about that last year, when Nitin''s patch first appeared on the ML). Regards, Andre. -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448-3567-12 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 12/04/2010 14:14, "Andre Przywara" <andre.przywara@amd.com> wrote:>> I''d hope that Xen could follow their lead on this rather >> than forge new ground, which may result in an incompatible >> implementation. > Until c/s 21118 Xen was just fine, reporting 8 nodes, 4 sockets and 48 > cores. It is just this socket per node (which got removed some years > ago), which is doing harm. (BTW: I already complained about that last > year, when Nitin''s patch first appeared on the ML).Okay, well I will revert the patch if we reach no further consensus. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir, Attached is the patch to take of the sockets_per_node information from the 21118 changeset. Signed-Off-By: Nitin A Kamble <nitin.a.kamble@intel.com> Thanks & Regards, Nitin> -----Original Message----- > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > Sent: Monday, April 12, 2010 6:25 AM > To: Andre Przywara; Dan Magenheimer > Cc: Christoph Egger; Kamble, Nitin A; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Re: c/s 21118: Magny-Coure breakage > > On 12/04/2010 14:14, "Andre Przywara" <andre.przywara@amd.com> wrote: > > >> I''d hope that Xen could follow their lead on this rather > >> than forge new ground, which may result in an incompatible > >> implementation. > > Until c/s 21118 Xen was just fine, reporting 8 nodes, 4 sockets and > 48 > > cores. It is just this socket per node (which got removed some years > > ago), which is doing harm. (BTW: I already complained about that last > > year, when Nitin''s patch first appeared on the ML). > > Okay, well I will revert the patch if we reach no further consensus. > > -- Keir >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kamble, Nitin A wrote:> Keir, > Attached is the patch to take of the sockets_per_node information from the 21118 changeset.Thanks Nitin, that looks better. Unfortunately the patch itself does not work for me, xend crashes (and keeps restarting) due to signal 11. It works though if I just revert your patch (rebuilt both hypervisor and tools). I will try to further debug this, but in general your patch looks clean to me. BTW: I found the output of xm info now a bit too verbose, can we introduce a -v option to show the NUMA information only when explicitly requested? On my 48 core/8 nodes machine I get a lot of text, this should be even worse with 128 threads, right? Regards, Andre. -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448-3567-12 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andre, We will look into the xend instability issue. I also find that lots of scrolling happens with xm info output. I think putting the detailed information in the -v option is a good idea. May be you can extend it in this matter. Thanks & Regards, Nitin> -----Original Message----- > From: Andre Przywara [mailto:andre.przywara@amd.com] > Sent: Wednesday, April 14, 2010 2:47 AM > To: Kamble, Nitin A > Cc: Keir Fraser; Dan Magenheimer; Christoph Egger; xen- > devel@lists.xensource.com > Subject: Re: [Xen-devel] Re: c/s 21118: Magny-Coure breakage > > Kamble, Nitin A wrote: > > Keir, > > Attached is the patch to take of the sockets_per_node information > from the 21118 changeset. > Thanks Nitin, that looks better. > Unfortunately the patch itself does not work for me, xend crashes (and > keeps restarting) due to signal 11. It works though if I just revert > your patch (rebuilt both hypervisor and tools). I will try to further > debug this, but in general your patch looks clean to me. > > BTW: I found the output of xm info now a bit too verbose, can we > introduce a -v option to show the NUMA information only when explicitly > requested? > On my 48 core/8 nodes machine I get a lot of text, this should be even > worse with 128 threads, right? > > Regards, > Andre. > > -- > Andre Przywara > AMD-Operating System Research Center (OSRC), Dresden, Germany > Tel: +49 351 448-3567-12_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andre Przywara
2010-Apr-15 11:33 UTC
[Xen-devel] [PATCH] xend: make NUMA in xm info optional
Kamble, Nitin A wrote:> Andre, > We will look into the xend instability issue.I will also do this next, I reverted to using xl for now ;-)> I also find that lots of scrolling happens with xm info output.> I think putting the detailed information in the -v option is a > good idea. May be you can extend it in this matter. OK, see the attached patch. I used -n (or --numa) for clarity. Tested against an older changeset (with still working xend). Signed-off-by: Andre Przywara <andre.przywara@amd.com> Regards, Andre. -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448-3567-12 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Apr-15 12:11 UTC
[Xen-devel] Re: [PATCH] xend: make NUMA in xm info optional
On 15/04/2010 12:33, "Andre Przywara" <andre.przywara@amd.com> wrote:>> We will look into the xend instability issue. > I will also do this next, I reverted to using xl for now ;-)This is fixed by xen-unstable:21183. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andre Przywara
2010-Apr-15 12:14 UTC
[Xen-devel] Re: [PATCH] xend: make NUMA in xm info optional
Keir Fraser wrote:> On 15/04/2010 12:33, "Andre Przywara" <andre.przywara@amd.com> wrote: > >>> We will look into the xend instability issue. >> I will also do this next, I reverted to using xl for now ;-) > > This is fixed by xen-unstable:21183.Yes, I just saw (and tested) this a few minutes ago. I will resend a rebased version of the patch along with a commit message. Regards, Andre. -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448-3567-12 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Apr-15 12:19 UTC
[Xen-devel] Re: [PATCH] xend: make NUMA in xm info optional
On 15/04/2010 13:14, "Andre Przywara" <andre.przywara@amd.com> wrote:>> This is fixed by xen-unstable:21183. > Yes, I just saw (and tested) this a few minutes ago. I will resend a > rebased version of the patch along with a commit message.I already checked your patch in, since it didn''t need rebasing, and you already sent a signed-off-by line. It''s xen-unstable:21186. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel