As we had a report of a problem booting Xen on an IBM x460, dying on the BUG_ON() in init_apic_ldr() in xen/include/asm-x86/mach-summit/mach_apic.h, I started comparing 32- and 64-bit APIC handling. Quickly I found that the same case is handled gracefully in 64-bits, by just tying any extra CPUs to the highest bit. (I suppose, will try to verify this with the originator, that the same machine also doesn''t boot with native 32-bit Linux, as the exact same issue should exist there). While doing the same generally shouldn''t be a problem, I wonder why this hasn''t been discovered so far and how many else differences there exist. Thanks, Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 17 Mar 2006, at 16:49, Jan Beulich wrote:> As we had a report of a problem booting Xen on an IBM x460, dying on > the BUG_ON() in init_apic_ldr() in > xen/include/asm-x86/mach-summit/mach_apic.h, I started comparing 32- > and 64-bit APIC handling. Quickly I found that the > same case is handled gracefully in 64-bits, by just tying any extra > CPUs to the highest bit. (I suppose, will try to > verify this with the originator, that the same machine also doesn''t > boot with native 32-bit Linux, as the exact same > issue should exist there). > While doing the same generally shouldn''t be a problem, I wonder why > this hasn''t been discovered so far and how many > else differences there exist.Differences between i386 and x86/64 native Linux APIC handling? A fair few, although mostly it''s because crufty old code has been removed from x86/64. I guess there are occasions where Andi Kleen has improved correctness at the same time as cleaning up. :-) This is the first time that the strategy of taking latest i386 APIC code has let us down I think. I guess we just patch it in Xen with a comment explaining the extra diff vs native i386 Linux version of the same file. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 03/17/06 6:06 PM >>> > >On 17 Mar 2006, at 16:49, Jan Beulich wrote: > >> As we had a report of a problem booting Xen on an IBM x460, dying on >> the BUG_ON() in init_apic_ldr() in >> xen/include/asm-x86/mach-summit/mach_apic.h, I started comparing 32- >> and 64-bit APIC handling. Quickly I found that the >> same case is handled gracefully in 64-bits, by just tying any extra >> CPUs to the highest bit. (I suppose, will try to >> verify this with the originator, that the same machine also doesn''t >> boot with native 32-bit Linux, as the exact same >> issue should exist there). >> While doing the same generally shouldn''t be a problem, I wonder why >> this hasn''t been discovered so far and how many >> else differences there exist. > >Differences between i386 and x86/64 native Linux APIC handling? A fair >few, although mostly it''s because crufty old code has been removed from >x86/64. I guess there are occasions where Andi Kleen has improved >correctness at the same time as cleaning up. :-) > >This is the first time that the strategy of taking latest i386 APIC >code has let us down I think. > >I guess we just patch it in Xen with a comment explaining the extra >diff vs native i386 Linux version of the same file.Actually, looking further, the originally mentioned adjustment can''t work. x86-64 Linux can do this because they deliver IPIs (on large systems) in physical destination mode, and hence they don''t really need to values written to DFR and specifically LDR. Since Xen inherited the 32-bit code, IPIs get sent in logical (cluster) mode, and then playing games like this with the LDR is only going to cause problems (you''ll hit multiple processors with a send that''s intended for a single one only). Hence I would think that more extensive changes are going to be needed; entirely taking x86-64''s model may not be feasible either (aside of the fact that this would mean quite extensive code changes), as they don''t need to care about pre-Pentium4 processors. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 19 Mar 2006, at 19:15, Jan Beulich wrote:> Actually, looking further, the originally mentioned adjustment can''t > work. x86-64 > Linux can do this because they deliver IPIs (on large systems) in > physical > destination mode, and hence they don''t really need to values written > to DFR and > specifically LDR. Since Xen inherited the 32-bit code, IPIs get sent > in logical > (cluster) mode, and then playing games like this with the LDR is only > going to > cause problems (you''ll hit multiple processors with a send that''s > intended for a > single one only). Hence I would think that more extensive changes are > going to > be needed; entirely taking x86-64''s model may not be feasible either > (aside of > the fact that this would mean quite extensive code changes), as they > don''t need > to care about pre-Pentium4 processors.Yes, that''s what stopped me taking x86/64 Linux code originally. I wonder if we can take just the IPI code from x86/64, plus whatever small changes we need to fix problems like this x460 bug. IPI code is reasonably self contained relative to the bulk of the apic code in apic.c. The x86/64 IPI code was a bunch cleaner than the i386 code last time I looked, and I don''t think there are lurking legacy 32-bit issues in that area that would bite us? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>Yes, that''s what stopped me taking x86/64 Linux code originally. I >wonder if we can take just the IPI code from x86/64, plus whatever >small changes we need to fix problems like this x460 bug. IPI code is >reasonably self contained relative to the bulk of the apic code in >apic.c. The x86/64 IPI code was a bunch cleaner than the i386 code last >time I looked, and I don''t think there are lurking legacy 32-bit issues >in that area that would bite us?Hmm, I''m not sure. Both because it wouldn''t address all issues (the I/O APIC redirection table entries also need to be programmed in physical mode) and because I''m not so certain about legacy 32-bit issues (on 32-bits you still have to account for the PentiumPro/II/III behavior, namely the only 4-bit wide IDs, as that at least seems to imply different cut-off criteria for the decision what destination mode to use). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 20 Mar 2006, at 13:53, Jan Beulich wrote:> Hmm, I''m not sure. Both because it wouldn''t address all issues (the > I/O APIC redirection table entries also need to be programmed in > physical mode) and because I''m not so certain about legacy 32-bit > issues (on 32-bits you still have to account for the PentiumPro/II/III > behavior, namely the only 4-bit wide IDs, as that at least seems to > imply different cut-off criteria for the decision what destination mode > to use).We can just steal x86/64 phys cluster code for summit. ES7000 already has its own phys cluster code that doesn''t look like it will have this issue, Bigsmp uses phys flat, and Default uses logical flat. So summit is the only subarch that needs fixing. Xen''s io_apic.c will program logical/phys dest mode appropriately (although it''s taken from i386 io_apic.c, those parts of the i386 file are identical to x86/64 io_apic.c). I''d also like to fix send_IPI_mask_sequence() to use physical destination mode. It scares me a bit that 2.6.16 native i386 bigsmp mode uses physical flat model, but uses send_IPI_mask_sequence with logical delivery. Seems weird to me. Both this and the summit fix will be after 3.0.2 now. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 20 Mar 2006, at 16:00, Keir Fraser wrote:> I''d also like to fix send_IPI_mask_sequence() to use physical > destination mode. It scares me a bit that 2.6.16 native i386 bigsmp > mode uses physical flat model, but uses send_IPI_mask_sequence with > logical delivery. Seems weird to me.Oh wow. I see that in fact mach-bigsmap and mach-es7000 both redefine APIC_DEST_LOGICAL to zero. That''s very sleazy. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>Xen''s io_apic.c will program logical/phys dest mode appropriately >(although it''s taken from i386 io_apic.c, those parts of the i386 file >are identical to x86/64 io_apic.c).Good - I didn''t check this explicitly.>I''d also like to fix send_IPI_mask_sequence() to use physical >destination mode. It scares me a bit that 2.6.16 native i386 bigsmp >mode uses physical flat model, but uses send_IPI_mask_sequence with >logical delivery. Seems weird to me.Haven''t been able to download 2.6.16, yet (the WLANs I''m currently moving in seem to be rather unreliable, and I have no copy of the unmodified sources of any of the RCs around), but in any case our bigsmp kernel already appears to be using consistent physical mode.>Both this and the summit fix will be after 3.0.2 now.We''ll have to pick them up anyway (if you get around to do it before I return from BrainShare, where I can''t really do any significant work, and namely no testing). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 21 Mar 2006, at 14:23, Jan Beulich wrote:>> Both this and the summit fix will be after 3.0.2 now. > > We''ll have to pick them up anyway (if you get around to do it before I > return > from BrainShare, where I can''t really do any significant work, and > namely no > testing).Testing is the scary thing for me. I have no access to Summit or ES7000 boxes, and I don''t know what external testing is being done on those systems and on what schedule. For all I know, bugs could be introduced and not fixed for an arbitrary time. If I know there is testing infrastructure I would like to move closer to x86/64''s genapic code (which uses generic mechanisms to determine what APIC mode to use) and keep the minimal amount possible of default/bigsmp/summit/es7000 specific subarchitecture code. Still, some incremental progress on that path is definitely achievable. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>Testing is the scary thing for me. I have no access to Summit or ES7000 >boxes, and I don''t know what external testing is being done on those >systems and on what schedule. For all I know, bugs could be introduced >and not fixed for an arbitrary time. If I know there is testing >infrastructure I would like to move closer to x86/64''s genapic code >(which uses generic mechanisms to determine what APIC mode to use) and >keep the minimal amount possible of default/bigsmp/summit/es7000 >specific subarchitecture code. Still, some incremental progress on that >path is definitely achievable.Whatever change we''re going to do here, we''ll have IBM test this on the x460-s they reported the problem on. While we supposedly have at least on x460 in our Lab, it''s only a 16-(logical-)processor one, and there it doesn''t matter whether clustered logical or physical modes get used. However, even if they disable HT on the 4-node system (reducing to 32 logical processors) they apparently also have a BIOS problem there in that, with the intention to make the OS use clustered logical mode, they assign processors to 16 clusters (2 processors each), ending up with two processors with APIC IDs of the form 0xFx, which to my understanding is valid only for physical mode. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tuesday 21 March 2006 08:34, Keir Fraser wrote:> Testing is the scary thing for me. I have no access to Summit or ES7000 > boxes, and I don''t know what external testing is being done on those > systems and on what schedule. For all I know, bugs could be introduced > and not fixed for an arbitrary time. If I know there is testing > infrastructure I would like to move closer to x86/64''s genapic code > (which uses generic mechanisms to determine what APIC mode to use) and > keep the minimal amount possible of default/bigsmp/summit/es7000 > specific subarchitecture code. Still, some incremental progress on that > path is definitely achievable.We have not seen this issue on our 2-node 460 or any of our other summit boxes. However we do have a couple of summit boxes we could use for testing if you have some code to throw at us. Thanks, Paul Larson _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Testing is the scary thing for me. I have no access to Summit orES7000> boxes, and I don''t know what external testing is being done on those > systems and on what schedule. For all I know, bugs could be introduced > and not fixed for an arbitrary time. If I know there is testing > infrastructure I would like to move closer to x86/64''s genapic code > (which uses generic mechanisms to determine what APIC mode to use) and > keep the minimal amount possible of default/bigsmp/summit/es7000 > specific subarchitecture code. Still, some incremental progress onthat> path is definitely achievable.Keir, We can help with the testing on ES7000 boxes. I have always wanted to move closer to the x86_64 genapic code so I will help in whatever possible way. Thanks, Aravindh Puthiyaparambil Xen Development Team Unisys Tredyffrin E240 E-Mail: aravindh.puthiyaparambil@unisys.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 21 Mar 2006, at 15:30, Puthiyaparambil, Aravindh wrote:> We can help with the testing on ES7000 boxes. I have always wanted to > move closer to the x86_64 genapic code so I will help in whatever > possible way.Thanks (also to Paul Larson re Summit testing). I''ll look into this after 3.0.2 then. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel