Hi, This is updated xen vMCE design foils, according to comments from community recently. This foils focus on vMCE part of Xen MCA, so as Keir said, it''s some dense. Later Will will present a document to elaborate more, including Intel MCA and surrounding features and Xen implementation. Thanks, Jinsong
>>> On 27.06.12 at 05:51, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: > This is updated xen vMCE design foils, according to comments from community > recently. > > This foils focus on vMCE part of Xen MCA, so as Keir said, it''s some dense. > Later Will will present a document to elaborate more, including Intel MCA > and surrounding features and Xen implementation.For MCi_CTL2 you probably meant to say "allow setting CMCI_EN and error count threshold"? The 2-bank approach also needs to be brought in line with the current host derived value in MCG_CAP, especially if the code to implement this new model doesn''t make it into 4.2 (which would generally save a larger value). Jan
Jan Beulich wrote:>>>> On 27.06.12 at 05:51, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: >> This is updated xen vMCE design foils, according to comments from >> community recently. >> >> This foils focus on vMCE part of Xen MCA, so as Keir said, it''s some >> dense. Later Will will present a document to elaborate more, >> including Intel MCA and surrounding features and Xen implementation. > > For MCi_CTL2 you probably meant to say "allow setting CMCI_EN > and error count threshold"?Yes, exactly! the words is w/ subtle but important different meaning.> > The 2-bank approach also needs to be brought in line with the > current host derived value in MCG_CAP, especially if the code to > implement this new model doesn''t make it into 4.2 (which would > generally save a larger value). > > JanLet me repeat in my word to avoid misunderstanding about your concern: Your concern rooted from the history patch (c/s 24887, as attached) which used to solve vMCE migration issue caused from bank number. I guess the patch was not in xen4.1.x but would be in xen 4.2 release recently (right? and when will xen 4.2 release?) Per my understanding, you want us to make sure our new vMCE model compatible w/ olde vMCE. For example if our patch in xen 4.3 release, you want to make sure a guest migrate from xen 4.2 to 4.3 would not broken. Is this your concern? Thanks, Jinsong
>>> On 28.06.12 at 10:54, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: > Jan Beulich wrote: >> The 2-bank approach also needs to be brought in line with the >> current host derived value in MCG_CAP, especially if the code to >> implement this new model doesn''t make it into 4.2 (which would >> generally save a larger value). > > Let me repeat in my word to avoid misunderstanding about your concern: > Your concern rooted from the history patch (c/s 24887, as attached) which > used to solve vMCE migration issue caused from bank number. I guess the patch > was not in xen4.1.x but would be in xen 4.2 release recently (right? and when > will xen 4.2 release?)4.2 is in feature freeze right now, preparing for the release.> Per my understanding, you want us to make sure our new vMCE model compatible > w/ olde vMCE. For example if our patch in xen 4.3 release, you want to make > sure a guest migrate from xen 4.2 to 4.3 would not broken. Is this your > concern?Yes. If we can''t get the save/restore records adjusted in time for 4.2, compatibility with 4.2 would be a requirement. If we manage to get the necessary adjustments done in time, and if they''re not too intrusive (i.e. would be acceptable at this late stage of the development cycle), then the concern could be dropped from an upstream perspective due to the lack of support in 4.1 (and hence no backward compatibility issues). BUT this isn''t as simple from a product usage point of view: As the save/restore code currently in -unstable was coded up to address a problem observed by SLE11 SP2 users, we already backported those patches. So compatibility will be a requirement in any case. Jan
Jan Beulich wrote:>>>> On 28.06.12 at 10:54, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: >> Jan Beulich wrote: >>> The 2-bank approach also needs to be brought in line with the >>> current host derived value in MCG_CAP, especially if the code to >>> implement this new model doesn''t make it into 4.2 (which would >>> generally save a larger value). >> >> Let me repeat in my word to avoid misunderstanding about your >> concern: >> Your concern rooted from the history patch (c/s 24887, as attached) >> which used to solve vMCE migration issue caused from bank number. I >> guess the patch was not in xen4.1.x but would be in xen 4.2 release >> recently (right? and when will xen 4.2 release?) > > 4.2 is in feature freeze right now, preparing for the release. > >> Per my understanding, you want us to make sure our new vMCE model >> compatible w/ olde vMCE. For example if our patch in xen 4.3 >> release, you want to make sure a guest migrate from xen 4.2 to 4.3 >> would not broken. Is this your concern? > > Yes. If we can''t get the save/restore records adjusted in time > for 4.2, compatibility with 4.2 would be a requirement. If we > manage to get the necessary adjustments done in time, and if > they''re not too intrusive (i.e. would be acceptable at this late > stage of the development cycle), then the concern could be > dropped from an upstream perspective due to the lack of > support in 4.1 (and hence no backward compatibility issues). > BUT this isn''t as simple from a product usage point of view: As > the save/restore code currently in -unstable was coded up to > address a problem observed by SLE11 SP2 users, we already > backported those patches. So compatibility will be a requirement > in any case. > > JanA basic point of new vMCE is, it get rid of old vMCE, start setting up a new model from the very beginning. From coding point of view, backward compatibility issue would be dirty and troublesome. The point is, old vMCE interface is host-based while new vMCE is pure s/w defined, hence troubles come from the interface heterogeneous (if need keep compatibility). The basic assumption of live migration from A to B is, A and B basically at same page, so it could be migrated by setting the smallest common feature/capability set (via cpuid, command line, etc.). However, old vMCE and new vMCE are quite different and cannot controlled effectively. For example, old vMCE has MCG_CTL but new vMCE doesn''t, and new vMCE has CMCI support (and MCi_CTL2) but old vMCE doesn''t. I even doubt the feasibility of keeping compatibility w/ old vMCE. An example is, it''s hard to migrate between Intel cpu and AMD cpu. So I would like to push new vMCE as quickly as possible. What''s the timeline of vMCE developing that xen 4.2 could accept? I wonder if we could make major components of vMCE done before xen 4.2 timeline, and leave the surrounding features and the corner cases done later? Thanks, Jinsong-- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
>>> On 28.06.12 at 11:40, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: > So I would like to push new vMCE as quickly as possible. What''s the timeline > of vMCE developing that xen 4.2 could accept?Weeks ago, really. See http://lists.xen.org/archives/html/xen-devel/2012-06/msg01619.html and follow-ups - we''d really only consider getting the save/restore interface into forward compatible shape as acceptable.> I wonder if we could make major > components of vMCE done before xen 4.2 timeline, and leave the surrounding > features and the corner cases done later?Unfortunately it''s likely going to be even less. However, if split that way, chances are things could go into e.g. 4.2.1. Jan
On Thu, 2012-06-28 at 10:55 +0100, Jan Beulich wrote:> >>> On 28.06.12 at 11:40, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: > > So I would like to push new vMCE as quickly as possible. What''s the timeline > > of vMCE developing that xen 4.2 could accept? > > Weeks ago, really. See http://lists.xen.org/archives/html/xen-devel/2012-06/msg01619.html > and follow-ups - we''d really only consider getting the save/restore > interface into forward compatible shape as acceptable.Yes it really is far to late to considering entire new features at this stage. Ian.> > > I wonder if we could make major > > components of vMCE done before xen 4.2 timeline, and leave the surrounding > > features and the corner cases done later? > > Unfortunately it''s likely going to be even less. However, if split > that way, chances are things could go into e.g. 4.2.1. > > Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Jan Beulich wrote:>>>> On 28.06.12 at 11:40, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: >> So I would like to push new vMCE as quickly as possible. What''s the >> timeline of vMCE developing that xen 4.2 could accept? > > Weeks ago, really. See > http://lists.xen.org/archives/html/xen-devel/2012-06/msg01619.html > and follow-ups - we''d really only consider getting the save/restore > interface into forward compatible shape as acceptable. > >> I wonder if we could make major >> components of vMCE done before xen 4.2 timeline, and leave the >> surrounding features and the corner cases done later? > > Unfortunately it''s likely going to be even less. However, if split > that way, chances are things could go into e.g. 4.2.1. > > JanSo let''s look at current vMCE status first: 1). functionally it work abnormally for guest (but tolerated by some guest like linux/solaris); 2). before xen 4.1 it blocks migration when migrate from big bank to small bank platform; We may try some middle steps, minimal adjusting for xen 4.2 release (to avoid futher compatible issue at xen 4.2.1, 4.3, ...): 1). we don''t handle vMCE function bugs, only make sure migration works OK; 2). update vMCE interface to a middle clean status: * remove MCG_CTL (otherwise we have to add this useless MSR at new vMCE); * stick all 1''s to MCi_CTL (avoid semantic difference); * for MCG_CAP, clear MCG_CTL_P, limit to 2 banks (otherwise dirty code have to be added at new vMCE); Thoughts? Thanks, Jinsong-- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
>>> On 28.06.12 at 15:38, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: > Jan Beulich wrote: >>>>> On 28.06.12 at 11:40, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: >>> So I would like to push new vMCE as quickly as possible. What''s the >>> timeline of vMCE developing that xen 4.2 could accept? >> >> Weeks ago, really. See >> http://lists.xen.org/archives/html/xen-devel/2012-06/msg01619.html >> and follow-ups - we''d really only consider getting the save/restore >> interface into forward compatible shape as acceptable. >> >>> I wonder if we could make major >>> components of vMCE done before xen 4.2 timeline, and leave the >>> surrounding features and the corner cases done later? >> >> Unfortunately it''s likely going to be even less. However, if split >> that way, chances are things could go into e.g. 4.2.1. >> >> Jan > > So let''s look at current vMCE status first: > 1). functionally it work abnormally for guest (but tolerated by some guest > like linux/solaris); > 2). before xen 4.1 it blocks migration when migrate from big bank to small > bank platform;Before 4.2 you mean (in 4.1 we only have this as a backport in SLE11).> We may try some middle steps, minimal adjusting for xen 4.2 release (to > avoid futher compatible issue at xen 4.2.1, 4.3, ...): > 1). we don''t handle vMCE function bugs, only make sure migration works OK;That''s the minimal goal.> 2). update vMCE interface to a middle clean status: > * remove MCG_CTL (otherwise we have to add this useless MSR at new > vMCE); > * stick all 1''s to MCi_CTL (avoid semantic difference); > * for MCG_CAP, clear MCG_CTL_P, limit to 2 banks (otherwise dirty code > have to be added at new vMCE);Whether that''s acceptable would need to be seen when code is ready. Jan
Jan Beulich wrote:>>>> On 28.06.12 at 15:38, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: >> Jan Beulich wrote: >>>>>> On 28.06.12 at 11:40, "Liu, Jinsong" <jinsong.liu@intel.com> >>>>>> wrote: >>>> So I would like to push new vMCE as quickly as possible. What''s the >>>> timeline of vMCE developing that xen 4.2 could accept? >>> >>> Weeks ago, really. See >>> http://lists.xen.org/archives/html/xen-devel/2012-06/msg01619.html >>> and follow-ups - we''d really only consider getting the save/restore >>> interface into forward compatible shape as acceptable. >>> >>>> I wonder if we could make major >>>> components of vMCE done before xen 4.2 timeline, and leave the >>>> surrounding features and the corner cases done later? >>> >>> Unfortunately it''s likely going to be even less. However, if split >>> that way, chances are things could go into e.g. 4.2.1. >>> >>> Jan >> >> So let''s look at current vMCE status first: >> 1). functionally it work abnormally for guest (but tolerated by some >> guest like linux/solaris); 2). before xen 4.1 it blocks migration >> when migrate from big bank to small bank platform; > > Before 4.2 you mean (in 4.1 we only have this as a backport in SLE11).Yes.> >> We may try some middle steps, minimal adjusting for xen 4.2 release >> (to avoid futher compatible issue at xen 4.2.1, 4.3, ...): >> 1). we don''t handle vMCE function bugs, only make sure migration >> works OK; > > That''s the minimal goal.You mean to fix current vMCE function bugs in xen 4.2? That would involve much work hence too late for xen 4.2. In fact the bugs currently tolerated by guest, so it''s important but non-urgent. What we need to do urgently is to adjust current vMCE interface a little so that 1). it would not block xen 4.2 live migration 2). it would not bring compatibility issues to new vMCE in the future These 2 points are our minimal targets for xen 4.2 Thanks, Jinsong> >> 2). update vMCE interface to a middle clean status: >> * remove MCG_CTL (otherwise we have to add this useless MSR at >> new vMCE); >> * stick all 1''s to MCi_CTL (avoid semantic difference); >> * for MCG_CAP, clear MCG_CTL_P, limit to 2 banks (otherwise >> dirty code have to be added at new vMCE); > > Whether that''s acceptable would need to be seen when code > is ready. > > Jan
Feedback from the AMD side: slide 2: - PV guests are supposed to install a MCE trap handler which reads the MSR values from struct mcinfo_bank. Hence it is unclear where the #GP should come from. Same for HVM guests which have a PV MCE "driver" (those are very rare in reality). slide 3: - unclear what "Weird per-domain MSRs" means - unclear what "Unnatural MCE injection semantics" means slide 4: - typo: interace -> interface :-) - enable UCR-related capabilities, but only on Intel machines - Filter non-SRAO/SRAR banks: Rename it to "Let guest see northbridge bank only to the guest" slide 7: - ignore/disable CMCI and CTL2 on AMD slide 8: - Filter non-SRAO/SRAR banks: Rename it to "Let guest see northbridge bank only to the guest" - Question: Should we allow the guest to inject errors? Does it make sense? - always disable MCi_CTL2 on AMD slide 9: - Model specific issue: Also affects AMD as some models have l3 cache and some do not. E.g. it does not make sense to report l3 cache errors to guests -- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85689 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
Thanks AMD''s feedback :) This vMCE design foils is basically for Intel MCA, involving many details specific to Intel. I agree that for x86 Intel and AMD can share logic in many fields. However, for MCA logic Intel and AMD are quite different, like 1. MSRs interface, e.g. MCG_CAP, MCi_MICS, MCi_CTL2, etc; 2. error injection, AMD provide NMI/single MCE/broadcast MCE, while in our design only concern broadcast MCE# (and pretend to expose CMCI); 3. MCE handler: currently in xen Intel and AMD mce use different triggle method and mce handler; Considering the big difference, I suggest we separately provide Intel vMCE and AMD vMCE (i.e. vmce_intel.c and vmce_amd.c). Thanks, Jinsong Christoph Egger wrote:> Feedback from the AMD side: > > slide 2: > - PV guests are supposed to install a MCE trap handler > which reads the MSR values from struct mcinfo_bank. > Hence it is unclear where the #GP should come from. > Same for HVM guests which have a PV MCE "driver" > (those are very rare in reality). > > slide 3: > - unclear what "Weird per-domain MSRs" means > - unclear what "Unnatural MCE injection semantics" means > > slide 4: > - typo: interace -> interface :-) > - enable UCR-related capabilities, but only on Intel machines > - Filter non-SRAO/SRAR banks: > Rename it to "Let guest see northbridge bank only to the guest" > > slide 7: > - ignore/disable CMCI and CTL2 on AMD > > slide 8: > - Filter non-SRAO/SRAR banks: > Rename it to "Let guest see northbridge bank only to the guest" > - Question: Should we allow the guest to inject errors? Does it make > sense? > - always disable MCi_CTL2 on AMD > > slide 9: > - Model specific issue: Also affects AMD as some models have > l3 cache and some do not. > E.g. it does not make sense to report l3 cache errors to guests
>>> On 02.07.12 at 19:32, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: > Thanks AMD''s feedback :) > > This vMCE design foils is basically for Intel MCA, involving many details > specific to Intel. > I agree that for x86 Intel and AMD can share logic in many fields. However, > for MCA logic Intel and AMD are quite different, like > 1. MSRs interface, e.g. MCG_CAP, MCi_MICS, MCi_CTL2, etc; > 2. error injection, AMD provide NMI/single MCE/broadcast MCE, while in our > design only concern broadcast MCE# (and pretend to expose CMCI); > 3. MCE handler: currently in xen Intel and AMD mce use different triggle > method and mce handler; > > Considering the big difference, I suggest we separately provide Intel vMCE > and AMD vMCE (i.e. vmce_intel.c and vmce_amd.c).I''m not convinced of the need, and would prefer aiming at a shared implementation unless issues arise that make this impossible. Jan
On 07/03/12 09:16, Jan Beulich wrote:>>>> On 02.07.12 at 19:32, "Liu, Jinsong" <jinsong.liu@intel.com> wrote: >> Thanks AMD''s feedback :) >> >> This vMCE design foils is basically for Intel MCA, involving many details >> specific to Intel. >> I agree that for x86 Intel and AMD can share logic in many fields. However, >> for MCA logic Intel and AMD are quite different, like >> 1. MSRs interface, e.g. MCG_CAP, MCi_MICS, MCi_CTL2, etc; >> 2. error injection, AMD provide NMI/single MCE/broadcast MCE, while in our >> design only concern broadcast MCE# (and pretend to expose CMCI); >> 3. MCE handler: currently in xen Intel and AMD mce use different triggle >> method and mce handler; >> >> Considering the big difference, I suggest we separately provide Intel vMCE >> and AMD vMCE (i.e. vmce_intel.c and vmce_amd.c). > > I''m not convinced of the need, and would prefer aiming at a > shared implementation unless issues arise that make this > impossible.I have patches ready that do that. About 80% of mce_intel.c is not Intel specific. I am just waiting for the feature freeze to end... Christoph -- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85689 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
> I''m not convinced of the need, and would prefer aiming at a > shared implementation unless issues arise that make this > impossible.It does sound odd. Yes, Intel and AMD have differences around CMCI ... but we are never going to send a CMCI to a guest (there is no point, it can''t do anything useful with the information, it may do something pointlessly stupid like stop using a guest physical page). The only reason I suggested making MCG_CAP pretend that CMCI was supported was a small optimization ... if a Linux guest sees that CMCI is supported, it will not poll the machine check banks looking for corrected errors. -Tony
On 07/03/12 15:26, Luck, Tony wrote:>> I''m not convinced of the need, and would prefer aiming at a >> shared implementation unless issues arise that make this >> impossible. > > It does sound odd. Yes, Intel and AMD have differences around CMCI ... but we are never > going to send a CMCI to a guest (there is no point, it can''t do anything useful with the > information, it may do something pointlessly stupid like stop using a guest physical page). > The only reason I suggested making MCG_CAP pretend that CMCI was supported was a > small optimization ... if a Linux guest sees that CMCI is supported, it will not poll the machine > check banks looking for corrected errors.Are you talking about PV or HVM guest? For HVM guests yes it makes sense to disable CMCI in MCG_CAP for both AMD and Intel. PV guests never read MCE MSRs directly. They install a trap handler and use the hypercall to fetch the error telemetry. The xen interface is common to both AMD and Intel - it''s basically just an array of bytes. The first bytes tell you how you can cast and interpret the bytes. That *content* is specific to AMD or Intel. How much logic can be shared is almost a matter of software design and not hardware design. Christoph -- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85689 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
>>> On 03.07.12 at 16:50, Christoph Egger <Christoph.Egger@amd.com> wrote: > On 07/03/12 15:26, Luck, Tony wrote: > >>> I''m not convinced of the need, and would prefer aiming at a >>> shared implementation unless issues arise that make this >>> impossible. >> >> It does sound odd. Yes, Intel and AMD have differences around CMCI ... but > we are never >> going to send a CMCI to a guest (there is no point, it can''t do anything > useful with the >> information, it may do something pointlessly stupid like stop using a guest > physical page). >> The only reason I suggested making MCG_CAP pretend that CMCI was supported > was a >> small optimization ... if a Linux guest sees that CMCI is supported, it will > not poll the machine >> check banks looking for corrected errors. > > > Are you talking about PV or HVM guest? > > For HVM guests yes it makes sense to disable CMCI in MCG_CAP for both > AMD and Intel."enable" you mean? Jan
[This email is either empty or too large to be displayed at this time]
On 07/03/12 17:08, Jan Beulich wrote:>>>> On 03.07.12 at 16:50, Christoph Egger <Christoph.Egger@amd.com> wrote: >> On 07/03/12 15:26, Luck, Tony wrote: >> >>>> I''m not convinced of the need, and would prefer aiming at a >>>> shared implementation unless issues arise that make this >>>> impossible. >>> >>> It does sound odd. Yes, Intel and AMD have differences around CMCI ... but >> we are never >>> going to send a CMCI to a guest (there is no point, it can''t do anything >> useful with the >>> information, it may do something pointlessly stupid like stop using a guest >> physical page). >>> The only reason I suggested making MCG_CAP pretend that CMCI was supported >> was a >>> small optimization ... if a Linux guest sees that CMCI is supported, it will >> not poll the machine >>> check banks looking for corrected errors. >> >> >> Are you talking about PV or HVM guest? >> >> For HVM guests yes it makes sense to disable CMCI in MCG_CAP for both >> AMD and Intel. > > "enable" you mean?Oh, sorry. I misread one sentence. If CMCI for the guest is really just emulated (= 100% software) then yes, enable for both AMD and Intel is fine. Christoph -- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85689 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632