Dan Magenheimer
2008-Jul-01 17:33 UTC
[Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Various versions of Linux under various circumstances select TSC as the primary clocksource for the kernel. This is especially true for uniprocessor kernels, but also in some cases for multiprocessor kernels. In most cases, this is because a processor bit (tsc_invariant? constant_tsc?) is passed through directly from the hardware via Xen and tested by the hvm guest and the result implies that the TSC is "stable". I'd like to propose that, for a Xen hvm guest, TSC should NEVER be considered stable. For at least these reasons: 1) Often this test is done once at guest boot; if the guest migrates to another machine without the bit set, time will be erratic. 2) I *think* that in some cases even within the same system, TSC values will skew somewhat. Since a uniprocessor guest will often be rescheduled to a different pcpu by Xen, the underlying tsc may appear erratic. Comments (especially from Intel and AMD)? If agreed, could Intel and AMD provide patches so that hvm reads of the bits return "false"? Or will this cause other problems? Another alternative would be to trap all rdtsc's and emulate them but this probably will not be easy and may have significant performance implications. But perhaps it should be an option? Dan ==================================Thanks... for the memory I really could use more / My throughput's on the floor The balloon is flat / My swap disk's fat / I've OOM's in store Overcommitted so much (with apologies to the late great Bob Hope) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-01 17:40 UTC
Re: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
On 1/7/08 18:33, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> Another alternative would be to trap all rdtsc''s and emulate > them but this probably will not be easy and may have > significant performance implications. But perhaps it should > be an option?I think this would be useful, even if just for diagnosing guest time issues. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jul-01 21:43 UTC
[Xen-devel] RE: Guest TSC and Xen (Intel and AMD feedback please)
Hi Andy -- Thanks for the reply. No, I don''t think the TSC offset capabilities in VT are sufficient. If you are migrating from a TSC-synchronized SMP system and TSC was selected as the clocksource by the guest at boot *because* TSC is always synchronized, and then you migrate to a system where TSC is not synchronized, Xen can synchronize it once at migrate time but then the TSC''s on the target system will immediately start to diverge. So TSC might be a reasonable clocksource on the first system but not on the target system. One could of course use CPUID to disallow a TSC-unsynchronized host as a suitable target for a TSC-synchronized-assumed guest, but that seems overly restrictive, especially if TSC wasn''t selected as the clocksource for the guest and/or the guest (and its apps) isn''t particularly time-sensitive.> I don''t know why you want to single out TSC here.I''m singleing it out because it is a per-cpu clock rather than a platform timer... a platform timer can be (and indeed is) offset''ed on migration and that is sufficient if it is selected as the clocksource.> That is what Linux is testing for anyways. If it decides it is > ok it is fine.Not sure... if Linux thinks it is running on a uniprocessor, but Xen reschedules this uniprocessor Linux guest on a different processor on the same physical SMP system, does Xen adjust the potential TSC difference? I could be wrong, but I think not.> The reason why it is an advantage to try to make TSC btw > is that it is *much* faster than any other timer and there > are definitely workloads that are very timer intensive.Yes, understood, but if a timer-intensive application makes the assumption that TSC is synchronized and thus will never go backwards, but TSC is not synchronized and it DOES (apparently) go backwards due to Xen scheduler or migration, a slower timer might have been preferred. Dan> -----Original Message----- > From: Andi Kleen [mailto:andi@firstfloor.org] > Sent: Tuesday, July 01, 2008 2:42 PM > To: dan.magenheimer@oracle.com > Cc: Xen-Devel (E-mail) > Subject: Re: Guest TSC and Xen (Intel and AMD feedback please) > > > "Dan Magenheimer" <dan.magenheimer@oracle.com> writes: > > > > 1) Often this test is done once at guest boot; if the guest > > migrates to another machine without the bit set, time > > will be erratic. > > First it doesn''t have to be. Xen could use the TSC offset capabilities > in VT to simulate a relatively [you would likely get a small hickup > during the migration, but nothing really bad] stable TSC between > machines per guest (assuming the TSC does not diverge by itself). > That would require a TSC negotation phase during migration. > > Then the tests are just a special instance of a CPUID bit test > [although they sometimes check other things too, but from this > perspective they are the same] and when your CPUID and other > configuration values are not controlled between different systems you > migrate to you cannot safely migrate anyways because everything else > that depends on CPUID features will fail too. > > I don''t know why you want to single out TSC here. > > > 2) I *think* that in some cases even within the same system, > > TSC values will skew somewhat. Since a uniprocessor guest > > will often be rescheduled to a different pcpu by Xen, > > the underlying tsc may appear erratic. > > That is what Linux is testing for anyways. If it decides it is > ok it is fine. > > The reason why it is an advantage to try to make TSC btw > is that it is *much* faster than any other timer and there > are definitely workloads that are very timer intensive. > > -Andi >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2008-Jul-02 01:08 UTC
Re: [Xen-devel] RE: Guest TSC and Xen (Intel and AMD feedback please)
Dan Magenheimer wrote:> Hi Andy -- > > Thanks for the reply. > > No, I don''t think the TSC offset capabilities in VT are sufficient. > If you are migrating from a TSC-synchronized SMP system and TSC > was selected as the clocksource by the guest at boot *because* TSC > is always synchronized, and then you migrate to a system where TSC > is not synchronized, Xen can synchronize it once at migrate time > but then the TSC''s on the target system will immediately start to > diverge. So TSC might be a reasonable clocksource on the first > system but not on the target system. One could of course use > CPUID to disallow a TSC-unsynchronized host as a suitable target > for a TSC-synchronized-assumed guest, but that seems overly restrictive, > especially if TSC wasn''t selected as the clocksource for the guest > and/or the guest (and its apps) isn''t particularly time-sensitive. >Linux (as in, relatively modern kernels, at least) will downgrade the tsc if it starts misbehaving. This can happen, for example, if the tsc seems fine on initial inspection, but then starts misbehaving when the cpu frequency changes. On the other hand, if its a modern kernel you may as well use a pv linux kernel and get a proper pv clocksource. This is an area in which kernel behaviour has been changing a lot over the last few years, so its hard to guess what any given kernel version (+hacked by vendor) will do in this case... J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2008-Jul-02 02:52 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>From: Dan Magenheimer >Sent: 2008年7月2日 1:33 > >Various versions of Linux under various circumstances select >TSC as the primary clocksource for the kernel. This is >especially true for uniprocessor kernels, but also in some >cases for multiprocessor kernels. In most cases, this >is because a processor bit (tsc_invariant? constant_tsc?) >is passed through directly from the hardware via Xen and >tested by the hvm guest and the result implies that the >TSC is "stable".constant_tsc is one of the factors affecting tsc stability, and the keypoint is that one internal variable ''tsc_unstable'' is initialized to zero. That means, kernel considers TSC as stable by default, and you need trigger some factors upon which guest kernel''d like to mark tsc as unstable when checking them. However by far none of those factors seem appliable to all circumstances. For example, even by hidding constant_tsc bit from cpuid, UP guest will still consider tsc as stable, and so does 32bit SMP guest running on Intel processors. There''s one proposal in KVM side, by exposing a faked C2 states into guest. ACPI processor idle path will mark TSC as unstable when trying to enter C2. However that requires guest to enable ACPI processor support, and need some tricky logic to further mark C2 invalid to avoid performance influence after purpose is achieved... When saying this, I''m not opposing your proposal since it does be a light option for some type of guests. My point is just that no reliabe option to mark tsc as unstable, and whether we have done all what can be done in Xen side to tackle this issue... Sorry that I didn''t follow up your long discussion in this area before and thus may lose some context.> >I''d like to propose that, for a Xen hvm guest, TSC should >NEVER be considered stable. For at least these reasons: > >1) Often this test is done once at guest boot; if the guest > migrates to another machine without the bit set, time > will be erratic.Since it''s ''if'', we should treat it as an option, instead of ''NEVER'', right? User may not have migration requirement in some cases, and or the migratable backups are with same bits exactly... :-)>2) I *think* that in some cases even within the same system, > TSC values will skew somewhat. Since a uniprocessor guest > will often be rescheduled to a different pcpu by Xen, > the underlying tsc may appear erratic.Can''t be the tsc kept monotonic to guest, with some tweak on tsc offset? Also Xen is always trying to sync tsc among cores. As long as you don''t run on a box with multiple bus crystals, or a box without constant tsc upon freq change, the tsc drift among cores should be negligible considering the overhead of migration. Then for cases out of ''as long as'', we should mark TSC as unstable, still as an option.> >Comments (especially from Intel and AMD)? > >If agreed, could Intel and AMD provide patches so that hvm >reads of the bits return "false"? Or will this cause other >problems?I vaguely recalled some posts about CPUID virtualization framework. There''s no need to a seperate category, and it looks like this option can be aligned with that part? Thanks, Kevin> >Another alternative would be to trap all rdtsc''s and emulate >them but this probably will not be easy and may have >significant performance implications. But perhaps it should >be an option? >Agree. Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2008-Jul-02 03:21 UTC
RE: [Xen-devel] RE: Guest TSC and Xen (Intel and AMD feedback please)
>From: Dan Magenheimer >Sent: 2008年7月2日 5:43 > >Hi Andy -- > >Thanks for the reply. > >No, I don''t think the TSC offset capabilities in VT are sufficient. >If you are migrating from a TSC-synchronized SMP system and TSC >was selected as the clocksource by the guest at boot *because* TSC >is always synchronized, and then you migrate to a system where TSC >is not synchronized, Xen can synchronize it once at migrate time >but then the TSC''s on the target system will immediately start toIf dest and src boxes are with same bits, saying same tsc freq, then the divergence is a fixed value and TSC offset adjustment is sufficient. Or else TSC offset can''t help.>diverge. So TSC might be a reasonable clocksource on the first >system but not on the target system. One could of course use >CPUID to disallow a TSC-unsynchronized host as a suitable target >for a TSC-synchronized-assumed guest, but that seems overly >restrictive, >especially if TSC wasn''t selected as the clocksource for the guest >and/or the guest (and its apps) isn''t particularly time-sensitive.So if it''s an option instead of a ''NEVER'' clause, I agree as: a) user may mark TSC unstable to migrate among boxes with mismatching TSC bits (bus crystal, cpu freq impact, etc.) b) user may always use TSC as clocksource and then trap RDTSC when migrating to a box with mismatching TSC bits c) user may always use TSC as clocksource when migrating to a box with same TSC bits, by adjusting TSC offset d) migration may be prevented since no reliable methods to ensure a)''s effect. Such prevention then falls into generic CPUID comparison involved in migration> >> I don''t know why you want to single out TSC here. > >I''m singleing it out because it is a per-cpu clock rather >than a platform timer... a platform timer can be (and indeed >is) offset''ed on migration and that is sufficient if it is >selected as the clocksource.The problem is not per-cpu vs platform, IMO. Instead, it''s the problem that currently guest TSC is conveyed by host TSC plus an offset approach, without read trap. If you also virtualize a platform clocksource by a real one, like dedicating a HPET ch, same concern also raises.> >> That is what Linux is testing for anyways. If it decides it is >> ok it is fine. > >Not sure... if Linux thinks it is running on a uniprocessor, >but Xen reschedules this uniprocessor Linux guest on a different >processor on the same physical SMP system, does Xen adjust the >potential TSC difference? I could be wrong, but I think not.Xen can do and should be, since SMP system is driven by same crystal and thus host TSC is synced. But I guess by far Xen hasn''t do it, since the TSC drift (dozen of cycles) is smaller than the overhead to migrate a vcpu. Thus guest won''t observe a backward value in theory.> >> The reason why it is an advantage to try to make TSC btw >> is that it is *much* faster than any other timer and there >> are definitely workloads that are very timer intensive.Curiously, how much downgrade using a platform clock source may be, for a time-intensive workload?> >Yes, understood, but if a timer-intensive application makes >the assumption that TSC is synchronized and thus will never >go backwards, but TSC is not synchronized and it DOES (apparently) >go backwards due to Xen scheduler or migration, a slower timer >might have been preferred.Shouldn''t this be a software bug instead? Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jul-03 01:21 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Hi Kevin --> constant_tsc is one of the factors affecting tsc stability, and > the keypoint is that one internal variable 'tsc_unstable' is > initialized to zero. That means, kernel considers TSC as > stable by default, and you need trigger some factors upon > which guest kernel'd like to mark tsc as unstable when > checking them. However by far none of those factors seem > appliable to all circumstances. > > For example, even by hidding constant_tsc bit from cpuid, > UP guest will still consider tsc as stable, and so does 32bit > SMP guest running on Intel processors.Yes, looking over some Linux code, I see you are right. In one version of RH, if the processor vendor is Intel, it always assumes tsc is stable!> Since it's 'if', we should treat it as an option, instead of 'NEVER', > right? User may not have migration requirement in some cases, > and or the migratable backups are with same bits exactly... :-)Yes, if it could be done, I agree an option would be better than 'NEVER'. But Keir is not too keen on more time-related options.> >2) I *think* that in some cases even within the same system, > > TSC values will skew somewhat. Since a uniprocessor guest > > will often be rescheduled to a different pcpu by Xen, > > the underlying tsc may appear erratic. > > Can't be the tsc kept monotonic to guest, with some tweak on > tsc offset? Also Xen is always trying to sync tsc among cores. > As long as you don't run on a box with multiple bus crystals, > or a box without constant tsc upon freq change, the tsc drift > among cores should be negligible considering the overhead of > migration. Then for cases out of 'as long as', we should mark > TSC as unstable, still as an option.No Xen doesn't actally sync the tsc's, just maintains its own software offset variables. I suppose it could periodically check to make sure the tsc skew is within some reasonable value, but after the guest boots, it is probably too late.> I vaguely recalled some posts about CPUID virtualization > framework. There's no need to a seperate category, and it > looks like this option can be aligned with that part?Yes, good point. I wasn't following that closely either but this could be covered by it.> >Another alternative would be to trap all rdtsc's and emulate > >them but this probably will not be easy and may have > >significant performance implications. But perhaps it should > >be an option? > > Agree.Is this something that you (or Intel in general) could look at? I would be happy to participate but I don't think I understand VT well enough. Once the trap occurs, I suppose Xen system time could be used as the virtual TSC, possibly scaled up. Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jul-03 01:26 UTC
RE: [Xen-devel] RE: Guest TSC and Xen (Intel and AMD feedback please)
Hi Kevin --> a) user may mark TSC unstable to migrate among boxes with > mismatching TSC bits (bus crystal, cpu freq impact, etc.) > > b) user may always use TSC as clocksource and then trap RDTSC > when migrating to a box with mismatching TSC bits > > c) user may always use TSC as clocksource when migrating to a > box with same TSC bits, by adjusting TSC offset > > d) migration may be prevented since no reliable methods to ensure > a)'s effect. Such prevention then falls into generic CPUID comparison > involved in migrationI think this is an excellent summary of the choices.> >> I don't know why you want to single out TSC here. > > > >I'm singleing it out because it is a per-cpu clock rather > >than a platform timer... a platform timer can be (and indeed > >is) offset'ed on migration and that is sufficient if it is > >selected as the clocksource. > > The problem is not per-cpu vs platform, IMO. Instead, it's the > problem that currently guest TSC is conveyed by host TSC plus > an offset approach, without read trap. If you also virtualize a > platform clocksource by a real one, like dedicating a HPET ch, > same concern also raises.Yes, you are right.> >> That is what Linux is testing for anyways. If it decides it is > >> ok it is fine. > > > >Not sure... if Linux thinks it is running on a uniprocessor, > >but Xen reschedules this uniprocessor Linux guest on a different > >processor on the same physical SMP system, does Xen adjust the > >potential TSC difference? I could be wrong, but I think not. > > Xen can do and should be, since SMP system is driven by same > crystal and thus host TSC is synced. But I guess by far Xen hasn't > do it, since the TSC drift (dozen of cycles) is smaller than > the overhead > to migrate a vcpu. Thus guest won't observe a backward value in > theory.I'm not sure this is always the case (though the patch I posted earlier today may indicate there is something else going on that has led to my assumption that TSC was skewing worse than dozens of cycles).> >> The reason why it is an advantage to try to make TSC btw > >> is that it is *much* faster than any other timer and there > >> are definitely workloads that are very timer intensive. > > Curiously, how much downgrade using a platform clock source may be, > for a time-intensive workload?A good question. We have a workload that spends >10% of its time doing gettimeoffset_tsc()... not sure if that is realistic but it would be interesting to measure that if it used an hpet instead or if rdtsc was fully emulated.> >Yes, understood, but if a timer-intensive application makes > >the assumption that TSC is synchronized and thus will never > >go backwards, but TSC is not synchronized and it DOES (apparently) > >go backwards due to Xen scheduler or migration, a slower timer > >might have been preferred. > > Shouldn't this be a software bug instead?If the application is smart enough to check the TSC bits when it launches, but stability changes later due to migration/scheduling, I'm not sure htis is an application bug. Or did I misunderstand your question? Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2008-Jul-04 00:56 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >Sent: 2008年7月3日 9:21 >> >> Can''t be the tsc kept monotonic to guest, with some tweak on >> tsc offset? Also Xen is always trying to sync tsc among cores. >> As long as you don''t run on a box with multiple bus crystals, >> or a box without constant tsc upon freq change, the tsc drift >> among cores should be negligible considering the overhead of >> migration. Then for cases out of ''as long as'', we should mark >> TSC as unstable, still as an option. > >No Xen doesn''t actally sync the tsc''s, just maintains its own >software offset variables. I suppose it could periodically check >to make sure the tsc skew is within some reasonable value, >but after the guest boots, it is probably too late.You are right. Xen only sync TSCs at boot time and then calibrate TSC pace independently on each cpu. If some factors drag TSC in the middle, then we may observe larger drift. But now I''m not sure whether periodically sync among cpus is light enough, if taking a similar convergence algothrim as boot time. But still, if only TSC drift exists (caused by occasional events instead of a constantly increasing drift from freq difference), this monotonic can be easily exposed to the guest, like derived from xen system time to adjust TSC offset at migration...> >> >Another alternative would be to trap all rdtsc''s and emulate >> >them but this probably will not be easy and may have >> >significant performance implications. But perhaps it should >> >be an option? >> >> Agree. > >Is this something that you (or Intel in general) could look at? >I would be happy to participate but I don''t think I understand >VT well enough. Once the trap occurs, I suppose Xen system time >could be used as the virtual TSC, possibly scaled up. >There should be tiny related to VT, as only turning on some bit to allow RDTSC trapping and then the rest stuff should be common how to handle it. We''ll take a look, but can''t commit the time due to other scheduled bandwidth. But if you''d like to jump in early we definitely can help with VT side. Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jul-04 17:31 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
> >Is this something that you (or Intel in general) could look at? > >I would be happy to participate but I don't think I understand > >VT well enough. Once the trap occurs, I suppose Xen system time > >could be used as the virtual TSC, possibly scaled up. > > > > There should be tiny related to VT, as only turning on some bit to > allow RDTSC trapping and then the rest stuff should be common > how to handle it. We'll take a look, but can't commit the time due > to other scheduled bandwidth. But if you'd like to jump in early > we definitely can help with VT side.If you can post a patch with code that: 1) declares a boolean global variable: softtsc = 0 2) if the variable is set, a rdtsc instruction in any hvm domain causes a trap 3) the trap handler just does a physical rdtsc and returns then I could probably take it from there. Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2008-Jul-08 01:38 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >Sent: 2008年7月5日 1:32 > >> >Is this something that you (or Intel in general) could look at? >> >I would be happy to participate but I don''t think I understand >> >VT well enough. Once the trap occurs, I suppose Xen system time >> >could be used as the virtual TSC, possibly scaled up. >> > >> >> There should be tiny related to VT, as only turning on some bit to >> allow RDTSC trapping and then the rest stuff should be common >> how to handle it. We''ll take a look, but can''t commit the time due >> to other scheduled bandwidth. But if you''d like to jump in early >> we definitely can help with VT side. > >If you can post a patch with code that: > >1) declares a boolean global variable: softtsc = 0 >2) if the variable is set, a rdtsc instruction in any hvm > domain causes a trap >3) the trap handler just does a physical rdtsc and returns > >then I could probably take it from there.OK, and Li in CC will take a look and bake a patch for you. Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhang, Li
2008-Jul-08 02:28 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Hi, Dan I just add some code to produce vmexit. The function vmx_rdtsc_intercept() is not completed. It is needed to add some code to get the guest tsc. Thanks --Li> -----Original Message----- > From: Tian, Kevin > Sent: Tuesday, July 08, 2008 9:39 AM > To: ''dan.magenheimer@oracle.com''; Xen-Devel (E-mail) > Cc: Zhang, Li > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) > > >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >Sent: 2008年7月5日 1:32 > > > >> >Is this something that you (or Intel in general) could look at? > >> >I would be happy to participate but I don''t think I understand > >> >VT well enough. Once the trap occurs, I suppose Xen system time > >> >could be used as the virtual TSC, possibly scaled up. > >> > > >> > >> There should be tiny related to VT, as only turning on some bit to > >> allow RDTSC trapping and then the rest stuff should be common > >> how to handle it. We''ll take a look, but can''t commit the time due > >> to other scheduled bandwidth. But if you''d like to jump in early > >> we definitely can help with VT side. > > > >If you can post a patch with code that: > > > >1) declares a boolean global variable: softtsc = 0 > >2) if the variable is set, a rdtsc instruction in any hvm > > domain causes a trap > >3) the trap handler just does a physical rdtsc and returns > > > >then I could probably take it from there. > > OK, and Li in CC will take a look and bake a patch for you. > > Thanks, > Kevin_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhang, Li
2008-Jul-08 02:59 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Hi, Dan This seems not to be enough. I will cook another patch.> -----Original Message----- > From: Zhang, Li > Sent: Tuesday, July 08, 2008 10:28 AM > To: Tian, Kevin; ''dan.magenheimer@oracle.com''; ''Xen-Devel (E-mail)'' > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) > > Hi, Dan > > I just add some code to produce vmexit. The function vmx_rdtsc_intercept() > is not completed. It is needed to add some code to get the guest tsc. > > Thanks > --Li > > > -----Original Message----- > > From: Tian, Kevin > > Sent: Tuesday, July 08, 2008 9:39 AM > > To: ''dan.magenheimer@oracle.com''; Xen-Devel (E-mail) > > Cc: Zhang, Li > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) > > > > >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > >Sent: 2008年7月5日 1:32 > > > > > >> >Is this something that you (or Intel in general) could look at? > > >> >I would be happy to participate but I don''t think I understand > > >> >VT well enough. Once the trap occurs, I suppose Xen system time > > >> >could be used as the virtual TSC, possibly scaled up. > > >> > > > >> > > >> There should be tiny related to VT, as only turning on some bit to > > >> allow RDTSC trapping and then the rest stuff should be common > > >> how to handle it. We''ll take a look, but can''t commit the time due > > >> to other scheduled bandwidth. But if you''d like to jump in early > > >> we definitely can help with VT side. > > > > > >If you can post a patch with code that: > > > > > >1) declares a boolean global variable: softtsc = 0 > > >2) if the variable is set, a rdtsc instruction in any hvm > > > domain causes a trap > > >3) the trap handler just does a physical rdtsc and returns > > > > > >then I could probably take it from there. > > > > OK, and Li in CC will take a look and bake a patch for you. > > > > Thanks, > > Kevin_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jul-08 03:34 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Thanks Kevin and Li -- A couple of questions: 1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot option, would this work: static int opt_softtsc = 0; boolean_param("softtsc", opt_softtsc); : min = /* original code */ if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; 2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd regs->eax from tmptsc, are we almost done? 3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called to force the trap? 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code somewhere to emulate WRMSR of tsc? Sorry I am not very familar with the details of the ia32 instruction set. Thanks, Dan> -----Original Message----- > From: Zhang, Li [mailto:li.zhang@intel.com] > Sent: Monday, July 07, 2008 9:00 PM > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Tian, Kevin > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > feedback please) > > > Hi, Dan > This seems not to be enough. I will cook another patch. > > > -----Original Message----- > > From: Zhang, Li > > Sent: Tuesday, July 08, 2008 10:28 AM > > To: Tian, Kevin; 'dan.magenheimer@oracle.com'; 'Xen-Devel (E-mail)' > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > feedback please) > > > > Hi, Dan > > > > I just add some code to produce vmexit. The function > vmx_rdtsc_intercept() > > is not completed. It is needed to add some code to get the > guest tsc. > > > > Thanks > > --Li > > > > > -----Original Message----- > > > From: Tian, Kevin > > > Sent: Tuesday, July 08, 2008 9:39 AM > > > To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail) > > > Cc: Zhang, Li > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and > AMD feedback please) > > > > > > >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > >Sent: 2008年7月5日 1:32 > > > > > > > >> >Is this something that you (or Intel in general) > could look at? > > > >> >I would be happy to participate but I don't think I > understand > > > >> >VT well enough. Once the trap occurs, I suppose > Xen system time > > > >> >could be used as the virtual TSC, possibly scaled up. > > > >> > > > > >> > > > >> There should be tiny related to VT, as only turning > on some bit to > > > >> allow RDTSC trapping and then the rest stuff should be common > > > >> how to handle it. We'll take a look, but can't > commit the time due > > > >> to other scheduled bandwidth. But if you'd like to > jump in early > > > >> we definitely can help with VT side. > > > > > > > >If you can post a patch with code that: > > > > > > > >1) declares a boolean global variable: softtsc = 0 > > > >2) if the variable is set, a rdtsc instruction in any hvm > > > > domain causes a trap > > > >3) the trap handler just does a physical rdtsc and returns > > > > > > > >then I could probably take it from there. > > > > > > OK, and Li in CC will take a look and bake a patch for you. > > > > > > Thanks, > > > Kevin >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2008-Jul-08 04:21 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
>From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >Sent: 2008年7月8日 11:34 > >1) If the EXITING flag to be set in vmcs.c is to be controlled >by a xen boot option, would this work: > >static int opt_softtsc = 0; >boolean_param("softtsc", opt_softtsc); > : >min = /* original code */ >if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING;yes.> >2) In vmx_rdtsc_intercept(/* need regs as param*/) if we >change the rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and >then set regs->edx amd regs->eax from tmptsc, are we almost done?Yes, and similar example is under case EXIT_REASON_CPUID.> >3) Des Linux or Windows use the CR4.TSD flag and, if so, what >code gets called to force the trap?My feeling is that both Linux and Windows allows user directly read to TSC, and thus CR4.TSD is unlikely to be set. But it needs confirmation. If you really want to trap such setting, hvm_set_cr4 is the point and you may further reference vmx_update_guest_cr if it''s vmx specific (but I guess it should be common). But I doubt the point to support such model, since two levels trapping (Xen trap, inject fault, guest kernel read, xen trap, ...) which adds much delay on Xen maintained guest tsc.> >4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there >already code somewhere to emulate WRMSR of tsc?I''m not sure about this one, and others may answer. Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhang, Li
2008-Jul-08 06:49 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Hi, Dan This is the patch which I think it has been completed VT. But there is an instable issue. The reason may be from that this interception is not bound to timer mode yet. Thanks --Li> -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: Tuesday, July 08, 2008 11:34 AM > To: Zhang, Li; Xen-Devel (E-mail) > Cc: Tian, Kevin; dan.magenheimer@oracle.com > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) > > Thanks Kevin and Li -- > > A couple of questions: > > 1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen boot > option, would this work: > > static int opt_softtsc = 0; > boolean_param("softtsc", opt_softtsc); > : > min = /* original code */ > if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; > > 2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the > rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd > regs->eax from tmptsc, are we almost done? > > 3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets called > to force the trap? > > 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code > somewhere to emulate WRMSR of tsc? > > Sorry I am not very familar with the details of the ia32 instruction set. > > Thanks, > Dan > > > -----Original Message----- > > From: Zhang, Li [mailto:li.zhang@intel.com] > > Sent: Monday, July 07, 2008 9:00 PM > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > Cc: Tian, Kevin > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > feedback please) > > > > > > Hi, Dan > > This seems not to be enough. I will cook another patch. > > > > > -----Original Message----- > > > From: Zhang, Li > > > Sent: Tuesday, July 08, 2008 10:28 AM > > > To: Tian, Kevin; ''dan.magenheimer@oracle.com''; ''Xen-Devel (E-mail)'' > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > feedback please) > > > > > > Hi, Dan > > > > > > I just add some code to produce vmexit. The function > > vmx_rdtsc_intercept() > > > is not completed. It is needed to add some code to get the > > guest tsc. > > > > > > Thanks > > > --Li > > > > > > > -----Original Message----- > > > > From: Tian, Kevin > > > > Sent: Tuesday, July 08, 2008 9:39 AM > > > > To: ''dan.magenheimer@oracle.com''; Xen-Devel (E-mail) > > > > Cc: Zhang, Li > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and > > AMD feedback please) > > > > > > > > >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > > >Sent: 2008年7月5日 1:32 > > > > > > > > > >> >Is this something that you (or Intel in general) > > could look at? > > > > >> >I would be happy to participate but I don''t think I > > understand > > > > >> >VT well enough. Once the trap occurs, I suppose > > Xen system time > > > > >> >could be used as the virtual TSC, possibly scaled up. > > > > >> > > > > > >> > > > > >> There should be tiny related to VT, as only turning > > on some bit to > > > > >> allow RDTSC trapping and then the rest stuff should be common > > > > >> how to handle it. We''ll take a look, but can''t > > commit the time due > > > > >> to other scheduled bandwidth. But if you''d like to > > jump in early > > > > >> we definitely can help with VT side. > > > > > > > > > >If you can post a patch with code that: > > > > > > > > > >1) declares a boolean global variable: softtsc = 0 > > > > >2) if the variable is set, a rdtsc instruction in any hvm > > > > > domain causes a trap > > > > >3) the trap handler just does a physical rdtsc and returns > > > > > > > > > >then I could probably take it from there. > > > > > > > > OK, and Li in CC will take a look and bake a patch for you. > > > > > > > > Thanks, > > > > Kevin > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhang, Li
2008-Jul-08 06:58 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
In fact, the issue is from guest. In the guest''s terminal, it shows that "your time source seems to be instable..."> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li > Sent: Tuesday, July 08, 2008 2:49 PM > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Tian, Kevin > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) > > Hi, Dan > > This is the patch which I think it has been completed VT. But there is an > instable issue. The reason may be from that this interception is not bound > to timer mode yet. > > Thanks > --Li > > > -----Original Message----- > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > Sent: Tuesday, July 08, 2008 11:34 AM > > To: Zhang, Li; Xen-Devel (E-mail) > > Cc: Tian, Kevin; dan.magenheimer@oracle.com > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) > > > > Thanks Kevin and Li -- > > > > A couple of questions: > > > > 1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen > boot > > option, would this work: > > > > static int opt_softtsc = 0; > > boolean_param("softtsc", opt_softtsc); > > : > > min = /* original code */ > > if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; > > > > 2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the > > rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx amd > > regs->eax from tmptsc, are we almost done? > > > > 3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets > called > > to force the trap? > > > > 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code > > somewhere to emulate WRMSR of tsc? > > > > Sorry I am not very familar with the details of the ia32 instruction set. > > > > Thanks, > > Dan > > > > > -----Original Message----- > > > From: Zhang, Li [mailto:li.zhang@intel.com] > > > Sent: Monday, July 07, 2008 9:00 PM > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > Cc: Tian, Kevin > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > feedback please) > > > > > > > > > Hi, Dan > > > This seems not to be enough. I will cook another patch. > > > > > > > -----Original Message----- > > > > From: Zhang, Li > > > > Sent: Tuesday, July 08, 2008 10:28 AM > > > > To: Tian, Kevin; ''dan.magenheimer@oracle.com''; ''Xen-Devel (E-mail)'' > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > feedback please) > > > > > > > > Hi, Dan > > > > > > > > I just add some code to produce vmexit. The function > > > vmx_rdtsc_intercept() > > > > is not completed. It is needed to add some code to get the > > > guest tsc. > > > > > > > > Thanks > > > > --Li > > > > > > > > > -----Original Message----- > > > > > From: Tian, Kevin > > > > > Sent: Tuesday, July 08, 2008 9:39 AM > > > > > To: ''dan.magenheimer@oracle.com''; Xen-Devel (E-mail) > > > > > Cc: Zhang, Li > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and > > > AMD feedback please) > > > > > > > > > > >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > > > >Sent: 2008年7月5日 1:32 > > > > > > > > > > > >> >Is this something that you (or Intel in general) > > > could look at? > > > > > >> >I would be happy to participate but I don''t think I > > > understand > > > > > >> >VT well enough. Once the trap occurs, I suppose > > > Xen system time > > > > > >> >could be used as the virtual TSC, possibly scaled up. > > > > > >> > > > > > > >> > > > > > >> There should be tiny related to VT, as only turning > > > on some bit to > > > > > >> allow RDTSC trapping and then the rest stuff should be common > > > > > >> how to handle it. We''ll take a look, but can''t > > > commit the time due > > > > > >> to other scheduled bandwidth. But if you''d like to > > > jump in early > > > > > >> we definitely can help with VT side. > > > > > > > > > > > >If you can post a patch with code that: > > > > > > > > > > > >1) declares a boolean global variable: softtsc = 0 > > > > > >2) if the variable is set, a rdtsc instruction in any hvm > > > > > > domain causes a trap > > > > > >3) the trap handler just does a physical rdtsc and returns > > > > > > > > > > > >then I could probably take it from there. > > > > > > > > > > OK, and Li in CC will take a look and bake a patch for you. > > > > > > > > > > Thanks, > > > > > Kevin > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhang, Li
2008-Jul-08 09:46 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Hi, Dan Please ignore previous comments. The above attachment is OK. Sorry for that. :(> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li > Sent: Tuesday, July 08, 2008 2:58 PM > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Tian, Kevin > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) > > In fact, the issue is from guest. In the guest''s terminal, it shows that "your > time source seems to be instable..." > > > -----Original Message----- > > From: xen-devel-bounces@lists.xensource.com > > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li > > Sent: Tuesday, July 08, 2008 2:49 PM > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > Cc: Tian, Kevin > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) > > > > Hi, Dan > > > > This is the patch which I think it has been completed VT. But there is an > > instable issue. The reason may be from that this interception is not bound > > to timer mode yet. > > > > Thanks > > --Li > > > > > -----Original Message----- > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > Sent: Tuesday, July 08, 2008 11:34 AM > > > To: Zhang, Li; Xen-Devel (E-mail) > > > Cc: Tian, Kevin; dan.magenheimer@oracle.com > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback > please) > > > > > > Thanks Kevin and Li -- > > > > > > A couple of questions: > > > > > > 1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen > > boot > > > option, would this work: > > > > > > static int opt_softtsc = 0; > > > boolean_param("softtsc", opt_softtsc); > > > : > > > min = /* original code */ > > > if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; > > > > > > 2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the > > > rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx > amd > > > regs->eax from tmptsc, are we almost done? > > > > > > 3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets > > called > > > to force the trap? > > > > > > 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code > > > somewhere to emulate WRMSR of tsc? > > > > > > Sorry I am not very familar with the details of the ia32 instruction > set. > > > > > > Thanks, > > > Dan > > > > > > > -----Original Message----- > > > > From: Zhang, Li [mailto:li.zhang@intel.com] > > > > Sent: Monday, July 07, 2008 9:00 PM > > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > > Cc: Tian, Kevin > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > > feedback please) > > > > > > > > > > > > Hi, Dan > > > > This seems not to be enough. I will cook another patch. > > > > > > > > > -----Original Message----- > > > > > From: Zhang, Li > > > > > Sent: Tuesday, July 08, 2008 10:28 AM > > > > > To: Tian, Kevin; ''dan.magenheimer@oracle.com''; ''Xen-Devel > (E-mail)'' > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > > feedback please) > > > > > > > > > > Hi, Dan > > > > > > > > > > I just add some code to produce vmexit. The function > > > > vmx_rdtsc_intercept() > > > > > is not completed. It is needed to add some code to get the > > > > guest tsc. > > > > > > > > > > Thanks > > > > > --Li > > > > > > > > > > > -----Original Message----- > > > > > > From: Tian, Kevin > > > > > > Sent: Tuesday, July 08, 2008 9:39 AM > > > > > > To: ''dan.magenheimer@oracle.com''; Xen-Devel (E-mail) > > > > > > Cc: Zhang, Li > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and > > > > AMD feedback please) > > > > > > > > > > > > >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > > > > >Sent: 2008年7月5日 1:32 > > > > > > > > > > > > > >> >Is this something that you (or Intel in general) > > > > could look at? > > > > > > >> >I would be happy to participate but I don''t think I > > > > understand > > > > > > >> >VT well enough. Once the trap occurs, I suppose > > > > Xen system time > > > > > > >> >could be used as the virtual TSC, possibly scaled up. > > > > > > >> > > > > > > > >> > > > > > > >> There should be tiny related to VT, as only turning > > > > on some bit to > > > > > > >> allow RDTSC trapping and then the rest stuff should be common > > > > > > >> how to handle it. We''ll take a look, but can''t > > > > commit the time due > > > > > > >> to other scheduled bandwidth. But if you''d like to > > > > jump in early > > > > > > >> we definitely can help with VT side. > > > > > > > > > > > > > >If you can post a patch with code that: > > > > > > > > > > > > > >1) declares a boolean global variable: softtsc = 0 > > > > > > >2) if the variable is set, a rdtsc instruction in any hvm > > > > > > > domain causes a trap > > > > > > >3) the trap handler just does a physical rdtsc and returns > > > > > > > > > > > > > >then I could probably take it from there. > > > > > > > > > > > > OK, and Li in CC will take a look and bake a patch for you. > > > > > > > > > > > > Thanks, > > > > > > Kevin > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dave Winchell
2008-Jul-08 14:48 UTC
Re: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Hi Li, Shouldn''t this + regs->eax = tsc & 0xffff; + regs->edx = tsc >> 32 & 0xffff; be + regs->eax = tsc & 0xffffffff; + regs->edx = tsc >> 32 & 0xffffffff; Thanks, Dave Zhang, Li wrote:>Hi, Dan >Please ignore previous comments. The above attachment is OK. Sorry for that. :( > > > >> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com >> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li >> Sent: Tuesday, July 08, 2008 2:58 PM >> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >> Cc: Tian, Kevin >> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) >> >> In fact, the issue is from guest. In the guest''s terminal, it shows that "your >> time source seems to be instable..." >> >> > -----Original Message----- >> > From: xen-devel-bounces@lists.xensource.com >> > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li >> > Sent: Tuesday, July 08, 2008 2:49 PM >> > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >> > Cc: Tian, Kevin >> > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) >> > >> > Hi, Dan >> > >> > This is the patch which I think it has been completed VT. But there is an >> > instable issue. The reason may be from that this interception is not bound >> > to timer mode yet. >> > >> > Thanks >> > --Li >> > >> > > -----Original Message----- >> > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >> > > Sent: Tuesday, July 08, 2008 11:34 AM >> > > To: Zhang, Li; Xen-Devel (E-mail) >> > > Cc: Tian, Kevin; dan.magenheimer@oracle.com >> > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback >> please) >> > > >> > > Thanks Kevin and Li -- >> > > >> > > A couple of questions: >> > > >> > > 1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen >> > boot >> > > option, would this work: >> > > >> > > static int opt_softtsc = 0; >> > > boolean_param("softtsc", opt_softtsc); >> > > : >> > > min = /* original code */ >> > > if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; >> > > >> > > 2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the >> > > rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx >> amd >> > > regs->eax from tmptsc, are we almost done? >> > > >> > > 3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets >> > called >> > > to force the trap? >> > > >> > > 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code >> > > somewhere to emulate WRMSR of tsc? >> > > >> > > Sorry I am not very familar with the details of the ia32 instruction >> set. >> > > >> > > Thanks, >> > > Dan >> > > >> > > > -----Original Message----- >> > > > From: Zhang, Li [mailto:li.zhang@intel.com] >> > > > Sent: Monday, July 07, 2008 9:00 PM >> > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >> > > > Cc: Tian, Kevin >> > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD >> > > > feedback please) >> > > > >> > > > >> > > > Hi, Dan >> > > > This seems not to be enough. I will cook another patch. >> > > > >> > > > > -----Original Message----- >> > > > > From: Zhang, Li >> > > > > Sent: Tuesday, July 08, 2008 10:28 AM >> > > > > To: Tian, Kevin; ''dan.magenheimer@oracle.com''; ''Xen-Devel >> (E-mail)'' >> > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD >> > > > feedback please) >> > > > > >> > > > > Hi, Dan >> > > > > >> > > > > I just add some code to produce vmexit. The function >> > > > vmx_rdtsc_intercept() >> > > > > is not completed. It is needed to add some code to get the >> > > > guest tsc. >> > > > > >> > > > > Thanks >> > > > > --Li >> > > > > >> > > > > > -----Original Message----- >> > > > > > From: Tian, Kevin >> > > > > > Sent: Tuesday, July 08, 2008 9:39 AM >> > > > > > To: ''dan.magenheimer@oracle.com''; Xen-Devel (E-mail) >> > > > > > Cc: Zhang, Li >> > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and >> > > > AMD feedback please) >> > > > > > >> > > > > > >From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >> > > > > > >Sent: 2008年7月5日 1:32 >> > > > > > > >> > > > > > >> >Is this something that you (or Intel in general) >> > > > could look at? >> > > > > > >> >I would be happy to participate but I don''t think I >> > > > understand >> > > > > > >> >VT well enough. Once the trap occurs, I suppose >> > > > Xen system time >> > > > > > >> >could be used as the virtual TSC, possibly scaled up. >> > > > > > >> > >> > > > > > >> >> > > > > > >> There should be tiny related to VT, as only turning >> > > > on some bit to >> > > > > > >> allow RDTSC trapping and then the rest stuff should be common >> > > > > > >> how to handle it. We''ll take a look, but can''t >> > > > commit the time due >> > > > > > >> to other scheduled bandwidth. But if you''d like to >> > > > jump in early >> > > > > > >> we definitely can help with VT side. >> > > > > > > >> > > > > > >If you can post a patch with code that: >> > > > > > > >> > > > > > >1) declares a boolean global variable: softtsc = 0 >> > > > > > >2) if the variable is set, a rdtsc instruction in any hvm >> > > > > > > domain causes a trap >> > > > > > >3) the trap handler just does a physical rdtsc and returns >> > > > > > > >> > > > > > >then I could probably take it from there. >> > > > > > >> > > > > > OK, and Li in CC will take a look and bake a patch for you. >> > > > > > >> > > > > > Thanks, >> > > > > > Kevin >> > > > >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >> >>------------------------------------------------------------------------ >> >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xensource.com >>http://lists.xensource.com/xen-devel >> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-08 14:56 UTC
Re: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
I would go for (uint32_t)tsc and (uint32_t)(tsc >> 32) as being even clearer. -- Keir On 8/7/08 15:48, "Dave Winchell" <dwinchell@virtualiron.com> wrote:> Hi Li, > > Shouldn''t this > > + regs->eax = tsc & 0xffff; > + regs->edx = tsc >> 32 & 0xffff; > > be > > + regs->eax = tsc & 0xffffffff; > + regs->edx = tsc >> 32 & 0xffffffff; > > Thanks, > Dave > > > Zhang, Li wrote: > >> Hi, Dan >> Please ignore previous comments. The above attachment is OK. Sorry for that. >> :( >> >> >> >>> -----Original Message----- >>> From: xen-devel-bounces@lists.xensource.com >>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li >>> Sent: Tuesday, July 08, 2008 2:58 PM >>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >>> Cc: Tian, Kevin >>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) >>> >>> In fact, the issue is from guest. In the guest''s terminal, it shows that >>> "your >>> time source seems to be instable..." >>> >>>> -----Original Message----- >>>> From: xen-devel-bounces@lists.xensource.com >>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Li >>>> Sent: Tuesday, July 08, 2008 2:49 PM >>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >>>> Cc: Tian, Kevin >>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please) >>>> >>>> Hi, Dan >>>> >>>> This is the patch which I think it has been completed VT. But there is an >>>> instable issue. The reason may be from that this interception is not bound >>>> to timer mode yet. >>>> >>>> Thanks >>>> --Li >>>> >>>>> -----Original Message----- >>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>> Sent: Tuesday, July 08, 2008 11:34 AM >>>>> To: Zhang, Li; Xen-Devel (E-mail) >>>>> Cc: Tian, Kevin; dan.magenheimer@oracle.com >>>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback >>> please) >>>>> >>>>> Thanks Kevin and Li -- >>>>> >>>>> A couple of questions: >>>>> >>>>> 1) If the EXITING flag to be set in vmcs.c is to be controlled by a xen >>>> boot >>>>> option, would this work: >>>>> >>>>> static int opt_softtsc = 0; >>>>> boolean_param("softtsc", opt_softtsc); >>>>> : >>>>> min = /* original code */ >>>>> if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; >>>>> >>>>> 2) In vmx_rdtsc_intercept(/* need regs as param*/) if we change the >>>>> rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and then set regs->edx >>> amd >>>>> regs->eax from tmptsc, are we almost done? >>>>> >>>>> 3) Des Linux or Windows use the CR4.TSD flag and, if so, what code gets >>>> called >>>>> to force the trap? >>>>> >>>>> 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is there already code >>>>> somewhere to emulate WRMSR of tsc? >>>>> >>>>> Sorry I am not very familar with the details of the ia32 instruction >>> set. >>>>> >>>>> Thanks, >>>>> Dan >>>>> >>>>>> -----Original Message----- >>>>>> From: Zhang, Li [mailto:li.zhang@intel.com] >>>>>> Sent: Monday, July 07, 2008 9:00 PM >>>>>> To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) >>>>>> Cc: Tian, Kevin >>>>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD >>>>>> feedback please) >>>>>> >>>>>> >>>>>> Hi, Dan >>>>>> This seems not to be enough. I will cook another patch. >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Zhang, Li >>>>>>> Sent: Tuesday, July 08, 2008 10:28 AM >>>>>>> To: Tian, Kevin; ''dan.magenheimer@oracle.com''; ''Xen-Devel >>> (E-mail)'' >>>>>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD >>>>>> feedback please) >>>>>>> >>>>>>> Hi, Dan >>>>>>> >>>>>>> I just add some code to produce vmexit. The function >>>>>> vmx_rdtsc_intercept() >>>>>>> is not completed. It is needed to add some code to get the >>>>>> guest tsc. >>>>>>> >>>>>>> Thanks >>>>>>> --Li >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Tian, Kevin >>>>>>>> Sent: Tuesday, July 08, 2008 9:39 AM >>>>>>>> To: ''dan.magenheimer@oracle.com''; Xen-Devel (E-mail) >>>>>>>> Cc: Zhang, Li >>>>>>>> Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and >>>>>> AMD feedback please) >>>>>>>> >>>>>>>>> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>>>> Sent: 2008年7月5日 1:32 >>>>>>>>> >>>>>>>>>>> Is this something that you (or Intel in general) >>>>>> could look at? >>>>>>>>>>> I would be happy to participate but I don''t think I >>>>>> understand >>>>>>>>>>> VT well enough. Once the trap occurs, I suppose >>>>>> Xen system time >>>>>>>>>>> could be used as the virtual TSC, possibly scaled up. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> There should be tiny related to VT, as only turning >>>>>> on some bit to >>>>>>>>>> allow RDTSC trapping and then the rest stuff should be common >>>>>>>>>> how to handle it. We''ll take a look, but can''t >>>>>> commit the time due >>>>>>>>>> to other scheduled bandwidth. But if you''d like to >>>>>> jump in early >>>>>>>>>> we definitely can help with VT side. >>>>>>>>> >>>>>>>>> If you can post a patch with code that: >>>>>>>>> >>>>>>>>> 1) declares a boolean global variable: softtsc = 0 >>>>>>>>> 2) if the variable is set, a rdtsc instruction in any hvm >>>>>>>>> domain causes a trap >>>>>>>>> 3) the trap handler just does a physical rdtsc and returns >>>>>>>>> >>>>>>>>> then I could probably take it from there. >>>>>>>> >>>>>>>> OK, and Li in CC will take a look and bake a patch for you. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Kevin >>>>>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >>> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jul-09 00:29 UTC
RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please)
Looks great! I am continuing to code and test. Some preliminary results: 1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() seems to just work! The hvm guest believes it is running on a 1000.185 MHz clock. 2) On my test box, a guest rdtsc without softtsc takes about 30ns. With softtsc, rdtsc takes about 1us. 3) On my user-land test program, I am seeing rare "Time goes backwards", even with hvm_get_guest_time(). I'm trying to track this down. Dan> -----Original Message----- > From: Zhang, Li [mailto:li.zhang@intel.com] > Sent: Tuesday, July 08, 2008 3:47 AM > To: Zhang, Li; dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Tian, Kevin > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > feedback please) > > > Hi, Dan > Please ignore previous comments. The above attachment is OK. > Sorry for that. :( > > > -----Original Message----- > > From: xen-devel-bounces@lists.xensource.com > > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf > Of Zhang, Li > > Sent: Tuesday, July 08, 2008 2:58 PM > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > Cc: Tian, Kevin > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > feedback please) > > > > In fact, the issue is from guest. In the guest's terminal, > it shows that "your > > time source seems to be instable..." > > > > > -----Original Message----- > > > From: xen-devel-bounces@lists.xensource.com > > > [mailto:xen-devel-bounces@lists.xensource.com] On > Behalf Of Zhang, Li > > > Sent: Tuesday, July 08, 2008 2:49 PM > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > Cc: Tian, Kevin > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and > AMD feedback please) > > > > > > Hi, Dan > > > > > > This is the patch which I think it has been completed > VT. But there is an > > > instable issue. The reason may be from that this > interception is not bound > > > to timer mode yet. > > > > > > Thanks > > > --Li > > > > > > > -----Original Message----- > > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > > Sent: Tuesday, July 08, 2008 11:34 AM > > > > To: Zhang, Li; Xen-Devel (E-mail) > > > > Cc: Tian, Kevin; dan.magenheimer@oracle.com > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel > and AMD feedback > > please) > > > > > > > > Thanks Kevin and Li -- > > > > > > > > A couple of questions: > > > > > > > > 1) If the EXITING flag to be set in vmcs.c is to be > controlled by a xen > > > boot > > > > option, would this work: > > > > > > > > static int opt_softtsc = 0; > > > > boolean_param("softtsc", opt_softtsc); > > > > : > > > > min = /* original code */ > > > > if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; > > > > > > > > 2) In vmx_rdtsc_intercept(/* need regs as param*/) > if we change the > > > > rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and > then set regs->edx > > amd > > > > regs->eax from tmptsc, are we almost done? > > > > > > > > 3) Des Linux or Windows use the CR4.TSD flag and, if > so, what code gets > > > called > > > > to force the trap? > > > > > > > > 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is > there already code > > > > somewhere to emulate WRMSR of tsc? > > > > > > > > Sorry I am not very familar with the details of the > ia32 instruction > > set. > > > > > > > > Thanks, > > > > Dan > > > > > > > > > -----Original Message----- > > > > > From: Zhang, Li [mailto:li.zhang@intel.com] > > > > > Sent: Monday, July 07, 2008 9:00 PM > > > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > > > Cc: Tian, Kevin > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > > > feedback please) > > > > > > > > > > > > > > > Hi, Dan > > > > > This seems not to be enough. I will cook another patch. > > > > > > > > > > > -----Original Message----- > > > > > > From: Zhang, Li > > > > > > Sent: Tuesday, July 08, 2008 10:28 AM > > > > > > To: Tian, Kevin; 'dan.magenheimer@oracle.com'; > 'Xen-Devel > > (E-mail)' > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen > (Intel and AMD > > > > > feedback please) > > > > > > > > > > > > Hi, Dan > > > > > > > > > > > > I just add some code to produce vmexit. The function > > > > > vmx_rdtsc_intercept() > > > > > > is not completed. It is needed to add some code > to get the > > > > > guest tsc. > > > > > > > > > > > > Thanks > > > > > > --Li > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Tian, Kevin > > > > > > > Sent: Tuesday, July 08, 2008 9:39 AM > > > > > > > To: 'dan.magenheimer@oracle.com'; Xen-Devel (E-mail) > > > > > > > Cc: Zhang, Li > > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and > > > > > AMD feedback please) > > > > > > > > > > > > > > >From: Dan Magenheimer > [mailto:dan.magenheimer@oracle.com] > > > > > > > >Sent: 2008年7月5日 1:32 > > > > > > > > > > > > > > > >> >Is this something that you (or Intel in general) > > > > > could look at? > > > > > > > >> >I would be happy to participate but I > don't think I > > > > > understand > > > > > > > >> >VT well enough. Once the trap occurs, I suppose > > > > > Xen system time > > > > > > > >> >could be used as the virtual TSC, > possibly scaled up. > > > > > > > >> > > > > > > > > >> > > > > > > > >> There should be tiny related to VT, as > only turning > > > > > on some bit to > > > > > > > >> allow RDTSC trapping and then the rest > stuff should be common > > > > > > > >> how to handle it. We'll take a look, but can't > > > > > commit the time due > > > > > > > >> to other scheduled bandwidth. But if you'd like to > > > > > jump in early > > > > > > > >> we definitely can help with VT side. > > > > > > > > > > > > > > > >If you can post a patch with code that: > > > > > > > > > > > > > > > >1) declares a boolean global variable: softtsc = 0 > > > > > > > >2) if the variable is set, a rdtsc > instruction in any hvm > > > > > > > > domain causes a trap > > > > > > > >3) the trap handler just does a physical > rdtsc and returns > > > > > > > > > > > > > > > >then I could probably take it from there. > > > > > > > > > > > > > > OK, and Li in CC will take a look and bake a > patch for you. > > > > > > > > > > > > > > Thanks, > > > > > > > Kevin > > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jul-09 21:32 UTC
[PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please))
The attached patch adds a new Xen boot option "softtsc" that causes all hvm domains to trap and emulate all rdtsc instructions and fetch Xen system time instead of the hardware tsc. As a result, the processor will appear to be running at 1000.xyz MHz (where xyz indicates some fraction that seems to vary slightly across different boots). I've also added a line of output to "xm debug-key t" to report softsc and tsc-related CPU features. (Keir, though 3.3 functionality freeze is past, this is more of a diagnostic aid and defaults off, so I don't know if you would consider it for 3.3.) Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com> Dan P.S. Thanks to Li Zhang for coding the foundation for this patch. Li, do you want to provide a signed-off-by line too?> -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: Tuesday, July 08, 2008 6:30 PM > To: 'Zhang, Li'; 'Xen-Devel (E-mail)' > Cc: 'Tian, Kevin' > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > feedback please) > > > Looks great! I am continuing to code and test. Some > preliminary results: > > 1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() > seems to just work! The hvm > guest believes it is running on a 1000.185 MHz clock. > > 2) On my test box, a guest rdtsc without softtsc takes about > 30ns. With softtsc, > rdtsc takes about 1us. > > 3) On my user-land test program, I am seeing rare "Time goes > backwards", even with > hvm_get_guest_time(). I'm trying to track this down. > > Dan > > > -----Original Message----- > > From: Zhang, Li [mailto:li.zhang@intel.com] > > Sent: Tuesday, July 08, 2008 3:47 AM > > To: Zhang, Li; dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > Cc: Tian, Kevin > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > feedback please) > > > > > > Hi, Dan > > Please ignore previous comments. The above attachment is OK. > > Sorry for that. :( > > > > > -----Original Message----- > > > From: xen-devel-bounces@lists.xensource.com > > > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf > > Of Zhang, Li > > > Sent: Tuesday, July 08, 2008 2:58 PM > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > Cc: Tian, Kevin > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > feedback please) > > > > > > In fact, the issue is from guest. In the guest's terminal, > > it shows that "your > > > time source seems to be instable..." > > > > > > > -----Original Message----- > > > > From: xen-devel-bounces@lists.xensource.com > > > > [mailto:xen-devel-bounces@lists.xensource.com] On > > Behalf Of Zhang, Li > > > > Sent: Tuesday, July 08, 2008 2:49 PM > > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > > Cc: Tian, Kevin > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and > > AMD feedback please) > > > > > > > > Hi, Dan > > > > > > > > This is the patch which I think it has been completed > > VT. But there is an > > > > instable issue. The reason may be from that this > > interception is not bound > > > > to timer mode yet. > > > > > > > > Thanks > > > > --Li > > > > > > > > > -----Original Message----- > > > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > > > Sent: Tuesday, July 08, 2008 11:34 AM > > > > > To: Zhang, Li; Xen-Devel (E-mail) > > > > > Cc: Tian, Kevin; dan.magenheimer@oracle.com > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel > > and AMD feedback > > > please) > > > > > > > > > > Thanks Kevin and Li -- > > > > > > > > > > A couple of questions: > > > > > > > > > > 1) If the EXITING flag to be set in vmcs.c is to be > > controlled by a xen > > > > boot > > > > > option, would this work: > > > > > > > > > > static int opt_softtsc = 0; > > > > > boolean_param("softtsc", opt_softtsc); > > > > > : > > > > > min = /* original code */ > > > > > if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; > > > > > > > > > > 2) In vmx_rdtsc_intercept(/* need regs as param*/) > > if we change the > > > > > rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and > > then set regs->edx > > > amd > > > > > regs->eax from tmptsc, are we almost done? > > > > > > > > > > 3) Des Linux or Windows use the CR4.TSD flag and, if > > so, what code gets > > > > called > > > > > to force the trap? > > > > > > > > > > 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is > > there already code > > > > > somewhere to emulate WRMSR of tsc? > > > > > > > > > > Sorry I am not very familar with the details of the > > ia32 instruction > > > set. > > > > > > > > > > Thanks, > > > > > Dan > > > > > > > > > > > -----Original Message----- > > > > > > From: Zhang, Li [mailto:li.zhang@intel.com] > > > > > > Sent: Monday, July 07, 2008 9:00 PM > > > > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > > > > Cc: Tian, Kevin > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > > > > feedback please) > > > > > > > > > > > > > > > > > > Hi, Dan > > > > > > This seems not to be enough. I will cook another patch. > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Zhang, Li > > > > > > > Sent: Tuesday, July 08, 2008 10:28 AM > > > > > > > To: Tian, Kevin; 'dan.magenheimer@oracle.com'; > > 'Xen-Devel > > > (E-mail)' > > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen > > (Intel and AMD > > > > > > feedback please) > > > > > > > > > > > > > > Hi, Dan > > > > > > > > > > > > > > I just add some code to produce vmexit. The function > > > > > > vmx_rdtsc_intercept() > > > > > > > is not completed. It is needed to add some code > > to get the > > > > > > guest tsc. > > > > > > > > > > > > > > Thanks > > > > > > > --Li > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Tian, Kevin > > > > > > > > Sent: Tuesday, July 08, 2008 9:39 AM > > > > > > > > To: 'dan.magenheimer@oracle.com'; > Xen-Devel (E-mail) > > > > > > > > Cc: Zhang, Li > > > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen > (Intel and > > > > > > AMD feedback please) > > > > > > > > > > > > > > > > >From: Dan Magenheimer > > [mailto:dan.magenheimer@oracle.com] > > > > > > > > >Sent: 2008年7月5日 1:32 > > > > > > > > > > > > > > > > > >> >Is this something that you (or Intel > in general) > > > > > > could look at? > > > > > > > > >> >I would be happy to participate but I > > don't think I > > > > > > understand > > > > > > > > >> >VT well enough. Once the trap occurs, > I suppose > > > > > > Xen system time > > > > > > > > >> >could be used as the virtual TSC, > > possibly scaled up. > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> There should be tiny related to VT, as > > only turning > > > > > > on some bit to > > > > > > > > >> allow RDTSC trapping and then the rest > > stuff should be common > > > > > > > > >> how to handle it. We'll take a look, but can't > > > > > > commit the time due > > > > > > > > >> to other scheduled bandwidth. But if > you'd like to > > > > > > jump in early > > > > > > > > >> we definitely can help with VT side. > > > > > > > > > > > > > > > > > >If you can post a patch with code that: > > > > > > > > > > > > > > > > > >1) declares a boolean global variable: softtsc = 0 > > > > > > > > >2) if the variable is set, a rdtsc > > instruction in any hvm > > > > > > > > > domain causes a trap > > > > > > > > >3) the trap handler just does a physical > > rdtsc and returns > > > > > > > > > > > > > > > > > >then I could probably take it from there. > > > > > > > > > > > > > > > > OK, and Li in CC will take a look and bake a > > patch for you. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Kevin > > > > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhang, Li
2008-Jul-10 01:48 UTC
RE: [PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please))
OK. Signed-off-by: Li Zhang <li.zhang@intel.com> Thanks> -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: Thursday, July 10, 2008 5:33 AM > To: dan.magenheimer@oracle.com; Zhang, Li; Xen-Devel (E-mail) > Cc: Tian, Kevin > Subject: [PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and > AMD feedback please)) > > The attached patch adds a new Xen boot option "softtsc" that > causes all hvm domains to trap and emulate all rdtsc instructions > and fetch Xen system time instead of the hardware tsc. As > a result, the processor will appear to be running at 1000.xyz MHz > (where xyz indicates some fraction that seems to vary slightly > across different boots). I''ve also added a line of output to > "xm debug-key t" to report softsc and tsc-related CPU features. > > (Keir, though 3.3 functionality freeze is past, this is more of a > diagnostic aid and defaults off, so I don''t know if you would > consider it for 3.3.) > > Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com> > > Dan > > P.S. Thanks to Li Zhang for coding the foundation for this patch. > Li, do you want to provide a signed-off-by line too? > > > -----Original Message----- > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > Sent: Tuesday, July 08, 2008 6:30 PM > > To: ''Zhang, Li''; ''Xen-Devel (E-mail)'' > > Cc: ''Tian, Kevin'' > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > feedback please) > > > > > > Looks great! I am continuing to code and test. Some > > preliminary results: > > > > 1) Substituting hvm_get_guest_time() for hvm_get_guest_tsc() > > seems to just work! The hvm > > guest believes it is running on a 1000.185 MHz clock. > > > > 2) On my test box, a guest rdtsc without softtsc takes about > > 30ns. With softtsc, > > rdtsc takes about 1us. > > > > 3) On my user-land test program, I am seeing rare "Time goes > > backwards", even with > > hvm_get_guest_time(). I''m trying to track this down. > > > > Dan > > > > > -----Original Message----- > > > From: Zhang, Li [mailto:li.zhang@intel.com] > > > Sent: Tuesday, July 08, 2008 3:47 AM > > > To: Zhang, Li; dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > Cc: Tian, Kevin > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > feedback please) > > > > > > > > > Hi, Dan > > > Please ignore previous comments. The above attachment is OK. > > > Sorry for that. :( > > > > > > > -----Original Message----- > > > > From: xen-devel-bounces@lists.xensource.com > > > > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf > > > Of Zhang, Li > > > > Sent: Tuesday, July 08, 2008 2:58 PM > > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > > Cc: Tian, Kevin > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > feedback please) > > > > > > > > In fact, the issue is from guest. In the guest''s terminal, > > > it shows that "your > > > > time source seems to be instable..." > > > > > > > > > -----Original Message----- > > > > > From: xen-devel-bounces@lists.xensource.com > > > > > [mailto:xen-devel-bounces@lists.xensource.com] On > > > Behalf Of Zhang, Li > > > > > Sent: Tuesday, July 08, 2008 2:49 PM > > > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > > > Cc: Tian, Kevin > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and > > > AMD feedback please) > > > > > > > > > > Hi, Dan > > > > > > > > > > This is the patch which I think it has been completed > > > VT. But there is an > > > > > instable issue. The reason may be from that this > > > interception is not bound > > > > > to timer mode yet. > > > > > > > > > > Thanks > > > > > --Li > > > > > > > > > > > -----Original Message----- > > > > > > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > > > > > Sent: Tuesday, July 08, 2008 11:34 AM > > > > > > To: Zhang, Li; Xen-Devel (E-mail) > > > > > > Cc: Tian, Kevin; dan.magenheimer@oracle.com > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel > > > and AMD feedback > > > > please) > > > > > > > > > > > > Thanks Kevin and Li -- > > > > > > > > > > > > A couple of questions: > > > > > > > > > > > > 1) If the EXITING flag to be set in vmcs.c is to be > > > controlled by a xen > > > > > boot > > > > > > option, would this work: > > > > > > > > > > > > static int opt_softtsc = 0; > > > > > > boolean_param("softtsc", opt_softtsc); > > > > > > : > > > > > > min = /* original code */ > > > > > > if (opt_softtsc) min |= CPU_BASED_RDTSC_EXITING; > > > > > > > > > > > > 2) In vmx_rdtsc_intercept(/* need regs as param*/) > > > if we change the > > > > > > rdtscll(tsc) to be tmptsc = hvm_get_guest_tsc() and > > > then set regs->edx > > > > amd > > > > > > regs->eax from tmptsc, are we almost done? > > > > > > > > > > > > 3) Des Linux or Windows use the CR4.TSD flag and, if > > > so, what code gets > > > > > called > > > > > > to force the trap? > > > > > > > > > > > > 4) Does Linux or Windows use RDMSR/WRMSR of tsc? Is > > > there already code > > > > > > somewhere to emulate WRMSR of tsc? > > > > > > > > > > > > Sorry I am not very familar with the details of the > > > ia32 instruction > > > > set. > > > > > > > > > > > > Thanks, > > > > > > Dan > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Zhang, Li [mailto:li.zhang@intel.com] > > > > > > > Sent: Monday, July 07, 2008 9:00 PM > > > > > > > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > > > > > > > Cc: Tian, Kevin > > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen (Intel and AMD > > > > > > > feedback please) > > > > > > > > > > > > > > > > > > > > > Hi, Dan > > > > > > > This seems not to be enough. I will cook another patch. > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Zhang, Li > > > > > > > > Sent: Tuesday, July 08, 2008 10:28 AM > > > > > > > > To: Tian, Kevin; ''dan.magenheimer@oracle.com''; > > > ''Xen-Devel > > > > (E-mail)'' > > > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen > > > (Intel and AMD > > > > > > > feedback please) > > > > > > > > > > > > > > > > Hi, Dan > > > > > > > > > > > > > > > > I just add some code to produce vmexit. The function > > > > > > > vmx_rdtsc_intercept() > > > > > > > > is not completed. It is needed to add some code > > > to get the > > > > > > > guest tsc. > > > > > > > > > > > > > > > > Thanks > > > > > > > > --Li > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: Tian, Kevin > > > > > > > > > Sent: Tuesday, July 08, 2008 9:39 AM > > > > > > > > > To: ''dan.magenheimer@oracle.com''; > > Xen-Devel (E-mail) > > > > > > > > > Cc: Zhang, Li > > > > > > > > > Subject: RE: [Xen-devel] Guest TSC and Xen > > (Intel and > > > > > > > AMD feedback please) > > > > > > > > > > > > > > > > > > >From: Dan Magenheimer > > > [mailto:dan.magenheimer@oracle.com] > > > > > > > > > >Sent: 2008年7月5日 1:32 > > > > > > > > > > > > > > > > > > > >> >Is this something that you (or Intel > > in general) > > > > > > > could look at? > > > > > > > > > >> >I would be happy to participate but I > > > don''t think I > > > > > > > understand > > > > > > > > > >> >VT well enough. Once the trap occurs, > > I suppose > > > > > > > Xen system time > > > > > > > > > >> >could be used as the virtual TSC, > > > possibly scaled up. > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> There should be tiny related to VT, as > > > only turning > > > > > > > on some bit to > > > > > > > > > >> allow RDTSC trapping and then the rest > > > stuff should be common > > > > > > > > > >> how to handle it. We''ll take a look, but can''t > > > > > > > commit the time due > > > > > > > > > >> to other scheduled bandwidth. But if > > you''d like to > > > > > > > jump in early > > > > > > > > > >> we definitely can help with VT side. > > > > > > > > > > > > > > > > > > > >If you can post a patch with code that: > > > > > > > > > > > > > > > > > > > >1) declares a boolean global variable: softtsc = 0 > > > > > > > > > >2) if the variable is set, a rdtsc > > > instruction in any hvm > > > > > > > > > > domain causes a trap > > > > > > > > > >3) the trap handler just does a physical > > > rdtsc and returns > > > > > > > > > > > > > > > > > > > >then I could probably take it from there. > > > > > > > > > > > > > > > > > > OK, and Li in CC will take a look and bake a > > > patch for you. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Kevin > > > > > > > > > > > > > > > _______________________________________________ > > > > Xen-devel mailing list > > > > Xen-devel@lists.xensource.com > > > > http://lists.xensource.com/xen-devel > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-10 09:18 UTC
Re: [PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please))
On 9/7/08 22:32, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> The attached patch adds a new Xen boot option "softtsc" that > causes all hvm domains to trap and emulate all rdtsc instructions > and fetch Xen system time instead of the hardware tsc. As > a result, the processor will appear to be running at 1000.xyz MHz > (where xyz indicates some fraction that seems to vary slightly > across different boots). I''ve also added a line of output to > "xm debug-key t" to report softsc and tsc-related CPU features. > > (Keir, though 3.3 functionality freeze is past, this is more of a > diagnostic aid and defaults off, so I don''t know if you would > consider it for 3.3.)Yes, I will take it, but have the following comments. 1. Why do you define new boolean ''constant_tsc''? Can you just use test_bit(X86_FEATURE_CONSTANT_TSC)? 2. Your change in common/keyhandler.c breaks ia64. :-) 3. Your change to arch/x86/time.c looks unnecessary. 4. Should you catch SVM''s RDTSCP vmexit as well as RDTSC? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jul-10 14:19 UTC
RE: [PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please))
> Yes, I will take it, but have the following comments. > 2. Your change in common/keyhandler.c breaks ia64. :-)Oops! ;-) What''s the best way to handle this? It would be unfortunate to lose valuable debug data just because its arch-dependent but I don''t see any other arch-dependent code in keyhandler.c and I''ll bet you don''t want to start adding ifdef''s nor introduce arch/xxx/keyhandler.c just for this.> 1. Why do you define new boolean ''constant_tsc''? Can you just use > test_bit(X86_FEATURE_CONSTANT_TSC)? > 3. Your change to arch/x86/time.c looks unnecessary.I was thinking that the tests for these features should probably be abstracted (e.g. static inline in a header file or a global function), but wasn''t sure about the best way to deal with the datatypes (e.g. struct cpuinfo_x86) so defaulted to global variables. Both globals are simply for debug output in keyhandler.c so depending on the answer to (2) above, those patch-parts could just go away.> 4. Should you catch SVM''s RDTSCP vmexit as well as RDTSC?I thought I remembered seeing code that reported/lied to guests that the rdtscp feature was not present? Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-10 14:29 UTC
Re: [PATCH] softtsc (was RE: [Xen-devel] Guest TSC and Xen (Intel and AMD feedback please))
Hmm.. okay I think I will throw away the debug-output pieces of the patch. They are logically separate and distinct anyway. -- Keir On 10/7/08 15:19, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:>> Yes, I will take it, but have the following comments. >> 2. Your change in common/keyhandler.c breaks ia64. :-) > > Oops! ;-) What''s the best way to handle this? It would > be unfortunate to lose valuable debug data just because its > arch-dependent but I don''t see any other arch-dependent code > in keyhandler.c and I''ll bet you don''t want to start adding > ifdef''s nor introduce arch/xxx/keyhandler.c just for this. > >> 1. Why do you define new boolean ''constant_tsc''? Can you just use >> test_bit(X86_FEATURE_CONSTANT_TSC)? >> 3. Your change to arch/x86/time.c looks unnecessary. > > I was thinking that the tests for these features should probably > be abstracted (e.g. static inline in a header file or a > global function), but wasn''t sure about the best way to deal > with the datatypes (e.g. struct cpuinfo_x86) so defaulted to > global variables. > > Both globals are simply for debug output in keyhandler.c > so depending on the answer to (2) above, those patch-parts > could just go away. > >> 4. Should you catch SVM''s RDTSCP vmexit as well as RDTSC? > > I thought I remembered seeing code that reported/lied to guests > that the rdtscp feature was not present? > > Thanks, > Dan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel