Carsten Schiers
2009-Mar-07 09:41 UTC
[Xen-devel] Still Time went backwards problems Xen 3.3.1/2.6.18.8
Still have problems with "Time went backwards" messages. Now I am nearly completely at Xen 3.3.1 and Xen kernel 2.6.18.8 for Dom0 and nearly all DomUs. There is only one DomU That is on a customized special kernel for Endian (2.6.21.7-2). Q1: Can a DomU kernel version anyhow be responsible for that effect? Attached you can see the last two days. I noticed that they occur every five minutes, so it''s probably connected with the munin cron job that collects xen domain usage and cpufreq-stats. Nothing more. Delta is still quite huge, around 40 ms. Value is negative. Q2: Is that any better than if it''s positive? Q3: Will the delta be reset or will it go up and down all the time? Currently I use both cpufreq=dom0 and cpuidle. Q4: Can the problem eventually be related to either of them only? Right now, I disabled cpuidle for the time being, to see whether there is a difference. If not, I intend to use xen-unstable next. I attached the current xm dmesg and xm info, so don''t be surprised to miss cpuidle. Well, I am not sure whether I do too much work on this issue, but I think the log message is there for a reason. Best Regards, Carsten. P.S.: There are combinations that are much worse that this. Especially running Xen 3.3.1 and actual Lenny kernel seems to be worse. -----Ursprüngliche Nachricht----- Von: Langsdorf, Mark [mailto:mark.langsdorf@amd.com] Gesendet: Donnerstag, 15. Januar 2009 20:27 An: Carsten Schiers Betreff: RE: RE: RE: RE: [Xen-devel] AMD P-States not recognized for Xen 3.3 and 3.4 If you have a single socket system, then fixing the out of sync issue will probably not fix the TSC issue. If you have a dual or quad socket system, fixing the out of sync error should reduce the frequency and magnitude of the TSC issues. At least, that''s the best I can decipher from my notes. Also, most of the work I did on cpufreq was in 2007, before Dan M. greatly redid Xen hypervisor time-keeping. If an up-to-date kernel/hypervisor still has TSC problems, it may be possible to fix them. I don''t really have time to research it myself, but if you do have problems, please contact me and I''ll see what can be done. -Mark Langsdorf Operating System Research Center AMD> -----Original Message----- > From: Carsten Schiers [mailto:carsten@schiers.de] > Sent: Thursday, January 15, 2009 1:17 PM > To: Langsdorf, Mark > Subject: AW: RE: RE: RE: [Xen-devel] AMD P-States not > recognized for Xen 3.3 and 3.4 > > Thanks Mark, don''t want to get on your nerve, but this will > only fix the > powernow-k8 out of synch issue but not the fact that my 4050e > will have > TSC issues between the two cores when switching frequencies, right? > > Am I right that there is no cure for that currently? And that > I am not > safe > to ignore them, because some kernel wizard is synchronizing > them? ''Cause > up to now I only found a synch in boot.c. > > Sorry for all those questions, it''s quite complicated. My > best times as > code producer habe been some 15 years ago... > > Thanks, > Carsten. > > -----Ursprüngliche Nachricht----- > Von: Langsdorf, Mark [mailto:mark.langsdorf@amd.com] > Gesendet: Donnerstag, 15. Januar 2009 18:23 > An: Carsten Schiers > Betreff: RE: RE: RE: [Xen-devel] AMD P-States not recognized > for Xen 3.3 > and 3.4 > > > Thank you Mark. Please excuse my small knowledge in > > kernel/xen hacking. I understood that I should either > > > > a) create a build of the actual xen-unstable.hg together with > > linux-2.6.18-xen.hg kernel (where you think to have fixed > the problem) > > > > or > > > > b) to include the mentioned fixes into my environment > (which is the > > 3.2.1 Xen and Bastian Blank''s (waldi) kernel (based on 2.6.18, too), > > right? > > > > I''ll try out what will be more easy, but I think it should be a). > > Right. Using the latest unstable kernel & hypervisor gives > you all the patches. If you''re backporting support into > the waldi kernel, you need to make sure you backport all > the relevant patches. > > -Mark Langsdorf > Operating System Research Center > AMD > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Carsten Schiers
2009-Mar-07 22:13 UTC
AW: [Xen-devel] Still Time went backwards problems Xen 3.3.1/2.6.18.8
Leaving out cpuidle doesn''t make any difference: Mar 7 22:45:04 data kernel: Timer ISR/0: Time went backwards: delta=-23283686 ... I think unstable is going to got frozen soon. I guess this is next to try. BTW: again three seconds after munin-cron. BR, Carsten. -----Ursprüngliche Nachricht----- Von: Carsten Schiers Gesendet: Samstag, 7. März 2009 10:42 An: xen-devel Cc: mark.langsdorf; keir.fraser Betreff: [Xen-devel] Still Time went backwards problems Xen 3.3.1/2.6.18.8 Still have problems with "Time went backwards" messages. Now I am nearly completely at Xen 3.3.1 and Xen kernel 2.6.18.8 for Dom0 and nearly all DomUs. There is only one DomU That is on a customized special kernel for Endian (2.6.21.7-2). Q1: Can a DomU kernel version anyhow be responsible for that effect? Attached you can see the last two days. I noticed that they occur every five minutes, so it''s probably connected with the munin cron job that collects xen domain usage and cpufreq-stats. Nothing more. Delta is still quite huge, around 40 ms. Value is negative. Q2: Is that any better than if it''s positive? Q3: Will the delta be reset or will it go up and down all the time? Currently I use both cpufreq=dom0 and cpuidle. Q4: Can the problem eventually be related to either of them only? Right now, I disabled cpuidle for the time being, to see whether there is a difference. If not, I intend to use xen-unstable next. I attached the current xm dmesg and xm info, so don''t be surprised to miss cpuidle. Well, I am not sure whether I do too much work on this issue, but I think the log message is there for a reason. Best Regards, Carsten. P.S.: There are combinations that are much worse that this. Especially running Xen 3.3.1 and actual Lenny kernel seems to be worse. -----Ursprüngliche Nachricht----- Von: Langsdorf, Mark [mailto:mark.langsdorf@amd.com] Gesendet: Donnerstag, 15. Januar 2009 20:27 An: Carsten Schiers Betreff: RE: RE: RE: RE: [Xen-devel] AMD P-States not recognized for Xen 3.3 and 3.4 If you have a single socket system, then fixing the out of sync issue will probably not fix the TSC issue. If you have a dual or quad socket system, fixing the out of sync error should reduce the frequency and magnitude of the TSC issues. At least, that''s the best I can decipher from my notes. Also, most of the work I did on cpufreq was in 2007, before Dan M. greatly redid Xen hypervisor time-keeping. If an up-to-date kernel/hypervisor still has TSC problems, it may be possible to fix them. I don''t really have time to research it myself, but if you do have problems, please contact me and I''ll see what can be done. -Mark Langsdorf Operating System Research Center AMD> -----Original Message----- > From: Carsten Schiers [mailto:carsten@schiers.de] > Sent: Thursday, January 15, 2009 1:17 PM > To: Langsdorf, Mark > Subject: AW: RE: RE: RE: [Xen-devel] AMD P-States not recognized for > Xen 3.3 and 3.4 > > Thanks Mark, don''t want to get on your nerve, but this will only fix > the > powernow-k8 out of synch issue but not the fact that my 4050e will > have TSC issues between the two cores when switching frequencies, > right? > > Am I right that there is no cure for that currently? And that I am not> safe to ignore them, because some kernel wizard is synchronizing them?> ''Cause up to now I only found a synch in boot.c. > > Sorry for all those questions, it''s quite complicated. My best times > as code producer habe been some 15 years ago... > > Thanks, > Carsten. > > -----Ursprüngliche Nachricht----- > Von: Langsdorf, Mark [mailto:mark.langsdorf@amd.com] > Gesendet: Donnerstag, 15. Januar 2009 18:23 > An: Carsten Schiers > Betreff: RE: RE: RE: [Xen-devel] AMD P-States not recognized for Xen > 3.3 and 3.4 > > > Thank you Mark. Please excuse my small knowledge in kernel/xen > > hacking. I understood that I should either > > > > a) create a build of the actual xen-unstable.hg together with > > linux-2.6.18-xen.hg kernel (where you think to have fixed > the problem) > > > > or > > > > b) to include the mentioned fixes into my environment > (which is the > > 3.2.1 Xen and Bastian Blank''s (waldi) kernel (based on 2.6.18, too),> > right? > > > > I''ll try out what will be more easy, but I think it should be a). > > Right. Using the latest unstable kernel & hypervisor gives you all > the patches. If you''re backporting support into the waldi kernel, you> need to make sure you backport all the relevant patches. > > -Mark Langsdorf > Operating System Research Center > AMD > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Carsten Schiers
2009-Mar-08 12:56 UTC
AW: [Xen-devel] Still Time went backwards problems Xen 3.3.1/2.6.18.8
Another finding: it seems as if - when munin starts - there is a transition directly from 1.0 GHz towards 2.1 GHz. At least I see only one transition state counted up and ticks on 2.1 GHz in cpufreq-stats. Hmmm. Is that right? BR, Carsten. -----Ursprüngliche Nachricht----- Von: Carsten Schiers Gesendet: Samstag, 7. März 2009 23:13 An: xen-devel Cc: mark.langsdorf; keir.fraser Betreff: AW: [Xen-devel] Still Time went backwards problems Xen 3.3.1/2.6.18.8 Leaving out cpuidle doesn''t make any difference: Mar 7 22:45:04 data kernel: Timer ISR/0: Time went backwards: delta=-23283686 ... I think unstable is going to got frozen soon. I guess this is next to try. BTW: again three seconds after munin-cron. BR, Carsten. -----Ursprüngliche Nachricht----- Von: Carsten Schiers Gesendet: Samstag, 7. März 2009 10:42 An: xen-devel Cc: mark.langsdorf; keir.fraser Betreff: [Xen-devel] Still Time went backwards problems Xen 3.3.1/2.6.18.8 Still have problems with "Time went backwards" messages. Now I am nearly completely at Xen 3.3.1 and Xen kernel 2.6.18.8 for Dom0 and nearly all DomUs. There is only one DomU That is on a customized special kernel for Endian (2.6.21.7-2). Q1: Can a DomU kernel version anyhow be responsible for that effect? Attached you can see the last two days. I noticed that they occur every five minutes, so it''s probably connected with the munin cron job that collects xen domain usage and cpufreq-stats. Nothing more. Delta is still quite huge, around 40 ms. Value is negative. Q2: Is that any better than if it''s positive? Q3: Will the delta be reset or will it go up and down all the time? Currently I use both cpufreq=dom0 and cpuidle. Q4: Can the problem eventually be related to either of them only? Right now, I disabled cpuidle for the time being, to see whether there is a difference. If not, I intend to use xen-unstable next. I attached the current xm dmesg and xm info, so don''t be surprised to miss cpuidle. Well, I am not sure whether I do too much work on this issue, but I think the log message is there for a reason. Best Regards, Carsten. P.S.: There are combinations that are much worse that this. Especially running Xen 3.3.1 and actual Lenny kernel seems to be worse. -----Ursprüngliche Nachricht----- Von: Langsdorf, Mark [mailto:mark.langsdorf@amd.com] Gesendet: Donnerstag, 15. Januar 2009 20:27 An: Carsten Schiers Betreff: RE: RE: RE: RE: [Xen-devel] AMD P-States not recognized for Xen 3.3 and 3.4 If you have a single socket system, then fixing the out of sync issue will probably not fix the TSC issue. If you have a dual or quad socket system, fixing the out of sync error should reduce the frequency and magnitude of the TSC issues. At least, that''s the best I can decipher from my notes. Also, most of the work I did on cpufreq was in 2007, before Dan M. greatly redid Xen hypervisor time-keeping. If an up-to-date kernel/hypervisor still has TSC problems, it may be possible to fix them. I don''t really have time to research it myself, but if you do have problems, please contact me and I''ll see what can be done. -Mark Langsdorf Operating System Research Center AMD> -----Original Message----- > From: Carsten Schiers [mailto:carsten@schiers.de] > Sent: Thursday, January 15, 2009 1:17 PM > To: Langsdorf, Mark > Subject: AW: RE: RE: RE: [Xen-devel] AMD P-States not recognized for > Xen 3.3 and 3.4 > > Thanks Mark, don''t want to get on your nerve, but this will only fix > the > powernow-k8 out of synch issue but not the fact that my 4050e will > have TSC issues between the two cores when switching frequencies, > right? > > Am I right that there is no cure for that currently? And that I am not> safe to ignore them, because some kernel wizard is synchronizing them?> ''Cause up to now I only found a synch in boot.c. > > Sorry for all those questions, it''s quite complicated. My best times > as code producer habe been some 15 years ago... > > Thanks, > Carsten. > > -----Ursprüngliche Nachricht----- > Von: Langsdorf, Mark [mailto:mark.langsdorf@amd.com] > Gesendet: Donnerstag, 15. Januar 2009 18:23 > An: Carsten Schiers > Betreff: RE: RE: RE: [Xen-devel] AMD P-States not recognized for Xen > 3.3 and 3.4 > > > Thank you Mark. Please excuse my small knowledge in kernel/xen > > hacking. I understood that I should either > > > > a) create a build of the actual xen-unstable.hg together with > > linux-2.6.18-xen.hg kernel (where you think to have fixed > the problem) > > > > or > > > > b) to include the mentioned fixes into my environment > (which is the > > 3.2.1 Xen and Bastian Blank''s (waldi) kernel (based on 2.6.18, too),> > right? > > > > I''ll try out what will be more easy, but I think it should be a). > > Right. Using the latest unstable kernel & hypervisor gives you all > the patches. If you''re backporting support into the waldi kernel, you> need to make sure you backport all the relevant patches. > > -Mark Langsdorf > Operating System Research Center > AMD > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel