I''ve been running some heavy testing on a recent Xen 4.0 snapshot and seeing a strange scheduling anomaly that I thought I should report. I don''t know if this is a regression... I suspect not. System is a Core 2 Duo (Conroe). Load is four 2-VCPU EL5u4 guests, two of which are 64-bit and two of which are 32-bit. Otherwise they are identical. All four are running a sequence of three Linux compiles with (make -j8 clean; make -j8). All are started approximately concurrently: I synchronize the start of the test after all domains are launched with an external NFS semaphore file that is checked every 30 seconds. What I am seeing is a rather large discrepancy in the amount of time consumed "underway" by the four domains as reported by xentop and xm list. I have seen this repeatedly, but the numbers in front of me right now are: 1191s dom0 3182s 64-bit #1 2577s 64-bit #2 <-- 20% less! 4316s 32-bit #1 2667s 32-bit #2 <-- 40% less! Again these are identical workloads and the pairs are identical released kernels running from identical "file"-based virtual block devices containing released distros. Much of my testing had been with tmem and self-ballooning so I had blamed them for awhile, but I have reproduced it multiple times with both of those turned off. At start and after each kernel compile, I record a timestamp, so I know the same work is being done. Eventually the workload finishes on each domain and intentionally crashes the kernel so measurement is stopped. At the conclusion, the 64-bit pair have very similar total CPU sec and the 32-bit pair have very similar total CPU sec so eventually (presumably when the #1''s are done hogging CPU), the "slower" domains do finish the same amount of work. As a result, it is hard to tell from just the final results that the four domains are getting scheduled at very different rates. Does this seem like a scheduler problem, or are there other explanations? Anybody care to try to reproduce it? Unfortunately, I have to use the machine now for other work. P.S. According to xentop, there is almost no network activity, so it is all CPU and VBD. And the ratio of VBD activity looks to be approximately the same ratio as CPU(sec). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Is it possible that Linux is just favoring one vcpu over the other for some reason? Did you try running the same test but with only one VM? Another theory would be that most interrupts are delivered to vcpu 0, so it may end up in "boost" priority more often. I''ll re-post the credit2 series shortly; Keir said he''d accept it post-4.0. You could try it with that and see what the performance is like. -George On Fri, Apr 2, 2010 at 5:48 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:> I''ve been running some heavy testing on a recent Xen 4.0 > snapshot and seeing a strange scheduling anomaly that > I thought I should report. I don''t know if this is > a regression... I suspect not. > > System is a Core 2 Duo (Conroe). Load is four 2-VCPU > EL5u4 guests, two of which are 64-bit and two of which > are 32-bit. Otherwise they are identical. All four > are running a sequence of three Linux compiles with > (make -j8 clean; make -j8). All are started approximately > concurrently: I synchronize the start of the test after > all domains are launched with an external NFS semaphore > file that is checked every 30 seconds. > > What I am seeing is a rather large discrepancy in the > amount of time consumed "underway" by the four domains > as reported by xentop and xm list. I have seen this > repeatedly, but the numbers in front of me right now are: > > 1191s dom0 > 3182s 64-bit #1 > 2577s 64-bit #2 <-- 20% less! > 4316s 32-bit #1 > 2667s 32-bit #2 <-- 40% less! > > Again these are identical workloads and the pairs > are identical released kernels running from identical > "file"-based virtual block devices containing released > distros. Much of my testing had been with tmem and > self-ballooning so I had blamed them for awhile, > but I have reproduced it multiple times with both > of those turned off. > > At start and after each kernel compile, I record > a timestamp, so I know the same work is being done. > Eventually the workload finishes on each domain and > intentionally crashes the kernel so measurement is > stopped. At the conclusion, the 64-bit pair have > very similar total CPU sec and the 32-bit pair have > very similar total CPU sec so eventually (presumably > when the #1''s are done hogging CPU), the "slower" > domains do finish the same amount of work. As a > result, it is hard to tell from just the final > results that the four domains are getting scheduled > at very different rates. > > Does this seem like a scheduler problem, or are there > other explanations? Anybody care to try to reproduce it? > Unfortunately, I have to use the machine now for other > work. > > P.S. According to xentop, there is almost no network > activity, so it is all CPU and VBD. And the ratio > of VBD activity looks to be approximately the same > ratio as CPU(sec). > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks for the reply! Well I''m now seeing something a little more alarming: Running an identical but CPU-overcommitted workload (just normal PV domains, no tmem or ballooning or anything), what would you expect the variance to be between successive identical measured runs on identical hardware? I am seeing total runtimes, both measured by elapsed time and by sum-of-CPUsec across all domains (incl dom0), vary by 6-7% or more. This seems a bit unusual/excessive to me and makes it very hard to measure improvements (e.g. by tmem, for an upcoming Xen summit presentation) or benchmark anything complex.> Is it possible that Linux is just favoring one vcpu over the other for > some reason? Did you try running the same test but with only one VM?Well "make -j8" will likely be single-threaded part of the time, but I wouldn''t expect that to make that big a difference between two identical workloads. I''m not sure I understand how I would run the same test with only one VM when the observation of the strangeness requires two VMs (and even then must be observed at random points during execution).> Another theory would be that most interrupts are delivered to vcpu 0, > so it may end up in "boost" priority more often.Hmmm... I''m not sure I get that, but what about _physical_ cpu 0 for Xen? If all physical cpu''s are not the same and one VM has an affinity for vcpu0-on-pcpu0 and the other has an affinity for vcpu1-in-pcpu0, would that make a difference? But still, 40% seems very large and almost certainly a bug, especially given the new observations above.> -----Original Message----- > From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > Sent: Monday, April 05, 2010 8:44 AM > To: Dan Magenheimer > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > > Is it possible that Linux is just favoring one vcpu over the other for > some reason? Did you try running the same test but with only one VM? > > Another theory would be that most interrupts are delivered to vcpu 0, > so it may end up in "boost" priority more often. > > I''ll re-post the credit2 series shortly; Keir said he''d accept it > post-4.0. You could try it with that and see what the performance is > like. > > -George > > On Fri, Apr 2, 2010 at 5:48 PM, Dan Magenheimer > <dan.magenheimer@oracle.com> wrote: > > I''ve been running some heavy testing on a recent Xen 4.0 > > snapshot and seeing a strange scheduling anomaly that > > I thought I should report. I don''t know if this is > > a regression... I suspect not. > > > > System is a Core 2 Duo (Conroe). Load is four 2-VCPU > > EL5u4 guests, two of which are 64-bit and two of which > > are 32-bit. Otherwise they are identical. All four > > are running a sequence of three Linux compiles with > > (make -j8 clean; make -j8). All are started approximately > > concurrently: I synchronize the start of the test after > > all domains are launched with an external NFS semaphore > > file that is checked every 30 seconds. > > > > What I am seeing is a rather large discrepancy in the > > amount of time consumed "underway" by the four domains > > as reported by xentop and xm list. I have seen this > > repeatedly, but the numbers in front of me right now are: > > > > 1191s dom0 > > 3182s 64-bit #1 > > 2577s 64-bit #2 <-- 20% less! > > 4316s 32-bit #1 > > 2667s 32-bit #2 <-- 40% less! > > > > Again these are identical workloads and the pairs > > are identical released kernels running from identical > > "file"-based virtual block devices containing released > > distros. Much of my testing had been with tmem and > > self-ballooning so I had blamed them for awhile, > > but I have reproduced it multiple times with both > > of those turned off. > > > > At start and after each kernel compile, I record > > a timestamp, so I know the same work is being done. > > Eventually the workload finishes on each domain and > > intentionally crashes the kernel so measurement is > > stopped. At the conclusion, the 64-bit pair have > > very similar total CPU sec and the 32-bit pair have > > very similar total CPU sec so eventually (presumably > > when the #1''s are done hogging CPU), the "slower" > > domains do finish the same amount of work. As a > > result, it is hard to tell from just the final > > results that the four domains are getting scheduled > > at very different rates. > > > > Does this seem like a scheduler problem, or are there > > other explanations? Anybody care to try to reproduce it? > > Unfortunately, I have to use the machine now for other > > work. > > > > P.S. According to xentop, there is almost no network > > activity, so it is all CPU and VBD. And the ratio > > of VBD activity looks to be approximately the same > > ratio as CPU(sec). > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
For the record, I am seeing the same problem (first one, haven''t yet got multiple runs) with vcpus=1 for all domains. Only on 32-bit this time and only 20%, but those may be random scheduling factors. This is also with tap:aio instead of file so as to eliminate dom0 page cacheing effects. 394s dom0 2265s 64-bit #1 2275s 64-bit #2 2912s 32-bit #1 2247s 32-bit #2 <-- 20% less! I''m going to try a dom0_vcpus=1 run next.> -----Original Message----- > From: Dan Magenheimer > Sent: Monday, April 05, 2010 2:18 PM > To: George Dunlap > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > > Thanks for the reply! > > Well I''m now seeing something a little more alarming: Running > an identical but CPU-overcommitted workload (just normal PV domains, > no tmem or ballooning or anything), what would you expect the > variance to be between successive identical measured runs > on identical hardware? > > I am seeing total runtimes, both measured by elapsed time and by > sum-of-CPUsec across all domains (incl dom0), vary by 6-7% or more. > This seems a bit unusual/excessive to me and makes it very hard > to measure improvements (e.g. by tmem, for an upcoming Xen summit > presentation) or benchmark anything complex. > > > Is it possible that Linux is just favoring one vcpu over the other > for > > some reason? Did you try running the same test but with only one VM? > > Well "make -j8" will likely be single-threaded part of the time, > but I wouldn''t expect that to make that big a difference between > two identical workloads. > > I''m not sure I understand how I would run the same test with > only one VM when the observation of the strangeness requires > two VMs (and even then must be observed at random points during > execution). > > > Another theory would be that most interrupts are delivered to vcpu 0, > > so it may end up in "boost" priority more often. > > Hmmm... I''m not sure I get that, but what about _physical_ cpu 0 > for Xen? If all physical cpu''s are not the same and one VM > has an affinity for vcpu0-on-pcpu0 and the other has an affinity > for vcpu1-in-pcpu0, would that make a difference? > > But still, 40% seems very large and almost certainly a bug, > especially given the new observations above. > > > -----Original Message----- > > From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > > Sent: Monday, April 05, 2010 8:44 AM > > To: Dan Magenheimer > > Cc: xen-devel@lists.xensource.com > > Subject: Re: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > > > > Is it possible that Linux is just favoring one vcpu over the other > for > > some reason? Did you try running the same test but with only one VM? > > > > Another theory would be that most interrupts are delivered to vcpu 0, > > so it may end up in "boost" priority more often. > > > > I''ll re-post the credit2 series shortly; Keir said he''d accept it > > post-4.0. You could try it with that and see what the performance is > > like. > > > > -George > > > > On Fri, Apr 2, 2010 at 5:48 PM, Dan Magenheimer > > <dan.magenheimer@oracle.com> wrote: > > > I''ve been running some heavy testing on a recent Xen 4.0 > > > snapshot and seeing a strange scheduling anomaly that > > > I thought I should report. I don''t know if this is > > > a regression... I suspect not. > > > > > > System is a Core 2 Duo (Conroe). Load is four 2-VCPU > > > EL5u4 guests, two of which are 64-bit and two of which > > > are 32-bit. Otherwise they are identical. All four > > > are running a sequence of three Linux compiles with > > > (make -j8 clean; make -j8). All are started approximately > > > concurrently: I synchronize the start of the test after > > > all domains are launched with an external NFS semaphore > > > file that is checked every 30 seconds. > > > > > > What I am seeing is a rather large discrepancy in the > > > amount of time consumed "underway" by the four domains > > > as reported by xentop and xm list. I have seen this > > > repeatedly, but the numbers in front of me right now are: > > > > > > 1191s dom0 > > > 3182s 64-bit #1 > > > 2577s 64-bit #2 <-- 20% less! > > > 4316s 32-bit #1 > > > 2667s 32-bit #2 <-- 40% less! > > > > > > Again these are identical workloads and the pairs > > > are identical released kernels running from identical > > > "file"-based virtual block devices containing released > > > distros. Much of my testing had been with tmem and > > > self-ballooning so I had blamed them for awhile, > > > but I have reproduced it multiple times with both > > > of those turned off. > > > > > > At start and after each kernel compile, I record > > > a timestamp, so I know the same work is being done. > > > Eventually the workload finishes on each domain and > > > intentionally crashes the kernel so measurement is > > > stopped. At the conclusion, the 64-bit pair have > > > very similar total CPU sec and the 32-bit pair have > > > very similar total CPU sec so eventually (presumably > > > when the #1''s are done hogging CPU), the "slower" > > > domains do finish the same amount of work. As a > > > result, it is hard to tell from just the final > > > results that the four domains are getting scheduled > > > at very different rates. > > > > > > Does this seem like a scheduler problem, or are there > > > other explanations? Anybody care to try to reproduce it? > > > Unfortunately, I have to use the machine now for other > > > work. > > > > > > P.S. According to xentop, there is almost no network > > > activity, so it is all CPU and VBD. And the ratio > > > of VBD activity looks to be approximately the same > > > ratio as CPU(sec). > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi George -- Thanks again for the reply. Hope it''s OK if I go back on-list... I''m hoping others may be able to reproduce as my ability to experiment is limited now (see below).> From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > > 1) Make a large ramdisk in each VM, big enough for the whole kernel > tree and binaries. Do the build there, and see if you have the same > discrepancy.My test domains have 384MB each. Dom0 has 256MB. (Total physical RAM is only 2GB.) So this isn''t really an option.> 2) Play with the dom0 io scheduler and see if it has an effect. If > your current one is "noop", that''s suspicious; see if "cfq" works > better.On dom0, /sys/block/sda/queue/scheduler shows [cfq]. Don''t know if this matters but /sys/block/tapdev*/queue/scheduler show [noop].> 3) Take a trace of just the scheduling events, using xentrace...I lost about a week of test runs that I''m working on for Xen Summit and have to re-do those before I do much experimenting, but will try out some of your ideas when my (week of redo) test runs are done. In the meantime, I''m still monitoring the test runs that I am running now. (I need a reliable set of non-tmem runs as a base to compare various tmem runs against.) I reported two problems that we can call: 1) "racing ahead", where one of a pair of identical domains seems to get a lot more cycles than the other 2) "irreproducibility", where two seemingly identical and heavily overcommitted test runs have timing results that differ by an unreasonable amount (6-7%) After reducing my test domains to a single vcpu, the "irreproducibility" problem seems to be greatly reduced. I made three runs and they differ by <0.3%. So as best I can tell, this problem requires multi-vcpu domains. (Actually, I changed from "file" to "tap:aio" also so it could be that too.) However, with: a) vcpus=1 for the test domains (see previous post) and b) vcpus=1 for test domains and dom0_max_vcpus=1 I am still seeing the "racing ahead" problem. On a current run of (b) 142s dom0 479s 64-bit #1 454s 64-bit #2 <-- 6% less 536s 32-bit #1 447s 32-bit #2 <-- 16% less! Again, this is a transitory oddity that may shed some light... after completion of the workload, the runtimes are very similar THOUGH #2 seems to always be the slower of the two by a small amount (<0.5%). Thanks, Dan> -----Original Message----- > From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > Sent: Tuesday, April 06, 2010 5:24 AM > To: Dan Magenheimer > Subject: Re: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > > How much memory does each VM have? Another possibility is that this > has to do with unfairness in the block driver servicing requests. > Three ways you could test this hypothesis. > > 1) Make a large ramdisk in each VM, big enough for the whole kernel > tree and binaries. Do the build there, and see if you have the same > discrepancy. > > 2) Play with the dom0 io scheduler and see if it has an effect. If > your current one is "noop", that''s suspicious; see if "cfq" works > better. > > 3) Take a trace of just the scheduling events, using xentrace, and use > xenalyze to see how much time each vcpu is spending running, runnable, > and blocked (waiting for the cpu). If the scheduler is being unfair, > then some vcpus will spend more time "runnable" than others. If it''s > something else (the dom0 disk scheduler being unfair, or the vm just > using different amounts of memory) then "runnable" will not be > considerably higher. > > To do #3: > > # xentrace -D -e 0x28000 -S 32 /tmp/filename.trace > > Then download: > http://xenbits.xensource.com/ext/xenalyze.hg > > Make it, and run the following command: > > $ xenalyze -s --cpu-hz [speed-in-gigahertz]G filename.trace > > filename.summary > > The summary file breaks information down by domain, then vcpu; look at > the "runstates" for each vcpu (running, runnable, blocked) and compare > them. > > -George > > On Tue, Apr 6, 2010 at 12:17 AM, Dan Magenheimer > <dan.magenheimer@oracle.com> wrote: > > For the record, I am seeing the same problem (first one, > > haven''t yet got multiple runs) with vcpus=1 for all domains. > > Only on 32-bit this time and only 20%, but those may > > be random scheduling factors. This is also with > > tap:aio instead of file so as to eliminate dom0 page > > cacheing effects. > > > > 394s dom0 > > 2265s 64-bit #1 > > 2275s 64-bit #2 > > 2912s 32-bit #1 > > 2247s 32-bit #2 <-- 20% less! > > > > I''m going to try a dom0_vcpus=1 run next. > > > >> -----Original Message----- > >> From: Dan Magenheimer > >> Sent: Monday, April 05, 2010 2:18 PM > >> To: George Dunlap > >> Cc: xen-devel@lists.xensource.com > >> Subject: RE: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > >> > >> Thanks for the reply! > >> > >> Well I''m now seeing something a little more alarming: Running > >> an identical but CPU-overcommitted workload (just normal PV domains, > >> no tmem or ballooning or anything), what would you expect the > >> variance to be between successive identical measured runs > >> on identical hardware? > >> > >> I am seeing total runtimes, both measured by elapsed time and by > >> sum-of-CPUsec across all domains (incl dom0), vary by 6-7% or more. > >> This seems a bit unusual/excessive to me and makes it very hard > >> to measure improvements (e.g. by tmem, for an upcoming Xen summit > >> presentation) or benchmark anything complex. > >> > >> > Is it possible that Linux is just favoring one vcpu over the other > >> for > >> > some reason? Did you try running the same test but with only one > VM? > >> > >> Well "make -j8" will likely be single-threaded part of the time, > >> but I wouldn''t expect that to make that big a difference between > >> two identical workloads. > >> > >> I''m not sure I understand how I would run the same test with > >> only one VM when the observation of the strangeness requires > >> two VMs (and even then must be observed at random points during > >> execution). > >> > >> > Another theory would be that most interrupts are delivered to vcpu > 0, > >> > so it may end up in "boost" priority more often. > >> > >> Hmmm... I''m not sure I get that, but what about _physical_ cpu 0 > >> for Xen? If all physical cpu''s are not the same and one VM > >> has an affinity for vcpu0-on-pcpu0 and the other has an affinity > >> for vcpu1-in-pcpu0, would that make a difference? > >> > >> But still, 40% seems very large and almost certainly a bug, > >> especially given the new observations above. > >> > >> > -----Original Message----- > >> > From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > >> > Sent: Monday, April 05, 2010 8:44 AM > >> > To: Dan Magenheimer > >> > Cc: xen-devel@lists.xensource.com > >> > Subject: Re: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > >> > > >> > Is it possible that Linux is just favoring one vcpu over the other > >> for > >> > some reason? Did you try running the same test but with only one > VM? > >> > > >> > Another theory would be that most interrupts are delivered to vcpu > 0, > >> > so it may end up in "boost" priority more often. > >> > > >> > I''ll re-post the credit2 series shortly; Keir said he''d accept it > >> > post-4.0. You could try it with that and see what the performance > is > >> > like. > >> > > >> > -George > >> > > >> > On Fri, Apr 2, 2010 at 5:48 PM, Dan Magenheimer > >> > <dan.magenheimer@oracle.com> wrote: > >> > > I''ve been running some heavy testing on a recent Xen 4.0 > >> > > snapshot and seeing a strange scheduling anomaly that > >> > > I thought I should report. I don''t know if this is > >> > > a regression... I suspect not. > >> > > > >> > > System is a Core 2 Duo (Conroe). Load is four 2-VCPU > >> > > EL5u4 guests, two of which are 64-bit and two of which > >> > > are 32-bit. Otherwise they are identical. All four > >> > > are running a sequence of three Linux compiles with > >> > > (make -j8 clean; make -j8). All are started approximately > >> > > concurrently: I synchronize the start of the test after > >> > > all domains are launched with an external NFS semaphore > >> > > file that is checked every 30 seconds. > >> > > > >> > > What I am seeing is a rather large discrepancy in the > >> > > amount of time consumed "underway" by the four domains > >> > > as reported by xentop and xm list. I have seen this > >> > > repeatedly, but the numbers in front of me right now are: > >> > > > >> > > 1191s dom0 > >> > > 3182s 64-bit #1 > >> > > 2577s 64-bit #2 <-- 20% less! > >> > > 4316s 32-bit #1 > >> > > 2667s 32-bit #2 <-- 40% less! > >> > > > >> > > Again these are identical workloads and the pairs > >> > > are identical released kernels running from identical > >> > > "file"-based virtual block devices containing released > >> > > distros. Much of my testing had been with tmem and > >> > > self-ballooning so I had blamed them for awhile, > >> > > but I have reproduced it multiple times with both > >> > > of those turned off. > >> > > > >> > > At start and after each kernel compile, I record > >> > > a timestamp, so I know the same work is being done. > >> > > Eventually the workload finishes on each domain and > >> > > intentionally crashes the kernel so measurement is > >> > > stopped. At the conclusion, the 64-bit pair have > >> > > very similar total CPU sec and the 32-bit pair have > >> > > very similar total CPU sec so eventually (presumably > >> > > when the #1''s are done hogging CPU), the "slower" > >> > > domains do finish the same amount of work. As a > >> > > result, it is hard to tell from just the final > >> > > results that the four domains are getting scheduled > >> > > at very different rates. > >> > > > >> > > Does this seem like a scheduler problem, or are there > >> > > other explanations? Anybody care to try to reproduce it? > >> > > Unfortunately, I have to use the machine now for other > >> > > work. > >> > > > >> > > P.S. According to xentop, there is almost no network > >> > > activity, so it is all CPU and VBD. And the ratio > >> > > of VBD activity looks to be approximately the same > >> > > ratio as CPU(sec). > >> > > > >> > > _______________________________________________ > >> > > Xen-devel mailing list > >> > > Xen-devel@lists.xensource.com > >> > > http://lists.xensource.com/xen-devel > >> > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Apr 6, 2010 at 5:51 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:> Thanks again for the reply. Hope it''s OK if I go back > on-list... I''m hoping others may be able to reproduce > as my ability to experiment is limited now (see below).Oops -- I meant to reply-to-all in that mail, so going back on the list is actively preferred. :-) -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
One more followup on this: It appears that I AM seeing the "irreproduciblity" problem as well, even with vcpus=1 for all guests. The range is a bit smaller, more like 5%, though. Sum of cpusec across all domains plus dom0 seems to be very reproducible (within < 0.5%). Total elapsed time for the whole workload is what is widely varying. So, either I/O is taking much longer in some cases (despite an identical workload on identical hardware); or the scheduler is selecting the idle domain much more frequently; or... something else?> -----Original Message----- > From: Dan Magenheimer > Sent: Tuesday, April 06, 2010 10:52 AM > To: George Dunlap > Cc: Xen-Devel (xen-devel@lists.xensource.com) > Subject: RE: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > > Hi George -- > > Thanks again for the reply. Hope it''s OK if I go back > on-list... I''m hoping others may be able to reproduce > as my ability to experiment is limited now (see below). > > > From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > > > > 1) Make a large ramdisk in each VM, big enough for the whole kernel > > tree and binaries. Do the build there, and see if you have the same > > discrepancy. > > My test domains have 384MB each. Dom0 has 256MB. > (Total physical RAM is only 2GB.) So this isn''t > really an option. > > > 2) Play with the dom0 io scheduler and see if it has an effect. If > > your current one is "noop", that''s suspicious; see if "cfq" works > > better. > > On dom0, /sys/block/sda/queue/scheduler shows [cfq]. > Don''t know if this matters but /sys/block/tapdev*/queue/scheduler > show [noop]. > > > 3) Take a trace of just the scheduling events, using xentrace... > > I lost about a week of test runs that I''m working on for > Xen Summit and have to re-do those before I do much > experimenting, but will try out some of your ideas when > my (week of redo) test runs are done. In the meantime, I''m > still monitoring the test runs that I am running now. > (I need a reliable set of non-tmem runs as a base > to compare various tmem runs against.) > > I reported two problems that we can call: > 1) "racing ahead", where one of a pair of identical domains > seems to get a lot more cycles than the other > 2) "irreproducibility", where two seemingly identical > and heavily overcommitted test runs have timing results > that differ by an unreasonable amount (6-7%) > > After reducing my test domains to a single vcpu, the > "irreproducibility" problem seems to be greatly reduced. > I made three runs and they differ by <0.3%. So as > best I can tell, this problem requires multi-vcpu domains. > (Actually, I changed from "file" to "tap:aio" also so > it could be that too.) > > However, with: > > a) vcpus=1 for the test domains (see previous post) and > b) vcpus=1 for test domains and dom0_max_vcpus=1 > > I am still seeing the "racing ahead" problem. On > a current run of (b) > > 142s dom0 > 479s 64-bit #1 > 454s 64-bit #2 <-- 6% less > 536s 32-bit #1 > 447s 32-bit #2 <-- 16% less! > > Again, this is a transitory oddity that may shed some > light... after completion of the workload, the runtimes > are very similar THOUGH #2 seems to always be the > slower of the two by a small amount (<0.5%). > > Thanks, > Dan > > > -----Original Message----- > > From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > > Sent: Tuesday, April 06, 2010 5:24 AM > > To: Dan Magenheimer > > Subject: Re: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > > > > How much memory does each VM have? Another possibility is that this > > has to do with unfairness in the block driver servicing requests. > > Three ways you could test this hypothesis. > > > > 1) Make a large ramdisk in each VM, big enough for the whole kernel > > tree and binaries. Do the build there, and see if you have the same > > discrepancy. > > > > 2) Play with the dom0 io scheduler and see if it has an effect. If > > your current one is "noop", that''s suspicious; see if "cfq" works > > better. > > > > 3) Take a trace of just the scheduling events, using xentrace, and > use > > xenalyze to see how much time each vcpu is spending running, > runnable, > > and blocked (waiting for the cpu). If the scheduler is being unfair, > > then some vcpus will spend more time "runnable" than others. If it''s > > something else (the dom0 disk scheduler being unfair, or the vm just > > using different amounts of memory) then "runnable" will not be > > considerably higher. > > > > To do #3: > > > > # xentrace -D -e 0x28000 -S 32 /tmp/filename.trace > > > > Then download: > > http://xenbits.xensource.com/ext/xenalyze.hg > > > > Make it, and run the following command: > > > > $ xenalyze -s --cpu-hz [speed-in-gigahertz]G filename.trace > > > filename.summary > > > > The summary file breaks information down by domain, then vcpu; look > at > > the "runstates" for each vcpu (running, runnable, blocked) and > compare > > them. > > > > -George > > > > On Tue, Apr 6, 2010 at 12:17 AM, Dan Magenheimer > > <dan.magenheimer@oracle.com> wrote: > > > For the record, I am seeing the same problem (first one, > > > haven''t yet got multiple runs) with vcpus=1 for all domains. > > > Only on 32-bit this time and only 20%, but those may > > > be random scheduling factors. This is also with > > > tap:aio instead of file so as to eliminate dom0 page > > > cacheing effects. > > > > > > 394s dom0 > > > 2265s 64-bit #1 > > > 2275s 64-bit #2 > > > 2912s 32-bit #1 > > > 2247s 32-bit #2 <-- 20% less! > > > > > > I''m going to try a dom0_vcpus=1 run next. > > > > > >> -----Original Message----- > > >> From: Dan Magenheimer > > >> Sent: Monday, April 05, 2010 2:18 PM > > >> To: George Dunlap > > >> Cc: xen-devel@lists.xensource.com > > >> Subject: RE: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > > >> > > >> Thanks for the reply! > > >> > > >> Well I''m now seeing something a little more alarming: Running > > >> an identical but CPU-overcommitted workload (just normal PV > domains, > > >> no tmem or ballooning or anything), what would you expect the > > >> variance to be between successive identical measured runs > > >> on identical hardware? > > >> > > >> I am seeing total runtimes, both measured by elapsed time and by > > >> sum-of-CPUsec across all domains (incl dom0), vary by 6-7% or > more. > > >> This seems a bit unusual/excessive to me and makes it very hard > > >> to measure improvements (e.g. by tmem, for an upcoming Xen summit > > >> presentation) or benchmark anything complex. > > >> > > >> > Is it possible that Linux is just favoring one vcpu over the > other > > >> for > > >> > some reason? Did you try running the same test but with only > one > > VM? > > >> > > >> Well "make -j8" will likely be single-threaded part of the time, > > >> but I wouldn''t expect that to make that big a difference between > > >> two identical workloads. > > >> > > >> I''m not sure I understand how I would run the same test with > > >> only one VM when the observation of the strangeness requires > > >> two VMs (and even then must be observed at random points during > > >> execution). > > >> > > >> > Another theory would be that most interrupts are delivered to > vcpu > > 0, > > >> > so it may end up in "boost" priority more often. > > >> > > >> Hmmm... I''m not sure I get that, but what about _physical_ cpu 0 > > >> for Xen? If all physical cpu''s are not the same and one VM > > >> has an affinity for vcpu0-on-pcpu0 and the other has an affinity > > >> for vcpu1-in-pcpu0, would that make a difference? > > >> > > >> But still, 40% seems very large and almost certainly a bug, > > >> especially given the new observations above. > > >> > > >> > -----Original Message----- > > >> > From: George Dunlap [mailto:George.Dunlap@eu.citrix.com] > > >> > Sent: Monday, April 05, 2010 8:44 AM > > >> > To: Dan Magenheimer > > >> > Cc: xen-devel@lists.xensource.com > > >> > Subject: Re: [Xen-devel] Scheduling anomaly with 4.0.0 (rc6) > > >> > > > >> > Is it possible that Linux is just favoring one vcpu over the > other > > >> for > > >> > some reason? Did you try running the same test but with only > one > > VM? > > >> > > > >> > Another theory would be that most interrupts are delivered to > vcpu > > 0, > > >> > so it may end up in "boost" priority more often. > > >> > > > >> > I''ll re-post the credit2 series shortly; Keir said he''d accept > it > > >> > post-4.0. You could try it with that and see what the > performance > > is > > >> > like. > > >> > > > >> > -George > > >> > > > >> > On Fri, Apr 2, 2010 at 5:48 PM, Dan Magenheimer > > >> > <dan.magenheimer@oracle.com> wrote: > > >> > > I''ve been running some heavy testing on a recent Xen 4.0 > > >> > > snapshot and seeing a strange scheduling anomaly that > > >> > > I thought I should report. I don''t know if this is > > >> > > a regression... I suspect not. > > >> > > > > >> > > System is a Core 2 Duo (Conroe). Load is four 2-VCPU > > >> > > EL5u4 guests, two of which are 64-bit and two of which > > >> > > are 32-bit. Otherwise they are identical. All four > > >> > > are running a sequence of three Linux compiles with > > >> > > (make -j8 clean; make -j8). All are started approximately > > >> > > concurrently: I synchronize the start of the test after > > >> > > all domains are launched with an external NFS semaphore > > >> > > file that is checked every 30 seconds. > > >> > > > > >> > > What I am seeing is a rather large discrepancy in the > > >> > > amount of time consumed "underway" by the four domains > > >> > > as reported by xentop and xm list. I have seen this > > >> > > repeatedly, but the numbers in front of me right now are: > > >> > > > > >> > > 1191s dom0 > > >> > > 3182s 64-bit #1 > > >> > > 2577s 64-bit #2 <-- 20% less! > > >> > > 4316s 32-bit #1 > > >> > > 2667s 32-bit #2 <-- 40% less! > > >> > > > > >> > > Again these are identical workloads and the pairs > > >> > > are identical released kernels running from identical > > >> > > "file"-based virtual block devices containing released > > >> > > distros. Much of my testing had been with tmem and > > >> > > self-ballooning so I had blamed them for awhile, > > >> > > but I have reproduced it multiple times with both > > >> > > of those turned off. > > >> > > > > >> > > At start and after each kernel compile, I record > > >> > > a timestamp, so I know the same work is being done. > > >> > > Eventually the workload finishes on each domain and > > >> > > intentionally crashes the kernel so measurement is > > >> > > stopped. At the conclusion, the 64-bit pair have > > >> > > very similar total CPU sec and the 32-bit pair have > > >> > > very similar total CPU sec so eventually (presumably > > >> > > when the #1''s are done hogging CPU), the "slower" > > >> > > domains do finish the same amount of work. As a > > >> > > result, it is hard to tell from just the final > > >> > > results that the four domains are getting scheduled > > >> > > at very different rates. > > >> > > > > >> > > Does this seem like a scheduler problem, or are there > > >> > > other explanations? Anybody care to try to reproduce it? > > >> > > Unfortunately, I have to use the machine now for other > > >> > > work. > > >> > > > > >> > > P.S. According to xentop, there is almost no network > > >> > > activity, so it is all CPU and VBD. And the ratio > > >> > > of VBD activity looks to be approximately the same > > >> > > ratio as CPU(sec). > > >> > > > > >> > > _______________________________________________ > > >> > > Xen-devel mailing list > > >> > > Xen-devel@lists.xensource.com > > >> > > http://lists.xensource.com/xen-devel > > >> > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel