Christoph Egger
2008-Aug-15 11:29 UTC
[Xen-devel] unfair VCPU scheduling: slow HVM guest boot
Hi, Launch a HVM guest with twice as many VCPUs as physical CPUs are in the machine. You will notice the guest boots slow. With xentop you see, the first VCPU is rarely scheduled once the other VCPUs are up in the guest. If the boot process is just waiting for the first VCPU to finish something (e.g. handling an interrupt), then the whole boot process "freezes" until the first VCPU gets scheduled. Here is an xentop line showing how unfair the VCPUs are scheduled: VCPUs(sec): 0: 44s 1: 94s 2: 96s 3: 140s Christoph -- AMD Saxony, Dresden, Germany Operating System Research Center Legal Information: AMD Saxony Limited Liability Company & Co. KG Sitz (Geschäftsanschrift): Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland Registergericht Dresden: HRA 4896 vertretungsberechtigter Komplementär: AMD Saxony LLC (Sitz Wilmington, Delaware, USA) Geschäftsführer der AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2008-Aug-15 11:39 UTC
Re: [Xen-devel] unfair VCPU scheduling: slow HVM guest boot
The fact that there''s different amounts of cpu time isn''t evidence that the scheduler is unfair. The vcpus may be blocked, or may be coming up at different times. Is there a particular reason you want to run with more VCPUs than physical cpus? Given that cpu synchronization primitives like spinlocks and IPIs were generally designed with the assumption that they''re running on bare metal and are not pre-empted, it''s not surprising that when vcpus are trying to work together but not able to run at the same time, there will be performance problems. -George On Fri, Aug 15, 2008 at 12:29 PM, Christoph Egger <Christoph.Egger@amd.com> wrote:> > Hi, > > Launch a HVM guest with twice as many VCPUs as physical CPUs are in the > machine. You will notice the guest boots slow. > > With xentop you see, the first VCPU is rarely scheduled once the other > VCPUs are up in the guest. > If the boot process is just waiting for the first VCPU to finish something > (e.g. handling an interrupt), then the whole boot process "freezes" until the > first VCPU gets scheduled. > > Here is an xentop line showing how unfair the VCPUs are scheduled: > > VCPUs(sec): 0: 44s 1: 94s 2: 96s 3: 140s > > > Christoph > > > -- > AMD Saxony, Dresden, Germany > Operating System Research Center > > Legal Information: > AMD Saxony Limited Liability Company & Co. KG > Sitz (Geschäftsanschrift): > Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland > Registergericht Dresden: HRA 4896 > vertretungsberechtigter Komplementär: > AMD Saxony LLC (Sitz Wilmington, Delaware, USA) > Geschäftsführer der AMD Saxony LLC: > Dr. Hans-R. Deppe, Thomas McCoy > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christoph Egger
2008-Aug-15 11:50 UTC
Re: [Xen-devel] unfair VCPU scheduling: slow HVM guest boot
On Friday 15 August 2008 13:39:45 George Dunlap wrote:> The fact that there''s different amounts of cpu time isn''t evidence > that the scheduler is unfair. The vcpus may be blocked, or may be > coming up at different times. > > Is there a particular reason you want to run with more VCPUs than physical > cpus?Yes. You can have a virtual test/crash box for some development work, for example.> Given that cpu synchronization primitives like spinlocks and IPIs were > generally designed with the assumption that they''re running on bare > metal and are not pre-empted, it''s not surprising that when vcpus are > trying to work together but not able to run at the same time, there > will be performance problems.Yes, overcommitting always causes performance problems. The thing I am observing with xentop is that the last activated VCPU seems to be prefered over the others and the virtual BSP runs very rarely. That is what I mean by "unfair". Christoph> -George > > On Fri, Aug 15, 2008 at 12:29 PM, Christoph Egger > > <Christoph.Egger@amd.com> wrote: > > Hi, > > > > Launch a HVM guest with twice as many VCPUs as physical CPUs are in the > > machine. You will notice the guest boots slow. > > > > With xentop you see, the first VCPU is rarely scheduled once the other > > VCPUs are up in the guest. > > If the boot process is just waiting for the first VCPU to finish > > something (e.g. handling an interrupt), then the whole boot process > > "freezes" until the first VCPU gets scheduled. > > > > Here is an xentop line showing how unfair the VCPUs are scheduled: > > > > VCPUs(sec): 0: 44s 1: 94s 2: 96s 3: > > 140s > > > > > > Christoph-- AMD Saxony, Dresden, Germany Operating System Research Center Legal Information: AMD Saxony Limited Liability Company & Co. KG Sitz (Geschäftsanschrift): Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland Registergericht Dresden: HRA 4896 vertretungsberechtigter Komplementär: AMD Saxony LLC (Sitz Wilmington, Delaware, USA) Geschäftsführer der AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel