It appears that in the new cpupools code: * pdata_allocate is called from alloc_vcpu() with that cpu for the first time. * this happens usually when dom0''s cpus are allocated, as it cycles through all of the cpus This seems a bit strange to me. Wouldn''t it make more sense to pass, or to make available, a cpumask of the cpus in a pool to the init function? Also, is there documentation of the cpupool lifecycle anywhere, to help scheduler writers understand what''s going on? -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 05/10/2010 07:47 PM, George Dunlap wrote:> It appears that in the new cpupools code: > * pdata_allocate is called from alloc_vcpu() with that cpu for the first time. > * this happens usually when dom0''s cpus are allocated, as it cycles > through all of the cpusNo. It happens when idle vcpus are allocated. At this time there is no cpupool existing, all physical cpus are marked as "free", e.g. they are in no pool at all. Dom0 vcpus are allocated in Pool-0. This pool is created after allocation of the idle vcpus.> > This seems a bit strange to me. Wouldn''t it make more sense to pass, > or to make available, a cpumask of the cpus in a pool to the init > function?No. A cpupool is created without any cpu in the beginning. Whenever a cpu is added to a cpupool, alloc_pdata is called for this cpu. BTW: Allocating the percpu data of the scheduler during the allocation of the first vcpu on this cpu was in sched_credit.c before cpupools were introduced.> > Also, is there documentation of the cpupool lifecycle anywhere, to > help scheduler writers understand what''s going on?Not yet. I''ll write something up in the next days. Juergen -- Juergen Gross Principal Developer Operating Systems TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967 Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com Domagkstr. 28 Internet: ts.fujitsu.com D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, May 10, 2010 at 11:48 PM, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:> No. It happens when idle vcpus are allocated. At this time there is no > cpupool > existing, all physical cpus are marked as "free", e.g. they are in no pool > at > all. > Dom0 vcpus are allocated in Pool-0. This pool is created after allocation of > the idle vcpus.Yeah, I spent some time tracing through the init code yesterday and figured that out. So it appears that at init time, regarding cpupools and schedulers: * init_idle_domain() calls schedule_init(), which calls ops->init() with a sort of "default" ops pointer. It also calls sched_init_vcpu() for idle domain''s vcpu 0, which will call ops->alloc_pdata for cpu 0. * smp_prepare_cpus will eventually call do_boot_cpu for each online cpu. do_boot_cpu will initialize the idle_domain vcpu for that cpu, which will call ops->alloc_pdata for that cpu (again, with cpu 0 - At this point, all online cpus have had alloc_pdata called, albeit for the "default" ops structure in the scheduler * cpupool_create will create cpupool 0, calling sched_init(), which calls ops->init with the cpupool0 ops structure. * cpupool0_cpu_assign will then un-assign all online cpus from the "default" ops structure and re-assign them into cpupool 0. Re-assigning looks like this: - First call alloc_pdata and then alloc_vdata for the new cpupool ops structure, for the physical cpu and idle vcpu respectively. - Ticks will be disabled on the old ops structure, then resumed on the new ops structure - The idle vcpu is added to the new pool - Calls free_vdata and free_pdata on the old cpupool ops structure for the idle vcpu and physical cpu, respectively. Now all online cpus have idle vcpus and pdatas initialized, and set up for cpupool 0. Is that a pretty accurate picture?> BTW: Allocating the percpu data of the scheduler during the allocation of > the > first vcpu on this cpu was in sched_credit.c before cpupools were > introduced.Yes, I remember that. IIRC it had something to do with the timer infrastructure not being ready during sched_init, so init_cpu was used as a hook to set up the per-cpu tick after the timer infrastructure had been initialized. Although, looking at the code again, it''s more likely that I thought that because when sched_init was called, only one cpu was online. Kier, out of curiosity, is there a reason init_idle_domain() (and thus schedule_init()) is called so early, before all of the cpus are up? Is it so that adding a cpu dynamically and at boot (which needs to do in it, add an idle vcpu, &c) all take the same codepath?> Not yet. > I''ll write something up in the next days.Cool, thanks. -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/05/2010 18:25, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote:> Kier, out of curiosity, is there a reason init_idle_domain() (and thus > schedule_init()) is called so early, before all of the cpus are up? > Is it so that adding a cpu dynamically and at boot (which needs to do > in it, add an idle vcpu, &c) all take the same codepath?Well, once secondary CPUs are fully up they are of course running their respective idle VCPUs, so we can''t call schedule_init() very late. The right thing to do is dynamically allocate state as CPUs are brought online -- and potentially free that state when CPUs are taken offline. Which is pretty much the direction the code points in already. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, May 11, 2010 at 1:41 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> Well, once secondary CPUs are fully up they are of course running their > respective idle VCPUs, so we can''t call schedule_init() very late. The right > thing to do is dynamically allocate state as CPUs are brought online -- and > potentially free that state when CPUs are taken offline. Which is pretty > much the direction the code points in already.Except that it''s done a little indirectly, via initializing the idle vcpu, rather than calling something like sched_init_pcpu() when the cpu is brought up before creating an idle vcpu for it. :-) Anyway, maybe once I get the credit2 stuff working again, I may think about writing a patch to clean it up a bit... mostly so that it''s easier to follow what''s going on. -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/05/2010 19:52, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote:> On Tue, May 11, 2010 at 1:41 PM, Keir Fraser <keir.fraser@eu.citrix.com> > wrote: >> Well, once secondary CPUs are fully up they are of course running their >> respective idle VCPUs, so we can''t call schedule_init() very late. The right >> thing to do is dynamically allocate state as CPUs are brought online -- and >> potentially free that state when CPUs are taken offline. Which is pretty >> much the direction the code points in already. > > Except that it''s done a little indirectly, via initializing the idle > vcpu, rather than calling something like sched_init_pcpu() when the > cpu is brought up before creating an idle vcpu for it. :-) Anyway, > maybe once I get the credit2 stuff working again, I may think about > writing a patch to clean it up a bit... mostly so that it''s easier to > follow what''s going on.Splitting it into an explicit separate call into the scheduling subsystem would be acceptable, I think. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 05/11/2010 07:25 PM, George Dunlap wrote:> On Mon, May 10, 2010 at 11:48 PM, Juergen Gross > <juergen.gross@ts.fujitsu.com> wrote: >> No. It happens when idle vcpus are allocated. At this time there is no >> cpupool >> existing, all physical cpus are marked as "free", e.g. they are in no pool >> at >> all. >> Dom0 vcpus are allocated in Pool-0. This pool is created after allocation of >> the idle vcpus. > > Yeah, I spent some time tracing through the init code yesterday and > figured that out. So it appears that at init time, regarding cpupools > and schedulers: > * init_idle_domain() calls schedule_init(), which calls ops->init() > with a sort of "default" ops pointer. It also calls sched_init_vcpu() > for idle domain''s vcpu 0, which will call ops->alloc_pdata for cpu 0. > * smp_prepare_cpus will eventually call do_boot_cpu for each online > cpu. do_boot_cpu will initialize the idle_domain vcpu for that cpu, > which will call ops->alloc_pdata for that cpu (again, with cpu 0 > - At this point, all online cpus have had alloc_pdata called, albeit > for the "default" ops structure in the scheduler > * cpupool_create will create cpupool 0, calling sched_init(), which > calls ops->init with the cpupool0 ops structure. > * cpupool0_cpu_assign will then un-assign all online cpus from the > "default" ops structure and re-assign them into cpupool 0. > Re-assigning looks like this: > - First call alloc_pdata and then alloc_vdata for the new cpupool ops > structure, for the physical cpu and idle vcpu respectively. > - Ticks will be disabled on the old ops structure, then resumed on > the new ops structure > - The idle vcpu is added to the new pool > - Calls free_vdata and free_pdata on the old cpupool ops structure > for the idle vcpu and physical cpu, respectively. > > Now all online cpus have idle vcpus and pdatas initialized, and set up > for cpupool 0. > > Is that a pretty accurate picture?Yes, this sounds correct. Juergen -- Juergen Gross Principal Developer Operating Systems TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967 Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com Domagkstr. 28 Internet: ts.fujitsu.com D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel