Magenheimer, Dan (HP Labs Fort Collins)
2005-Mar-05 00:08 UTC
[Xen-devel] Scheduler guts question
I''m trying to track down a problem on Xen/ia64 and to help find it, I need to better understand the common scheduler code and probably the Xen/x86 implementation. The question is: What forces a domain to give up the CPU? At a high-level, it probably works something like this: 1) A timer interrupt occurs which causes an interrupt handler (do_IRQ()->timer_interrupt()?) to be launched. 2) Some counter is incremented 3) The counter exceeds some threshold which results in some routine being called which "evicts" the currently running domain (or in -unstable the running exec_domain). 4) The scheduler is called (__enter_scheduler()?) which looks at a queue to determine what tasks are runnable and chooses one and launches it. Looking at the code, I am missing some key connection between the interrupt routines and the scheduler routines. Perhaps there is some hidden call in a macro or in a function pointer indirection that I am missing. Could someone walk me through at a low level the sequence of Xen/x86 calls that occur when a timer interrupt results in some timer expiration that results in the currently running domain getting "paused" and another domain launched? Thanks for any help! Dan ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Magenheimer, Dan (HP Labs Fort Collins) <dan.magenheimer@hp.com> wrote:> Looking at the code, I am missing some key connection > between the interrupt routines and the scheduler routines. > Perhaps there is some hidden call in a macro or in > a function pointer indirection that I am missing. > > Could someone walk me through at a low level the sequence > of Xen/x86 calls that occur when a timer interrupt results in > some timer expiration that results in the currently running > domain getting "paused" and another domain launched?There are certainly better people to address this, since I just started looking at this code last week, but here''s what I''ve figured out so far. start_of_day() calls scheduler_init(), in which an ac_timer named s_timer is initialized for each CPU''s schedule_data. Each time one of these goes off the s_timer_fn() function is invoked which raises the SCHEDULER_SOFTIRQ. Later, setup() calls schedulers_start(), which invokes s_timer_fn() for the first time to raise the first scheduler softirq. __enter_scheduler(), which is attached to the scheduler softirq by the call to open_softirq() in scheduler_init(), removes the ac_timer when starting, then adds a new one based on the time slice provided by the scheduler''s do_schedule() routine. See the "reprogram the timer" comment in __enter_scheduler(). When that timer goes off the scheduler softirq will be raised by s_timer_fn() again, causing __enter_scheduler() to be invoked again. The softirqs are handled in entry.S. ret_from_intr checks whether there are events pending and, if so, jumps to test_all_events. (It appears that several places jump to test_all_events and it naturally follows a hypercall invocation.) test_all_events checks for pending softirqs, and jumps to process_softirqs to handle them. This calls do_softirq() (in softirq.c), which indirectly invokes __enter_scheduler() through the softirq_handlers method table. The ac_timer functions (s_timer_fn for the scheduler timer) are invoked by ac_timer_softirq_action() which is invoked (also by do_softirq()) when AC_TIMER_SOFTIRQ is raised. This ac_timer softirq is raised by smp_apic_timer_interrupt() (and timer_interrupt() if there is no apic). Note that smp_apic_timer_interrupt() is invoked by the apic_timer_interrupt() function created by the BUILD_SMP_TIMER_INTERRUPT() macro in irq.h (the use of the macro is in i8259.c) and registered in the timer''s interrupt gate in init_IRQ() (which is also called by start_of_day()). So, a timer interrupt gives: o timer interrupt -> apic_timer_interrupt() -> smp_apic_timer_interrupt() -> raises AC_TIMER_SOFTIRQ Then, next time leaving Xen (through entry.S): o test_all_events: -> process_softirqs: -> do_softirq() -> softirq_handlers[AC_TIMER_SOFTIRQ] == ac_timer_softirq_action() -> s_timer_fn() -> raises SCHEDULER_SOFTIRQ Then, since AC_TIMER_SOFTIRQ == SCHEDULER_SOFTIRQ - 1, the next iteration of do_softirq()''s loop invokes the scheduler: o do_softirq() -> softirq_handlers[SCHEDULER_SOFTIRQ] == __enter_scheduler() Note that I''m looking at 2.0.4. Also, there''s a good chance that some of this is not quite right or complete, in which case I''d appreciate being corrected by someone who has a better understanding of the code base. Andrew ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Magenheimer, Dan (HP Labs Fort Collins)
2005-Mar-05 21:12 UTC
RE: [Xen-devel] Scheduler guts question
Perfect! Thanks! Because Linux/ia64 uses the architectural timer (ar.itc) for all timer events, I missed the critical dependency on the APIC timer.> -----Original Message----- > From: Andrew Biggadike [mailto:biggadike@cmu.edu] > Sent: Friday, March 04, 2005 7:21 PM > To: Magenheimer, Dan (HP Labs Fort Collins) > Cc: xen-devel@lists.sourceforge.net > Subject: Re: [Xen-devel] Scheduler guts question > > Magenheimer, Dan (HP Labs Fort Collins) > <dan.magenheimer@hp.com> wrote: > > Looking at the code, I am missing some key connection > > between the interrupt routines and the scheduler routines. > > Perhaps there is some hidden call in a macro or in > > a function pointer indirection that I am missing. > > > > Could someone walk me through at a low level the sequence > > of Xen/x86 calls that occur when a timer interrupt results in > > some timer expiration that results in the currently running > > domain getting "paused" and another domain launched? > > There are certainly better people to address this, since I > just started > looking at this code last week, but here''s what I''ve figured > out so far. > > start_of_day() calls scheduler_init(), in which an ac_timer named > s_timer is initialized for each CPU''s schedule_data. Each time one of > these goes off the s_timer_fn() function is invoked which raises the > SCHEDULER_SOFTIRQ. Later, setup() calls schedulers_start(), which > invokes s_timer_fn() for the first time to raise the first scheduler > softirq. > > __enter_scheduler(), which is attached to the scheduler softirq by the > call to open_softirq() in scheduler_init(), removes the ac_timer when > starting, then adds a new one based on the time slice provided by the > scheduler''s do_schedule() routine. See the "reprogram the timer" > comment in __enter_scheduler(). When that timer goes off the > scheduler > softirq will be raised by s_timer_fn() again, causing > __enter_scheduler() to be invoked again. > > The softirqs are handled in entry.S. ret_from_intr checks > whether there > are events pending and, if so, jumps to test_all_events. (It appears > that several places jump to test_all_events and it naturally follows a > hypercall invocation.) test_all_events checks for pending > softirqs, and > jumps to process_softirqs to handle them. This calls do_softirq() (in > softirq.c), which indirectly invokes __enter_scheduler() through the > softirq_handlers method table. > > The ac_timer functions (s_timer_fn for the scheduler timer) > are invoked > by ac_timer_softirq_action() which is invoked (also by do_softirq()) > when AC_TIMER_SOFTIRQ is raised. This ac_timer softirq is raised by > smp_apic_timer_interrupt() (and timer_interrupt() if there is > no apic). > Note that smp_apic_timer_interrupt() is invoked by the > apic_timer_interrupt() function created by the > BUILD_SMP_TIMER_INTERRUPT() macro in irq.h (the use of the macro is in > i8259.c) and registered in the timer''s interrupt gate in init_IRQ() > (which is also called by start_of_day()). > > > So, a timer interrupt gives: > o timer interrupt -> > apic_timer_interrupt() -> > smp_apic_timer_interrupt() -> raises AC_TIMER_SOFTIRQ > > Then, next time leaving Xen (through entry.S): > o test_all_events: -> > process_softirqs: -> > do_softirq() -> > softirq_handlers[AC_TIMER_SOFTIRQ] == > ac_timer_softirq_action() -> > s_timer_fn() -> raises SCHEDULER_SOFTIRQ > > Then, since AC_TIMER_SOFTIRQ == SCHEDULER_SOFTIRQ - 1, the next > iteration of do_softirq()''s loop invokes the scheduler: > o do_softirq() -> > softirq_handlers[SCHEDULER_SOFTIRQ] == __enter_scheduler() > > > Note that I''m looking at 2.0.4. Also, there''s a good chance that some > of this is not quite right or complete, in which case I''d appreciate > being corrected by someone who has a better understanding of the code > base. > > Andrew >------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel