Ian Campbell
2009-Nov-24 10:16 UTC
[Xen-devel] [PATCH] xen: register timer interrupt with IRQF_TIMER
Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents correct operation of stop_machine on multi-processor systems and breaks suspend. Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Cc: Jeremy Fitzhardinge <jeremy@goop.org> Cc: xen-devel@lists.xensource.com --- arch/x86/xen/time.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c index 6bbff94..9d1f853 100644 --- a/arch/x86/xen/time.c +++ b/arch/x86/xen/time.c @@ -434,7 +434,7 @@ void xen_setup_timer(int cpu) name = "<timer kasprintf failed>"; irq = bind_virq_to_irqhandler(VIRQ_TIMER, cpu, xen_timer_interrupt, - IRQF_DISABLED|IRQF_PERCPU|IRQF_NOBALANCING, + IRQF_DISABLED|IRQF_PERCPU|IRQF_NOBALANCING|IRQF_TIMER, name, NULL); evt = &per_cpu(xen_clock_events, cpu); -- 1.5.6.5 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2009-Nov-24 10:41 UTC
[Xen-devel] Re: [PATCH] xen: register timer interrupt with IRQF_TIMER
On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote:> Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents > correct operation of stop_machine on multi-processor systems and breaks > suspend.This appears to have been broken since after 2.6.29 sometime (I see failures with 2.6.30). I suspect this commit, or one related to it, made IRQF_TIMER a hard requirement: commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f Author: Rafael J. Wysocki <rjw@sisk.pl> Date: Mon Mar 16 22:33:49 2009 +0100 PM: Introduce functions for suspending and resuming device interrupts Introduce helper functions allowing us to prevent device drivers from getting any interrupts (without disabling interrupts on the CPU) during suspend (or hibernation) and to make them start to receive interrupts again during the subsequent resume. These functions make it possible to keep timer interrupts enabled while the "late" suspend and "early" resume callbacks provided by device drivers are being executed. In turn, this allows device drivers'' "late" suspend and "early" resume callbacks to sleep, execute ACPI callbacks etc. The functions introduced here will be used to rework the handling of interrupts during suspend (hibernation) and resume. Namely, interrupts will only be disabled on the CPU right before suspending sysdevs, while device drivers will be prevented from receiving interrupts, with the help of the new helper function, before their "late" suspend callbacks run (and analogously during resume). Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Valtteri Kiviniemi
2009-Nov-24 12:51 UTC
Re: [Xen-devel] Re: [PATCH] xen: register timer interrupt with IRQF_TIMER
Hi, This is not still working, when I try to save a domain: file: root@xen:/# du -h db.save 64K db.save domU console. Ignoring shutdown request: suspend xend.log: [2009-11-24 14:47:57 2131] DEBUG (XendCheckpoint:124) [xc_save]: /usr/lib/xen/bin/xc_save 54 1 0 0 0 [2009-11-24 14:47:57 2131] INFO (XendCheckpoint:424) xc_save: failed to get the suspend evtchn port domU config: kernel = "/boot/bzImage-domU" builder = "linux" memory = "1024" name = "db" vcpus = "8" cpus = [ "0", "1", "2", "3", "4", "5", "6", "7" ] tsc_native="1" vif = [ "mac=00:50:56:13:27:d9, bridge=eth0" ] disk = [ "phy:/dev/virtuals/db,xvda1,w" ] root = "/dev/xvda1 ro" extra = "console=hvc0" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" Latest xen-unstable (compiled 1 hour ago) latest 2.6.31.6 kernel with this patch applied. - Valtteri Kiviniemi Ian Campbell kirjoitti:> On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote: >> Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents >> correct operation of stop_machine on multi-processor systems and breaks >> suspend. > > This appears to have been broken since after 2.6.29 sometime (I see > failures with 2.6.30). I suspect this commit, or one related to it, made > IRQF_TIMER a hard requirement: > > commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f > Author: Rafael J. Wysocki <rjw@sisk.pl> > Date: Mon Mar 16 22:33:49 2009 +0100 > > PM: Introduce functions for suspending and resuming device interrupts > > Introduce helper functions allowing us to prevent device drivers from > getting any interrupts (without disabling interrupts on the CPU) > during suspend (or hibernation) and to make them start to receive > interrupts again during the subsequent resume. These functions make it > possible to keep timer interrupts enabled while the "late" suspend and > "early" resume callbacks provided by device drivers are being > executed. In turn, this allows device drivers'' "late" suspend and > "early" resume callbacks to sleep, execute ACPI callbacks etc. > > The functions introduced here will be used to rework the handling of > interrupts during suspend (hibernation) and resume. Namely, > interrupts will only be disabled on the CPU right before suspending > sysdevs, while device drivers will be prevented from receiving > interrupts, with the help of the new helper function, before their > "late" suspend callbacks run (and analogously during resume). > > Ian. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2009-Nov-24 13:11 UTC
Re: [Xen-devel] Re: [PATCH] xen: register timer interrupt with IRQF_TIMER
On Tue, 2009-11-24 at 12:51 +0000, Valtteri Kiviniemi wrote:> Hi, > > This is not still working, when I try to save a domain: > > file: > > root@xen:/# du -h db.save > 64K db.save > > domU console. > > Ignoring shutdown request: suspendAs I said before - enable CONFIG_PM_SLEEP, you won''t get anywhere without it. Ian.> > xend.log: > > [2009-11-24 14:47:57 2131] DEBUG (XendCheckpoint:124) [xc_save]: > /usr/lib/xen/bin/xc_save 54 1 0 0 0 > [2009-11-24 14:47:57 2131] INFO (XendCheckpoint:424) xc_save: failed to > get the suspend evtchn port > > domU config: > > kernel = "/boot/bzImage-domU" > builder = "linux" > memory = "1024" > name = "db" > vcpus = "8" > cpus = [ "0", "1", "2", "3", "4", "5", "6", "7" ] > tsc_native="1" > vif = [ "mac=00:50:56:13:27:d9, bridge=eth0" ] > disk = [ "phy:/dev/virtuals/db,xvda1,w" ] > root = "/dev/xvda1 ro" > extra = "console=hvc0" > on_poweroff = "destroy" > on_reboot = "restart" > on_crash = "restart" > > Latest xen-unstable (compiled 1 hour ago) latest 2.6.31.6 kernel with > this patch applied. > > - Valtteri Kiviniemi > > Ian Campbell kirjoitti: > > On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote: > >> Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents > >> correct operation of stop_machine on multi-processor systems and breaks > >> suspend. > > > > This appears to have been broken since after 2.6.29 sometime (I see > > failures with 2.6.30). I suspect this commit, or one related to it, made > > IRQF_TIMER a hard requirement: > > > > commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f > > Author: Rafael J. Wysocki <rjw@sisk.pl> > > Date: Mon Mar 16 22:33:49 2009 +0100 > > > > PM: Introduce functions for suspending and resuming device interrupts > > > > Introduce helper functions allowing us to prevent device drivers from > > getting any interrupts (without disabling interrupts on the CPU) > > during suspend (or hibernation) and to make them start to receive > > interrupts again during the subsequent resume. These functions make it > > possible to keep timer interrupts enabled while the "late" suspend and > > "early" resume callbacks provided by device drivers are being > > executed. In turn, this allows device drivers'' "late" suspend and > > "early" resume callbacks to sleep, execute ACPI callbacks etc. > > > > The functions introduced here will be used to rework the handling of > > interrupts during suspend (hibernation) and resume. Namely, > > interrupts will only be disabled on the CPU right before suspending > > sysdevs, while device drivers will be prevented from receiving > > interrupts, with the help of the new helper function, before their > > "late" suspend callbacks run (and analogously during resume). > > > > Ian. > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Valtteri Kiviniemi
2009-Nov-24 14:14 UTC
Re: [Xen-devel] Re: [PATCH] xen: register timer interrupt with IRQF_TIMER
Hi, Ok, sorry didn not notice that. my domU kernels are quite small, they dont contain anything that is not necessary and since power management options were not necessary on Xen 3.4 I did not enable them on unstable either. Gonna now compile new domU kernels and test again. Thanks. - Valtteri Kiviniemi Ian Campbell kirjoitti:> On Tue, 2009-11-24 at 12:51 +0000, Valtteri Kiviniemi wrote: >> Hi, >> >> This is not still working, when I try to save a domain: >> >> file: >> >> root@xen:/# du -h db.save >> 64K db.save >> >> domU console. >> >> Ignoring shutdown request: suspend > > As I said before - enable CONFIG_PM_SLEEP, you won''t get anywhere > without it. > > Ian. > >> xend.log: >> >> [2009-11-24 14:47:57 2131] DEBUG (XendCheckpoint:124) [xc_save]: >> /usr/lib/xen/bin/xc_save 54 1 0 0 0 >> [2009-11-24 14:47:57 2131] INFO (XendCheckpoint:424) xc_save: failed to >> get the suspend evtchn port >> >> domU config: >> >> kernel = "/boot/bzImage-domU" >> builder = "linux" >> memory = "1024" >> name = "db" >> vcpus = "8" >> cpus = [ "0", "1", "2", "3", "4", "5", "6", "7" ] >> tsc_native="1" >> vif = [ "mac=00:50:56:13:27:d9, bridge=eth0" ] >> disk = [ "phy:/dev/virtuals/db,xvda1,w" ] >> root = "/dev/xvda1 ro" >> extra = "console=hvc0" >> on_poweroff = "destroy" >> on_reboot = "restart" >> on_crash = "restart" >> >> Latest xen-unstable (compiled 1 hour ago) latest 2.6.31.6 kernel with >> this patch applied. >> >> - Valtteri Kiviniemi >> >> Ian Campbell kirjoitti: >>> On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote: >>>> Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents >>>> correct operation of stop_machine on multi-processor systems and breaks >>>> suspend. >>> This appears to have been broken since after 2.6.29 sometime (I see >>> failures with 2.6.30). I suspect this commit, or one related to it, made >>> IRQF_TIMER a hard requirement: >>> >>> commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f >>> Author: Rafael J. Wysocki <rjw@sisk.pl> >>> Date: Mon Mar 16 22:33:49 2009 +0100 >>> >>> PM: Introduce functions for suspending and resuming device interrupts >>> >>> Introduce helper functions allowing us to prevent device drivers from >>> getting any interrupts (without disabling interrupts on the CPU) >>> during suspend (or hibernation) and to make them start to receive >>> interrupts again during the subsequent resume. These functions make it >>> possible to keep timer interrupts enabled while the "late" suspend and >>> "early" resume callbacks provided by device drivers are being >>> executed. In turn, this allows device drivers'' "late" suspend and >>> "early" resume callbacks to sleep, execute ACPI callbacks etc. >>> >>> The functions introduced here will be used to rework the handling of >>> interrupts during suspend (hibernation) and resume. Namely, >>> interrupts will only be disabled on the CPU right before suspending >>> sysdevs, while device drivers will be prevented from receiving >>> interrupts, with the help of the new helper function, before their >>> "late" suspend callbacks run (and analogously during resume). >>> >>> Ian. >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Valtteri Kiviniemi
2009-Nov-24 14:23 UTC
Re: [Xen-devel] Re: [PATCH] xen: register timer interrupt with IRQF_TIMER
Hi, I can confirm that save works, but restore does not work. It will restore the domU, but the domU wont respond to anything. In the console I can see root@xen:/# xm console db Using V2 grant tables. And then its stuck, cant login. xend.log from restore: [2009-11-24 16:20:11 14312] DEBUG (balloon:181) Balloon: 16069048 KiB free; need 1048576; done. [2009-11-24 16:20:11 14312] DEBUG (XendCheckpoint:305) [xc_restore]: /usr/lib/xen/bin/xc_restore 4 2 1 2 0 0 0 0 [2009-11-24 16:20:11 14312] INFO (XendCheckpoint:424) xc_domain_restore start: p2m_size = 40000 [2009-11-24 16:20:11 14312] INFO (XendCheckpoint:424) Reloading memory pages: 0% [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) ERROR Internal error: Error when reading batch size [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) ERROR Internal error: error when buffering batch, finishing [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:4100% [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) Memory reloaded (0 pages) [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 0 [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 1 [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 2 [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 3 [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 4 [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 5 [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 6 [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 7 [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) Completed checkpoint load [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) Domain ready to be built. [2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) Restore exit with rc=0 [2009-11-24 16:20:48 14312] DEBUG (XendCheckpoint:395) store-mfn 4037254 [2009-11-24 16:20:48 14312] DEBUG (XendCheckpoint:395) console-mfn 4037253 [2009-11-24 16:20:48 14312] DEBUG (XendDomainInfo:2883) XendDomainInfo.completeRestore - Valtteri Kiviniemi Valtteri Kiviniemi kirjoitti:> Hi, > > Ok, sorry didn not notice that. my domU kernels are quite small, they > dont contain anything that is not necessary and since power management > options were not necessary on Xen 3.4 I did not enable them on unstable > either. Gonna now compile new domU kernels and test again. > > Thanks. > > - Valtteri Kiviniemi > > Ian Campbell kirjoitti: >> On Tue, 2009-11-24 at 12:51 +0000, Valtteri Kiviniemi wrote: >>> Hi, >>> >>> This is not still working, when I try to save a domain: >>> >>> file: >>> >>> root@xen:/# du -h db.save >>> 64K db.save >>> >>> domU console. >>> >>> Ignoring shutdown request: suspend >> >> As I said before - enable CONFIG_PM_SLEEP, you won''t get anywhere >> without it. >> >> Ian. >> >>> xend.log: >>> >>> [2009-11-24 14:47:57 2131] DEBUG (XendCheckpoint:124) [xc_save]: >>> /usr/lib/xen/bin/xc_save 54 1 0 0 0 >>> [2009-11-24 14:47:57 2131] INFO (XendCheckpoint:424) xc_save: failed >>> to get the suspend evtchn port >>> >>> domU config: >>> >>> kernel = "/boot/bzImage-domU" >>> builder = "linux" >>> memory = "1024" >>> name = "db" >>> vcpus = "8" >>> cpus = [ "0", "1", "2", "3", "4", "5", "6", "7" ] >>> tsc_native="1" >>> vif = [ "mac=00:50:56:13:27:d9, bridge=eth0" ] >>> disk = [ "phy:/dev/virtuals/db,xvda1,w" ] >>> root = "/dev/xvda1 ro" >>> extra = "console=hvc0" >>> on_poweroff = "destroy" >>> on_reboot = "restart" >>> on_crash = "restart" >>> >>> Latest xen-unstable (compiled 1 hour ago) latest 2.6.31.6 kernel with >>> this patch applied. >>> >>> - Valtteri Kiviniemi >>> >>> Ian Campbell kirjoitti: >>>> On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote: >>>>> Otherwise the timer is disabled by dpm_suspend_noirq() which in >>>>> turn prevents >>>>> correct operation of stop_machine on multi-processor systems and >>>>> breaks >>>>> suspend. >>>> This appears to have been broken since after 2.6.29 sometime (I see >>>> failures with 2.6.30). I suspect this commit, or one related to it, >>>> made >>>> IRQF_TIMER a hard requirement: >>>> >>>> commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f >>>> Author: Rafael J. Wysocki <rjw@sisk.pl> >>>> Date: Mon Mar 16 22:33:49 2009 +0100 >>>> PM: Introduce functions for suspending and >>>> resuming device interrupts >>>> Introduce helper functions allowing us to >>>> prevent device drivers from >>>> getting any interrupts (without disabling interrupts on >>>> the CPU) >>>> during suspend (or hibernation) and to make them start >>>> to receive >>>> interrupts again during the subsequent resume. These >>>> functions make it >>>> possible to keep timer interrupts enabled while the >>>> "late" suspend and >>>> "early" resume callbacks provided by device drivers are >>>> being >>>> executed. In turn, this allows device drivers'' "late" >>>> suspend and >>>> "early" resume callbacks to sleep, execute ACPI >>>> callbacks etc. >>>> The functions introduced here will be used >>>> to rework the handling of >>>> interrupts during suspend (hibernation) and resume. >>>> Namely, >>>> interrupts will only be disabled on the CPU right before >>>> suspending >>>> sysdevs, while device drivers will be prevented from >>>> receiving >>>> interrupts, with the help of the new helper function, >>>> before their >>>> "late" suspend callbacks run (and analogously during >>>> resume). >>>> >>>> Ian. >>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel