Hi,
latest xen-unstable crashed when I tried to remove a cpu from a cpupool via
xl cpupool-cpu-remove
Serial log shows:
(XEN) Assertion ''!(((*({ unsigned long __ptr; __asm__ ("" :
"=r"(__ptr) :
"0"(&per_cpu____preempt_count));
(typeof(&per_cpu____preempt_count)) (__ptr +
(get_cpu_info()->per_cpu_offset)); }))) ||
((irq_stat[(((get_cpu_info()->processor_id)))].__local_irq_count) != 0) ||
!local_irq_is_enabled())'' failed at softirq.c:61
(XEN) ----[ Xen-4.1-unstable x86_64 debug=y Tainted: C ]----
(XEN) CPU: 5
(XEN) RIP: e008:[<ffff82c4801212f9>] do_softirq+0x51/0x7a
(XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor
(XEN) rax: ffff82c4802ca2a0 rbx: ffff8300780da000 rcx: ffff82c4802a7080
(XEN) rdx: 00000043b9aefd80 rsi: ffff830839db7ec0 rdi: ffff82c48024d0e0
(XEN) rbp: ffff830839db7f08 rsp: ffff830839db7f08 r8: 0000000000000005
(XEN) r9: 0000ffff0000ffff r10: 00000000deadbeef r11: 0f0f0f0f0f0f0f0f
(XEN) r12: ffff8803f8f61440 r13: ffff8803fdf4fe58 r14: 0000000000000000
(XEN) r15: 00007fff335d7150 cr0: 0000000080050033 cr4: 00000000000026f0
(XEN) cr3: 0000000846990000 cr2: 0000000000627048
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
(XEN) Xen stack trace from rsp=ffff830839db7f08:
(XEN) 00007cf7c62480c7 ffff82c480202736 00007fff335d7150 0000000000000000
(XEN) ffff8803fdf4fe58 ffff8803f8f61440 ffffffffffffffda ffff8803fd491438
(XEN) 0000000000000216 000000000061e004 000000000061d050 00007f6d701d9f6c
(XEN) 0000000000000000 ffffffff8000346a 000000000061c2e0 0000000000624004
(XEN) 00000000deadbeef 0000010000000000 ffffffff8000346a 000000000000e033
(XEN) 0000000000000216 ffff8803fdf4fd80 000000000000e02b 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000005
(XEN) ffff8300780da000 00000043b9aefd80 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82c4801212f9>] do_softirq+0x51/0x7a
(XEN)
I suspect the waitqueue stuff broke the continue_hypercall_on_cpu() stuff.
Juergen
--
Juergen Gross Principal Developer Operating Systems
TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28 Internet: ts.fujitsu.com
D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2010-Nov-25 17:14 UTC
Re: [Xen-devel] hypervisor crash with latest xen-unstable
On 25/11/2010 10:27, "Juergen Gross" <juergen.gross@ts.fujitsu.com> wrote:> (XEN) Xen call trace: > (XEN) [<ffff82c4801212f9>] do_softirq+0x51/0x7a > (XEN) > > I suspect the waitqueue stuff broke the continue_hypercall_on_cpu() stuff.Then please bisect to confirm that. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Juergen Gross
2010-Nov-26 05:32 UTC
Re: [Xen-devel] hypervisor crash with latest xen-unstable
On 11/25/10 18:14, Keir Fraser wrote:> On 25/11/2010 10:27, "Juergen Gross"<juergen.gross@ts.fujitsu.com> wrote: > >> (XEN) Xen call trace: >> (XEN) [<ffff82c4801212f9>] do_softirq+0x51/0x7a >> (XEN) >> >> I suspect the waitqueue stuff broke the continue_hypercall_on_cpu() stuff. > > Then please bisect to confirm that.Removing the ASSERT(!in_atomic()) in do_softirq() and schedule() makes it work again. And these assertions were added with cs22395. The question now is, whether these assertions are wrong or if it is just working by chance without them. It seems to be necessary to be able to call schedule() in this case, as adding if (in_atomic()) return; to do_softirq() made the hypercall hang. Juergen -- Juergen Gross Principal Developer Operating Systems TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967 Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com Domagkstr. 28 Internet: ts.fujitsu.com D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Nov-26 10:12 UTC
Re: [Xen-devel] hypervisor crash with latest xen-unstable
Fixed by xen-unstable:22342. K. On 25/11/2010 10:27, "Juergen Gross" <juergen.gross@ts.fujitsu.com> wrote:> Hi, > > latest xen-unstable crashed when I tried to remove a cpu from a cpupool via > > xl cpupool-cpu-remove > > Serial log shows: > > (XEN) Assertion ''!(((*({ unsigned long __ptr; __asm__ ("" : "=r"(__ptr) : > "0"(&per_cpu____preempt_count)); (typeof(&per_cpu____preempt_count)) (__ptr + > (get_cpu_info()->per_cpu_offset)); }))) || > ((irq_stat[(((get_cpu_info()->processor_id)))].__local_irq_count) != 0) || > !local_irq_is_enabled())'' failed at softirq.c:61 > (XEN) ----[ Xen-4.1-unstable x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 5 > (XEN) RIP: e008:[<ffff82c4801212f9>] do_softirq+0x51/0x7a > (XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor > (XEN) rax: ffff82c4802ca2a0 rbx: ffff8300780da000 rcx: ffff82c4802a7080 > (XEN) rdx: 00000043b9aefd80 rsi: ffff830839db7ec0 rdi: ffff82c48024d0e0 > (XEN) rbp: ffff830839db7f08 rsp: ffff830839db7f08 r8: 0000000000000005 > (XEN) r9: 0000ffff0000ffff r10: 00000000deadbeef r11: 0f0f0f0f0f0f0f0f > (XEN) r12: ffff8803f8f61440 r13: ffff8803fdf4fe58 r14: 0000000000000000 > (XEN) r15: 00007fff335d7150 cr0: 0000000080050033 cr4: 00000000000026f0 > (XEN) cr3: 0000000846990000 cr2: 0000000000627048 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from rsp=ffff830839db7f08: > (XEN) 00007cf7c62480c7 ffff82c480202736 00007fff335d7150 0000000000000000 > (XEN) ffff8803fdf4fe58 ffff8803f8f61440 ffffffffffffffda ffff8803fd491438 > (XEN) 0000000000000216 000000000061e004 000000000061d050 00007f6d701d9f6c > (XEN) 0000000000000000 ffffffff8000346a 000000000061c2e0 0000000000624004 > (XEN) 00000000deadbeef 0000010000000000 ffffffff8000346a 000000000000e033 > (XEN) 0000000000000216 ffff8803fdf4fd80 000000000000e02b 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000005 > (XEN) ffff8300780da000 00000043b9aefd80 0000000000000000 > (XEN) Xen call trace: > (XEN) [<ffff82c4801212f9>] do_softirq+0x51/0x7a > (XEN) > > I suspect the waitqueue stuff broke the continue_hypercall_on_cpu() stuff. > > > Juergen_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Juergen Gross
2010-Nov-26 10:24 UTC
Re: [Xen-devel] hypervisor crash with latest xen-unstable
Thanks, is working again! Juergen On 11/26/10 11:12, Keir Fraser wrote:> Fixed by xen-unstable:22342. > > K. > > On 25/11/2010 10:27, "Juergen Gross"<juergen.gross@ts.fujitsu.com> wrote: > >> Hi, >> >> latest xen-unstable crashed when I tried to remove a cpu from a cpupool via >> >> xl cpupool-cpu-remove >> >> Serial log shows: >> >> (XEN) Assertion ''!(((*({ unsigned long __ptr; __asm__ ("" : "=r"(__ptr) : >> "0"(&per_cpu____preempt_count)); (typeof(&per_cpu____preempt_count)) (__ptr + >> (get_cpu_info()->per_cpu_offset)); }))) || >> ((irq_stat[(((get_cpu_info()->processor_id)))].__local_irq_count) != 0) || >> !local_irq_is_enabled())'' failed at softirq.c:61 >> (XEN) ----[ Xen-4.1-unstable x86_64 debug=y Tainted: C ]---- >> (XEN) CPU: 5 >> (XEN) RIP: e008:[<ffff82c4801212f9>] do_softirq+0x51/0x7a >> (XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor >> (XEN) rax: ffff82c4802ca2a0 rbx: ffff8300780da000 rcx: ffff82c4802a7080 >> (XEN) rdx: 00000043b9aefd80 rsi: ffff830839db7ec0 rdi: ffff82c48024d0e0 >> (XEN) rbp: ffff830839db7f08 rsp: ffff830839db7f08 r8: 0000000000000005 >> (XEN) r9: 0000ffff0000ffff r10: 00000000deadbeef r11: 0f0f0f0f0f0f0f0f >> (XEN) r12: ffff8803f8f61440 r13: ffff8803fdf4fe58 r14: 0000000000000000 >> (XEN) r15: 00007fff335d7150 cr0: 0000000080050033 cr4: 00000000000026f0 >> (XEN) cr3: 0000000846990000 cr2: 0000000000627048 >> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 >> (XEN) Xen stack trace from rsp=ffff830839db7f08: >> (XEN) 00007cf7c62480c7 ffff82c480202736 00007fff335d7150 0000000000000000 >> (XEN) ffff8803fdf4fe58 ffff8803f8f61440 ffffffffffffffda ffff8803fd491438 >> (XEN) 0000000000000216 000000000061e004 000000000061d050 00007f6d701d9f6c >> (XEN) 0000000000000000 ffffffff8000346a 000000000061c2e0 0000000000624004 >> (XEN) 00000000deadbeef 0000010000000000 ffffffff8000346a 000000000000e033 >> (XEN) 0000000000000216 ffff8803fdf4fd80 000000000000e02b 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000005 >> (XEN) ffff8300780da000 00000043b9aefd80 0000000000000000 >> (XEN) Xen call trace: >> (XEN) [<ffff82c4801212f9>] do_softirq+0x51/0x7a >> (XEN) >> >> I suspect the waitqueue stuff broke the continue_hypercall_on_cpu() stuff. >> >> >> Juergen > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > >-- Juergen Gross Principal Developer Operating Systems TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967 Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com Domagkstr. 28 Internet: ts.fujitsu.com D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel