I haven''t made any further progress on this [1]bug, even with adding in
some extra tracing.
When we put xm create following by xm destroy in a loop, eventually xen
hits a hard reboot dumping out a BUG_ON()
(XEN) BUG at domain.c:1054
(XEN) CPU: 0
(XEN) EIP: e008:[<ff12a6b6>] domain_relinquish_resources+0x43/0x1c8
(XEN) EFLAGS: 00010282 CONTEXT: hypervisor
(XEN) eax: ff187fb8 ebx: ffbf1080 ecx: ffbf4000 edx: 00000000
(XEN) esi: 00000007 edi: ff103fac ebp: ff103b0c esp: ff103af4
(XEN) cr0: 8005003b cr3: db629000
(XEN) ds: e010 es: e010 fs: 0000 gs: 0000 ss: e010 cs: e008
(XEN) Xen stack trace from esp=ff103af4:
(XEN) ff17ba49 ff17bb67 0000041e ff11cfa9 ffbd9094 00000000 ff103b2c ff109027
(XEN) ffbda080 ffbda308 00000001 ffbda080 ffbda080 00000000 ff103f8c ff107f88
(XEN) ffbda080 ff103dbc 00000000 ffbda310 00000001 00000001 ff103b5c ff13f087
(XEN) 00000003 00000000 ffbc7080 ffbda080 ff103b74 00000020 ff103b8c ff10c368
(XEN) 00000000 ffbf10d4 ff103b9c ff130211 00000001 00000001 00000001 00000004
(XEN) ffbf1e00 ff103cc8 ff111a6c ff103c2c 00000001 00000009 ff103bbc 00000004
(XEN) 00000004 ff103fb4 ff103bcc ff1233a3 00000004 80000003 80000003 00000004
(XEN) 80000002 80000003 fd8e671c ff12fb9a ff103be8 80000004 ffbf2080 ffbf2080
(XEN) fd750ef4 ff103fac ff103c5c 80000003 00ef0000 80000003 80000003 80000003
(XEN) 80000002 ff103c48 ff1352eb fd8e6710 ff103c34 00000020 ff103c4c 00000001
(XEN) 80000003 00000001 ffbf2080 ffbf2080 ffbf2080 80000003 80000004 80000003
(XEN) ffbf2080 00000000 ff103c8c ff1350ed fd70c770 ffbf2080 ffbf2080 00000540
(XEN) c9e72063 00000000 00000001 00000001 00000000 000001d1 ff103cbc ff135407
(XEN) fedd1000 ffbf2080 00000400 00000001 00000004 ffbf1e00 ff103ddc ff111a6c
(XEN) ff103d40 00000001 00000008 ff135de0 fd6edab0 00000001 ff103cdc ff13b552
(XEN) 20000000 00000000 0000b400 000004c4 c4b40000 00000004 00008000 00009b42
(XEN) 000007dc 00000000 f0000000 04c4b400 00000004 00000001 33ef003c 33ef003c
(XEN) 00000000 00000000 00000000 00000000 000007dc 8000003e 8000003f 8000003e
(XEN) ffbf2080 00000000 00000000 00009b42 00008000 33ef003c 00000000 33ef003c
(XEN) 00000000 00000000 00000000 0000755a 00000000 23ef0000 ff103d7c ff12315c
(XEN) Xen call trace:
(XEN) [<ff12a6b6>] domain_relinquish_resources+0x43/0x1c8
(XEN) [<ff109027>] domain_kill+0x62/0x9e
(XEN) [<ff107f88>] do_dom0_op+0x54d/0x103b
(XEN) [<ff155d8f>] hypercall+0x8f/0xaf
the line in question is:
BUG_ON(!cpus_empty(d->cpumask));
This says to me, bail if the domain''s cpumask is NOT empty. AFAICT,
the only places that cpumask for a domain are modified are:
1. startup_cpu_idle_loop() in xen/arch/x86/domain.c
cpu_set(smp_processor_id(),v->domain->cpumask);
2. __context_switch() in xen/arch/x86/domain.c
if ( p->domain != n->domain )
cpu_set(cpu, n->domain->cpumask);
...
if ( p->domain != n->domain )
cpu_clear(cpu, p->domain->cpumask);
If we are hitting that assert, then one or more of the vcpus in the
domain are still running? Any help on pointing out where I''ve
misunderstood whats happening or good places to insert some debugging
would be of great help.
1. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=197
--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
ryanh@us.ibm.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
I don''t think this is the right fix, but it does highlight the issue.
While killing a domain, the vcpus are descheduled, but every
now and then, one of the cpus is still running one of the vcpus, which
means d->cpumask is not empty. This triggers the BUG_ON() in
xen/arch/x86/domain.c:domain_relinquish_resources(). The patch puts in
some printks and a cpu_relax() loop till the cpumask is empty before
calling domain_relinquish_resources(). With this patch, I''ve gone
through several thousand iterations of create/destroy without crashing.
--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
ryanh@us.ibm.com
diffstat output:
domain.c | 8 ++++++++
1 files changed, 8 insertions(+)
Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
---
diff -r 413c911e5780 xen/common/domain.c
--- a/xen/common/domain.c Mon Sep 12 12:48:33 2005
+++ b/xen/common/domain.c Mon Sep 12 13:25:07 2005
@@ -112,6 +112,14 @@
{
for_each_vcpu(d, v)
sched_rem_domain(v);
+ if(!cpus_empty(d->cpumask)) {
+ printk("ACK! DOM%d still running, waiting before
dying\n",
+ d->domain_id);
+ while(!cpus_empty(d->cpumask))
+ cpu_relax();
+ printk("DOM%d cpumask clear, relinquishing resources\n",
+ d->domain_id);
+ }
domain_relinquish_resources(d);
put_domain(d);
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
RH> With this patch, I''ve gone through several thousand iterations of RH> create/destroy without crashing. The test 10_create_fastdestroy in the xm-test suite pokes this bug every time for me. After applying this patch, I get the following on a serial console while running the test: (XEN) ACK! DOM1 still running, waiting before dying After that, the system is completely hung. -- Dan Smith IBM Linux Technology Center Open Hypervisor Team email: danms@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks for trying out the last patch. While the previous workaround worked for
me, it did not for others. Looking into domain_pause(), we can see that
vcpu_sleep_sync() is called on each vcpu:
/*
* We can be sure that the VCPU is finally descheduled after the running
* flag is cleared and the scheduler lock is released.
*/
while ( test_bit(_VCPUF_running, &v->vcpu_flags)
&& !domain_runnable(v)
&&
spin_is_locked(&schedule_data[v->processor].schedule_lock) )
cpu_relax();
If we are to believe the comment, (which makes sense), then the while loop code
is broken. That is, this function will spin until *any* of the three tests
returns false rather than waiting until *all* tests are false. This patch
switches the &&s to ||s and inverts the domain_runnable() check. I
believe we
want to spin while 1) vcpu_running flag is up 2) the domain is runnable and 3)
the scheduler lock is held.
--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
ryanh@us.ibm.com
diffstat output:
schedule.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
---
#
# fix logic to match comments. ie. we want to spin until
# 1) the running flag is down,
# 2) the domain isnt runnable (pausecnt > 0)
# 3) the scheduler lock isnt held
#
# Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
#
diff -r 413c911e5780 xen/common/schedule.c
--- a/xen/common/schedule.c Mon Sep 12 12:48:33 2005
+++ b/xen/common/schedule.c Tue Sep 13 09:46:36 2005
@@ -214,8 +214,8 @@
* flag is cleared and the scheduler lock is released.
*/
while ( test_bit(_VCPUF_running, &v->vcpu_flags)
- && !domain_runnable(v)
- &&
spin_is_locked(&schedule_data[v->processor].schedule_lock) )
+ || domain_runnable(v)
+ ||
spin_is_locked(&schedule_data[v->processor].schedule_lock) )
cpu_relax();
sync_vcpu_execstate(v);
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2005-Sep-13 19:04 UTC
Re: [Xen-devel] [PATCH] workaround for bug#197: second try
On 13 Sep 2005, at 19:22, Ryan Harper wrote:> > If we are to believe the comment, (which makes sense), then the while > loop code > is broken. That is, this function will spin until *any* of the three > tests > returns false rather than waiting until *all* tests are false. This > patch > switches the &&s to ||s and inverts the domain_runnable() check. I > believe we > want to spin while 1) vcpu_running flag is up 2) the domain is > runnable and 3) > the scheduler lock is held.Good catch. Actually we want to spin only while the domain is non-runnable (just to catch the very unlikely event that someone else wakes the domain while we spin). I updated the comment and the loop predicate and checked the patch in. Thanks! -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Smith
2005-Sep-13 19:23 UTC
Re: [Xen-devel] [PATCH] workaround for bug#197: second try
RH> This patch switches the &&s to ||s and inverts the RH> domain_runnable() check. I believe we want to spin while 1) RH> vcpu_running flag is up 2) the domain is runnable and 3) the RH> scheduler lock is held. This fixed it for me as well... Thanks Ryan! -- Dan Smith IBM Linux Technology Center Open Hypervisor Team email: danms@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel