Hi Keir & Jan:
Good news is, the patch for Xen panic bug works. Tests on two servers
are running happily
for almost two days. I will stop test if it doesn''t fail until tomorrow
evening.
Appreciate for all help you and Jan offered.
Well, till now, I think I''ve collected enough information for
VM hang problem to have a discussion
with you two. Basically, we have two situation of VM hang(Those VM are all HVMs,
and all are able to
run well for some time before actually hung).
1. We have two VMs in this situation. And under this situation, what we know
clear are
(1) The *times* column in "xm ls" command never changed after VM hangs
In below three VMs, E2EZYXVM-56-W2.786.92 is hang(its "times" 9339.3
freezed), and the other two work well.
E2EZYXVM-56-W1.786.92 2 1024 2 -b----
29009.0
E2EZYXVM-56-W2.786.92 3 1024 2 ------
9339.3
E2EZYXVM-56-W3.786.92 4 1024 2 -b----
27538.6
(2) From Xenctx output it call trace is same, and never change on every xenctx
run.
Call trace likes:
Call Trace:
[<80708a5a>] <--
[<f76f1789>]
[<85f3f1f0>]
[<861fb0e8>]
[<861fb370>]
[<80558188>]
[<f76f3c1f>]
[<861fb370>]
[<85f3f1f0>]
[<861fb0e8>]
2. We have another two VMS on this situation, what we know is
(1) The *times* column in "xm ls" command are much higher than other
VMS, and become larger very fast
In below three VMs, E2EZYXVM-138-W5.827.92is hang(its "times" 58262.8
grows every seconds),
and the other two work well.
E2EZYXVM-138-W4.827.92 5 1024 2 r----- 27692.5
E2EZYXVM-138-W5.827.92 6 1024 2 r----- 58262.8
E2EZYXVM-138-W6.827.92 7 1024 2 r----- 26954.3
(2) From Xenctx output it call trace is same, and never change on every xenctx
run.
Call Trace:
[<80708a66>] <--
[<f7c2f072>]
[<861fa9dc>]
[<805582a8>]
[<f7c318a5>]
[<861fa9dc>]
[<861fa9dc>]
[<861fa0e0>]
[<861faa08>]
In addition, we have another VM which is in black screen and hung. But there is
no abnormal information
I can get form "xm li" and xenctx.
Early in this afternoon, I was trying to decode those back trace to get symbols
(HVMS is windows XP) but failed.
I am wondering If I could trigger a Domain U crash, and have the dump analyzed
on windbg. Basically I am trying
to find out what did VM do exactly before hung.
Looking forward to your suggestion, thanks.
> Date: Wed, 8 Sep 2010 06:11:07 -0700
> Subject: Re: [Xen-devel] Xen-unstable panic: FATAL PAGE FAULT
> From: keir.fraser@eu.citrix.com
> To: tinnycloud@hotmail.com; JBeulich@novell.com
>
> On 08/09/2010 02:03, "MaoXiaoyun" <tinnycloud@hotmail.com>
wrote:
>
> > Here is my plan. I hope I could find a way to make it reproduced
eaiser(Right
> > now hang shows
> > in a very small possibility). Also, I will learn to use xentrace,
xenanyle to
> > help locate the bug.
> > I wonder if there exists a way that I can dump the guest Domain info,
or at
> > least find out
> > where VM hang on, or have the backtrace.
>
> There is a tool called xenctx in tools/xentrace/ directory. This will dump
> registers and stack for a specified domain and vcpu. I think it may even be
> able to dump symbolic call traces for Linux kernels, if youc an pass it the
> vmlinux file.
>
> -- Keir
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Hi Keir:
Regards to HVM hang , according to our recent test, it turns out this
issue still exists.
When I go through the code, I obseved something abnormal and need your
help.
We''ve noticed when VM hang, its VCPU flags is always 4, which
indicates _VPF_blocked_in_xen,
and it is invoked in prepare_wait_on_xen_event_channel. I''ve
noticed that Domain U has setup
a event channel with domain 0 for each VCPU and qemu-dm select on the
event fd.
notify_via_xen_event_channel is called when Domain U issue a request. And
in qemu-dm it will
get the event, and invoke
cpu_handle_ioreq(/xen-4.0.0/tools/ioemu-qemu-xen/i386-dm/helper2.c)
->cpu_get_ioreq()->xc_evtchn_unmask(). In evtchn_unmask it will has
operation on evtchn_pending,
evtchn_mask, or evtchn_pending_sel.
My confusion is on notify_via_xen_event_channel()->evtchn_set_pending,
the **evtchn_set_pending here
in not locked**, while inside it also have operation on evtchn_pending,
evtchn_mask, or evtchn_pending_sel.
I''m afried this access competition might cause event undeliverd
from dom U to qemu-dm, but I am not sure,
since I still not fully understand where event_mask and is set, and where
event_pending is cleared.
-------------------------notify_via_xen_event_channel-------------------------------------
989 void notify_via_xen_event_channel(int lport)
990 {
991 struct evtchn *lchn, *rchn;
992 struct domain *ld = current->domain, *rd;
993 int rport;
994
995 spin_lock(&ld->event_lock);
996
997 ASSERT(port_is_valid(ld, lport));
998 lchn = evtchn_from_port(ld, lport);
999 ASSERT(lchn->consumer_is_xen);
1000
1001 if ( likely(lchn->state == ECS_INTERDOMAIN) )
1002 {
1003 rd = lchn->u.interdomain.remote_dom;
1004 rport = lchn->u.interdomain.remote_port;
1005 rchn = evtchn_from_port(rd, rport);
1006 evtchn_set_pending(rd->vcpu[rchn->notify_vcpu_id], rport);
1007 }
1008
1009 spin_unlock(&ld->event_lock);
1010 }
----------------------------evtchn_set_pending----------------------
535 static int evtchn_set_pending(struct vcpu *v, int port)
536 {
537 struct domain *d = v->domain;
538 int vcpuid;
539
540 /*
541 * The following bit operations must happen in strict order.
542 * NB. On x86, the atomic bit operations also act as memory barriers.
543 * There is therefore sufficiently strict ordering for this
architecture --
544 * others may require explicit memory barriers.
545 */
546
547 if ( test_and_set_bit(port, &shared_info(d, evtchn_pending)) )
548 return 1;
549
550 if ( !test_bit (port, &shared_info(d, evtchn_mask))
&&
551 !test_and_set_bit(port / BITS_PER_EVTCHN_WORD(d),
552 &vcpu_info(v, evtchn_pending_sel)) )
553 {
554 vcpu_mark_events_pending(v);
555 }
556
557 /* Check if some VCPU might be polling for this event. */
558 if ( likely(bitmap_empty(d->poll_mask, d->max_vcpus)) )
559 return 0;
560
561 /* Wake any interested (or potentially interested) pollers. */
562 for ( vcpuid = find_first_bit(d->poll_mask, d->max_vcpus);
563 vcpuid < d->max_vcpus;
564 vcpuid = find_next_bit(d->poll_mask, d->max_vcpus,
vcpuid+1) )
565 {
566 v = d->vcpu[vcpuid];
567 if ( ((v->poll_evtchn <= 0) || (v->poll_evtchn == port))
&&
568 test_and_clear_bit(vcpuid, d->poll_mask) )
569 {
570 v->poll_evtchn = 0;
571 vcpu_unblock(v);
--------------------------------------evtchn_unmask------------------------------
764
765 int evtchn_unmask(unsigned int port)
766 {
767 struct domain *d = current->domain;
768 struct vcpu *v;
769
770 spin_lock(&d->event_lock);
771
772 if ( unlikely(!port_is_valid(d, port)) )
773 {
774 spin_unlock(&d->event_lock);
775 return -EINVAL;
776 }
777
778 v = d->vcpu[evtchn_from_port(d, port)->notify_vcpu_id];
779
780 /*
781 * These operations must happen in strict order. Based on
782 * include/xen/event.h:evtchn_set_pending().
783 */
784 if ( test_and_clear_bit(port, &shared_info(d, evtchn_mask))
&&
785 test_bit (port, &shared_info(d, evtchn_pending))
&&
786 !test_and_set_bit (port / BITS_PER_EVTCHN_WORD(d),
787 &vcpu_info(v, evtchn_pending_sel)) )
788 {
789 vcpu_mark_events_pending(v);
790 }
791
792 spin_unlock(&d->event_lock);
793
794 return 0;
795 }
----------------------------cpu_get_ioreq-------------------------
260 static ioreq_t *cpu_get_ioreq(void)
261 {
262 int i;
263 evtchn_port_t port;
264
265 port = xc_evtchn_pending(xce_handle);
266 if (port != -1) {
267 for ( i = 0; i < vcpus; i++ )
268 if ( ioreq_local_port[i] == port )
269 break;
270
271 if ( i == vcpus ) {
272 fprintf(logfile, "Fatal error while trying to get io
event!\n");
273 exit(1);
274 }
275
276 // unmask the wanted port again
277 xc_evtchn_unmask(xce_handle, port);
278
279 //get the io packet from shared memory
280 send_vcpu = i;
281 return __cpu_get_ioreq(i);
282 }
283
284 //read error or read nothing
285 return NULL;
286 }
287
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 19/09/2010 11:37, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote:> Hi Keir: > > Regards to HVM hang , according to our recent test, it turns out this > issue still exists. > When I go through the code, I obseved something abnormal and need your > help. > > We''ve noticed when VM hang, its VCPU flags is always 4, which indicates > _VPF_blocked_in_xen, > and it is invoked in prepare_wait_on_xen_event_channel. I''ve noticed > that Domain U has setup > a event channel with domain 0 for each VCPU and qemu-dm select on the > event fd. > > notify_via_xen_event_channel is called when Domain U issue a request. > And in qemu-dm it will > get the event, and invoke > cpu_handle_ioreq(/xen-4.0.0/tools/ioemu-qemu-xen/i386-dm/helper2.c) > ->cpu_get_ioreq()->xc_evtchn_unmask(). In evtchn_unmask it will has > operation on evtchn_pending, > evtchn_mask, or evtchn_pending_sel. > > My confusion is on notify_via_xen_event_channel()->evtchn_set_pending, > the **evtchn_set_pending here > in not locked**, while inside it also have operation on evtchn_pending, > evtchn_mask, or evtchn_pending_sel.Atomic ops are used to make the operations on evtchn_pending, evtchn_mask, and evtchn_sel concurrency safe. Note that the locking from notify_via_xen_event_channel() is just the same as, say, from evtchn_send(): the local domain''s (ie. DomU''s, in this case) event_lock is held, while the remote domain''s (ie. dom0''s, in this case) does not need to be held. If your domU is stuck in state _VPF_blocked_in_xen, it probably means qemu-dm is toast. I would investigate whether the qemu-dm process is still present, still doing useful work, etc etc. -- Keir> I''m afried this access competition might cause event undeliverd from dom > U to qemu-dm, but I am not sure, > since I still not fully understand where event_mask and is set, and > where event_pending is cleared. > > -------------------------notify_via_xen_event_channel------------------------- > ------------ > 989 void notify_via_xen_event_channel(int lport) > 990 { > 991 struct evtchn *lchn, *rchn; > 992 struct domain *ld = current->domain, *rd; > 993 int rport; > 994 > 995 spin_lock(&ld->event_lock); > 996 > 997 ASSERT(port_is_valid(ld, lport)); > 998 lchn = evtchn_from_port(ld, lport); > 999 ASSERT(lchn->consumer_is_xen); > 1000 > 1001 if ( likely(lchn->state == ECS_INTERDOMAIN) ) > 1002 { > 1003 rd = lchn->u.interdomain.remote_dom; > 1004 rport = lchn->u.interdomain.remote_port; > 1005 rchn = evtchn_from_port(rd, rport); > 1006 evtchn_set_pending(rd->vcpu[rchn->notify_vcpu_id], rport); > 1007 } > 1008 > 1009 spin_unlock(&ld->event_lock); > 1010 } > > ----------------------------evtchn_set_pending---------------------- > 535 static int evtchn_set_pending(struct vcpu *v, int port) > 536 { > 537 struct domain *d = v->domain; > 538 int vcpuid; > 539 > 540 /* > 541 * The following bit operations must happen in strict order. > 542 * NB. On x86, the atomic bit operations also act as memory barriers. > 543 * There is therefore sufficiently strict ordering for this > architecture -- > 544 * others may require explicit memory barriers. > 545 */ > 546 > 547 if ( test_and_set_bit(port, &shared_info(d, evtchn_pending)) ) > 548 return 1; > 549 > 550 if ( !test_bit (port, &shared_info(d, evtchn_mask)) && > 551 !test_and_set_bit(port / BITS_PER_EVTCHN_WORD(d), > 552 &vcpu_info(v, evtchn_pending_sel)) ) > 553 { > 554 vcpu_mark_events_pending(v); > 555 } > 556 > 557 /* Check if some VCPU might be polling for this event. */ > 558 if ( likely(bitmap_empty(d->poll_mask, d->max_vcpus)) ) > 559 return 0; > 560 > 561 /* Wake any interested (or potentially interested) pollers. */ > 562 for ( vcpuid = find_first_bit(d->poll_mask, d->max_vcpus); > 563 vcpuid < d->max_vcpus; > 564 vcpuid = find_next_bit(d->poll_mask, d->max_vcpus, vcpuid+1) ) > 565 { > 566 v = d->vcpu[vcpuid]; > 567 if ( ((v->poll_evtchn <= 0) || (v->poll_evtchn == port)) && > 568 test_and_clear_bit(vcpuid, d->poll_mask) ) > 569 { > 570 v->poll_evtchn = 0; > 571 vcpu_unblock(v); > > --------------------------------------evtchn_unmask--------------------------- > --- > 764 > 765 int evtchn_unmask(unsigned int port) > 766 { > 767 struct domain *d = current->domain; > 768 struct vcpu *v; > 769 > 770 spin_lock(&d->event_lock); > 771 > 772 if ( unlikely(!port_is_valid(d, port)) ) > 773 { > 774 spin_unlock(&d->event_lock); > 775 return -EINVAL; > 776 } > 777 > 778 v = d->vcpu[evtchn_from_port(d, port)->notify_vcpu_id]; > 779 > 780 /* > 781 * These operations must happen in strict order. Based on > 782 * include/xen/event.h:evtchn_set_pending(). > 783 */ > 784 if ( test_and_clear_bit(port, &shared_info(d, evtchn_mask)) && > 785 test_bit (port, &shared_info(d, evtchn_pending)) && > 786 !test_and_set_bit (port / BITS_PER_EVTCHN_WORD(d), > 787 &vcpu_info(v, evtchn_pending_sel)) ) > 788 { > 789 vcpu_mark_events_pending(v); > 790 } > 791 > 792 spin_unlock(&d->event_lock); > 793 > 794 return 0; > 795 } > ----------------------------cpu_get_ioreq------------------------- > 260 static ioreq_t *cpu_get_ioreq(void) > 261 { > 262 int i; > 263 evtchn_port_t port; > 264 > 265 port = xc_evtchn_pending(xce_handle); > 266 if (port != -1) { > 267 for ( i = 0; i < vcpus; i++ ) > 268 if ( ioreq_local_port[i] == port ) > 269 break; > 270 > 271 if ( i == vcpus ) { > 272 fprintf(logfile, "Fatal error while trying to get io > event!\n"); > 273 exit(1); > 274 } > 275 > 276 // unmask the wanted port again > 277 xc_evtchn_unmask(xce_handle, port); > 278 > 279 //get the io packet from shared memory > 280 send_vcpu = i; > 281 return __cpu_get_ioreq(i); > 282 } > 283 > 284 //read error or read nothing > 285 return NULL; > 286 } > 287 > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I also meet HVM guest hang in our stress testing. For detail, pls see the bugzilla: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1664 best regards yang> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser > Sent: Sunday, September 19, 2010 7:50 PM > To: MaoXiaoyun > Cc: xen devel > Subject: [Xen-devel] Re: VM hung after running sometime > > On 19/09/2010 11:37, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > > Hi Keir: > > > > Regards to HVM hang , according to our recent test, it turns out > this > > issue still exists. > > When I go through the code, I obseved something abnormal and > need your > > help. > > > > We''ve noticed when VM hang, its VCPU flags is always 4, which > indicates > > _VPF_blocked_in_xen, > > and it is invoked in prepare_wait_on_xen_event_channel. I''ve noticed > > that Domain U has setup > > a event channel with domain 0 for each VCPU and qemu-dm select > on the > > event fd. > > > > notify_via_xen_event_channel is called when Domain U issue a > request. > > And in qemu-dm it will > > get the event, and invoke > > cpu_handle_ioreq(/xen-4.0.0/tools/ioemu-qemu-xen/i386-dm/helper2.c) > > ->cpu_get_ioreq()->xc_evtchn_unmask(). In evtchn_unmask it will has > > operation on evtchn_pending, > > evtchn_mask, or evtchn_pending_sel. > > > > My confusion is on > notify_via_xen_event_channel()->evtchn_set_pending, > > the **evtchn_set_pending here > > in not locked**, while inside it also have operation on > evtchn_pending, > > evtchn_mask, or evtchn_pending_sel. > > Atomic ops are used to make the operations on evtchn_pending, evtchn_mask, > and evtchn_sel concurrency safe. Note that the locking from > notify_via_xen_event_channel() is just the same as, say, from evtchn_send(): > the local domain''s (ie. DomU''s, in this case) event_lock is held, while the > remote domain''s (ie. dom0''s, in this case) does not need to be held. > > If your domU is stuck in state _VPF_blocked_in_xen, it probably means > qemu-dm is toast. I would investigate whether the qemu-dm process is still > present, still doing useful work, etc etc. > > -- Keir > > > I''m afried this access competition might cause event undeliverd from > dom > > U to qemu-dm, but I am not sure, > > since I still not fully understand where event_mask and is set, and > > where event_pending is cleared. > > > > -------------------------notify_via_xen_event_channel------------------------- > > ------------ > > 989 void notify_via_xen_event_channel(int lport) > > 990 { > > 991 struct evtchn *lchn, *rchn; > > 992 struct domain *ld = current->domain, *rd; > > 993 int rport; > > 994 > > 995 spin_lock(&ld->event_lock); > > 996 > > 997 ASSERT(port_is_valid(ld, lport)); > > 998 lchn = evtchn_from_port(ld, lport); > > 999 ASSERT(lchn->consumer_is_xen); > > 1000 > > 1001 if ( likely(lchn->state == ECS_INTERDOMAIN) ) > > 1002 { > > 1003 rd = lchn->u.interdomain.remote_dom; > > 1004 rport = lchn->u.interdomain.remote_port; > > 1005 rchn = evtchn_from_port(rd, rport); > > 1006 evtchn_set_pending(rd->vcpu[rchn->notify_vcpu_id], rport); > > 1007 } > > 1008 > > 1009 spin_unlock(&ld->event_lock); > > 1010 } > > > > ----------------------------evtchn_set_pending---------------------- > > 535 static int evtchn_set_pending(struct vcpu *v, int port) > > 536 { > > 537 struct domain *d = v->domain; > > 538 int vcpuid; > > 539 > > 540 /* > > 541 * The following bit operations must happen in strict order. > > 542 * NB. On x86, the atomic bit operations also act as memory > barriers. > > 543 * There is therefore sufficiently strict ordering for this > > architecture -- > > 544 * others may require explicit memory barriers. > > 545 */ > > 546 > > 547 if ( test_and_set_bit(port, &shared_info(d, evtchn_pending)) ) > > 548 return 1; > > 549 > > 550 if ( !test_bit (port, &shared_info(d, evtchn_mask)) && > > 551 !test_and_set_bit(port / BITS_PER_EVTCHN_WORD(d), > > 552 &vcpu_info(v, evtchn_pending_sel)) ) > > 553 { > > 554 vcpu_mark_events_pending(v); > > 555 } > > 556 > > 557 /* Check if some VCPU might be polling for this event. */ > > 558 if ( likely(bitmap_empty(d->poll_mask, d->max_vcpus)) ) > > 559 return 0; > > 560 > > 561 /* Wake any interested (or potentially interested) pollers. */ > > 562 for ( vcpuid = find_first_bit(d->poll_mask, d->max_vcpus); > > 563 vcpuid < d->max_vcpus; > > 564 vcpuid = find_next_bit(d->poll_mask, d->max_vcpus, > vcpuid+1) ) > > 565 { > > 566 v = d->vcpu[vcpuid]; > > 567 if ( ((v->poll_evtchn <= 0) || (v->poll_evtchn == port)) && > > 568 test_and_clear_bit(vcpuid, d->poll_mask) ) > > 569 { > > 570 v->poll_evtchn = 0; > > 571 vcpu_unblock(v); > > > > --------------------------------------evtchn_unmask--------------------------- > > --- > > 764 > > 765 int evtchn_unmask(unsigned int port) > > 766 { > > 767 struct domain *d = current->domain; > > 768 struct vcpu *v; > > 769 > > 770 spin_lock(&d->event_lock); > > 771 > > 772 if ( unlikely(!port_is_valid(d, port)) ) > > 773 { > > 774 spin_unlock(&d->event_lock); > > 775 return -EINVAL; > > 776 } > > 777 > > 778 v = d->vcpu[evtchn_from_port(d, port)->notify_vcpu_id]; > > 779 > > 780 /* > > 781 * These operations must happen in strict order. Based on > > 782 * include/xen/event.h:evtchn_set_pending(). > > 783 */ > > 784 if ( test_and_clear_bit(port, &shared_info(d, evtchn_mask)) && > > 785 test_bit (port, &shared_info(d, > evtchn_pending)) && > > 786 !test_and_set_bit (port / BITS_PER_EVTCHN_WORD(d), > > 787 &vcpu_info(v, > evtchn_pending_sel)) ) > > 788 { > > 789 vcpu_mark_events_pending(v); > > 790 } > > 791 > > 792 spin_unlock(&d->event_lock); > > 793 > > 794 return 0; > > 795 } > > ----------------------------cpu_get_ioreq------------------------- > > 260 static ioreq_t *cpu_get_ioreq(void) > > 261 { > > 262 int i; > > 263 evtchn_port_t port; > > 264 > > 265 port = xc_evtchn_pending(xce_handle); > > 266 if (port != -1) { > > 267 for ( i = 0; i < vcpus; i++ ) > > 268 if ( ioreq_local_port[i] == port ) > > 269 break; > > 270 > > 271 if ( i == vcpus ) { > > 272 fprintf(logfile, "Fatal error while trying to get io > > event!\n"); > > 273 exit(1); > > 274 } > > 275 > > 276 // unmask the wanted port again > > 277 xc_evtchn_unmask(xce_handle, port); > > 278 > > 279 //get the io packet from shared memory > > 280 send_vcpu = i; > > 281 return __cpu_get_ioreq(i); > > 282 } > > 283 > > 284 //read error or read nothing > > 285 return NULL; > > 286 } > > 287 > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Keir:
Appreciate for your kindly help.
Just now I notice another possiblity of event missed and need your
verification.
As we known, when do IO, domain U will write those requests into ring
buffer, and
notice to qemu-dm (which is waiting on select) throught event channel. And when
qemu is
actived it will notify back( helper2.c line 548) to clean possible wait on
_VPF_blocked_in_xen.
When IO is not ready, domain U in VMEXIT->hvm_do_resume might invoke
wait_on_xen_event_channel
(where it is blocked in _VPF_blocked_in_xen).
Here is my assumption of event missed.
step 1: hvm_do_resume execute 260, and suppose p->state is STATE_IOREQ_READY
or STATE_IOREQ_INPROCESS
step 2: then in cpu_handle_ioreq is in line 547, it execute line 548 so quickly
before hvm_do_resume execute line 270.
Well, the event is missed.
In other words, the _VPF_blocked_in_xen is cleared before it is actually setted,
and Domian U who is blocked
might never get unblocked, it this possible?
thx.
-------------------------------xen/arch/x86/hvm/hvm.c---------------
252 void hvm_do_resume(struct vcpu *v)
253 {
254 ioreq_t *p;
255 static int i;
256
257 pt_restore_timer(v);
258
259 /* NB. Optimised for common case (p->state == STATE_IOREQ_NONE). */
260 p = get_ioreq(v);
261 while ( p->state != STATE_IOREQ_NONE )
262 {
263 switch ( p->state )
264 {
265 case STATE_IORESP_READY: /* IORESP_READY -> NONE */
266 hvm_io_assist();
267 break;
268 case STATE_IOREQ_READY: /* IOREQ_{READY,INPROCESS} ->
IORESP_READY */
269 case STATE_IOREQ_INPROCESS:
270 wait_on_xen_event_channel(v->arch.hvm_vcpu.xen_port,
271 (p->state != STATE_IOREQ_READY)
&&
272 (p->state !=
STATE_IOREQ_INPROCESS));
273 break;
274 default:
275 gdprintk(XENLOG_ERR, "Weird HVM iorequest state
%d.\n", p->state);
276 domain_crash(v->domain);
277 return; /* bail */
278 }
279 }
280 }
--------------tools/ioemu-qemu-xen/i386-dm/helper2.c--------
507 static void cpu_handle_ioreq(void *opaque)
508 {
509 extern int shutdown_requested;
510 CPUState *env = opaque;
511 ioreq_t *req = cpu_get_ioreq();
512 static int i = 0;
513
514 __handle_buffered_iopage(env);
515 if (req) {
516 __handle_ioreq(env, req);
517
518 if (req->state != STATE_IOREQ_INPROCESS) {
519 fprintf(logfile, "Badness in I/O request ... not in
service?!: "
520 "%x, ptr: %x, port: %"PRIx64", "
521 "data: %"PRIx64", count: %u, size:
%u\n",
522 req->state, req->data_is_ptr, req->addr,
523 req->data, req->count, req->size);
524 destroy_hvm_domain();
525 return;
526 }
527
528 xen_wmb(); /* Update ioreq contents /then/ update state. */
529
530 /*
531 * We do this before we send the response so that the tools
532 * have the opportunity to pick up on the reset before the
533 * guest resumes and does a hlt with interrupts disabled which
534 * causes Xen to powerdown the domain.
535 */
536 if (vm_running) {
537 if (qemu_shutdown_requested()) {
538 fprintf(logfile, "shutdown requested in
cpu_handle_ioreq\n");
539 destroy_hvm_domain();
540 }
541 if (qemu_reset_requested()) {
542 fprintf(logfile, "reset requested in
cpu_handle_ioreq.\n");
543 qemu_system_reset();
544 }
545 }
546
547 req->state = STATE_IORESP_READY;
548 xc_evtchn_notify(xce_handle, ioreq_local_port[send_vcpu]);
549 }
550 }
> Date: Sun, 19 Sep 2010 12:49:44 +0100
> Subject: Re: VM hung after running sometime
> From: keir.fraser@eu.citrix.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; jbeulich@novell.com
>
> On 19/09/2010 11:37, "MaoXiaoyun" <tinnycloud@hotmail.com>
wrote:
>
> > Hi Keir:
> >
> > Regards to HVM hang , according to our recent test, it turns out this
> > issue still exists.
> > When I go through the code, I obseved something abnormal and need your
> > help.
> >
> > We''ve noticed when VM hang, its VCPU flags is always 4, which
indicates
> > _VPF_blocked_in_xen,
> > and it is invoked in prepare_wait_on_xen_event_channel. I''ve
noticed
> > that Domain U has setup
> > a event channel with domain 0 for each VCPU and qemu-dm select on the
> > event fd.
> >
> > notify_via_xen_event_channel is called when Domain U issue a request.
> > And in qemu-dm it will
> > get the event, and invoke
> > cpu_handle_ioreq(/xen-4.0.0/tools/ioemu-qemu-xen/i386-dm/helper2.c)
> > ->cpu_get_ioreq()->xc_evtchn_unmask(). In evtchn_unmask it will
has
> > operation on evtchn_pending,
> > evtchn_mask, or evtchn_pending_sel.
> >
> > My confusion is on
notify_via_xen_event_channel()->evtchn_set_pending,
> > the **evtchn_set_pending here
> > in not locked**, while inside it also have operation on
evtchn_pending,
> > evtchn_mask, or evtchn_pending_sel.
>
> Atomic ops are used to make the operations on evtchn_pending, evtchn_mask,
> and evtchn_sel concurrency safe. Note that the locking from
> notify_via_xen_event_channel() is just the same as, say, from
evtchn_send():
> the local domain''s (ie. DomU''s, in this case) event_lock
is held, while the
> remote domain''s (ie. dom0''s, in this case) does not need
to be held.
>
> If your domU is stuck in state _VPF_blocked_in_xen, it probably means
> qemu-dm is toast. I would investigate whether the qemu-dm process is still
> present, still doing useful work, etc etc.
>
> -- Keir
>
> > I''m afried this access competition might cause event
undeliverd from dom
> > U to qemu-dm, but I am not sure,
> > since I still not fully understand where event_mask and is set, and
> > where event_pending is cleared.
> >
> >
-------------------------notify_via_xen_event_channel-------------------------
> > ------------
> > 989 void notify_via_xen_event_channel(int lport)
> > 990 {
> > 991 struct evtchn *lchn, *rchn;
> > 992 struct domain *ld = current->domain, *rd;
> > 993 int rport;
> > 994
> > 995 spin_lock(&ld->event_lock);
> > 996
> > 997 ASSERT(port_is_valid(ld, lport));
> > 998 lchn = evtchn_from_port(ld, lport);
> > 999 ASSERT(lchn->consumer_is_xen);
> > 1000
> > 1001 if ( likely(lchn->state == ECS_INTERDOMAIN) )
> > 1002 {
> > 1003 rd = lchn->u.interdomain.remote_dom;
> > 1004 rport = lchn->u.interdomain.remote_port;
> > 1005 rchn = evtchn_from_port(rd, rport);
> > 1006 evtchn_set_pending(rd->vcpu[rchn->notify_vcpu_id], rport);
> > 1007 }
> > 1008
> > 1009 spin_unlock(&ld->event_lock);
> > 1010 }
> >
> > ----------------------------evtchn_set_pending----------------------
> > 535 static int evtchn_set_pending(struct vcpu *v, int port)
> > 536 {
> > 537 struct domain *d = v->domain;
> > 538 int vcpuid;
> > 539
> > 540 /*
> > 541 * The following bit operations must happen in strict order.
> > 542 * NB. On x86, the atomic bit operations also act as memory
barriers.
> > 543 * There is therefore sufficiently strict ordering for this
> > architecture --
> > 544 * others may require explicit memory barriers.
> > 545 */
> > 546
> > 547 if ( test_and_set_bit(port, &shared_info(d, evtchn_pending)) )
> > 548 return 1;
> > 549
> > 550 if ( !test_bit (port, &shared_info(d, evtchn_mask)) &&
> > 551 !test_and_set_bit(port / BITS_PER_EVTCHN_WORD(d),
> > 552 &vcpu_info(v, evtchn_pending_sel)) )
> > 553 {
> > 554 vcpu_mark_events_pending(v);
> > 555 }
> > 556
> > 557 /* Check if some VCPU might be polling for this event. */
> > 558 if ( likely(bitmap_empty(d->poll_mask, d->max_vcpus)) )
> > 559 return 0;
> > 560
> > 561 /* Wake any interested (or potentially interested) pollers. */
> > 562 for ( vcpuid = find_first_bit(d->poll_mask, d->max_vcpus);
> > 563 vcpuid < d->max_vcpus;
> > 564 vcpuid = find_next_bit(d->poll_mask, d->max_vcpus, vcpuid+1)
)
> > 565 {
> > 566 v = d->vcpu[vcpuid];
> > 567 if ( ((v->poll_evtchn <= 0) || (v->poll_evtchn == port))
&&
> > 568 test_and_clear_bit(vcpuid, d->poll_mask) )
> > 569 {
> > 570 v->poll_evtchn = 0;
> > 571 vcpu_unblock(v);
> >
> >
--------------------------------------evtchn_unmask---------------------------
> > ---
> > 764
> > 765 int evtchn_unmask(unsigned int port)
> > 766 {
> > 767 struct domain *d = current->domain;
> > 768 struct vcpu *v;
> > 769
> > 770 spin_lock(&d->event_lock);
> > 771
> > 772 if ( unlikely(!port_is_valid(d, port)) )
> > 773 {
> > 774 spin_unlock(&d->event_lock);
> > 775 return -EINVAL;
> > 776 }
> > 777
> > 778 v = d->vcpu[evtchn_from_port(d, port)->notify_vcpu_id];
> > 779
> > 780 /*
> > 781 * These operations must happen in strict order. Based on
> > 782 * include/xen/event.h:evtchn_set_pending().
> > 783 */
> > 784 if ( test_and_clear_bit(port, &shared_info(d, evtchn_mask))
&&
> > 785 test_bit (port, &shared_info(d, evtchn_pending)) &&
> > 786 !test_and_set_bit (port / BITS_PER_EVTCHN_WORD(d),
> > 787 &vcpu_info(v, evtchn_pending_sel)) )
> > 788 {
> > 789 vcpu_mark_events_pending(v);
> > 790 }
> > 791
> > 792 spin_unlock(&d->event_lock);
> > 793
> > 794 return 0;
> > 795 }
> > ----------------------------cpu_get_ioreq-------------------------
> > 260 static ioreq_t *cpu_get_ioreq(void)
> > 261 {
> > 262 int i;
> > 263 evtchn_port_t port;
> > 264
> > 265 port = xc_evtchn_pending(xce_handle);
> > 266 if (port != -1) {
> > 267 for ( i = 0; i < vcpus; i++ )
> > 268 if ( ioreq_local_port[i] == port )
> > 269 break;
> > 270
> > 271 if ( i == vcpus ) {
> > 272 fprintf(logfile, "Fatal error while trying to get io
> > event!\n");
> > 273 exit(1);
> > 274 }
> > 275
> > 276 // unmask the wanted port again
> > 277 xc_evtchn_unmask(xce_handle, port);
> > 278
> > 279 //get the io packet from shared memory
> > 280 send_vcpu = i;
> > 281 return __cpu_get_ioreq(i);
> > 282 }
> > 283
> > 284 //read error or read nothing
> > 285 return NULL;
> > 286 }
> > 287
> >
> >
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote:> When IO is not ready, domain U in VMEXIT->hvm_do_resume might invoke > wait_on_xen_event_channel > (where it is blocked in _VPF_blocked_in_xen). > > Here is my assumption of event missed. > > step 1: hvm_do_resume execute 260, and suppose p->state is STATE_IOREQ_READY > or STATE_IOREQ_INPROCESS > step 2: then in cpu_handle_ioreq is in line 547, it execute line 548 so > quickly before hvm_do_resume execute line 270. > Well, the event is missed. > In other words, the _VPF_blocked_in_xen is cleared before it is actually > setted, and Domian U who is blocked > might never get unblocked, it this possible?Firstly, that code is very paranoid and it should never actually be the case that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in hvm_do_resume(). Secondly, even if you do, take a look at the implementation of wait_on_xen_event_channel() -- it is smart enough to avoid the race you mention. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Keir. I will kick off a test based on my assumption. Actually Domian U is always blocked in _VPF_blocked_in_xen, which is set only in two functions, wait_on_xen_event_channel, prepare_wait_on_xen_event_channel. Both two functions first set the bit and will givp up or try to give up the schedule. What I want to do is modify these functions not to set the bit, and only give up the schedule. Under this situation, the CPU will never be blocked. If the bug is due to the race I imagined, domian U will never hang since even event is missed, VCPU still knows when IO is ready by further schedule. If the bug is something else, say in qemu, domain U will still confront hang since IO never ready. Am I right?> Date: Mon, 20 Sep 2010 08:45:21 +0100 > Subject: Re: VM hung after running sometime > From: keir.fraser@eu.citrix.com > To: tinnycloud@hotmail.com > CC: xen-devel@lists.xensource.com; jbeulich@novell.com > > On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > > When IO is not ready, domain U in VMEXIT->hvm_do_resume might invoke > > wait_on_xen_event_channel > > (where it is blocked in _VPF_blocked_in_xen). > > > > Here is my assumption of event missed. > > > > step 1: hvm_do_resume execute 260, and suppose p->state is STATE_IOREQ_READY > > or STATE_IOREQ_INPROCESS > > step 2: then in cpu_handle_ioreq is in line 547, it execute line 548 so > > quickly before hvm_do_resume execute line 270. > > Well, the event is missed. > > In other words, the _VPF_blocked_in_xen is cleared before it is actually > > setted, and Domian U who is blocked > > might never get unblocked, it this possible? > > Firstly, that code is very paranoid and it should never actually be the case > that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in hvm_do_resume(). > Secondly, even if you do, take a look at the implementation of > wait_on_xen_event_channel() -- it is smart enough to avoid the race you > mention. > > -- Keir > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Keir. You''re right, after I deeply looked into the wait_on_xen_event_channel, it is smart enough to avoid the race I assumed. How about prepare_wait_on_xen_event_channel ? Currently Istill don''t know when it will be invoked. Could enlighten me?> Date: Mon, 20 Sep 2010 08:45:21 +0100 > Subject: Re: VM hung after running sometime > From: keir.fraser@eu.citrix.com > To: tinnycloud@hotmail.com > CC: xen-devel@lists.xensource.com; jbeulich@novell.com > > On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > > When IO is not ready, domain U in VMEXIT->hvm_do_resume might invoke > > wait_on_xen_event_channel > > (where it is blocked in _VPF_blocked_in_xen). > > > > Here is my assumption of event missed. > > > > step 1: hvm_do_resume execute 260, and suppose p->state is STATE_IOREQ_READY > > or STATE_IOREQ_INPROCESS > > step 2: then in cpu_handle_ioreq is in line 547, it execute line 548 so > > quickly before hvm_do_resume execute line 270. > > Well, the event is missed. > > In other words, the _VPF_blocked_in_xen is cleared before it is actually > > setted, and Domian U who is blocked > > might never get unblocked, it this possible? > > Firstly, that code is very paranoid and it should never actually be the case > that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in hvm_do_resume(). > Secondly, even if you do, take a look at the implementation of > wait_on_xen_event_channel() -- it is smart enough to avoid the race you > mention. > > -- Keir > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 20/09/2010 10:15, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote:> Thanks Keir. > > You''re right, after I deeply looked into the wait_on_xen_event_channel, it is > smart enough > to avoid the race I assumed. > > How about prepare_wait_on_xen_event_channel ? > Currently Istill don''t know when it will be invoked. > Could enlighten me?As you can see it is called from hvm_send_assist_req(), hence it is called whenever an ioreq is sent to qemu-dm. Note that it is called *before* qemu-dm is notified -- hence it cannot race the wakeup from qemu, as we will not get woken until qemu-dm has done the work, and it cannot start the work until it is notified, and it is not notified until after prepare_wait_on_xen_event_channel has been executed. -- Keir> >> Date: Mon, 20 Sep 2010 08:45:21 +0100 >> Subject: Re: VM hung after running sometime >> From: keir.fraser@eu.citrix.com >> To: tinnycloud@hotmail.com >> CC: xen-devel@lists.xensource.com; jbeulich@novell.com >> >> On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: >> >>> When IO is not ready, domain U in VMEXIT->hvm_do_resume might invoke >>> wait_on_xen_event_channel >>> (where it is blocked in _VPF_blocked_in_xen). >>> >>> Here is my assumption of event missed. >>> >>> step 1: hvm_do_resume execute 260, and suppose p->state is STATE_IOREQ_READY >>> or STATE_IOREQ_INPROCESS >>> step 2: then in cpu_handle_ioreq is in line 547, it execute line 548 so >>> quickly before hvm_do_resume execute line 270. >>> Well, the event is missed. >>> In other words, the _VPF_blocked_in_xen is cleared before it is actually >>> setted, and Domian U who is blocked >>> might never get unblocked, it this possible? >> >> Firstly, that code is very paranoid and it should never actually be the case >> that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in hvm_do_resume(). >> Secondly, even if you do, take a look at the implementation of >> wait_on_xen_event_channel() -- it is smart enough to avoid the race you >> mention. >> >> -- Keir >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Keir:
I spent more time on how event channel works. And now I know that event
is bind to
irq with call of request_irq. When event is sent, the other side of the channel
will run into
asm_do_IRQ->generic_handle_irq->generic_handle_irq_desc->handle_level_irq(
here it actually invokes desc->handle_irq, and for evtchn this is
handle_level_irq).
I noticed that in handle_level_irq the event mask and pending is cleared.
Well I have one more analysis to be discussed.
Attached is the evtchn when a VM is hang in physical server. Domain 10 is hang.
We can see domain 10 CPU info on the bottem the log, its has flags = 4 which
means
_VPF_blocked_in_xen.
(XEN) VCPU information and callbacks for domain 10:
(XEN) VCPU0: CPU11 [has=F] flags=4 poll=0 upcall_pend = 00, upcall_mask = 00
dirty_cpus={} cpu_affinity={4-15}
(XEN) paging assistance: shadowed 2-on-3
(XEN) No periodic timer
(XEN) Notifying guest (virq 1, port 0, stat 0/-1/0)
(XEN) VCPU1: CPU9 [has=T] flags=0 poll=0 upcall_pend = 00, upcall_mask = 00
dirty_cpus={9} cpu_affinity={4-15}
(XEN) paging assistance: shadowed 2-on-3
(XEN) No periodic timer
(XEN) Notifying guest (virq 1, port 0, stat 0/-1/0)
And its domain event info is :
(XEN) Domain 10 polling vCPUs: {No periodic timer}
(XEN) Event channel information for domain 10:
(XEN) port [p/m]
(XEN) 1 [0/1]: s=3 n=0 d=0 p=105 x=1
(XEN) 2 [0/1]: s=3 n=1 d=0 p=106 x=1
(XEN) 3 [0/0]: s=3 n=0 d=0 p=104 x=0
(XEN) 4 [0/1]: s=2 n=0 d=0 x=0
(XEN) 5 [0/0]: s=6 n=0 x=0
(XEN) 6 [0/0]: s=2 n=0 d=0 x=0
(XEN) 7 [0/0]: s=3 n=0 d=0 p=107 x=0
(XEN) 8 [0/0]: s=3 n=0 d=0 p=108 x=0
(XEN) 9 [0/0]: s=3 n=0 d=0 p=109 x=0
(XEN) 10 [0/0]: s=3 n=0 d=0 p=110 x=0
Base on our situation, we only interest in the event channel which
consumer_is_xen is 1,
and here "x=1", that is port 1 and 2. According to the log, the other
side of the channel
is domain 0, port 105, and 106.
Take a look at domain 0 event channel with port 105,106, I find on port 105, it
pending is
1.(in [1,0], first bit refer to pending, and is 1, second bit refer to mask, is
0).
(XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0
(XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0
In all, we have domain U cpu blocking on _VPF_blocked_in_xen, and it must set
the pending bit.
Consider pending is 1, it looks like the irq is not triggered, am I right ?
Since if it is triggerred, it should clear the pending bit. (line 361).
------------------------------/linux-2.6-pvops.git/kernel/irq/chip.c---
354 void
355 handle_level_irq(unsigned int irq, struct irq_desc *desc)
356 {
357 struct irqaction *action;
358 irqreturn_t action_ret;
359
360 spin_lock(&desc->lock);
361 mask_ack_irq(desc, irq);
362
363 if (unlikely(desc->status & IRQ_INPROGRESS))
364 goto out_unlock;
365 desc->status &= ~(IRQ_REPLAY | IRQ_WAITING);
366 kstat_incr_irqs_this_cpu(irq, desc);
367
BTW, the qemu still works fine when VM is hang. Below is it strace output.
No much difference between other well worked qemu instance, other than select
all Timeout.
-------------------
select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {673470, 59535265}) = 0
clock_gettime(CLOCK_MONOTONIC, {673470, 59629728}) = 0
clock_gettime(CLOCK_MONOTONIC, {673470, 59717700}) = 0
clock_gettime(CLOCK_MONOTONIC, {673470, 59806552}) = 0
select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {673470, 70234406}) = 0
clock_gettime(CLOCK_MONOTONIC, {673470, 70332116}) = 0
clock_gettime(CLOCK_MONOTONIC, {673470, 70419835}) = 0
> Date: Mon, 20 Sep 2010 10:35:46 +0100
> Subject: Re: VM hung after running sometime
> From: keir.fraser@eu.citrix.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; jbeulich@novell.com
>
> On 20/09/2010 10:15, "MaoXiaoyun" <tinnycloud@hotmail.com>
wrote:
>
> > Thanks Keir.
> >
> > You''re right, after I deeply looked into the
wait_on_xen_event_channel, it is
> > smart enough
> > to avoid the race I assumed.
> >
> > How about prepare_wait_on_xen_event_channel ?
> > Currently Istill don''t know when it will be invoked.
> > Could enlighten me?
>
> As you can see it is called from hvm_send_assist_req(), hence it is called
> whenever an ioreq is sent to qemu-dm. Note that it is called *before*
> qemu-dm is notified -- hence it cannot race the wakeup from qemu, as we
will
> not get woken until qemu-dm has done the work, and it cannot start the work
> until it is notified, and it is not notified until after
> prepare_wait_on_xen_event_channel has been executed.
>
> -- Keir
>
> >
> >> Date: Mon, 20 Sep 2010 08:45:21 +0100
> >> Subject: Re: VM hung after running sometime
> >> From: keir.fraser@eu.citrix.com
> >> To: tinnycloud@hotmail.com
> >> CC: xen-devel@lists.xensource.com; jbeulich@novell.com
> >>
> >> On 20/09/2010 07:00, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> >>
> >>> When IO is not ready, domain U in VMEXIT->hvm_do_resume
might invoke
> >>> wait_on_xen_event_channel
> >>> (where it is blocked in _VPF_blocked_in_xen).
> >>>
> >>> Here is my assumption of event missed.
> >>>
> >>> step 1: hvm_do_resume execute 260, and suppose p->state is
STATE_IOREQ_READY
> >>> or STATE_IOREQ_INPROCESS
> >>> step 2: then in cpu_handle_ioreq is in line 547, it execute
line 548 so
> >>> quickly before hvm_do_resume execute line 270.
> >>> Well, the event is missed.
> >>> In other words, the _VPF_blocked_in_xen is cleared before it
is actually
> >>> setted, and Domian U who is blocked
> >>> might never get unblocked, it this possible?
> >>
> >> Firstly, that code is very paranoid and it should never actually
be the case
> >> that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in
hvm_do_resume().
> >> Secondly, even if you do, take a look at the implementation of
> >> wait_on_xen_event_channel() -- it is smart enough to avoid the
race you
> >> mention.
> >>
> >> -- Keir
> >>
> >>
> >
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 21/09/2010 06:02, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote:> Take a look at domain 0 event channel with port 105,106, I find on port 105, > it pending is > 1.(in [1,0], first bit refer to pending, and is 1, second bit refer to mask, > is 0). > > (XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0 > (XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0 > > In all, we have domain U cpu blocking on _VPF_blocked_in_xen, and it must set > the pending bit. > Consider pending is 1, it looks like the irq is not triggered, am I right ? > Since if it is triggerred, it should clear the pending bit. (line 361).Yes it looks like dom0 is not handling the event for some reason. Qemu looks like it still works and is waiting for a notification via select(). But that won''t happen until dom0 kernel handles the event as an IRQ and calls the relevant irq handler (drivers/xen/evtchn.c:evtchn_interrupt()). I think you''re on the right track in your debugging. I don''t know much about the pv_ops irq handling path, except to say that this aspect is different than non-pv_ops kernels which special-case handling of events bound to user-space rather more. So at the moment my best guess would be that the bug is in the pv_ops kernel irq handling for this type of user-space-bound event. -- Keir> ------------------------------/linux-2.6-pvops.git/kernel/irq/chip.c--- > 354 void > 355 handle_level_irq(unsigned int irq, struct irq_desc *desc) > 356 { > 357 struct irqaction *action; > 358 irqreturn_t action_ret; > 359 > 360 spin_lock(&desc->lock); > 361 mask_ack_irq(desc, irq); > 362 > 363 if (unlikely(desc->status & IRQ_INPROGRESS)) > 364 goto out_unlock; > 365 desc->status &= ~(IRQ_REPLAY | IRQ_WAITING); > 366 kstat_incr_irqs_this_cpu(irq, desc); > 367 > > BTW, the qemu still works fine when VM is hang. Below is it strace output. > No much difference between other well worked qemu instance, other than select > all Timeout. > ------------------- > select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) > clock_gettime(CLOCK_MONOTONIC, {673470, 59535265}) = 0 > clock_gettime(CLOCK_MONOTONIC, {673470, 59629728}) = 0 > clock_gettime(CLOCK_MONOTONIC, {673470, 59717700}) = 0 > clock_gettime(CLOCK_MONOTONIC, {673470, 59806552}) = 0 > select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) > clock_gettime(CLOCK_MONOTONIC, {673470, 70234406}) = 0 > clock_gettime(CLOCK_MONOTONIC, {673470, 70332116}) = 0 > clock_gettime(CLOCK_MONOTONIC, {673470, 70419835}) = 0 > > > > >> Date: Mon, 20 Sep 2010 10:35:46 +0100 >> Subject: Re: VM hung after running sometime >> From: keir.fraser@eu.citrix.com >> To: tinnycloud@hotmail.com >> CC: xen-devel@lists.xensource.com; jbeulich@novell.com >> >> On 20/09/2010 10:15, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: >> >>> Thanks Keir. >>> >>> You''re right, after I deeply looked into the wait_on_xen_event_channel, it >>> is >>> smart enough >>> to avoid the race I assumed. >>> >>> How about prepare_wait_on_xen_event_channel ? >>> Currently Istill don''t know when it will be invoked. >>> Could enlighten me? >> >> As you can see it is called from hvm_send_assist_req(), hence it is called >> whenever an ioreq is sent to qemu-dm. Note that it is called *before* >> qemu-dm is notified -- hence it cannot race the wakeup from qemu, as we will >> not get woken until qemu-dm has done the work, and it cannot start the work >> until it is notified, and it is not notified until after >> prepare_wait_on_xen_event_channel has been executed. >> >> -- Keir >> >>> >>>> Date: Mon, 20 Sep 2010 08:45:21 +0100 >>>> Subject: Re: VM hung after running sometime >>>> From: keir.fraser@eu.citrix.com >>>> To: tinnycloud@hotmail.com >>>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com >>>> >>>> On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: >>>> >>>>> When IO is not ready, domain U in VMEXIT->hvm_do_resume might invoke >>>>> wait_on_xen_event_channel >>>>> (where it is blocked in _VPF_blocked_in_xen). >>>>> >>>>> Here is my assumption of event missed. >>>>> >>>>> step 1: hvm_do_resume execute 260, and suppose p->state is >>>>> STATE_IOREQ_READY >>>>> or STATE_IOREQ_INPROCESS >>>>> step 2: then in cpu_handle_ioreq is in line 547, it execute line 548 so >>>>> quickly before hvm_do_resume execute line 270. >>>>> Well, the event is missed. >>>>> In other words, the _VPF_blocked_in_xen is cleared before it is actually >>>>> setted, and Domian U who is blocked >>>>> might never get unblocked, it this possible? >>>> >>>> Firstly, that code is very paranoid and it should never actually be the >>>> case >>>> that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in hvm_do_resume(). >>>> Secondly, even if you do, take a look at the implementation of >>>> wait_on_xen_event_channel() -- it is smart enough to avoid the race you >>>> mention. >>>> >>>> -- Keir >>>> >>>> >>> >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I also met this issue, especially running high work load on HVM vms with xen 4.0.0 + pvops 2.6.31.13xen. I noticed port 1 of VCPU1 always be blocked on this port, the system of vcpu0 is normal but stopped on vcpu1. Jeremy, could you please take a look on this issue? Could you give some idea on it? thanks, James 2010/9/21 Keir Fraser <keir.fraser@eu.citrix.com>> On 21/09/2010 06:02, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > > Take a look at domain 0 event channel with port 105,106, I find on port > 105, > > it pending is > > 1.(in [1,0], first bit refer to pending, and is 1, second bit refer to > mask, > > is 0). > > > > (XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0 > > (XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0 > > > > In all, we have domain U cpu blocking on _VPF_blocked_in_xen, and it must > set > > the pending bit. > > Consider pending is 1, it looks like the irq is not triggered, am I > right ? > > Since if it is triggerred, it should clear the pending bit. (line 361). > > Yes it looks like dom0 is not handling the event for some reason. Qemu > looks > like it still works and is waiting for a notification via select(). But > that > won''t happen until dom0 kernel handles the event as an IRQ and calls the > relevant irq handler (drivers/xen/evtchn.c:evtchn_interrupt()). > > I think you''re on the right track in your debugging. I don''t know much > about > the pv_ops irq handling path, except to say that this aspect is different > than non-pv_ops kernels which special-case handling of events bound to > user-space rather more. So at the moment my best guess would be that the > bug > is in the pv_ops kernel irq handling for this type of user-space-bound > event. > > -- Keir > > > ------------------------------/linux-2.6-pvops.git/kernel/irq/chip.c--- > > 354 void > > 355 handle_level_irq(unsigned int irq, struct irq_desc *desc) > > 356 { > > 357 struct irqaction *action; > > 358 irqreturn_t action_ret; > > 359 > > 360 spin_lock(&desc->lock); > > 361 mask_ack_irq(desc, irq); > > 362 > > 363 if (unlikely(desc->status & IRQ_INPROGRESS)) > > 364 goto out_unlock; > > 365 desc->status &= ~(IRQ_REPLAY | IRQ_WAITING); > > 366 kstat_incr_irqs_this_cpu(irq, desc); > > 367 > > > > BTW, the qemu still works fine when VM is hang. Below is it strace > output. > > No much difference between other well worked qemu instance, other than > select > > all Timeout. > > ------------------- > > select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) > > clock_gettime(CLOCK_MONOTONIC, {673470, 59535265}) = 0 > > clock_gettime(CLOCK_MONOTONIC, {673470, 59629728}) = 0 > > clock_gettime(CLOCK_MONOTONIC, {673470, 59717700}) = 0 > > clock_gettime(CLOCK_MONOTONIC, {673470, 59806552}) = 0 > > select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) > > clock_gettime(CLOCK_MONOTONIC, {673470, 70234406}) = 0 > > clock_gettime(CLOCK_MONOTONIC, {673470, 70332116}) = 0 > > clock_gettime(CLOCK_MONOTONIC, {673470, 70419835}) = 0 > > > > > > > > > >> Date: Mon, 20 Sep 2010 10:35:46 +0100 > >> Subject: Re: VM hung after running sometime > >> From: keir.fraser@eu.citrix.com > >> To: tinnycloud@hotmail.com > >> CC: xen-devel@lists.xensource.com; jbeulich@novell.com > >> > >> On 20/09/2010 10:15, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > >> > >>> Thanks Keir. > >>> > >>> You''re right, after I deeply looked into the wait_on_xen_event_channel, > it > >>> is > >>> smart enough > >>> to avoid the race I assumed. > >>> > >>> How about prepare_wait_on_xen_event_channel ? > >>> Currently Istill don''t know when it will be invoked. > >>> Could enlighten me? > >> > >> As you can see it is called from hvm_send_assist_req(), hence it is > called > >> whenever an ioreq is sent to qemu-dm. Note that it is called *before* > >> qemu-dm is notified -- hence it cannot race the wakeup from qemu, as we > will > >> not get woken until qemu-dm has done the work, and it cannot start the > work > >> until it is notified, and it is not notified until after > >> prepare_wait_on_xen_event_channel has been executed. > >> > >> -- Keir > >> > >>> > >>>> Date: Mon, 20 Sep 2010 08:45:21 +0100 > >>>> Subject: Re: VM hung after running sometime > >>>> From: keir.fraser@eu.citrix.com > >>>> To: tinnycloud@hotmail.com > >>>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com > >>>> > >>>> On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > >>>> > >>>>> When IO is not ready, domain U in VMEXIT->hvm_do_resume might invoke > >>>>> wait_on_xen_event_channel > >>>>> (where it is blocked in _VPF_blocked_in_xen). > >>>>> > >>>>> Here is my assumption of event missed. > >>>>> > >>>>> step 1: hvm_do_resume execute 260, and suppose p->state is > >>>>> STATE_IOREQ_READY > >>>>> or STATE_IOREQ_INPROCESS > >>>>> step 2: then in cpu_handle_ioreq is in line 547, it execute line 548 > so > >>>>> quickly before hvm_do_resume execute line 270. > >>>>> Well, the event is missed. > >>>>> In other words, the _VPF_blocked_in_xen is cleared before it is > actually > >>>>> setted, and Domian U who is blocked > >>>>> might never get unblocked, it this possible? > >>>> > >>>> Firstly, that code is very paranoid and it should never actually be > the > >>>> case > >>>> that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in > hvm_do_resume(). > >>>> Secondly, even if you do, take a look at the implementation of > >>>> wait_on_xen_event_channel() -- it is smart enough to avoid the race > you > >>>> mention. > >>>> > >>>> -- Keir > >>>> > >>>> > >>> > >> > >> > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I also noticed that there are only one port(number 2) bind to vcpu1, I
wonder that what this port use to do?
(XEN) [2010-09-21 17:09:04] Domain 3 polling vCPUs: {}
(XEN) [2010-09-21 17:09:04] Event channel information for domain 3:
(XEN) [2010-09-21 17:09:04] port [p/m]
(XEN) [2010-09-21 17:09:04] 1 [0/1]: s=3 n=0 d=0 p=42 x=1
(XEN) [2010-09-21 17:09:04] 2 [0/1]: s=3 n=1 d=0 p=43 x=1
(XEN) [2010-09-21 17:09:04] 3 [0/0]: s=3 n=0 d=0 p=41 x=0
(XEN) [2010-09-21 17:09:04] 4 [0/1]: s=2 n=0 d=0 x=0
(XEN) [2010-09-21 17:09:04] 5 [0/0]: s=6 n=0 x=0
(XEN) [2010-09-21 17:09:04] 6 [0/0]: s=2 n=0 d=0 x=0
(XEN) [2010-09-21 17:09:04] 7 [0/0]: s=3 n=0 d=0 p=44 x=0
(XEN) [2010-09-21 17:09:04] 8 [0/0]: s=3 n=0 d=0 p=45 x=0
regards,
2010/9/21 wei song <james.songwei@gmail.com>
> I also met this issue, especially running high work load on HVM vms with
> xen 4.0.0 + pvops 2.6.31.13xen. I noticed port 1 of VCPU1 always be blocked
> on this port, the system of vcpu0 is normal but stopped on vcpu1. Jeremy,
> could you please take a look on this issue? Could you give some idea on it?
>
> thanks,
> James
>
> 2010/9/21 Keir Fraser <keir.fraser@eu.citrix.com>
>
> On 21/09/2010 06:02, "MaoXiaoyun" <tinnycloud@hotmail.com>
wrote:
>>
>> > Take a look at domain 0 event channel with port 105,106, I find on
port
>> 105,
>> > it pending is
>> > 1.(in [1,0], first bit refer to pending, and is 1, second bit
refer to
>> mask,
>> > is 0).
>> >
>> > (XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0
>> > (XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0
>> >
>> > In all, we have domain U cpu blocking on _VPF_blocked_in_xen, and
it
>> must set
>> > the pending bit.
>> > Consider pending is 1, it looks like the irq is not triggered, am
I
>> right ?
>> > Since if it is triggerred, it should clear the pending bit. (line
361).
>>
>> Yes it looks like dom0 is not handling the event for some reason. Qemu
>> looks
>> like it still works and is waiting for a notification via select(). But
>> that
>> won''t happen until dom0 kernel handles the event as an IRQ and
calls the
>> relevant irq handler (drivers/xen/evtchn.c:evtchn_interrupt()).
>>
>> I think you''re on the right track in your debugging. I
don''t know much
>> about
>> the pv_ops irq handling path, except to say that this aspect is
different
>> than non-pv_ops kernels which special-case handling of events bound to
>> user-space rather more. So at the moment my best guess would be that
the
>> bug
>> is in the pv_ops kernel irq handling for this type of user-space-bound
>> event.
>>
>> -- Keir
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Sep-21 17:28 UTC
Re: [Xen-devel] Re: VM hung after running sometime
On 09/21/2010 12:53 AM, Keir Fraser wrote:> On 21/09/2010 06:02, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > >> Take a look at domain 0 event channel with port 105,106, I find on port 105, >> it pending is >> 1.(in [1,0], first bit refer to pending, and is 1, second bit refer to mask, >> is 0). >> >> (XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0 >> (XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0 >> >> In all, we have domain U cpu blocking on _VPF_blocked_in_xen, and it must set >> the pending bit. >> Consider pending is 1, it looks like the irq is not triggered, am I right ? >> Since if it is triggerred, it should clear the pending bit. (line 361). > Yes it looks like dom0 is not handling the event for some reason. Qemu looks > like it still works and is waiting for a notification via select(). But that > won''t happen until dom0 kernel handles the event as an IRQ and calls the > relevant irq handler (drivers/xen/evtchn.c:evtchn_interrupt()). > > I think you''re on the right track in your debugging. I don''t know much about > the pv_ops irq handling path, except to say that this aspect is different > than non-pv_ops kernels which special-case handling of events bound to > user-space rather more. So at the moment my best guess would be that the bug > is in the pv_ops kernel irq handling for this type of user-space-bound > event.We no longer use handle_level_irq because there''s a race which loses events when interrupt migration is enabled. Current xen/stable-2.6.32.x has a proper fix for this, but the quick workaround is to disable irqbalanced. J> -- Keir > >> ------------------------------/linux-2.6-pvops.git/kernel/irq/chip.c--- >> 354 void >> 355 handle_level_irq(unsigned int irq, struct irq_desc *desc) >> 356 { >> 357 struct irqaction *action; >> 358 irqreturn_t action_ret; >> 359 >> 360 spin_lock(&desc->lock); >> 361 mask_ack_irq(desc, irq); >> 362 >> 363 if (unlikely(desc->status & IRQ_INPROGRESS)) >> 364 goto out_unlock; >> 365 desc->status &= ~(IRQ_REPLAY | IRQ_WAITING); >> 366 kstat_incr_irqs_this_cpu(irq, desc); >> 367 >> >> BTW, the qemu still works fine when VM is hang. Below is it strace output. >> No much difference between other well worked qemu instance, other than select >> all Timeout. >> ------------------- >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) >> clock_gettime(CLOCK_MONOTONIC, {673470, 59535265}) = 0 >> clock_gettime(CLOCK_MONOTONIC, {673470, 59629728}) = 0 >> clock_gettime(CLOCK_MONOTONIC, {673470, 59717700}) = 0 >> clock_gettime(CLOCK_MONOTONIC, {673470, 59806552}) = 0 >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) >> clock_gettime(CLOCK_MONOTONIC, {673470, 70234406}) = 0 >> clock_gettime(CLOCK_MONOTONIC, {673470, 70332116}) = 0 >> clock_gettime(CLOCK_MONOTONIC, {673470, 70419835}) = 0 >> >> >> >> >>> Date: Mon, 20 Sep 2010 10:35:46 +0100 >>> Subject: Re: VM hung after running sometime >>> From: keir.fraser@eu.citrix.com >>> To: tinnycloud@hotmail.com >>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com >>> >>> On 20/09/2010 10:15, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: >>> >>>> Thanks Keir. >>>> >>>> You''re right, after I deeply looked into the wait_on_xen_event_channel, it >>>> is >>>> smart enough >>>> to avoid the race I assumed. >>>> >>>> How about prepare_wait_on_xen_event_channel ? >>>> Currently Istill don''t know when it will be invoked. >>>> Could enlighten me? >>> As you can see it is called from hvm_send_assist_req(), hence it is called >>> whenever an ioreq is sent to qemu-dm. Note that it is called *before* >>> qemu-dm is notified -- hence it cannot race the wakeup from qemu, as we will >>> not get woken until qemu-dm has done the work, and it cannot start the work >>> until it is notified, and it is not notified until after >>> prepare_wait_on_xen_event_channel has been executed. >>> >>> -- Keir >>> >>>>> Date: Mon, 20 Sep 2010 08:45:21 +0100 >>>>> Subject: Re: VM hung after running sometime >>>>> From: keir.fraser@eu.citrix.com >>>>> To: tinnycloud@hotmail.com >>>>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com >>>>> >>>>> On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: >>>>> >>>>>> When IO is not ready, domain U in VMEXIT->hvm_do_resume might invoke >>>>>> wait_on_xen_event_channel >>>>>> (where it is blocked in _VPF_blocked_in_xen). >>>>>> >>>>>> Here is my assumption of event missed. >>>>>> >>>>>> step 1: hvm_do_resume execute 260, and suppose p->state is >>>>>> STATE_IOREQ_READY >>>>>> or STATE_IOREQ_INPROCESS >>>>>> step 2: then in cpu_handle_ioreq is in line 547, it execute line 548 so >>>>>> quickly before hvm_do_resume execute line 270. >>>>>> Well, the event is missed. >>>>>> In other words, the _VPF_blocked_in_xen is cleared before it is actually >>>>>> setted, and Domian U who is blocked >>>>>> might never get unblocked, it this possible? >>>>> Firstly, that code is very paranoid and it should never actually be the >>>>> case >>>>> that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in hvm_do_resume(). >>>>> Secondly, even if you do, take a look at the implementation of >>>>> wait_on_xen_event_channel() -- it is smart enough to avoid the race you >>>>> mention. >>>>> >>>>> -- Keir >>>>> >>>>> >>> >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Jeremy.
Regards to fix you mentioned, did you mean the patch I searched and pasted
below, if so, it this all what I need?
For irqbalance disabled, I am afried it might have negative performance impact,
right?
-------------------------------------------------------
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 32f4a2c..06fc991 100644 (file)
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -368,7 +368,7 @@ int bind_evtchn_to_irq(unsigned int evtchn)
irq = find_unbound_irq();
set_irq_chip_and_handler_name(irq, &xen_dynamic_chip,
- handle_level_irq,
"event");
+ handle_edge_irq,
"event");
evtchn_to_irq[evtchn] = irq;
irq_info[irq] = mk_evtchn_info(evtchn);
> Date: Tue, 21 Sep 2010 10:28:34 -0700> From: jeremy@goop.org
> To: keir.fraser@eu.citrix.com
> CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Re: VM hung after running sometime
>
> On 09/21/2010 12:53 AM, Keir Fraser wrote:
> > On 21/09/2010 06:02, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> >
> >> Take a look at domain 0 event channel with port 105,106, I find on
port 105,
> >> it pending is
> >> 1.(in [1,0], first bit refer to pending, and is 1, second bit
refer to mask,
> >> is 0).
> >>
> >> (XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0
> >> (XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0
> >>
> >> In all, we have domain U cpu blocking on _VPF_blocked_in_xen, and
it must set
> >> the pending bit.
> >> Consider pending is 1, it looks like the irq is not triggered, am
I right ?
> >> Since if it is triggerred, it should clear the pending bit. (line
361).
> > Yes it looks like dom0 is not handling the event for some reason. Qemu
looks
> > like it still works and is waiting for a notification via select().
But that
> > won''t happen until dom0 kernel handles the event as an IRQ
and calls the
> > relevant irq handler (drivers/xen/evtchn.c:evtchn_interrupt()).
> >
> > I think you''re on the right track in your debugging. I
don''t know much about
> > the pv_ops irq handling path, except to say that this aspect is
different
> > than non-pv_ops kernels which special-case handling of events bound to
> > user-space rather more. So at the moment my best guess would be that
the bug
> > is in the pv_ops kernel irq handling for this type of user-space-bound
> > event.
>
> We no longer use handle_level_irq because there''s a race which
loses
> events when interrupt migration is enabled. Current xen/stable-2.6.32.x
> has a proper fix for this, but the quick workaround is to disable
> irqbalanced.
>
> J
>
> > -- Keir
> >
> >>
------------------------------/linux-2.6-pvops.git/kernel/irq/chip.c---
> >> 354 void
> >> 355 handle_level_irq(unsigned int irq, struct irq_desc *desc)
> >> 356 {
> >> 357 struct irqaction *action;
> >> 358 irqreturn_t action_ret;
> >> 359
> >> 360 spin_lock(&desc->lock);
> >> 361 mask_ack_irq(desc, irq);
> >> 362
> >> 363 if (unlikely(desc->status & IRQ_INPROGRESS))
> >> 364 goto out_unlock;
> >> 365 desc->status &= ~(IRQ_REPLAY | IRQ_WAITING);
> >> 366 kstat_incr_irqs_this_cpu(irq, desc);
> >> 367
> >>
> >> BTW, the qemu still works fine when VM is hang. Below is it strace
output.
> >> No much difference between other well worked qemu instance, other
than select
> >> all Timeout.
> >> -------------------
> >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout)
> >> clock_gettime(CLOCK_MONOTONIC, {673470, 59535265}) = 0
> >> clock_gettime(CLOCK_MONOTONIC, {673470, 59629728}) = 0
> >> clock_gettime(CLOCK_MONOTONIC, {673470, 59717700}) = 0
> >> clock_gettime(CLOCK_MONOTONIC, {673470, 59806552}) = 0
> >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout)
> >> clock_gettime(CLOCK_MONOTONIC, {673470, 70234406}) = 0
> >> clock_gettime(CLOCK_MONOTONIC, {673470, 70332116}) = 0
> >> clock_gettime(CLOCK_MONOTONIC, {673470, 70419835}) = 0
> >>
> >>
> >>
> >>
> >>> Date: Mon, 20 Sep 2010 10:35:46 +0100
> >>> Subject: Re: VM hung after running sometime
> >>> From: keir.fraser@eu.citrix.com
> >>> To: tinnycloud@hotmail.com
> >>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com
> >>>
> >>> On 20/09/2010 10:15, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> >>>
> >>>> Thanks Keir.
> >>>>
> >>>> You''re right, after I deeply looked into the
wait_on_xen_event_channel, it
> >>>> is
> >>>> smart enough
> >>>> to avoid the race I assumed.
> >>>>
> >>>> How about prepare_wait_on_xen_event_channel ?
> >>>> Currently Istill don''t know when it will be
invoked.
> >>>> Could enlighten me?
> >>> As you can see it is called from hvm_send_assist_req(), hence
it is called
> >>> whenever an ioreq is sent to qemu-dm. Note that it is called
*before*
> >>> qemu-dm is notified -- hence it cannot race the wakeup from
qemu, as we will
> >>> not get woken until qemu-dm has done the work, and it cannot
start the work
> >>> until it is notified, and it is not notified until after
> >>> prepare_wait_on_xen_event_channel has been executed.
> >>>
> >>> -- Keir
> >>>
> >>>>> Date: Mon, 20 Sep 2010 08:45:21 +0100
> >>>>> Subject: Re: VM hung after running sometime
> >>>>> From: keir.fraser@eu.citrix.com
> >>>>> To: tinnycloud@hotmail.com
> >>>>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com
> >>>>>
> >>>>> On 20/09/2010 07:00, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> >>>>>
> >>>>>> When IO is not ready, domain U in
VMEXIT->hvm_do_resume might invoke
> >>>>>> wait_on_xen_event_channel
> >>>>>> (where it is blocked in _VPF_blocked_in_xen).
> >>>>>>
> >>>>>> Here is my assumption of event missed.
> >>>>>>
> >>>>>> step 1: hvm_do_resume execute 260, and suppose
p->state is
> >>>>>> STATE_IOREQ_READY
> >>>>>> or STATE_IOREQ_INPROCESS
> >>>>>> step 2: then in cpu_handle_ioreq is in line 547,
it execute line 548 so
> >>>>>> quickly before hvm_do_resume execute line 270.
> >>>>>> Well, the event is missed.
> >>>>>> In other words, the _VPF_blocked_in_xen is cleared
before it is actually
> >>>>>> setted, and Domian U who is blocked
> >>>>>> might never get unblocked, it this possible?
> >>>>> Firstly, that code is very paranoid and it should
never actually be the
> >>>>> case
> >>>>> that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS
in hvm_do_resume().
> >>>>> Secondly, even if you do, take a look at the
implementation of
> >>>>> wait_on_xen_event_channel() -- it is smart enough to
avoid the race you
> >>>>> mention.
> >>>>>
> >>>>> -- Keir
> >>>>>
> >>>>>
> >>>
> >>
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Sep-22 00:17 UTC
Re: [Xen-devel] Re: VM hung after running sometime
On 09/21/2010 05:02 PM, MaoXiaoyun wrote:> Thanks Jeremy. > > Regards to fix you mentioned, did you mean the patch I searched and > pasted below, if so, it this all what I need?No, you need more than that. There are quite a few changes from multiple branches, so I''d recommend just using a current kernel.> For irqbalance disabled, I am afried it might have negative > performance impact, right?I doubt it. Unless you have so many interrupts that they can''t all be handled on one cpu, it shouldn''t make much difference. After all, the interrupts have to be handled *somewhere*, but it doesn''t matter much where - who cares if cpu0 is mostly handling interrupts if it leaves the other cpus free for other work? irqbalanced is primarily concerned with migrating interrupts according to the CPU topology to save power and (maybe) handle interrupts closer to the interrupting device. But that''s meaningless in a domain where the vcpus can be mapped to different pcpus from moment to moment. J> > ------------------------------------------------------- > diff --git a/drivers/xen/events.c > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856> > b/drivers/xen/events.c > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608> > index 32f4a2c > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856>..06fc991 > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608> > 100644 (file) > --- a/drivers/xen/events.c > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856> > +++ b/drivers/xen/events.c > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608> > @@ -368,7 > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856#l368> > +368,7 > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608#l368> > @@ int bind_evtchn_to_irq(unsigned int evtchn) > irq = find_unbound_irq(); > set_irq_chip_and_handler_name(irq, &xen_dynamic_chip, > - handle_level_irq, "event"); > + handle_edge_irq, "event"); > evtchn_to_irq[evtchn] = irq; > irq_info[irq] = mk_evtchn_info(evtchn); > > Date: Tue, 21 Sep 2010 10:28:34 -0700 > > From: jeremy@goop.org > > To: keir.fraser@eu.citrix.com > > CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com > > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > > > On 09/21/2010 12:53 AM, Keir Fraser wrote: > > > On 21/09/2010 06:02, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > > > > >> Take a look at domain 0 event channel with port 105,106, I find > on port 105, > > >> it pending is > > >> 1.(in [1,0], first bit refer to pending, and is 1, second bit > refer to mask, > > >> is 0). > > >> > > >> (XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0 > > >> (XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0 > > >> > > >> In all, we have domain U cpu blocking on _VPF_blocked_in_xen, and > it must set > > >> the pending bit. > > >> Consider pending is 1, it looks like the irq is not triggered, am > I right ? > > >> Since if it is triggerred, it should clear the pending bit. (line > 361). > > > Yes it looks like dom0 is not handling the event for some reason. > Qemu looks > > > like it still works and is waiting for a notification via > select(). But that > > > won''t happen until dom0 kernel handles the event as an IRQ and > calls the > > > relevant irq handler (drivers/xen/evtchn.c:evtchn_interrupt()). > > > > > > I think you''re on the right track in your debugging. I don''t know > much about > > > the pv_ops irq handling path, except to say that this aspect is > different > > > than non-pv_ops kernels which special-case handling of events bound to > > > user-space rather more. So at the moment my best guess would be > that the bug > > > is in the pv_ops kernel irq handling for this type of user-space-bound > > > event. > > > > We no longer use handle_level_irq because there''s a race which loses > > events when interrupt migration is enabled. Current xen/stable-2.6.32.x > > has a proper fix for this, but the quick workaround is to disable > > irqbalanced. > > > > J > > > > > -- Keir > > > > > >> > ------------------------------/linux-2.6-pvops.git/kernel/irq/chip.c--- > > >> 354 void > > >> 355 handle_level_irq(unsigned int irq, struct irq_desc *desc) > > >> 356 { > > >> 357 struct irqaction *action; > > >> 358 irqreturn_t action_ret; > > >> 359 > > >> 360 spin_lock(&desc->lock); > > >> 361 mask_ack_irq(desc, irq); > > >> 362 > > >> 363 if (unlikely(desc->status & IRQ_INPROGRESS)) > > >> 364 goto out_unlock; > > >> 365 desc->status &= ~(IRQ_REPLAY | IRQ_WAITING); > > >> 366 kstat_incr_irqs_this_cpu(irq, desc); > > >> 367 > > >> > > >> BTW, the qemu still works fine when VM is hang. Below is it > strace output. > > >> No much difference between other well worked qemu instance, other > than select > > >> all Timeout. > > >> ------------------- > > >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59535265}) = 0 > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59629728}) = 0 > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59717700}) = 0 > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59806552}) = 0 > > >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70234406}) = 0 > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70332116}) = 0 > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70419835}) = 0 > > >> > > >> > > >> > > >> > > >>> Date: Mon, 20 Sep 2010 10:35:46 +0100 > > >>> Subject: Re: VM hung after running sometime > > >>> From: keir.fraser@eu.citrix.com > > >>> To: tinnycloud@hotmail.com > > >>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com > > >>> > > >>> On 20/09/2010 10:15, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > >>> > > >>>> Thanks Keir. > > >>>> > > >>>> You''re right, after I deeply looked into the > wait_on_xen_event_channel, it > > >>>> is > > >>>> smart enough > > >>>> to avoid the race I assumed. > > >>>> > > >>>> How about prepare_wait_on_xen_event_channel ? > > >>>> Currently Istill don''t know when it will be invoked. > > >>>> Could enlighten me? > > >>> As you can see it is called from hvm_send_assist_req(), hence it > is called > > >>> whenever an ioreq is sent to qemu-dm. Note that it is called > *before* > > >>> qemu-dm is notified -- hence it cannot race the wakeup from > qemu, as we will > > >>> not get woken until qemu-dm has done the work, and it cannot > start the work > > >>> until it is notified, and it is not notified until after > > >>> prepare_wait_on_xen_event_channel has been executed. > > >>> > > >>> -- Keir > > >>> > > >>>>> Date: Mon, 20 Sep 2010 08:45:21 +0100 > > >>>>> Subject: Re: VM hung after running sometime > > >>>>> From: keir.fraser@eu.citrix.com > > >>>>> To: tinnycloud@hotmail.com > > >>>>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com > > >>>>> > > >>>>> On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > >>>>> > > >>>>>> When IO is not ready, domain U in VMEXIT->hvm_do_resume might > invoke > > >>>>>> wait_on_xen_event_channel > > >>>>>> (where it is blocked in _VPF_blocked_in_xen). > > >>>>>> > > >>>>>> Here is my assumption of event missed. > > >>>>>> > > >>>>>> step 1: hvm_do_resume execute 260, and suppose p->state is > > >>>>>> STATE_IOREQ_READY > > >>>>>> or STATE_IOREQ_INPROCESS > > >>>>>> step 2: then in cpu_handle_ioreq is in line 547, it execute > line 548 so > > >>>>>> quickly before hvm_do_resume execute line 270. > > >>>>>> Well, the event is missed. > > >>>>>> In other words, the _VPF_blocked_in_xen is cleared before it > is actually > > >>>>>> setted, and Domian U who is blocked > > >>>>>> might never get unblocked, it this possible? > > >>>>> Firstly, that code is very paranoid and it should never > actually be the > > >>>>> case > > >>>>> that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in > hvm_do_resume(). > > >>>>> Secondly, even if you do, take a look at the implementation of > > >>>>> wait_on_xen_event_channel() -- it is smart enough to avoid the > race you > > >>>>> mention. > > >>>>> > > >>>>> -- Keir > > >>>>> > > >>>>> > > >>> > > >> > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks for the details.
Currently guest VM hang in our heavy IO stress test, (In detail, we have created
more than 12 HVMS on our 16cores physical server,
and each of HVM inside, iometer and ab regard as heavy IO periodically run).
Guest hang shows up in 1 or 2 days. So the IO is very
heavy, so as the interrupts, I think.
According to the hang log, the domain blocked in _VPF_blocked_in_xen, indicates
"x=1" in log file below, and that is port 1, 2. And
all our HVM are have PVdriver installed, one thing I am not clear right now is
the IO event in these two ports. Does it only include
"mouse, vga"event, or it also includes hard disk events? (If it has
hard disk events included, the interrupt would be very heavy, right?
and right now we have 4 physical CPU allocated to domain 0, is it appropriate ?
)
Anyway, I think I can have irqbalance disabled for a quick test.
Meanwhile, I will spent some time on the patch merge.
Many thanks.
And its domain event info is :
(XEN) Domain 10 polling vCPUs: {No periodic timer}
(XEN) Event channel information for domain 10:
(XEN) port [p/m]
(XEN) 1 [0/1]: s=3 n=0 d=0 p=105 x=1
(XEN) 2 [0/1]: s=3 n=1 d=0 p=106 x=1
(XEN) 3 [0/0]: s=3 n=0 d=0 p=104 x=0
(XEN) 4 [0/1]: s=2 n=0 d=0 x=0
(XEN) 5 [0/0]: s=6 n=0 x=0
(XEN) 6 [0/0]: s=2 n=0 d=0 x=0
(XEN) 7 [0/0]: s=3 n=0 d=0 p=107 x=0
(XEN) 8 [0/0]: s=3 n=0 d=0 p=108 x=0
(XEN) 9 [0/0]: s=3 n=0 d=0 p=109 x=0
(XEN) 10 [0/0]: s=3 n=0 d=0 p=110 x=0
> Date: Tue, 21 Sep 2010 17:17:12 -0700
> From: jeremy@goop.org
> To: tinnycloud@hotmail.com
> CC: keir.fraser@eu.citrix.com; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Re: VM hung after running sometime
>
> On 09/21/2010 05:02 PM, MaoXiaoyun wrote:
> > Thanks Jeremy.
> >
> > Regards to fix you mentioned, did you mean the patch I searched and
> > pasted below, if so, it this all what I need?
>
> No, you need more than that. There are quite a few changes from multiple
> branches, so I''d recommend just using a current kernel.
>
> > For irqbalance disabled, I am afried it might have negative
> > performance impact, right?
>
> I doubt it. Unless you have so many interrupts that they can''t all
be
> handled on one cpu, it shouldn''t make much difference. After all,
the
> interrupts have to be handled *somewhere*, but it doesn''t matter
much
> where - who cares if cpu0 is mostly handling interrupts if it leaves the
> other cpus free for other work?
>
> irqbalanced is primarily concerned with migrating interrupts according
> to the CPU topology to save power and (maybe) handle interrupts closer
> to the interrupting device. But that''s meaningless in a domain
where the
> vcpus can be mapped to different pcpus from moment to moment.
>
> J
>
>
> >
> > -------------------------------------------------------
> > diff --git a/drivers/xen/events.c
> >
<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856>
> > b/drivers/xen/events.c
> >
<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608>
> > index 32f4a2c
> >
<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856>..06fc991
> >
<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608>
> > 100644 (file)
> > --- a/drivers/xen/events.c
> >
<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856>
> > +++ b/drivers/xen/events.c
> >
<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608>
> > @@ -368,7
> >
<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856#l368>
> > +368,7
> >
<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608#l368>
> > @@ int bind_evtchn_to_irq(unsigned int evtchn)
> > irq = find_unbound_irq();
> > set_irq_chip_and_handler_name(irq, &xen_dynamic_chip,
> > - handle_level_irq, "event");
> > + handle_edge_irq, "event");
> > evtchn_to_irq[evtchn] = irq;
> > irq_info[irq] = mk_evtchn_info(evtchn);
> > > Date: Tue, 21 Sep 2010 10:28:34 -0700
> > > From: jeremy@goop.org
> > > To: keir.fraser@eu.citrix.com
> > > CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> > > Subject: Re: [Xen-devel] Re: VM hung after running sometime
> > >
> > > On 09/21/2010 12:53 AM, Keir Fraser wrote:
> > > > On 21/09/2010 06:02, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> > > >
> > > >> Take a look at domain 0 event channel with port 105,106,
I find
> > on port 105,
> > > >> it pending is
> > > >> 1.(in [1,0], first bit refer to pending, and is 1,
second bit
> > refer to mask,
> > > >> is 0).
> > > >>
> > > >> (XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0
> > > >> (XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0
> > > >>
> > > >> In all, we have domain U cpu blocking on
_VPF_blocked_in_xen, and
> > it must set
> > > >> the pending bit.
> > > >> Consider pending is 1, it looks like the irq is not
triggered, am
> > I right ?
> > > >> Since if it is triggerred, it should clear the pending
bit. (line
> > 361).
> > > > Yes it looks like dom0 is not handling the event for some
reason.
> > Qemu looks
> > > > like it still works and is waiting for a notification via
> > select(). But that
> > > > won''t happen until dom0 kernel handles the event as
an IRQ and
> > calls the
> > > > relevant irq handler
(drivers/xen/evtchn.c:evtchn_interrupt()).
> > > >
> > > > I think you''re on the right track in your
debugging. I don''t know
> > much about
> > > > the pv_ops irq handling path, except to say that this aspect
is
> > different
> > > > than non-pv_ops kernels which special-case handling of
events bound to
> > > > user-space rather more. So at the moment my best guess would
be
> > that the bug
> > > > is in the pv_ops kernel irq handling for this type of
user-space-bound
> > > > event.
> > >
> > > We no longer use handle_level_irq because there''s a race
which loses
> > > events when interrupt migration is enabled. Current
xen/stable-2.6.32.x
> > > has a proper fix for this, but the quick workaround is to disable
> > > irqbalanced.
> > >
> > > J
> > >
> > > > -- Keir
> > > >
> > > >>
> >
------------------------------/linux-2.6-pvops.git/kernel/irq/chip.c---
> > > >> 354 void
> > > >> 355 handle_level_irq(unsigned int irq, struct irq_desc
*desc)
> > > >> 356 {
> > > >> 357 struct irqaction *action;
> > > >> 358 irqreturn_t action_ret;
> > > >> 359
> > > >> 360 spin_lock(&desc->lock);
> > > >> 361 mask_ack_irq(desc, irq);
> > > >> 362
> > > >> 363 if (unlikely(desc->status & IRQ_INPROGRESS))
> > > >> 364 goto out_unlock;
> > > >> 365 desc->status &= ~(IRQ_REPLAY | IRQ_WAITING);
> > > >> 366 kstat_incr_irqs_this_cpu(irq, desc);
> > > >> 367
> > > >>
> > > >> BTW, the qemu still works fine when VM is hang. Below is
it
> > strace output.
> > > >> No much difference between other well worked qemu
instance, other
> > than select
> > > >> all Timeout.
> > > >> -------------------
> > > >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0
(Timeout)
> > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59535265}) = 0
> > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59629728}) = 0
> > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59717700}) = 0
> > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59806552}) = 0
> > > >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0
(Timeout)
> > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70234406}) = 0
> > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70332116}) = 0
> > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70419835}) = 0
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> Date: Mon, 20 Sep 2010 10:35:46 +0100
> > > >>> Subject: Re: VM hung after running sometime
> > > >>> From: keir.fraser@eu.citrix.com
> > > >>> To: tinnycloud@hotmail.com
> > > >>> CC: xen-devel@lists.xensource.com;
jbeulich@novell.com
> > > >>>
> > > >>> On 20/09/2010 10:15, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> > > >>>
> > > >>>> Thanks Keir.
> > > >>>>
> > > >>>> You''re right, after I deeply looked
into the
> > wait_on_xen_event_channel, it
> > > >>>> is
> > > >>>> smart enough
> > > >>>> to avoid the race I assumed.
> > > >>>>
> > > >>>> How about prepare_wait_on_xen_event_channel ?
> > > >>>> Currently Istill don''t know when it
will be invoked.
> > > >>>> Could enlighten me?
> > > >>> As you can see it is called from
hvm_send_assist_req(), hence it
> > is called
> > > >>> whenever an ioreq is sent to qemu-dm. Note that it
is called
> > *before*
> > > >>> qemu-dm is notified -- hence it cannot race the
wakeup from
> > qemu, as we will
> > > >>> not get woken until qemu-dm has done the work, and
it cannot
> > start the work
> > > >>> until it is notified, and it is not notified until
after
> > > >>> prepare_wait_on_xen_event_channel has been executed.
> > > >>>
> > > >>> -- Keir
> > > >>>
> > > >>>>> Date: Mon, 20 Sep 2010 08:45:21 +0100
> > > >>>>> Subject: Re: VM hung after running sometime
> > > >>>>> From: keir.fraser@eu.citrix.com
> > > >>>>> To: tinnycloud@hotmail.com
> > > >>>>> CC: xen-devel@lists.xensource.com;
jbeulich@novell.com
> > > >>>>>
> > > >>>>> On 20/09/2010 07:00, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> > > >>>>>
> > > >>>>>> When IO is not ready, domain U in
VMEXIT->hvm_do_resume might
> > invoke
> > > >>>>>> wait_on_xen_event_channel
> > > >>>>>> (where it is blocked in
_VPF_blocked_in_xen).
> > > >>>>>>
> > > >>>>>> Here is my assumption of event missed.
> > > >>>>>>
> > > >>>>>> step 1: hvm_do_resume execute 260, and
suppose p->state is
> > > >>>>>> STATE_IOREQ_READY
> > > >>>>>> or STATE_IOREQ_INPROCESS
> > > >>>>>> step 2: then in cpu_handle_ioreq is in
line 547, it execute
> > line 548 so
> > > >>>>>> quickly before hvm_do_resume execute
line 270.
> > > >>>>>> Well, the event is missed.
> > > >>>>>> In other words, the _VPF_blocked_in_xen
is cleared before it
> > is actually
> > > >>>>>> setted, and Domian U who is blocked
> > > >>>>>> might never get unblocked, it this
possible?
> > > >>>>> Firstly, that code is very paranoid and it
should never
> > actually be the
> > > >>>>> case
> > > >>>>> that we see STATE_IOREQ_READY or
STATE_IOREQ_INPROCESS in
> > hvm_do_resume().
> > > >>>>> Secondly, even if you do, take a look at the
implementation of
> > > >>>>> wait_on_xen_event_channel() -- it is smart
enough to avoid the
> > race you
> > > >>>>> mention.
> > > >>>>>
> > > >>>>> -- Keir
> > > >>>>>
> > > >>>>>
> > > >>>
> > > >>
> > > >
> > > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > > >
> > >
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Sep-22 18:31 UTC
Re: [Xen-devel] Re: VM hung after running sometime
On 09/21/2010 06:19 PM, MaoXiaoyun wrote:> Thanks for the details. > > Currently guest VM hang in our heavy IO stress test, (In detail, we > have created more than 12 HVMS on our 16cores physical server, > and each of HVM inside, iometer and ab regard as heavy IO periodically > run). Guest hang shows up in 1 or 2 days. So the IO is very > heavy, so as the interrupts, I think.What does /proc/interrupts look like?> > According to the hang log, the domain blocked in _VPF_blocked_in_xen, > indicates "x=1" in log file below, and that is port 1, 2. And > all our HVM are have PVdriver installed, one thing I am not clear > right now is the IO event in these two ports. Does it only include > "mouse, vga"event, or it also includes hard disk events? (If it has > hard disk events included, the interrupt would be very heavy, right? > and right now we have 4 physical CPU allocated to domain 0, is it > appropriate ? )I''m not sure of the details of how qemu<->hvm interaction works, but it was hangs in blkfront in PV domains which brought the lost event problem to light. At the basic event channel level, they will both look the same, and suffer from the same problems.> > Anyway, I think I can have irqbalance disabled for a quick test.Thanks; that should confirm the diagnosis.> Meanwhile, I will spent some time on the patch merge.If you''re not willing to go to the current kernel, I can help you with the minimal set of patches to backport. J> Many thanks. > > And its domain event info is : > (XEN) Domain 10 polling vCPUs: {No periodic timer} > (XEN) Event channel information for domain 10: > (XEN) port [p/m] > (XEN) 1 [0/1]: s=3 n=0 d=0 p=105 x=1 > (XEN) 2 [0/1]: s=3 n=1 d=0 p=106 x=1 > (XEN) 3 [0/0]: s=3 n=0 d=0 p=104 x=0 > (XEN) 4 [0/1]: s=2 n=0 d=0 x=0 > (XEN) 5 [0/0]: s=6 n=0 x=0 > (XEN) 6 [0/0]: s=2 n=0 d=0 x=0 > (XEN) 7 [0/0]: s=3 n=0 d=0 p=107 x=0 > (XEN) 8 [0/0]: s=3 n=0 d=0 p=108 x=0 > (XEN) 9 [0/0]: s=3 n=0 d=0 p=109 x=0 > (XEN) 10 [0/0]: s=3 n=0 d=0 p=110 x=0 > > > Date: Tue, 21 Sep 2010 17:17:12 -0700 > > From: jeremy@goop.org > > To: tinnycloud@hotmail.com > > CC: keir.fraser@eu.citrix.com; xen-devel@lists.xensource.com > > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > > > On 09/21/2010 05:02 PM, MaoXiaoyun wrote: > > > Thanks Jeremy. > > > > > > Regards to fix you mentioned, did you mean the patch I searched and > > > pasted below, if so, it this all what I need? > > > > No, you need more than that. There are quite a few changes from multiple > > branches, so I''d recommend just using a current kernel. > > > > > For irqbalance disabled, I am afried it might have negative > > > performance impact, right? > > > > I doubt it. Unless you have so many interrupts that they can''t all be > > handled on one cpu, it shouldn''t make much difference. After all, the > > interrupts have to be handled *somewhere*, but it doesn''t matter much > > where - who cares if cpu0 is mostly handling interrupts if it leaves the > > other cpus free for other work? > > > > irqbalanced is primarily concerned with migrating interrupts according > > to the CPU topology to save power and (maybe) handle interrupts closer > > to the interrupting device. But that''s meaningless in a domain where the > > vcpus can be mapped to different pcpus from moment to moment. > > > > J > > > > > > > > > > ------------------------------------------------------- > > > diff --git a/drivers/xen/events.c > > > > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856> > > > b/drivers/xen/events.c > > > > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608> > > > index 32f4a2c > > > > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856>..06fc991 > > > > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608> > > > 100644 (file) > > > --- a/drivers/xen/events.c > > > > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856> > > > +++ b/drivers/xen/events.c > > > > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608> > > > @@ -368,7 > > > > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=32f4a2cfe11e342104b9e568c230f2f17b5ae856;hb=32f4a2cfe11e342104b9e568c230f2f17b5ae856#l368> > > > +368,7 > > > > <http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/xen/events.c;h=06fc9915176cdceca49f554c7a108c0fc3c5e608;hb=06fc9915176cdceca49f554c7a108c0fc3c5e608#l368> > > > @@ int bind_evtchn_to_irq(unsigned int evtchn) > > > irq = find_unbound_irq(); > > > set_irq_chip_and_handler_name(irq, &xen_dynamic_chip, > > > - handle_level_irq, "event"); > > > + handle_edge_irq, "event"); > > > evtchn_to_irq[evtchn] = irq; > > > irq_info[irq] = mk_evtchn_info(evtchn); > > > > Date: Tue, 21 Sep 2010 10:28:34 -0700 > > > > From: jeremy@goop.org > > > > To: keir.fraser@eu.citrix.com > > > > CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com > > > > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > > > > > > > On 09/21/2010 12:53 AM, Keir Fraser wrote: > > > > > On 21/09/2010 06:02, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > > > > > > > > >> Take a look at domain 0 event channel with port 105,106, I find > > > on port 105, > > > > >> it pending is > > > > >> 1.(in [1,0], first bit refer to pending, and is 1, second bit > > > refer to mask, > > > > >> is 0). > > > > >> > > > > >> (XEN) 105 [1/0]: s=3 n=2 d=10 p=1 x=0 > > > > >> (XEN) 106 [0/0]: s=3 n=2 d=10 p=2 x=0 > > > > >> > > > > >> In all, we have domain U cpu blocking on _VPF_blocked_in_xen, and > > > it must set > > > > >> the pending bit. > > > > >> Consider pending is 1, it looks like the irq is not triggered, am > > > I right ? > > > > >> Since if it is triggerred, it should clear the pending bit. (line > > > 361). > > > > > Yes it looks like dom0 is not handling the event for some reason. > > > Qemu looks > > > > > like it still works and is waiting for a notification via > > > select(). But that > > > > > won''t happen until dom0 kernel handles the event as an IRQ and > > > calls the > > > > > relevant irq handler (drivers/xen/evtchn.c:evtchn_interrupt()). > > > > > > > > > > I think you''re on the right track in your debugging. I don''t know > > > much about > > > > > the pv_ops irq handling path, except to say that this aspect is > > > different > > > > > than non-pv_ops kernels which special-case handling of events > bound to > > > > > user-space rather more. So at the moment my best guess would be > > > that the bug > > > > > is in the pv_ops kernel irq handling for this type of > user-space-bound > > > > > event. > > > > > > > > We no longer use handle_level_irq because there''s a race which loses > > > > events when interrupt migration is enabled. Current > xen/stable-2.6.32.x > > > > has a proper fix for this, but the quick workaround is to disable > > > > irqbalanced. > > > > > > > > J > > > > > > > > > -- Keir > > > > > > > > > >> > > > > ------------------------------/linux-2.6-pvops.git/kernel/irq/chip.c--- > > > > >> 354 void > > > > >> 355 handle_level_irq(unsigned int irq, struct irq_desc *desc) > > > > >> 356 { > > > > >> 357 struct irqaction *action; > > > > >> 358 irqreturn_t action_ret; > > > > >> 359 > > > > >> 360 spin_lock(&desc->lock); > > > > >> 361 mask_ack_irq(desc, irq); > > > > >> 362 > > > > >> 363 if (unlikely(desc->status & IRQ_INPROGRESS)) > > > > >> 364 goto out_unlock; > > > > >> 365 desc->status &= ~(IRQ_REPLAY | IRQ_WAITING); > > > > >> 366 kstat_incr_irqs_this_cpu(irq, desc); > > > > >> 367 > > > > >> > > > > >> BTW, the qemu still works fine when VM is hang. Below is it > > > strace output. > > > > >> No much difference between other well worked qemu instance, other > > > than select > > > > >> all Timeout. > > > > >> ------------------- > > > > >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) > > > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59535265}) = 0 > > > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59629728}) = 0 > > > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59717700}) = 0 > > > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 59806552}) = 0 > > > > >> select(14, [3 7 11 12 13], [], [], {0, 10000}) = 0 (Timeout) > > > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70234406}) = 0 > > > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70332116}) = 0 > > > > >> clock_gettime(CLOCK_MONOTONIC, {673470, 70419835}) = 0 > > > > >> > > > > >> > > > > >> > > > > >> > > > > >>> Date: Mon, 20 Sep 2010 10:35:46 +0100 > > > > >>> Subject: Re: VM hung after running sometime > > > > >>> From: keir.fraser@eu.citrix.com > > > > >>> To: tinnycloud@hotmail.com > > > > >>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com > > > > >>> > > > > >>> On 20/09/2010 10:15, "MaoXiaoyun" <tinnycloud@hotmail.com> > wrote: > > > > >>> > > > > >>>> Thanks Keir. > > > > >>>> > > > > >>>> You''re right, after I deeply looked into the > > > wait_on_xen_event_channel, it > > > > >>>> is > > > > >>>> smart enough > > > > >>>> to avoid the race I assumed. > > > > >>>> > > > > >>>> How about prepare_wait_on_xen_event_channel ? > > > > >>>> Currently Istill don''t know when it will be invoked. > > > > >>>> Could enlighten me? > > > > >>> As you can see it is called from hvm_send_assist_req(), hence it > > > is called > > > > >>> whenever an ioreq is sent to qemu-dm. Note that it is called > > > *before* > > > > >>> qemu-dm is notified -- hence it cannot race the wakeup from > > > qemu, as we will > > > > >>> not get woken until qemu-dm has done the work, and it cannot > > > start the work > > > > >>> until it is notified, and it is not notified until after > > > > >>> prepare_wait_on_xen_event_channel has been executed. > > > > >>> > > > > >>> -- Keir > > > > >>> > > > > >>>>> Date: Mon, 20 Sep 2010 08:45:21 +0100 > > > > >>>>> Subject: Re: VM hung after running sometime > > > > >>>>> From: keir.fraser@eu.citrix.com > > > > >>>>> To: tinnycloud@hotmail.com > > > > >>>>> CC: xen-devel@lists.xensource.com; jbeulich@novell.com > > > > >>>>> > > > > >>>>> On 20/09/2010 07:00, "MaoXiaoyun" <tinnycloud@hotmail.com> > wrote: > > > > >>>>> > > > > >>>>>> When IO is not ready, domain U in VMEXIT->hvm_do_resume might > > > invoke > > > > >>>>>> wait_on_xen_event_channel > > > > >>>>>> (where it is blocked in _VPF_blocked_in_xen). > > > > >>>>>> > > > > >>>>>> Here is my assumption of event missed. > > > > >>>>>> > > > > >>>>>> step 1: hvm_do_resume execute 260, and suppose p->state is > > > > >>>>>> STATE_IOREQ_READY > > > > >>>>>> or STATE_IOREQ_INPROCESS > > > > >>>>>> step 2: then in cpu_handle_ioreq is in line 547, it execute > > > line 548 so > > > > >>>>>> quickly before hvm_do_resume execute line 270. > > > > >>>>>> Well, the event is missed. > > > > >>>>>> In other words, the _VPF_blocked_in_xen is cleared before it > > > is actually > > > > >>>>>> setted, and Domian U who is blocked > > > > >>>>>> might never get unblocked, it this possible? > > > > >>>>> Firstly, that code is very paranoid and it should never > > > actually be the > > > > >>>>> case > > > > >>>>> that we see STATE_IOREQ_READY or STATE_IOREQ_INPROCESS in > > > hvm_do_resume(). > > > > >>>>> Secondly, even if you do, take a look at the implementation of > > > > >>>>> wait_on_xen_event_channel() -- it is smart enough to avoid the > > > race you > > > > >>>>> mention. > > > > >>>>> > > > > >>>>> -- Keir > > > > >>>>> > > > > >>>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > _______________________________________________ > > > > > Xen-devel mailing list > > > > > Xen-devel@lists.xensource.com > > > > > http://lists.xensource.com/xen-devel > > > > > > > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The interrputs file is attached. The server has 24 HVM domains runnning about 40 hours. Well, we may upgrade to the new kernel in the further, but currently we prefer the fix has least impact on our present server. So it is really nice of you if you could offer the sets of patches, also, it would be our fisrt choice. Later I will kick off the irqbalance disabled test in different servers, will keep you noticed. Thanks for your kindly assitance.> Date: Wed, 22 Sep 2010 11:31:22 -0700 > From: jeremy@goop.org > To: tinnycloud@hotmail.com > CC: keir.fraser@eu.citrix.com; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > On 09/21/2010 06:19 PM, MaoXiaoyun wrote: > > Thanks for the details. > > > > Currently guest VM hang in our heavy IO stress test, (In detail, we > > have created more than 12 HVMS on our 16cores physical server, > > and each of HVM inside, iometer and ab regard as heavy IO periodically > > run). Guest hang shows up in 1 or 2 days. So the IO is very > > heavy, so as the interrupts, I think. > > What does /proc/interrupts look like? > > > > > According to the hang log, the domain blocked in _VPF_blocked_in_xen, > > indicates "x=1" in log file below, and that is port 1, 2. And > > all our HVM are have PVdriver installed, one thing I am not clear > > right now is the IO event in these two ports. Does it only include > > "mouse, vga"event, or it also includes hard disk events? (If it has > > hard disk events included, the interrupt would be very heavy, right? > > and right now we have 4 physical CPU allocated to domain 0, is it > > appropriate ? ) > > I''m not sure of the details of how qemu<->hvm interaction works, but it > was hangs in blkfront in PV domains which brought the lost event problem > to light. At the basic event channel level, they will both look the > same, and suffer from the same problems. > > > > > Anyway, I think I can have irqbalance disabled for a quick test. > > Thanks; that should confirm the diagnosis. > > > Meanwhile, I will spent some time on the patch merge. > > If you''re not willing to go to the current kernel, I can help you with > the minimal set of patches to backport. > > J_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Sep-23 23:20 UTC
Re: [Xen-devel] Re: VM hung after running sometime
On 09/22/2010 05:55 PM, MaoXiaoyun wrote:> The interrputs file is attached. The server has 24 HVM domains > runnning about 40 hours. > > Well, we may upgrade to the new kernel in the further, but currently > we prefer the fix has least impact on our present server. > So it is really nice of you if you could offer the sets of patches, > also, it would be our fisrt choice.Try cherry-picking: 8401e9b96f80f9c0128e7c8fc5a01abfabbfa021 xen: use percpu interrupts for IPIs and VIRQs 66fd3052fec7e7c21a9d88ba1a03bc062f5fb53d xen: handle events as edge-triggered 29a2e2a7bd19233c62461b104c69233f15ce99ec xen/apic: use handle_edge_irq for pirq events f61692642a2a2b83a52dd7e64619ba3bb29998af xen/pirq: do EOI properly for pirq events 0672fb44a111dfb6386022071725c5b15c9de584 xen/events: change to using fasteoi 2789ef00cbe2cdb38deb30ee4085b88befadb1b0 xen: make pirq interrupts use fasteoi d0936845a856816af2af48ddf019366be68e96ba xen/evtchn: rename enable/disable_dynirq -> unmask/mask_irq c6a16a778f86699b339585ba5b9197035d77c40f xen/evtchn: rename retrigger_dynirq -> irq f4526f9a78ffb3d3fc9f81636c5b0357fc1beccd xen/evtchn: make pirq enable/disable unmask/mask 43d8a5030a502074f3c4aafed4d6095ebd76067c xen/evtchn: pirq_eoi does unmask cb23e8d58ca35b6f9e10e1ea5682bd61f2442ebd xen/evtchn: correction, pirq hypercall does not unmask 2390c371ecd32d9f06e22871636185382bf70ab7 xen/events: use PHYSDEVOP_pirq_eoi_gmfn to get pirq need-EOI info 158d6550716687486000a828c601706b55322ad0 xen/pirq: use eoi as enable d2ea486300ca6e207ba178a425fbd023b8621bb1 xen/pirq: use fasteoi for MSI too f0d4a0552f03b52027fb2c7958a1cbbe210cf418 xen/apic: fix pirq_eoi_gmfn resume> Later I will kick off the irqbalance disabled test in different > servers, will keep you noticed.Thanks, J> > Thanks for your kindly assitance. > > > Date: Wed, 22 Sep 2010 11:31:22 -0700 > > From: jeremy@goop.org > > To: tinnycloud@hotmail.com > > CC: keir.fraser@eu.citrix.com; xen-devel@lists.xensource.com > > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > > > On 09/21/2010 06:19 PM, MaoXiaoyun wrote: > > > Thanks for the details. > > > > > > Currently guest VM hang in our heavy IO stress test, (In detail, we > > > have created more than 12 HVMS on our 16cores physical server, > > > and each of HVM inside, iometer and ab regard as heavy IO periodically > > > run). Guest hang shows up in 1 or 2 days. So the IO is very > > > heavy, so as the interrupts, I think. > > > > What does /proc/interrupts look like? > > > > > > > > According to the hang log, the domain blocked in _VPF_blocked_in_xen, > > > indicates "x=1" in log file below, and that is port 1, 2. And > > > all our HVM a re have PVdriver installed, one thing I am not clear > > > right now is the IO event in these two ports. Does it only include > > > "mouse, vga"event, or it also includes hard disk events? (If it has > > > hard disk events included, the interrupt would be very heavy, right? > > > and right now we have 4 physical CPU allocated to domain 0, is it > > > appropriate ? ) > > > > I''m not sure of the details of how qemu<->hvm interaction works, but it > > was hangs in blkfront in PV domains which brought the lost event problem > > to light. At the basic event channel level, they will both look the > > same, and suffer from the same problems. > > > > > > > > Anyway, I think I can have irqbalance disabled for a quick test. > > > > Thanks; that should confirm the diagnosis. > > > > > Meanwhile, I will spent some time on the patch merge. > > > > If you''re not willing to go to t he current kernel, I can help you with > > the minimal set of patches to backport. > > > > J > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thank you very much, Jeremy. I will have a try.> Date: Thu, 23 Sep 2010 16:20:09 -0700 > From: jeremy@goop.org > To: tinnycloud@hotmail.com > CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > On 09/22/2010 05:55 PM, MaoXiaoyun wrote: > > The interrputs file is attached. The server has 24 HVM domains > > runnning about 40 hours. > > > > Well, we may upgrade to the new kernel in the further, but currently > > we prefer the fix has least impact on our present server. > > So it is really nice of you if you could offer the sets of patches, > > also, it would be our fisrt choice. > > Try cherry-picking: > 8401e9b96f80f9c0128e7c8fc5a01abfabbfa021 xen: use percpu interrupts for > IPIs and VIRQs > 66fd3052fec7e7c21a9d88ba1a03bc062f5fb53d xen: handle events as > edge-triggered > 29a2e2a7bd19233c62461b104c69233f15ce99ec xen/apic: use handle_edge_irq > for pirq events > f61692642a2a2b83a52dd7e64619ba3bb29998af xen/pirq: do EOI properly for > pirq events > 0672fb44a111dfb6386022071725c5b15c9de584 xen/events: change to using fasteoi > 2789ef00cbe2cdb38deb30ee4085b88befadb1b0 xen: make pirq interrupts use > fasteoi > d0936845a856816af2af48ddf019366be68e96ba xen/evtchn: rename > enable/disable_dynirq -> unmask/mask_irq > c6a16a778f86699b339585ba5b9197035d77c40f xen/evtchn: rename > retrigger_dynirq -> irq > f4526f9a78ffb3d3fc9f81636c5b0357fc1beccd xen/evtchn: make pirq > enable/disable unmask/mask > 43d8a5030a502074f3c4aafed4d6095ebd76067c xen/evtchn: pirq_eoi does unmask > cb23e8d58ca35b6f9e10e1ea5682bd61f2442ebd xen/evtchn: correction, pirq > hypercall does not unmask > 2390c371ecd32d9f06e22871636185382bf70ab7 xen/events: use > PHYSDEVOP_pirq_eoi_gmfn to get pirq need-EOI info > 158d6550716687486000a828c601706b55322ad0 xen/pirq: use eoi as enable > d2ea486300ca6e207ba178a425fbd023b8621bb1 xen/pirq: use fasteoi for MSI too > f0d4a0552f03b52027fb2c7958a1cbbe210cf418 xen/apic: fix pirq_eoi_gmfn resume > > > Later I will kick off the irqbalance disabled test in different > > servers, will keep you noticed. > > Thanks, > J > > > > > Thanks for your kindly assitance. > > > > > Date: Wed, 22 Sep 2010 11:31:22 -0700 > > > From: jeremy@goop.org > > > To: tinnycloud@hotmail.com > > > CC: keir.fraser@eu.citrix.com; xen-devel@lists.xensource.com > > > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > > > > > On 09/21/2010 06:19 PM, MaoXiaoyun wrote: > > > > Thanks for the details. > > > > > > > > Currently guest VM hang in our heavy IO stress test, (In detail, we > > > > have created more than 12 HVMS on our 16cores physical server, > > > > and each of HVM inside, iometer and ab regard as heavy IO periodically > > > > run). Guest hang shows up in 1 or 2 days. So the IO is very > > > > heavy, so as the interrupts, I think. > > > > > > What does /proc/interrupts look like? > > > > > > > > > > > According to the hang log, the domain blocked in _VPF_blocked_in_xen, > > > > indicates "x=1" in log file below, and that is port 1, 2. And > > > > all our HVM a re have PVdriver installed, one thing I am not clear > > > > right now is the IO event in these two ports. Does it only include > > > > "mouse, vga"event, or it also includes hard disk events? (If it has > > > > hard disk events included, the interrupt would be very heavy, right? > > > > and right now we have 4 physical CPU allocated to domain 0, is it > > > > appropriate ? ) > > > > > > I''m not sure of the details of how qemu<->hvm interaction works, but it > > > was hangs in blkfront in PV domains which brought the lost event problem > > > to light. At the basic event channel level, they will both look the > > > same, and suffer from the same problems. > > > > > > > > > > > Anyway, I think I can have irqbalance disabled for a quick test. > > > > > > Thanks; that should confirm the diagnosis. > > > > > > > Meanwhile, I will spent some time on the patch merge. > > > > > > If you''re not willing to go to t he current kernel, I can help you with > > > the minimal set of patches to backport. > > > > > > J > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Jeremy:
The test of irqbalance disabled is running. Currently one server was
crashed on NIC.
Trace.jpg in attachments is the screenshot from serial port, and trace.txt
is from /varl/log/message.
Do you think it has connection with irqbalance disabled, or some other
possibilities?
In addition, I find in /proc/interrupts, all interrupts are happend on
cpu0(please refer to interrputs.txt
attached). Could it be a possible cause of server crash, and is there a way I
can configure manually to
distribute those interrupts evenly?
Meanwhile, I wil start the new test with kernel patched soon. Thanks.
> Date: Thu, 23 Sep 2010 16:20:09 -0700
> From: jeremy@goop.org
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com
> Subject: Re: [Xen-devel] Re: VM hung after running sometime
>
> On 09/22/2010 05:55 PM, MaoXiaoyun wrote:
> > The interrputs file is attached. The server has 24 HVM domains
> > runnning about 40 hours.
> >
> > Well, we may upgrade to the new kernel in the further, but currently
> > we prefer the fix has least impact on our present server.
> > So it is really nice of you if you could offer the sets of patches,
> > also, it would be our fisrt choice.
>
> Try cherry-picking:
> 8401e9b96f80f9c0128e7c8fc5a01abfabbfa021 xen: use percpu interrupts for
> IPIs and VIRQs
> 66fd3052fec7e7c21a9d88ba1a03bc062f5fb53d xen: handle events as
> edge-triggered
> 29a2e2a7bd19233c62461b104c69233f15ce99ec xen/apic: use handle_edge_irq
> for pirq events
> f61692642a2a2b83a52dd7e64619ba3bb29998af xen/pirq: do EOI properly for
> pirq events
> 0672fb44a111dfb6386022071725c5b15c9de584 xen/events: change to using
fasteoi
> 2789ef00cbe2cdb38deb30ee4085b88befadb1b0 xen: make pirq interrupts use
> fasteoi
> d0936845a856816af2af48ddf019366be68e96ba xen/evtchn: rename
> enable/disable_dynirq -> unmask/mask_irq
> c6a16a778f86699b339585ba5b9197035d77c40f xen/evtchn: rename
> retrigger_dynirq -> irq
> f4526f9a78ffb3d3fc9f81636c5b0357fc1beccd xen/evtchn: make pirq
> enable/disable unmask/mask
> 43d8a5030a502074f3c4aafed4d6095ebd76067c xen/evtchn: pirq_eoi does unmask
> cb23e8d58ca35b6f9e10e1ea5682bd61f2442ebd xen/evtchn: correction, pirq
> hypercall does not unmask
> 2390c371ecd32d9f06e22871636185382bf70ab7 xen/events: use
> PHYSDEVOP_pirq_eoi_gmfn to get pirq need-EOI info
> 158d6550716687486000a828c601706b55322ad0 xen/pirq: use eoi as enable
> d2ea486300ca6e207ba178a425fbd023b8621bb1 xen/pirq: use fasteoi for MSI too
> f0d4a0552f03b52027fb2c7958a1cbbe210cf418 xen/apic: fix pirq_eoi_gmfn resume
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Hi Jeremy,
Do you think this issue is cased of without CONFIG_X86_F00F_BUG.
thanks,
James
在 2010年9月25日 下午5:33，MaoXiaoyun <tinnycloud@hotmail.com>写道：
> Hi Jeremy:
>
> The test of irqbalance disabled is running. Currently one server was
> crashed on NIC.
> Trace.jpg in attachments is the screenshot from serial port, and
> trace.txt is from /varl/log/message.
> Do you think it has connection with irqbalance disabled, or some
> other possibilities?
>
> In addition, I find in /proc/interrupts, all interrupts are happend
> on cpu0(please refer to interrputs.txt
> attached). Could it be a possible cause of server crash, and is there a way
> I can configure manually to
> distribute those interrupts evenly?
>
> Meanwhile, I wil start the new test with kernel patched soon. Thanks.
>
>
> > Date: Thu, 23 Sep 2010 16:20:09 -0700
>
> > From: jeremy@goop.org
> > To: tinnycloud@hotmail.com
> > CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com
>
> > Subject: Re: [Xen-devel] Re: VM hung after running sometime
> >
> > On 09/22/2010 05:55 PM, MaoXiaoyun wrote:
> > > The interrputs file is attached. The server has 24 HVM domains
> > > runnning about 40 hours.
> > >
> > > Well, we may upgrade to the new kernel in the further, but
currently
> > > we prefer the fix has least impact on our present server.
> > > So it is really nice of you if you could offer the sets of
patches,
> > > also, it would be our fisrt choice.
> >
> > Try cherry-picking:
> > 8401e9b96f80f9c0128e7c8fc5a01abfabbfa021 xen: use percpu interrupts
for
> > IPIs and VIRQs
> > 66fd3052 fec7e7c21a9d88ba1a03bc062f5fb53d xen: handle events as
>
> > edge-triggered
> > 29a2e2a7bd19233c62461b104c69233f15ce99ec xen/apic: use handle_edge_irq
> > for pirq events
> > f61692642a2a2b83a52dd7e64619ba3bb29998af xen/pirq: do EOI properly for
> > pirq events
> > 0672fb44a111dfb6386022071725c5b15c9de584 xen/events: change to using
> fasteoi
> > 2789ef00cbe2cdb38deb30ee4085b88befadb1b0 xen: make pirq interrupts use
> > fasteoi
> > d0936845a856816af2af48ddf019366be68e96ba xen/evtchn: rename
> > enable/disable_dynirq -> unmask/mask_irq
> > c6a16a778f86699b339585ba5b9197035d77c40f xen/evtchn: rename
> > retrigger_dynirq -> irq
> > f4526f9a78ffb3d3fc9f81636c5b0357fc1beccd xen/evtchn: make pirq
> > enable/disable unmask/mask
> > 43d8a5030a502074f3c4aafed4d6095ebd76067c xen/evtchn: pirq_eoi does
unmask
> > cb23e8d58ca35b6f9e10e1ea5682bd61f2442ebd xen/evtchn: correction, pirq
> > hypercall does not unmask
> > 2390c371ecd32d9f06e22871636185382bf70ab7 xen/events: use
> > PHYSDEVOP_pirq_eoi_gmfn to get pirq need-EOI info
> > 158d6550716687486000a828c601706b55322ad0 xen/pirq: use eoi as enable
> > d2ea486300ca6e207ba178a425fbd023b8621bb1 xen/pirq: use fasteoi for MSI
> too
> > f0d4a0552f03b52027fb2c7958a1cbbe210cf418 xen/apic: fix pirq_eoi_gmfn
> resume
> >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Hi Jeremy:
About the NIC crash, it turns out to our NIC driver problem.
The crash no longer show up after the driver upgraded.
The irqbanlance disabled test is running smoothly so far.
Meanwhile, we had merged your patch to our current kernel(2.6.31), and
start the test.
Unfortunately, one of the VM hang in a few minutes after it started.
But this time some abnormal kernel backtrace logged in /var/log/message.
I wonder if the patch is compatible with our current kernel? Or some extra
modifications I need?
Consider the good result of irqbalance disabled test, I''m afried I
may commit some mistakes in patch
merge since I''m newer to git stuff(-_-!!).
So I attached the merged patch (only event.c), could you help to review -:)?
Thanks for your time.
Kernel backtrace below:
---------------------------------------------------------------------------------------------------------------------
14 Sep 27 18:36:10 pc1 kernel: ------------[ cut here ]------------
15 Sep 27 18:36:10 pc1 kernel: WARNING: at net/core/skbuff.c:475
skb_release_head_state+0x71/0xf8()
16 Sep 27 18:36:10 pc1 kernel: Hardware name: PowerEdge R710
18 Sep 27 18:36:10 pc1 kernel: Pid: 0, comm: swapper Tainted: G W
2.6.31.13xen #4
19 Sep 27 18:36:10 pc1 kernel: Call Trace:
20 Sep 27 18:36:10 pc1 kernel: <IRQ> [<ffffffff8136c751>] ?
skb_release_head_state+0x71/0xf8
21 Sep 27 18:36:10 pc1 kernel: [<ffffffff810535ba>]
warn_slowpath_common+0x7c/0x94
22 Sep 27 18:36:10 pc1 kernel: [<ffffffff810535e6>]
warn_slowpath_null+0x14/0x16
23 Sep 27 18:36:10 pc1 kernel: [<ffffffff8136c751>]
skb_release_head_state+0x71/0xf8
24 Sep 27 18:36:10 pc1 kernel: [<ffffffff8136c7ee>]
skb_release_all+0x16/0x22
25 Sep 27 18:36:10 pc1 kernel: [<ffffffff8136c837>] __kfree_skb+0x16/0x84
26 Sep 27 18:36:10 pc1 kernel: [<ffffffff8136c8d2>] consume_skb+0x2d/0x2f
27 Sep 27 18:36:10 pc1 kernel: [<ffffffffa0069aab>]
bnx2_poll_work+0x1b7/0xa0f [bnx2]
28 Sep 27 18:36:10 pc1 kernel: [<ffffffff81260f00>] ?
HYPERVISOR_event_channel_op+0x1a/0x4d
29 Sep 27 18:36:10 pc1 kernel: [<ffffffff8126102a>] ?
unmask_evtchn+0x4f/0xa3
30 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100eb71>] ?
xen_force_evtchn_callback+0xd/0xf
31 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f292>] ?
check_events+0x12/0x20
32 Sep 27 18:36:10 pc1 kernel: [<ffffffff81414226>] ?
_spin_lock_irqsave+0x1e/0x37
33 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f27f>] ?
xen_restore_fl_direct_end+0x0/0x1
34 Sep 27 18:36:10 pc1 kernel: [<ffffffffa006c8a7>]
bnx2_poll_msix+0x38/0x92 [bnx2]
35 Sep 27 18:36:10 pc1 kernel: [<ffffffff81382eaf>]
netpoll_poll+0xa3/0x38f
36 Sep 27 18:36:10 pc1 kernel: [<ffffffff810ec08b>] ?
__kmalloc_track_caller+0x11a/0x12c
37 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f27f>] ?
xen_restore_fl_direct_end+0x0/0x1
38 Sep 27 18:36:10 pc1 kernel: [<ffffffff813832b4>]
netpoll_send_skb+0x119/0x1f7
39 Sep 27 18:36:10 pc1 kernel: [<ffffffff8138360d>]
netpoll_send_udp+0x1e4/0x1f1
40 Sep 27 18:36:10 pc1 kernel: [<ffffffffa021d18f>] write_msg+0x8d/0xd2
[netconsole]
41 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f27f>] ?
xen_restore_fl_direct_end+0x0/0x1
42 Sep 27 18:36:10 pc1 kernel: [<ffffffff81053937>]
__call_console_drivers+0x6c/0x7e
43 Sep 27 18:36:10 pc1 kernel: [<ffffffff810539a9>]
_call_console_drivers+0x60/0x64
44 Sep 27 18:36:10 pc1 kernel: [<ffffffff81414226>] ?
_spin_lock_irqsave+0x1e/0x37
45 Sep 27 18:36:10 pc1 kernel: [<ffffffff81053df2>]
release_console_sem+0x11a/0x19c
46 Sep 27 18:36:10 pc1 kernel: [<ffffffff810543b9>] vprintk+0x2e1/0x31a
47 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f1a5>] ?
xen_clocksource_get_cycles+0x9/0x1c
48 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f0d6>] ?
xen_clocksource_read+0x21/0x23
49 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100eb71>] ?
xen_force_evtchn_callback+0xd/0xf
50 Sep 27 18:36:10 pc1 kernel: [<ffffffff81054499>] printk+0xa7/0xa9
51 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f27f>] ?
xen_restore_fl_direct_end+0x0/0x1
52 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f0d6>] ?
xen_clocksource_read+0x21/0x23
53 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100f1a5>] ?
xen_clocksource_get_cycles+0x9/0x1c
54 Sep 27 18:36:10 pc1 kernel: [<ffffffff81070fcf>] ?
clocksource_read+0xf/0x11
55 Sep 27 18:36:10 pc1 kernel: [<ffffffff81071695>] ?
getnstimeofday+0x5b/0xbb
56 Sep 27 18:36:10 pc1 kernel: [<ffffffff8126125d>] ?
cpumask_next+0x1e/0x20
57 Sep 27 18:36:10 pc1 kernel: [<ffffffff812627c1>]
xen_debug_interrupt+0x256/0x289
58 Sep 27 18:36:10 pc1 kernel: [<ffffffff81098276>]
handle_IRQ_event+0x66/0x120
59 Sep 27 18:36:10 pc1 kernel: [<ffffffff81099947>]
handle_percpu_irq+0x41/0x6e
60 Sep 27 18:36:10 pc1 kernel: [<ffffffff812624dd>]
xen_evtchn_do_upcall+0x102/0x190
61 Sep 27 18:36:10 pc1 kernel: [<ffffffff81014fbe>]
xen_do_hypervisor_callback+0x1e/0x30
62 Sep 27 18:36:10 pc1 kernel: <EOI> [<ffffffff810093aa>] ?
hypercall_page+0x3aa/0x1000
63 Sep 27 18:36:10 pc1 kernel: [<ffffffff810093aa>] ?
hypercall_page+0x3aa/0x1000
64 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100ebb7>] ?
xen_safe_halt+0x10/0x1a
65 Sep 27 18:36:10 pc1 kernel: [<ffffffff8100c0f5>] ? xen_idle+0x3b/0x52
66 Sep 27 18:36:10 pc1 kernel: [<ffffffff81012c9d>] ? cpu_idle+0x5d/0x8c
67 Sep 27 18:36:10 pc1 kernel: [<ffffffff8140aaa3>] ?
cpu_bringup_and_idle+0x13/0x15
68 Sep 27 18:36:10 pc1 kernel: ---[ end trace d83eb1ebe87fed96 ]---
From: tinnycloud@hotmail.com
To: jeremy@goop.org
CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com
Subject: RE: [Xen-devel] Re: VM hung after running sometime
Date: Sat, 25 Sep 2010 17:33:23 +0800
Hi Jeremy:
The test of irqbalance disabled is running. Currently one server was
crashed on NIC.
Trace.jpg in attachments is the screenshot from serial port, and trace.txt
is from /varl/log/message.
Do you think it has connection with irqbalance disabled, or some other
possibilities?
In addition, I find in /proc/interrupts, all interrupts are happend on
cpu0(please refer to interrputs.txt
attached). Could it be a possible cause of server crash, and is there a way I
can configure manually to
distribute those interrupts evenly?
Meanwhile, I wil start the new test with kernel patched soon. Thanks.
> Date: Thu, 23 Sep 2010 16:20:09 -0700
> From: jeremy@goop.org
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com
> Subject: Re: [Xen-devel] Re: VM hung after running sometime
>
> On 09/22/2010 05:55 PM, MaoXiaoyun wrote:
> > The interrputs file is attached. The server has 24 HVM domains
> > runnning about 40 hours.
> >
> > Well, we may upgrade to the new kernel in the further, but currently
> > we prefer the fix has least impact on our present server.
> > So it is really nice of you if you could offer the sets of patches,
> > also, it would be our fisrt choice.
>
> Try cherry-picking:
> 8401e9b96f80f9c0128e7c8fc5a01abfabbfa021 xen: use percpu interrupts for
> IPIs and VIRQs
> 66fd3052fec7e7c21a9d88ba1a03bc062f5fb53d xen: handle events as
> edge-triggered
> 29a2e2a7bd19233c62461b104c69233f15ce99ec xen/apic: use handle_edge_irq
> for pirq events
> f61692642a2a2b83a52dd7e64619ba3bb29998af xen/pirq: do EOI properly for
> pirq events
> 0672fb44a111dfb6386022071725c5b15c9de584 xen/events: change to using
fasteoi
> 2789ef00cbe2cdb38deb30ee4085b88befadb1b0 xen: make pirq interrupts use
> fasteoi
> d0936845a856816af2af48ddf019366be68e96ba xen/evtchn: rename
> enable/disable_dynirq -> unmask/mask_irq
> c6a16a778f86699b339585ba5b9197035d77c40f xen/evtchn: rename
> retrigger_dynirq -> irq
> f4526f9a78ffb3d3fc9f81636c5b0357fc1beccd xen/evtchn: make pirq
> enable/disable unmask/mask
> 43d8a5030a502074f3c4aafed4d6095ebd76067c xen/evtchn: pirq_eoi does unmask
> cb23e8d58ca35b6f9e10e1ea5682bd61f2442ebd xen/evtchn: correction, pirq
> hypercall does not unmask
> 2390c371ecd32d9f06e22871636185382bf70ab7 xen/events: use
> PHYSDEVOP_pirq_eoi_gmfn to get pirq need-EOI info
> 158d6550716687486000a828c601706b55322ad0 xen/pirq: use eoi as enable
> d2ea486300ca6e207ba178a425fbd023b8621bb1 xen/pirq: use fasteoi for MSI too
> f0d4a0552f03b52027fb2c7958a1cbbe210cf418 xen/apic: fix pirq_eoi_gmfn resume
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Sep-27 18:02 UTC
Re: [Xen-devel] Re: VM hung after running sometime
On 09/25/2010 03:40 AM, wei song wrote:> > Hi Jeremy, > > Do you think| this issue is cased of without CONFIG_X86_F00F_BUG|.F00F_BUG? Why would that be related? F00F itself should be irrelevant in any Xen situation, since the bug only affects P5 processors which I assume you''re not using (and I don''t think are supported under Xen). J> > thanks, > James > > 在 2010年9月25日 下午5:33，MaoXiaoyun <tinnycloud@hotmail.com > <mailto:tinnycloud@hotmail.com>>写 道： > > Hi Jeremy: > > The test of irqbalance disabled is running. Currently one server > was crashed on NIC. > Trace.jpg in attachments is the screenshot from serial port, and > trace.txt is from /varl/log/message. > Do you think it has connection with irqbalance disabled, or some > other possibilities? > > In addition, I find in /proc/interrupts, all interrupts are > happend on cpu0(please refer to interrputs.txt > attached). Could it be a possible cause of server crash, and is > there a way I can configure manually to > distribute those interrupts evenly? > > Meanwhile, I wil start the new test with kernel patched soon. Thanks. > > > Date: Thu, 23 Sep 2010 16:20:09 -0700 > > > From: jeremy@goop.org <mailto:jeremy@goop.org> > > To: tinnycloud@hotmail.com <mailto:tinnycloud@hotmail.com> > > CC: xen-devel@lists.xensource.com > <mailto:xen-devel@lists.xensource.com>; keir.fraser@eu.citrix.com > <mailto:keir.fraser@eu.citrix.com> > > > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > > > On 09/22/2010 05:55 PM, MaoXiaoyun wrote: > > > The interrputs file is attached. The server has 24 HVM domains > > > runnning about 40 hours. > > > > > > Well, we may upgrade to the new kernel in the further, but > currently > > > we prefer the fix has least impact on our present server. > > > So it is really nice of you if you could offer the sets of > patches, > > > also, it would be our fisrt choice. > > > > Try cherry-picking: > > 8401e9b96f80f9c0128e7c8fc5a01abfabbfa021 xen: use percpu > interrupts for > > IPIs and VIRQs > > 66fd3052 fec7e7c21a9d88ba1a03bc062f5fb53d xen: handle events as > > > edge-triggered > > 29a2e2a7bd19233c62461b104c69233f15ce99ec xen/apic: use > handle_edge_irq > > for pirq events > > f61692642a2a2b83a52dd7e64619ba3bb29998af xen/pirq: do EOI > properly for > > pirq events > > 0672fb44a111dfb6386022071725c5b15c9de584 xen/events: change to > using fasteoi > > 2789ef00cbe2cdb38deb30ee4085b88befadb1b0 xen: make pirq > interrupts use > > fasteoi > > d0936845a856816af2af48ddf019366be68e96ba xen/evtchn: rename > > enable/disable_dynirq -> unmask/mask_irq > > c6a16a778f86699b339585ba5b9197035d77c40f xen/evtchn: rename > > retrigger_dynirq -> irq > > f4526f9a78ffb3d3fc9f81636c5b0357fc1beccd xen/evtchn: make pirq > > enable/disable unmask/mask > > 43d8a5030a502074f3c4aafed4d6095ebd76067c xen/evtchn: pirq_eoi > does unmask > > cb23e8d58ca35b6f9e10e1ea5682bd61f2442ebd xen/evtchn: correction, > pirq > > hypercall does not unmask > > 2390c371ecd32d9f06e22871636185382bf70ab7 xen/events: use > > PHYSDEVOP_pirq_eoi_gmfn to get pirq need-EOI info > > 158d6550716687486000a828c601706b55322ad0 xen/pirq: use eoi as enable > > d2ea486300ca6e207ba178a425fbd023b8621bb1 xen/pirq: use fasteoi > for MSI too > > f0d4a0552f03b52027fb2c7958a1cbbe210cf418 xen/apic: fix > pirq_eoi_gmfn resume > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com <mailto:Xen-devel@lists.xensource.com> > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Jeremy:
I just found a patch you presented to Linus.
http://lists.xensource.com/archives/html/xen-devel/2010-08/msg01510.html
If I patch this one into 2.6.31. Suppose it will also fix the
handle_level_irq problem, right?
thanks.
> Date: Thu, 23 Sep 2010 16:20:09 -0700
> From: jeremy@goop.org
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com
> Subject: Re: [Xen-devel] Re: VM hung after running sometime
>
> On 09/22/2010 05:55 PM, MaoXiaoyun wrote:
> > The interrputs file is attached. The server has 24 HVM domains
> > runnning about 40 hours.
> >
> > Well, we may upgrade to the new kernel in the further, but currently
> > we prefer the fix has least impact on our present server.
> > So it is really nice of you if you could offer the sets of patches,
> > also, it would be our fisrt choice.
>
> Try cherry-picking:
> 8401e9b96f80f9c0128e7c8fc5a01abfabbfa021 xen: use percpu interrupts for
> IPIs and VIRQs
> 66fd3052fec7e7c21a9d88ba1a03bc062f5fb53d xen: handle events as
> edge-triggered
> 29a2e2a7bd19233c62461b104c69233f15ce99ec xen/apic: use handle_edge_irq
> for pirq events
> f61692642a2a2b83a52dd7e64619ba3bb29998af xen/pirq: do EOI properly for
> pirq events
> 0672fb44a111dfb6386022071725c5b15c9de584 xen/events: change to using
fasteoi
> 2789ef00cbe2cdb38deb30ee4085b88befadb1b0 xen: make pirq interrupts use
> fasteoi
> d0936845a856816af2af48ddf019366be68e96ba xen/evtchn: rename
> enable/disable_dynirq -> unmask/mask_irq
> c6a16a778f86699b339585ba5b9197035d77c40f xen/evtchn: rename
> retrigger_dynirq -> irq
> f4526f9a78ffb3d3fc9f81636c5b0357fc1beccd xen/evtchn: make pirq
> enable/disable unmask/mask
> 43d8a5030a502074f3c4aafed4d6095ebd76067c xen/evtchn: pirq_eoi does unmask
> cb23e8d58ca35b6f9e10e1ea5682bd61f2442ebd xen/evtchn: correction, pirq
> hypercall does not unmask
> 2390c371ecd32d9f06e22871636185382bf70ab7 xen/events: use
> PHYSDEVOP_pirq_eoi_gmfn to get pirq need-EOI info
> 158d6550716687486000a828c601706b55322ad0 xen/pirq: use eoi as enable
> d2ea486300ca6e207ba178a425fbd023b8621bb1 xen/pirq: use fasteoi for MSI too
> f0d4a0552f03b52027fb2c7958a1cbbe210cf418 xen/apic: fix pirq_eoi_gmfn resume
>
> > Later I will kick off the irqbalance disabled test in different
> > servers, will keep you noticed.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Hi Jeremy:
Is it safe to set irq affinity(including NIC, domain event, etc)
manually?
Will it cause irq lost?
Thanks.
From: tinnycloud@hotmail.com
To: jeremy@goop.org
CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com
Subject: RE: [Xen-devel] Re: VM hung after running sometime
Date: Tue, 28 Sep 2010 13:43:12 +0800
Hi Jeremy:
I just found a patch you presented to Linus.
http://lists.xensource.com/archives/html/xen-devel/2010-08/msg01510.html
If I patch this one into 2.6.31. Suppose it will also fix the
handle_level_irq problem, right?
thanks.
> Date: Thu, 23 Sep 2010 16:20:09 -0700
> From: jeremy@goop.org
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com
> Subject: Re: [Xen-devel] Re: VM hung after running sometime
>
> On 09/22/2010 05:55 PM, MaoXiaoyun wrote:
> > The interrputs file is attached. The server has 24 HVM domains
> > runnning about 40 hours.
> >
> > Well, we may upgrade to the new kernel in the further, but currently
> > we prefer the fix has least impact on our present server.
> > So it is really nice of you if you could offer the sets of patches,
> > also, it would be our fisrt choice.
>
> Try cherry-picking:
> 8401e9b96f80f9c0128e7c8fc5a01abfabbfa021 xen: use percpu interrupts for
> IPIs and VIRQs
> 66fd3052fec7e7c21a9d88ba1a03bc062f5fb53d xen: handle events as
> edge-triggered
> 29a2e2a7bd19233c62461b104c69233f15ce99ec xen/apic: use handle_edge_irq
> for pirq events
> f61692642a2a2b83a52dd7e64619ba3bb29998af xen/pirq: do EOI properly for
> pirq events
> 0672fb44a111dfb6386022071725c5b15c9de584 xen/events: change to using
fasteoi
> 2789ef00cbe2cdb38deb30ee4085b88befadb1b0 xen: make pirq interrupts use
> fasteoi
> d0936845a856816af2af48ddf019366be68e96ba xen/evtchn: rename
> enable/disable_dynirq -> unmask/mask_irq
> c6a16a778f86699b339585ba5b9197035d77c40f xen/evtchn: rename
> retrigger_dynirq -> irq
> f4526f9a78ffb3d3fc9f81636c5b0357fc1beccd xen/evtchn: make pirq
> enable/disable unmask/mask
> 43d8a5030a502074f3c4aafed4d6095ebd76067c xen/evtchn: pirq_eoi does unmask
> cb23e8d58ca35b6f9e10e1ea5682bd61f2442ebd xen/evtchn: correction, pirq
> hypercall does not unmask
> 2390c371ecd32d9f06e22871636185382bf70ab7 xen/events: use
> PHYSDEVOP_pirq_eoi_gmfn to get pirq need-EOI info
> 158d6550716687486000a828c601706b55322ad0 xen/pirq: use eoi as enable
> d2ea486300ca6e207ba178a425fbd023b8621bb1 xen/pirq: use fasteoi for MSI too
> f0d4a0552f03b52027fb2c7958a1cbbe210cf418 xen/apic: fix pirq_eoi_gmfn resume
>
> > Later I will kick off the irqbalance disabled test in different
> > servers, will keep you noticed.
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Sep-28 17:07 UTC
Re: [Xen-devel] Re: VM hung after running sometime
On 09/28/2010 04:23 AM, MaoXiaoyun wrote:> > Is it safe to set irq affinity(including NIC, domain event, etc) manually? > Will it cause irq lost?There''s only a very small chance of a lost interrupt, especially if the device is mostly idle at the time. The event can only be lost if: 1. it is handling a device interrupt 2. you migrate it to another cpu 3. another interrupt comes in before the first one has finished processing J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Well， I just go throught the source of irqbalance, it shows that it balances the irq through updating /proc/irq/$irq/smp_affinity. That, in my understanding, set irq affinity is almost equal to irq migration. I later find the NIC interrupt is not modified in dom0, so it is safe to set its affinity, but interrupt of xen event use handle_level_irq, set its affinity will subject to irq lost. Am I right?> Date: Tue, 28 Sep 2010 10:07:28 -0700 > From: jeremy@goop.org > To: tinnycloud@hotmail.com > CC: xen-devel@lists.xensource.com; keir.fraser@eu.citrix.com > Subject: Re: [Xen-devel] Re: VM hung after running sometime > > On 09/28/2010 04:23 AM, MaoXiaoyun wrote: > > > > Is it safe to set irq affinity(including NIC, domain event, etc) manually? > > Will it cause irq lost? > > There''s only a very small chance of a lost interrupt, especially if the > device is mostly idle at the time. The event can only be lost if: > > 1. it is handling a device interrupt > 2. you migrate it to another cpu > 3. another interrupt comes in before the first one has finished > processing > > J >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Sep-29 16:12 UTC
Re: [Xen-devel] Re: VM hung after running sometime
On 09/28/2010 11:01 PM, MaoXiaoyun wrote:> Well， I just go throught the source of irqbalance, it shows that it > balances the irq through > updating /proc/irq/$irq/smp_affinity. That, in my understanding, set > irq affinity is almost > equal to irq migration.Correct.> > I later find the NIC interrupt is not modified in dom0, so it is safe > to set its affinity, > but interrupt of xen event use handle_level_irq, set its affinity will > subject to irq > lost. > > Am I right?I can''t parse that sentence so I''m not sure. But if you have successfully migrated the interrupt/evtchn to a different vcpu, then there''s no subsequent risk of lost events. The event loss can only happen in the circumstance I mentioned earlier: when you migrate while an event is being handled, and a second event occurs on the new cpu before the first one has finished. So you need both bad timing and a fairly high event delivery rate to trigger the problem. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2010-Oct-15 12:43 UTC
[Xen-devel] Domain 0 stop response on frequently reboot VMS
Hi Keir:
First, I''d like to express my appreciation for the help your
offered before.
Well, recently we confront a rather nasty domain 0 no response problem.
We still have 12 HVMs almost continuously and concurrently reboot test
on a physical server.
A few hours later, the server looks like dead. We only can ping to the
server and get right response,
the Xen works fine since we can get debug info from serial port. Attached is the
full debug output.
After decode the domain 0 CPU stack, I find the CPU still works for domain 0
since the stack changed
info changed every time I dumped.
Could help to take a look at the attentchment to see whether there are
some hints for debugging this
problem. Thanks in advance.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2010-Oct-15 12:57 UTC
Re: [Xen-devel] Domain 0 stop response on frequently reboot VMS
You''ll probably want to see if you can get SysRq output from dom0 via serial line. It''s likely you can if it is alive enough to respond to ping. This might tell you things like what all processes are getting blocked on, and thus indicate what is stopping dom0 from making progress. -- Keir On 15/10/2010 13:43, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote:> > Hi Keir: > > First, I''d like to express my appreciation for the help your offered > before. > Well, recently we confront a rather nasty domain 0 no response > problem. > > We still have 12 HVMs almost continuously and concurrently reboot > test on a physical server. > A few hours later, the server looks like dead. We only can ping to > the server and get right response, > the Xen works fine since we can get debug info from serial port. Attached is > the full debug output. > After decode the domain 0 CPU stack, I find the CPU still works for domain 0 > since the stack changed > info changed every time I dumped. > > Could help to take a look at the attentchment to see whether there are > some hints for debugging this > problem. Thanks in advance. > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2010-Oct-16 05:39 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
Well, Thanks Keir.
Fortunately we caught the bug, it turned out to be a tapdisk problem.
A brief explaination for other guys might confront this issue.
Clear BLKTAP_DEFERRED on line 19 will lead to the concurrent access of
tap->deferred_queue between line 24 and 37, which will finally cause bad
pointer of tap->deferred_queue, and infinte loop in while clause in line 22.
Lock line 24 will be a simple fix.
/linux-2.6-pvops.git/drivers/xen/blktap/wait_queue.c
9 void
10 blktap_run_deferred(void)
11 {
12 LIST_HEAD(queue);
13 struct blktap *tap;
14 unsigned long flags;
15
16 spin_lock_irqsave(&deferred_work_lock, flags);
17 list_splice_init(&deferred_work_queue, &queue);
18 list_for_each_entry(tap, &queue, deferred_queue)
19 clear_bit(BLKTAP_DEFERRED, &tap->dev_inuse);
20 spin_unlock_irqrestore(&deferred_work_lock, flags);
21
22 while (!list_empty(&queue)) {
23 tap = list_entry(queue.next, struct blktap, deferred_queue);
24 list_del_init(&tap->deferred_queue);
25 blktap_device_restart(tap);
26 }
27 }
28
29 void
30 blktap_defer(struct blktap *tap)
31 {
32 unsigned long flags;
33
34 spin_lock_irqsave(&deferred_work_lock, flags);
35 if (!test_bit(BLKTAP_DEFERRED, &tap->dev_inuse)) {
36 set_bit(BLKTAP_DEFERRED, &tap->dev_inuse);
37 list_add_tail(&tap->deferred_queue,
&deferred_work_queue);
38 }
39 spin_unlock_irqrestore(&deferred_work_lock, flags);
40 }
> Date: Fri, 15 Oct 2010 13:57:09 +0100
> Subject: Re: [Xen-devel] Domain 0 stop response on frequently reboot VMS
> From: keir@xen.org
> To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
>
> You''ll probably want to see if you can get SysRq output from dom0
via serial
> line. It''s likely you can if it is alive enough to respond to
ping. This
> might tell you things like what all processes are getting blocked on, and
> thus indicate what is stopping dom0 from making progress.
>
> -- Keir
>
> On 15/10/2010 13:43, "MaoXiaoyun" <tinnycloud@hotmail.com>
wrote:
>
> >
> > Hi Keir:
> >
> > First, I''d like to express my appreciation for the help your
offered
> > before.
> > Well, recently we confront a rather nasty domain 0 no response
> > problem.
> >
> > We still have 12 HVMs almost continuously and concurrently reboot
> > test on a physical server.
> > A few hours later, the server looks like dead. We only can ping to
> > the server and get right response,
> > the Xen works fine since we can get debug info from serial port.
Attached is
> > the full debug output.
> > After decode the domain 0 CPU stack, I find the CPU still works for
domain 0
> > since the stack changed
> > info changed every time I dumped.
> >
> > Could help to take a look at the attentchment to see whether there are
> > some hints for debugging this
> > problem. Thanks in advance.
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2010-Oct-16 07:16 UTC
Re: [Xen-devel] Domain 0 stop response on frequently reboot VMS
Send a patch to the list, Cc Jeremy Fitzhardinge and also a blktap maintainer, which you should be able to derive from changeset histories and signed-off-by lines. Flag it clearly in the subject line as a proposed bugfix for pv_ops. -- Keir On 16/10/2010 06:39, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote:> Well, Thanks Keir. > Fortunately we caught the bug, it turned out to be a tapdisk problem. > A brief explaination for other guys might confront this issue. > > Clear BLKTAP_DEFERRED on line 19 will lead to the concurrent access of > tap->deferred_queue between line 24 and 37, which will finally cause bad > pointer of tap->deferred_queue, and infinte loop in while clause in line 22. > Lock line 24 will be a simple fix. > > /linux-2.6-pvops.git/drivers/xen/blktap/wait_queue.c > 9 void > 10 blktap_run_deferred(void) > 11 { > 12 LIST_HEAD(queue); > 13 struct blktap *tap; > 14 unsigned long flags; > 15 > 16 spin_lock_irqsave(&deferred_work_lock, flags); > 17 list_splice_init(&deferred_work_queue, &queue); > 18 list_for_each_entry(tap, &queue, deferred_queue) > 19 clear_bit(BLKTAP_DEFERRED, &tap->dev_inuse); > 20 spin_unlock_irqrestore(&deferred_work_lock, flags); > 21 > 22 while (!list_empty(&queue)) { > 23 tap = list_entry(queue.next, struct blktap, deferred_queue); > 24 &nb sp; list_del_init(&tap->deferred_queue); > 25 blktap_device_restart(tap); > 26 } > 27 } > 28 > 29 void > 30 blktap_defer(struct blktap *tap) > 31 { > 32 unsigned long flags; > 33 > 34 spin_lock_irqsave(&deferred_work_lock, flags); > 35 if (!test_bit(BLKTAP_DEFERRED, &tap->dev_inuse)) { > 36 set_bit(BLKTAP_DEFERRED, &tap->dev_inuse); > 37 list_add_tail(&tap->deferred_queue, &deferred_work_queue); > 38 } > 39 spin_unlock_irqrestore(&deferred_work_lock,! f lags); > 40 } > > >> Date: Fri, 15 Oct 2010 13:57:09 +0100 >> Subject: Re: [Xen-devel] Domain 0 stop response on frequently reboot VMS >> From: keir@xen.org >> To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com >> >> You''ll probably want to see if you can get SysRq output from dom0 via serial >> line. It''s likely you can if it is alive enough to respond to ping. This >> might tell you things like what all processes are getting blocked on, and >> thus indicate what is stopping dom0 from making progress. >> >> -- Keir >> >> On 15/10/2010 13:43, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: >> >>> >>> Hi Keir: >>> >>> First, I''d like to express my appreciation for the help your offered >>> before. >>> Well, recently we confront a rather nasty domain 0 no response >>> problem. >>> >>> We still have 12 HVMs almost continuously and con currently reboot >>> test on a physical server. >>> A few hours later, the server looks like dead. We only can ping to >>> the server and get right response, >>> the Xen works fine since we can get debug info from serial port. Attached is >>> the full debug output. >>> After decode the domain 0 CPU stack, I find the CPU still works for domain 0 >>> since the stack changed >>> info changed every time I dumped. >>> >>> Could help to take a look at the attentchment to see whether there are >>> some hints for debugging this >>> problem. Thanks in advance. >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >> >> > !_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Oct-18 21:17 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
I''d strongly suggest to try upgrading your kernel, or at least the blktap component. The condition below is new to me, but that wait_queue file and some related code was known to be buggy and has long since been removed. If you choose to only upgrade blktap from tip, let me know what kernel version you''re dealing with, you might need to backport some of the device queue macros to match your version''s needs. Daniel On Sat, 2010-10-16 at 01:39 -0400, MaoXiaoyun wrote:> Well, Thanks Keir. > Fortunately we caught the bug, it turned out to be a tapdisk problem. > A brief explaination for other guys might confront this issue. > > Clear BLKTAP_DEFERRED on line 19 will lead to the concurrent access > of > tap->deferred_queue between line 24 and 37, which will finally cause > bad > pointer of tap->deferred_queue, and infinte loop in while clause in > line 22. > Lock line 24 will be a simple fix. > > /linux-2.6-pvops.git/drivers/xen/blktap/wait_queue.c > 9 void > 10 blktap_run_deferred(void) > 11 { > 12 LIST_HEAD(queue); > 13 struct blktap *tap; > 14 unsigned long flags; > 15 > 16 spin_lock_irqsave(&deferred_work_lock, flags); > 17 list_splice_init(&deferred_work_queue, &queue); > 18 list_for_each_entry(tap, &queue, deferred_queue) > 19 clear_bit(BLKTAP_DEFERRED, &tap->dev_inuse); > 20 spin_unlock_irqrestore(&deferred_work_lock, flags); > 21 > 22 while (!list_empty(&queue)) { > 23 tap = list_entry(queue.next, struct blktap, > deferred_queue); > 24 &nb sp; list_del_init(&tap->deferred_queue); > 25 blktap_device_restart(tap); > 26 } > 27 } > 28 > 29 void > 30 blktap_defer(struct blktap *tap) > 31 { > 32 unsigned long flags; > 33 > 34 spin_lock_irqsave(&deferred_work_lock, flags); > 35 if (!test_bit(BLKTAP_DEFERRED, &tap->dev_inuse)) { > 36 set_bit(BLKTAP_DEFERRED, &tap->dev_inuse); > 37 list_add_tail(&tap->deferred_queue, &deferred_work_queue); > 38 } > 39 spin_unlock_irqrestore(&deferred_work_lock, f lags); > 40 } > > > > Date: Fri, 15 Oct 2010 13:57:09 +0100 > > Subject: Re: [Xen-devel] Domain 0 stop response on frequently reboot > VMS > > From: keir@xen.org > > To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com > > > > You''ll probably want to see if you can get SysRq output from dom0 > via serial > > line. It''s likely you can if it is alive enough to respond to ping. > This > > might tell you things like what all processes are getting blocked > on, and > > thus indicate what is stopping dom0 from making progress. > > > > -- Keir > > > > On 15/10/2010 13:43, "MaoXiaoyun" <tinnycloud@hotmail.com> wrote: > > > > > > > > Hi Keir: > > > > > > First, I''d like to express my appreciation for the help your > offered > > > before. > > > Well, recently we confront a rather nasty domain 0 no response > > > problem. > > > > > > We still have 12 HVMs almost continuously and con currently reboot > > > test on a physical server. > > > A few hours later, the server looks like dead. We only can ping to > > > the server and get right response, > > > the Xen works fine since we can get debug info from serial port. > Attached is > > > the full debug output. > > > After decode the domain 0 CPU stack, I find the CPU still works > for domain 0 > > > since the stack changed > > > info changed every time I dumped. > > > > > > Could help to take a look at the attentchment to see whether there > are > > > some hints for debugging this > > > problem. Thanks in advance. > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2010-Oct-24 05:48 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
Hi Daniel:
Sorry for tht late response, and really thanks for your kindly suggestion.
Well, I believe we will upgrade to the lastest kernel in the coming future,
but currently
we perfer to maintain for stable reason.
Our kernel version is 2.6.31. Now I am going through the change set of
blktap to get
more detail info.
thanks.
> Subject: RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
> From: daniel.stodden@citrix.com
> To: tinnycloud@hotmail.com; jeremy@goop.org
> CC: keir@xen.org; xen-devel@lists.xensource.com
> Date: Mon, 18 Oct 2010 14:17:50 -0700
>
>
> I''d strongly suggest to try upgrading your kernel, or at least the
> blktap component. The condition below is new to me, but that wait_queue
> file and some related code was known to be buggy and has long since been
> removed.
>
> If you choose to only upgrade blktap from tip, let me know what kernel
> version you''re dealing with, you might need to backport some of
the
> device queue macros to match your version''s needs.
>
> Daniel
>
>
> On Sat, 2010-10-16 at 01:39 -0400, MaoXiaoyun wrote:
> > Well, Thanks Keir.
> > Fortunately we caught the bug, it turned out to be a tapdisk problem.
> > A brief explaination for other guys might confront this issue.
> >
> > Clear BLKTAP_DEFERRED on line 19 will lead to the concurrent access
> > of
> > tap->deferred_queue between line 24 and 37, which will finally
cause
> > bad
> > pointer of tap->deferred_queue, and infinte loop in while clause in
> > line 22.
> > Lock line 24 will be a simple fix.
> >
> > /linux-2.6-pvops.git/drivers/xen/blktap/wait_queue.c
> > 9 void
> > 10 blktap_run_deferred(void)
> > 11 {
> > 12 LIST_HEAD(queue);
> > 13 struct blktap *tap;
> > 14 unsigned long flags;
> > 15
> > 16 spin_lock_irqsave(&deferred_work_lock, flags);
> > 17 list_splice_init(&deferred_work_queue, &queue);
> > 18 list_for_each_entry(tap, &queue, deferred_queue)
> > 19 clear_bit(BLKTAP_DEFERRED, &tap->dev_inuse);
> > 20 spin_unlock_irqrestore(&deferred_work_lock, flags);
> > 21
> > 22 while (!list_empty(&queue)) {
> > 23 tap = list_entry(queue.next, struct blktap,
> > deferred_queue);
> > 24 &nb sp; list_del_init(&tap->deferred_queue);
> > 25 blktap_device_restart(tap);
> > 26 }
> > 27 }
> > 28
> > 29 void
> > 30 blktap_defer(struct blktap *tap)
> > 31 {
> > 32 unsigned long flags;
> > 33
> > 34 spin_lock_irqsave(&deferred_work_lock, flags);
> > 35 if (!test_bit(BLKTAP_DEFERRED, &tap->dev_inuse)) {
> > 36 set_bit(BLKTAP_DEFERRED, &tap->dev_inuse);
> > 37 list_add_tail(&tap->deferred_queue,
&deferred_work_queue);
> > 38 }
> > 39 spin_unlock_irqrestore(&deferred_work_lock, f lags);
> > 40 }
> >
> >
> > > Date: Fri, 15 Oct 2010 13:57:09 +0100
> > > Subject: Re: [Xen-devel] Domain 0 stop response on frequently
reboot
> > VMS
> > > From: keir@xen.org
> > > To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> > >
> > > You''ll probably want to see if you can get SysRq output
from dom0
> > via serial
> > > line. It''s likely you can if it is alive enough to
respond to ping.
> > This
> > > might tell you things like what all processes are getting blocked
> > on, and
> > > thus indicate what is stopping dom0 from making progress.
> > >
> > > -- Keir
> > >
> > > On 15/10/2010 13:43, "MaoXiaoyun"
<tinnycloud@hotmail.com> wrote:
> > >
> > > >
> > > > Hi Keir:
> > > >
> > > > First, I''d like to express my appreciation for the
help your
> > offered
> > > > before.
> > > > Well, recently we confront a rather nasty domain 0 no
response
> > > > problem.
> > > >
> > > > We still have 12 HVMs almost continuously and con currently
reboot
> > > > test on a physical server.
> > > > A few hours later, the server looks like dead. We only can
ping to
> > > > the server and get right response,
> > > > the Xen works fine since we can get debug info from serial
port.
> > Attached is
> > > > the full debug output.
> > > > After decode the domain 0 CPU stack, I find the CPU still
works
> > for domain 0
> > > > since the stack changed
> > > > info changed every time I dumped.
> > > >
> > > > Could help to take a look at the attentchment to see whether
there
> > are
> > > > some hints for debugging this
> > > > problem. Thanks in advance.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xensource.com
> > > > http://lists.xensource.com/xen-devel
> > >
> > >
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Oct-24 05:56 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
On Sun, 2010-10-24 at 01:48 -0400, MaoXiaoyun wrote:> Hi Daniel: > > Sorry for tht late response, and really thanks for your kindly > suggestion. > Well, I believe we will upgrade to the lastest kernel in the > coming future, but currently > we perfer to maintain for stable reason. > > Our kernel version is 2.6.31. Now I am going through the change > set of blktap to get > more detail info.NP. Let me know if you have questions. Daniel> thanks. > > > Subject: RE: [Xen-devel] Domain 0 stop response on frequently reboot > VMS > > From: daniel.stodden@citrix.com > > To: tinnycloud@hotmail.com; jeremy@goop.org > > CC: keir@xen.org; xen-devel@lists.xensource.com > > Date: Mon, 18 Oct 2010 14:17:50 -0700 > > > > > > I''d strongly suggest to try upgrading your kernel, or at least the > > blktap component. The condition below is new to me, but that > wait_queue > > file and some related code was known to be buggy and has long since > been > > removed. > > > > If you choose to only upgrade blktap from tip, let me know what > kernel > > version you''re dealing with, you might need to backport some of the > > device queue macros to match your version''s needs. > > > > Daniel > > > > > > On Sat, 2010-10-16 at 01:39 -0400, MaoXiaoyun wrote: > > > Well, Thanks Keir. > > > Fortunately we caught the bug, it turned out to be a tapdisk > problem. > > > A brief explaination for other guys might confront this issue. > > > > > > Clear BLKTAP_DEFERRED on line 19 will lead to the concurrent > access > > > of > > > tap->deferred_queue between line 24 and 37, which will finally > cause > > > bad > > > pointer of tap->deferred_queue, and infinte loop in while clause > in > > > line 22. > > > Lock line 24 will be a simple fix. > > > > > > /linux-2.6-pvops.git/drivers/xen/blktap/wait_queue.c > > > 9 void > > > 10 blktap_run_deferred(void) > > > 11 { > > > 12 LIST_HEAD(queue); > > > 13 struct blktap *tap; > > > 14 unsigned long flags; > > > 15 > > > 16 spin_lock_irqsave(&deferred_work_lock, flags); > > > 17 list_splice_init(&deferred_work_queue, &queue); > > > 18 list_for_each_entry(tap, &queue, deferred_queue) > > > 19 clear_bit(BLKTAP_DEFERRED, &tap->dev_inuse); > > > 20 spin_unlock_irqrestore(&deferred_work_lock, flags); > > > 21 > > > 22 while (!list_empty(&queue)) { > > > 23 tap = list_entry(queue.next, struct blktap, > > > deferred_queue); > > > 24 &nb sp; list_del_init(&tap->deferred_queue); > > > 25 blktap_device_restart(tap); > > > 26 } > > > 27 } > > > 28 > > > 29 void > > > 30 blktap_defer(struct blktap *tap) > > > 31 { > > > 32 unsigned long flags; > > > 33 > > > 34 spin_lock_irqsave(&deferred_work_lock, flags); > > > 35 if (!test_bit(BLKTAP_DEFERRED, &tap->dev_inuse)) { > > > 36 set_bit(BLKTAP_DEFERRED, &tap->dev_inuse); > > > 37 list_add_tail(&tap->deferred_queue, &deferred_work_queue); > > > 38 } > > > 39 spin_unlock_irqrestore(&deferred_work_lock, f lags); > > > 40 } > > > > > > > > > > Date: Fri, 15 Oct 2010 13:57:09 +0100 > > > > Subject: Re: [Xen-devel] Domain 0 stop response on frequently > reboot > > > VMS > > > > From: keir@xen.org > > > > To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com > > > > > > > > You''ll probably want to see if you can get SysRq output from > dom0 > > > via serial > > > > line. It''s likely you can if it is alive enough to respond to > ping. > > > This > > > > might tell you things like what all processes are getting > blocked > > > on, and > > > > thus indicate what is stopping dom0 from making progress. > > > > > > > > -- Keir > > > > > > > > On 15/10/2010 13:43, "MaoXiaoyun" <tinnycloud@hotmail.com> > wrote: > > > > > > > > > > > > > > Hi Keir: > > > > > > > > > > First, I''d like to express my appreciation for the help your > > > offered > > > > > before. > > > > > Well, recently we confront a rather nasty domain 0 no response > > > > > problem. > > > > > > > > > > We still have 12 HVMs almost continuously and con currently > reboot > > > > > test on a physical server. > > > > > A few hours later, the server looks like dead. We only can > ping to > > > > > the server and get right response, > > > > > the Xen works fine since we can get debug info from serial > port. > > > Attached is > > > > > the full debug output. > > > > > After decode the domain 0 CPU stack, I find the CPU still > works > > > for domain 0 > > > > > since the stack changed > > > > > info changed every time I dumped. > > > > > > > > > > Could help to take a look at the attentchment to see whether > there > > > are > > > > > some hints for debugging this > > > > > problem. Thanks in advance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Xen-devel mailing list > > > > > Xen-devel@lists.xensource.com > > > > > http://lists.xensource.com/xen-devel > > > > > > > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2010-Oct-26 08:16 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
Hi Dainnel:
Well, where can I start if I want to maintain the current kernel(2.6.31),
and only update the blktap2?
As I go throught the git branch of xen/dom0/backend/blktap2, I found
wait_queue.c is removed.
It looks like blktap2 has changed a lot, right?
So I am courious the difference between the new and the old one.
Could you share some brief explainations, that would be very helpful.
Thanks in advance.
> Subject: RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
> From: daniel.stodden@citrix.com
> To: tinnycloud@hotmail.com
> CC: keir@xen.org; xen-devel@lists.xensource.com
> Date: Sat, 23 Oct 2010 22:56:51 -0700
>
> On Sun, 2010-10-24 at 01:48 -0400, MaoXiaoyun wrote:
> > Hi Daniel:
> >
> > Sorry for tht late response, and really thanks for your kindly
> > suggestion.
> > Well, I believe we will upgrade to the lastest kernel in the
> > coming future, but currently
> > we perfer to maintain for stable reason.
> >
> > Our kernel version is 2.6.31. Now I am going through the change
> > set of blktap to get
> > more detail info.
>
> NP. Let me know if you have questions.
>
> Daniel
>
> > thanks.
> >
> > > Subject: RE: [Xen-devel] Domain 0 stop response on frequently
reboot
> > VMS
> > > From: daniel.stodden@citrix.com
> > > To: tinnycloud@hotmail.com; jeremy@goop.org
> > > CC: keir@xen.org; xen-devel@lists.xensource.com
> > > Date: Mon, 18 Oct 2010 14:17:50 -0700
> > >
> > >
> > > I''d strongly suggest to try upgrading your kernel, or at
least the
> > > blktap component. The condition below is new to me, but that
> > wait_queue
> > > file and some related code was known to be buggy and has long
since
> > been
> > > removed.
> > >
> > > If you choose to only upgrade blktap from tip, let me know what
> > kernel
> > > version you''re dealing with, you might need to backport
some of the
> > > device queue macros to match your version''s needs.
> > >
> > > Daniel
> > >
> > >
> > > On Sat, 2010-10-16 at 01:39 -0400, MaoXiaoyun wrote:
> > > > Well, Thanks Keir.
> > > > Fortunately we caught the bug, it turned out to be a tapdisk
> > problem.
> > > > A brief explaination for other guys might confront this
issue.
> > > >
> > > > Clear BLKTAP_DEFERRED on line 19 will lead to the concurrent
> > access
> > > > of
> > > > tap->deferred_queue between line 24 and 37, which will
finally
> > cause
> > > > bad
> > > > pointer of tap->deferred_queue, and infinte loop in while
clause
> > in
> > > > line 22.
> > > > Lock line 24 will be a simple fix.
> > > >
> > > > /linux-2.6-pvops.git/drivers/xen/blktap/wait_queue.c
> > > > 9 void
> > > > 10 blktap_run_deferred(void)
> > > > 11 {
> > > > 12 LIST_HEAD(queue);
> > > > 13 struct blktap *tap;
> > > > 14 unsigned long flags;
> > > > 15
> > > > 16 spin_lock_irqsave(&deferred_work_lock, flags);
> > > > 17 list_splice_init(&deferred_work_queue, &queue);
> > > > 18 list_for_each_entry(tap, &queue, deferred_queue)
> > > > 19 clear_bit(BLKTAP_DEFERRED, &tap->dev_inuse);
> > > > 20 spin_unlock_irqrestore(&deferred_work_lock, flags);
> > > > 21
> > > > 22 while (!list_empty(&queue)) {
> > > > 23 tap = list_entry(queue.next, struct blktap,
> > > > deferred_queue);
> > > > 24 &nb sp; list_del_init(&tap->deferred_queue);
> > > > 25 blktap_device_restart(tap);
> > > > 26 }
> > > > 27 }
> > > > 28
> > > > 29 void
> > > > 30 blktap_defer(struct blktap *tap)
> > > > 31 {
> > > > 32 unsigned long flags;
> > > > 33
> > > > 34 spin_lock_irqsave(&deferred_work_lock, flags);
> > > > 35 if (!test_bit(BLKTAP_DEFERRED, &tap->dev_inuse)) {
> > > > 36 set_bit(BLKTAP_DEFERRED, &tap->dev_inuse);
> > > > 37 list_add_tail(&tap->deferred_queue,
&deferred_work_queue);
> > > > 38 }
> > > > 39 spin_unlock_irqrestore(&deferred_work_lock, f lags);
> > > > 40 }
> > > >
> > > >
> > > > > Date: Fri, 15 Oct 2010 13:57:09 +0100
> > > > > Subject: Re: [Xen-devel] Domain 0 stop response on
frequently
> > reboot
> > > > VMS
> > > > > From: keir@xen.org
> > > > > To: tinnycloud@hotmail.com;
xen-devel@lists.xensource.com
> > > > >
> > > > > You''ll probably want to see if you can get
SysRq output from
> > dom0
> > > > via serial
> > > > > line. It''s likely you can if it is alive
enough to respond to
> > ping.
> > > > This
> > > > > might tell you things like what all processes are
getting
> > blocked
> > > > on, and
> > > > > thus indicate what is stopping dom0 from making
progress.
> > > > >
> > > > > -- Keir
> > > > >
> > > > > On 15/10/2010 13:43, "MaoXiaoyun"
<tinnycloud@hotmail.com>
> > wrote:
> > > > >
> > > > > >
> > > > > > Hi Keir:
> > > > > >
> > > > > > First, I''d like to express my
appreciation for the help your
> > > > offered
> > > > > > before.
> > > > > > Well, recently we confront a rather nasty domain 0
no response
> > > > > > problem.
> > > > > >
> > > > > > We still have 12 HVMs almost continuously and con
currently
> > reboot
> > > > > > test on a physical server.
> > > > > > A few hours later, the server looks like dead. We
only can
> > ping to
> > > > > > the server and get right response,
> > > > > > the Xen works fine since we can get debug info
from serial
> > port.
> > > > Attached is
> > > > > > the full debug output.
> > > > > > After decode the domain 0 CPU stack, I find the
CPU still
> > works
> > > > for domain 0
> > > > > > since the stack changed
> > > > > > info changed every time I dumped.
> > > > > >
> > > > > > Could help to take a look at the attentchment to
see whether
> > there
> > > > are
> > > > > > some hints for debugging this
> > > > > > problem. Thanks in advance.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Xen-devel mailing list
> > > > > > Xen-devel@lists.xensource.com
> > > > > > http://lists.xensource.com/xen-devel
> > > > >
> > > > >
> > >
> > >
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Oct-26 09:09 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
On Tue, 2010-10-26 at 04:16 -0400, MaoXiaoyun wrote:> > Well, where can I start if I want to maintain the current > kernel(2.6.31), and only update the blktap2? > As I go throught the git branch of > xen/dom0/backend/blktap2<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=refs/heads/xen/dom0/backend/blktap2>, I found wait_queue.c is removed. > It looks like blktap2 has changed a lot, right? > So I am courious the difference between the new and the old one. > Could you share some brief explainations, that would be very > helpful. > Thanks in advance.For brief explanations but nothing particular I can only refer you to the commit messages. Because there have been plenty of them. :) They''re all still backward compatible as far as the userspace ABI for older tapdisk2s goes. My recommendation would be replacing that whole blktap/ directory, because I can''t support you with more than that. If you choose to do so: You might need some patches fixing request queue accesses, there might be slight differences to 2.6.32, that''s a bit of a moving target sometimes. But it''s not a big deal. Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2010-Oct-26 09:20 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
On Tue, 2010-10-26 at 09:16 +0100, MaoXiaoyun wrote:> Well, where can I start if I want to maintain the current > kernel(2.6.31)I don''t think 2.6.31 is the default in any current Xen tree, is it? xen/master in xen.git still points to a 2.6.31 based tree but that''s rather misleading. That branch hasn''t been updated since July. The currently maintained stable branch is xen/stable-2.6.32.x which is used by both xen-4.0-testing.hg and xen-unstable.hg Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2010-Oct-26 10:54 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
Thanks, Daniel. I think I can handle it myself.> Subject: RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS > From: daniel.stodden@citrix.com > To: tinnycloud@hotmail.com > CC: xen-devel@lists.xensource.com > Date: Tue, 26 Oct 2010 02:09:52 -0700 > > On Tue, 2010-10-26 at 04:16 -0400, MaoXiaoyun wrote: > > > > Well, where can I start if I want to maintain the current > > kernel(2.6.31), and only update the blktap2? > > As I go throught the git branch of > > xen/dom0/backend/blktap2<http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=refs/heads/xen/dom0/backend/blktap2>, I found wait_queue.c is removed. > > It looks like blktap2 has changed a lot, right? > > So I am courious the difference between the new and the old one. > > Could you share some brief explainations, that would be very > > helpful. > > Thanks in advance. > > For brief explanations but nothing particular I can only refer you to > the commit messages. Because there have been plenty of them. :) > > They''re all still backward compatible as far as the userspace ABI for > older tapdisk2s goes. My recommendation would be replacing that whole > blktap/ directory, because I can''t support you with more than that. > > If you choose to do so: You might need some patches fixing request queue > accesses, there might be slight differences to 2.6.32, that''s a bit of a > moving target sometimes. But it''s not a big deal. > > Daniel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2010-Oct-26 10:59 UTC
RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS
Thanks Ian. You are right. The code used in my server is from xen/master, also very old. And I am going to upgrade it.> Subject: RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS > From: Ian.Campbell@citrix.com > To: tinnycloud@hotmail.com > CC: Daniel.Stodden@citrix.com; xen-devel@lists.xensource.com > Date: Tue, 26 Oct 2010 10:20:06 +0100 > > On Tue, 2010-10-26 at 09:16 +0100, MaoXiaoyun wrote: > > Well, where can I start if I want to maintain the current > > kernel(2.6.31) > > I don''t think 2.6.31 is the default in any current Xen tree, is it? > > xen/master in xen.git still points to a 2.6.31 based tree but that''s > rather misleading. That branch hasn''t been updated since July. > > The currently maintained stable branch is xen/stable-2.6.32.x which is > used by both xen-4.0-testing.hg and xen-unstable.hg > > Ian. > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Oct-26 11:54 UTC
Re: [Xen-devel] Domain 0 stop response on frequently reboot VMS, fix xen/master link?
Hello, Jeremy: Maybe now it''s time to kill the xen/master link to 2.6.31 tree to avoid confusion? -- Pasi On Tue, Oct 26, 2010 at 06:59:00PM +0800, MaoXiaoyun wrote:> Thanks Ian. You are right. > The code used in my server is from xen/master, also very old. > And I am going to upgrade it. > > > Subject: RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS > > From: Ian.Campbell@citrix.com > > To: tinnycloud@hotmail.com > > CC: Daniel.Stodden@citrix.com; xen-devel@lists.xensource.com > > Date: Tue, 26 Oct 2010 10:20:06 +0100 > > > > On Tue, 2010-10-26 at 09:16 +0100, MaoXiaoyun wrote: > > > Well, where can I start if I want to maintain the current > > > kernel(2.6.31) > > > > I don''t think 2.6.31 is the default in any current Xen tree, is it? > > > > xen/master in xen.git still points to a 2.6.31 based tree but that''s > > rather misleading. That branch hasn''t been updated since July. > > > > The currently maintained stable branch is xen/stable-2.6.32.x which is > > used by both xen-4.0-testing.hg and xen-unstable.hg > > > > Ian. > > > >> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Oct-26 17:08 UTC
Re: [Xen-devel] Domain 0 stop response on frequently reboot VMS, fix xen/master link?
On 10/26/2010 04:54 AM, Pasi Kärkkäinen wrote:> Hello, > > Jeremy: Maybe now it''s time to kill the xen/master link to 2.6.31 tree > to avoid confusion?Yeah, esp since it is still the default branch for xen.git. OK, done, and default switched to xen/stable-2.6.32.x. Thanks, J> -- Pasi > > On Tue, Oct 26, 2010 at 06:59:00PM +0800, MaoXiaoyun wrote: >> Thanks Ian. You are right. >> The code used in my server is from xen/master, also very old. >> And I am going to upgrade it. >> >> > Subject: RE: [Xen-devel] Domain 0 stop response on frequently reboot VMS >> > From: Ian.Campbell@citrix.com >> > To: tinnycloud@hotmail.com >> > CC: Daniel.Stodden@citrix.com; xen-devel@lists.xensource.com >> > Date: Tue, 26 Oct 2010 10:20:06 +0100 >> > >> > On Tue, 2010-10-26 at 09:16 +0100, MaoXiaoyun wrote: >> > > Well, where can I start if I want to maintain the current >> > > kernel(2.6.31) >> > >> > I don''t think 2.6.31 is the default in any current Xen tree, is it? >> > >> > xen/master in xen.git still points to a 2.6.31 based tree but that''s >> > rather misleading. That branch hasn''t been updated since July. >> > >> > The currently maintained stable branch is xen/stable-2.6.32.x which is >> > used by both xen-4.0-testing.hg and xen-unstable.hg >> > >> > Ian. >> > >> > >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
MaoXiaoyun
2010-Nov-04 03:09 UTC
[Xen-devel] A Patch for modify DomU network transmit rate dynamically
Hi :
I''ve written a patch which supports dynamically update domU netif
transmit rate.
But after I finish it, I have some questiones on the patch itself.
1. It seems I don''t need to update netif->remaining_credit,
since in netback,c: tx_add_credit()
will update automatically on every transmit, so all I need to do is
update netif->credit_bytes
and netif->credit_usec, right? Also, am I do the right way?
2. I notice that netback is also a module, so I think it can be rmmod or
insmod, right? If so
I can apply this patch online(with no affect on the running VM)
Could someone help me to confirm this, many thanks.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
MaoXiaoyun
2010-Nov-04 03:43 UTC
[Xen-devel] RE: A Patch for modify DomU network transmit rate dynamically
Well, I tested, it is necessary to modify netif->remaining_credit.
So question left
1. currently I use atomic64_t, is it necessary
2. Could rmmmod netback and insmod netback when apply this patch?
thx
From: tinnycloud@hotmail.com
To: xen-devel@lists.xensource.com
CC: keir@xen.org; daniel.stodden@citrix.com; jeremy@goop.org
Subject: A Patch for modify DomU network transmit rate dynamically
Date: Thu, 4 Nov 2010 11:09:55 +0800
Hi :
I''ve written a patch which supports dynamically update domU netif
transmit rate.
But after I finish it, I have some questiones on the patch itself.
1. It seems I don''t need to update netif->remaining_credit,
since in netback,c: tx_add_credit()
will update automatically on every transmit, so all I need to do is
update netif->credit_bytes
and netif->credit_usec, right? Also, am I do the right way?
2. I notice that netback is also a module, so I think it can be rmmod or
insmod, right? If so
I can apply this patch online(with no affect on the running VM)
Could someone help me to confirm this, many thanks.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel