Tian, Kevin
2007-Feb-03 04:49 UTC
[Xen-devel] [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug
Irq nunmber for per-vcpu event (virq/ipi) needs kept
accross vcpu plug/unplug, once allocated. We just
reuse this irq number and bind it to a new event port.
Or else, /proc/interrupt exports messed statistics
like:
ot@localhost ~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
256: 423066 0 0 0 Dynamic-irq timer0
257: 3014 0 0 0 Dynamic-irq
resched0
258: 37 0 0 0 Dynamic-irq
callfunc0
259: 713 0 0 0 Dynamic-irq xenbus
260: 343 1323 65 50 Dynamic-irq
xencons
261: 270 62 0 0 Dynamic-irq blkif
262: 0 1053818 1220 855 Dynamic-irq
resched1
263: 0 0 2165 53 Dynamic-irq
callfunc1
264: 0 3218 69 0 Dynamic-irq timer1
265: 0 0 42768 228 Dynamic-irq
resched2
266: 0 45 0 1867 Dynamic-irq
callfunc2
267: 0 0 39543 256 Dynamic-irq timer2
268: 0 2244 46 33027 Dynamic-irq
resched3
269: 1585 0 0 0 Dynamic-irq
callfunc3
270: 11273 0 1885 1073 Dynamic-irq timer3
NMI: 0 0 0 0
LOC: 0 0 0 0
ERR: 0
MIS: 0
You can see from above that each virq/ipi have
statistics on several vcpus, when that data on
other vcpus belong to other virq/ipi before unplug.
This also applies to save/restore for same issue.
Signed-off-by Kevin Tian <kevin.tian@intel.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2007-Feb-04 04:51 UTC
Re: [Xen-devel] [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug
On 3/2/07 04:49, "Tian, Kevin" <kevin.tian@intel.com> wrote:> Irq nunmber for per-vcpu event (virq/ipi) needs kept > accross vcpu plug/unplug, once allocated. We just > reuse this irq number and bind it to a new event port. > Or else, /proc/interrupt exports messed statistics > like:Noone cares about absolute /proc/interrupt numbers, only rate of change. If pushed, I would argue that the stats should be zeroed when an interrupt line is freed (since the interrupt then stops appearing in /proc/interrupts, which logically implies that the stats lifetime has ended, and so a reuse of that interrupt is a new lifetime starting from zero). After all, all irq-evtchn bindings are dynamic running on Xen: a Linux irq may theoretically get used for all sorts of different devices during the lifetime of the Linux guest. Should all these uses get aggregated over time? Zeroing the stats would potentially be a patch for lkml, or we could do it for dynirqs ourselves in unbind_from_irq(). That''s a patch I would accept. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2007-Feb-04 12:28 UTC
RE: [Xen-devel] [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug
>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] >Sent: 2007年2月4日 12:52 > >On 3/2/07 04:49, "Tian, Kevin" <kevin.tian@intel.com> wrote: > >> Irq nunmber for per-vcpu event (virq/ipi) needs kept >> accross vcpu plug/unplug, once allocated. We just >> reuse this irq number and bind it to a new event port. >> Or else, /proc/interrupt exports messed statistics >> like: > >Noone cares about absolute /proc/interrupt numbers, only rate >of change. If >pushed, I would argue that the stats should be zeroed when an >interrupt line >is freed (since the interrupt then stops appearing in /proc/interrupts, >which logically implies that the stats lifetime has ended, and >so a reuse of >that interrupt is a new lifetime starting from zero). After all, all >irq-evtchn bindings are dynamic running on Xen: a Linux irq may >theoretically get used for all sorts of different devices during the >lifetime of the Linux guest. Should all these uses get >aggregated over time? > >Zeroing the stats would potentially be a patch for lkml, or we >could do it >for dynirqs ourselves in unbind_from_irq(). That''s a patch I >would accept. > > -- KeirBasically I agree with you, but I''m not sure about the usage model on such stats. Isn''t it up to application to decide? App may decide to do some action (like balance) by rate of change, so does the aggregated value. If both usage models exist, do we need consider both for compatibility to app''s assumption? Another point I''m not for sure is the save/restore and migration.>From user perspective, he has no knowledge that cpu1...N areunplugged and then plugged along the process. Thus will anyone feel strange about the fact that all cpus except cpu0 gets their per-vcpu interrupt stats empty after restored or migrated? This seems not an explicit lifetime restart... So I still think virq/ipi are a bit special, and it''s reasonable to zero stats for the rest. :-) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Feb-04 14:39 UTC
Re: [Xen-devel] [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug
On 4/2/07 12:28, "Tian, Kevin" <kevin.tian@intel.com> wrote:> Another point I''m not for sure is the save/restore and migration. > From user perspective, he has no knowledge that cpu1...N are > unplugged and then plugged along the process. Thus will anyone > feel strange about the fact that all cpus except cpu0 gets their > per-vcpu interrupt stats empty after restored or migrated? This > seems not an explicit lifetime restart......and the same is true for his network and block-device interrupts. Currently these may come back with different IRQs too, and ''fixing'' this would be somewhat painful (e.g., it would probably need some evtchn interface changes). Why do you consider the VIRQs/IPIs so special? Your original point that having non-zero counts for CPU!=n for an interrupt that is specific to CPU==n looks weird and may confuse people is a fair one. Zeroing stats is a nice simple solution to that. As you say /proc/interrupts is used for IRQ load balancing, where rate of change matters. I''m not aware of any other application uses of /proc/interrupts -- certainly none that are ''stateful'' in terms of requiring a given interrupt source to stick with its original IRQ, or require accurate aggregate counts. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2007-Feb-05 02:49 UTC
RE: [Xen-devel] [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug
So, attached patch zeros irq stats at bind_evtchn_to_cpu, since rate of change is the real concern. By this way, we don''t need to care whether stale stats are on multiple cpus at irq unbinding. Thanks, Kevin.>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] >Sent: 2007年2月4日 22:39 > >On 4/2/07 12:28, "Tian, Kevin" <kevin.tian@intel.com> wrote: > >> Another point I''m not for sure is the save/restore and migration. >> From user perspective, he has no knowledge that cpu1...N are >> unplugged and then plugged along the process. Thus will anyone >> feel strange about the fact that all cpus except cpu0 gets their >> per-vcpu interrupt stats empty after restored or migrated? This >> seems not an explicit lifetime restart... > >...and the same is true for his network and block-device interrupts. >Currently these may come back with different IRQs too, and >''fixing'' this >would be somewhat painful (e.g., it would probably need some evtchn >interface changes). Why do you consider the VIRQs/IPIs so special? > >Your original point that having non-zero counts for CPU!=n for >an interrupt >that is specific to CPU==n looks weird and may confuse people >is a fair one. >Zeroing stats is a nice simple solution to that. > >As you say /proc/interrupts is used for IRQ load balancing, >where rate of >change matters. I''m not aware of any other application uses of >/proc/interrupts -- certainly none that are ''stateful'' in >terms of requiring >a given interrupt source to stick with its original IRQ, or >require accurate >aggregate counts. > > -- Keir >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel