Dante Cinco
2010-Nov-10 01:18 UTC
[Xen-devel] IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
I have a Fibre Channel HBA that I''m passing through to the pvops domU.
In domU its IRQ 127 and I''ve affinitized it to VCPU0 (out of 16) by
setting /proc/irq/127/smp_affinity to 0001 (default setting was FFFF).
However, when I checked the interrupt bindings in Xen, it still shows
IRQ 127 as being affinitized to all CPUs (all F''s). I checked lspci in
both dom0 and domU and the address portion (00000000fee00000) of the
MSI address/data seem to match Xen''s interrupt binding report (I
expected the digits to the right of ''fee'' to be non-zero if
it''s
affinitized to one specific CPU).
I''ve tried using a recent Ubuntu HVM kernel for domU instead of
Konrad''s pcifront pvops kernel and with the HVM kernel, Xen''s
interrupt binding matches the IRQ smp_affinity in domU.
Am I not correctly interpreting the apparent affinity discrepancy
between Xen and pvops domU or is there a known limitation or problem
with affinitizing IRQs in Konrad''s pcifront pvops domU kernel?
from Konrad''s pcifront pvop domU:
lspci -vv -s 00:00.0
00:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
Interrupt: pin D routed to IRQ 127
Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+
Queue=0/1 Enable+
Address: 00000000fee00000 Data: 4043
Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9
Vector table: BAR=4 offset=00004100
PBA: BAR=4 offset=00004000
cat /proc/irq/127/smp_affinity
0001
uname -a
Linux kaan-40-dpm 2.6.36-rc7-pvops-kpcif-08-2-domu-5.8.dcinco-debug #1
SMP Tue Nov 9 10:36:45 PST 2010 x86_64 GNU/Linux
from 2.6.32.25 pvops dom0:
lspci -vv -s 11:00.3
11:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
Interrupt: pin D routed to IRQ 4426
Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+
Queue=0/1 Enable+
Address: 00000000fee00000 Data: 4043
Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9
Vector table: BAR=4 offset=00004100
PBA: BAR=4 offset=00004000
uname -a
Linux kaan-40 2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug #1 SMP
PREEMPT Fri Nov 5 16:13:32 PDT 2010 x86_64 GNU/Linux
(XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to
switch
input to DOM0)
(XEN) Guest interrupt information:
(XEN) IRQ: 67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:43
type=PCI-MSI status=00000010 in-flight=0
domain-list=2:127(----),
xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 1 r----- 43.6
domU 2 2048 16 -b---- 63.0
xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 r-- 44.7 0
domU 2 0 0 -b- 17.1 0
domU 2 1 1 -b- 4.9 1
domU 2 2 2 -b- 3.1 2
domU 2 3 3 -b- 4.7 3
domU 2 4 4 -b- 3.0 4
domU 2 5 5 -b- 3.9 5
domU 2 6 6 -b- 4.9 6
domU 2 7 7 -b- 1.9 7
domU 2 8 8 -b- 3.7 8
domU 2 9 9 -b- 1.8 9
domU 2 10 10 -b- 1.8 10
domU 2 11 11 -b- 4.3 11
domU 2 12 12 -b- 2.8 12
domU 2 13 13 -b- 2.7 13
domU 2 14 14 -b- 1.9 14
domU 2 15 15 -b- 3.9 15
xm info
release : 2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug
version : #1 SMP PREEMPT Fri Nov 5 16:13:32 PDT 2010
machine : x86_64
hw_caps :
bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000
max_node_id : 1
xen_major : 4
xen_minor : 0
xen_extra : .2-rc1-pre
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_changeset : Fri Sep 17 17:06:57 2010 +0100 21350:6e0ffcd2d9e0
xen_commandline : dummy=dummy dom0_mem=1024M dom0_max_vcpus=1
dom0_vcpus_pin=true iommu=1,passthrough,no-intremap loglvl=all
loglvl_guest=all loglevl=10 debug apic=on apic_verbosity=verbose
extra_guest_irqs=80 com1=115200,8n1 console=com1 console_to_ring
noirqbalance xen-pciback.permissive acpi=force numa=on
- Dante
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Nov-10 02:11 UTC
Re: [Xen-devel] IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
On 11/09/2010 05:18 PM, Dante Cinco wrote:> I have a Fibre Channel HBA that I''m passing through to the pvops domU. > In domU its IRQ 127 and I''ve affinitized it to VCPU0 (out of 16) by > setting /proc/irq/127/smp_affinity to 0001 (default setting was FFFF). > However, when I checked the interrupt bindings in Xen, it still shows > IRQ 127 as being affinitized to all CPUs (all F''s). I checked lspci in > both dom0 and domU and the address portion (00000000fee00000) of the > MSI address/data seem to match Xen''s interrupt binding report (I > expected the digits to the right of ''fee'' to be non-zero if it''s > affinitized to one specific CPU). > > I''ve tried using a recent Ubuntu HVM kernel for domU instead of > Konrad''s pcifront pvops kernel and with the HVM kernel, Xen''s > interrupt binding matches the IRQ smp_affinity in domU. > > Am I not correctly interpreting the apparent affinity discrepancy > between Xen and pvops domU or is there a known limitation or problem > with affinitizing IRQs in Konrad''s pcifront pvops domU kernel?A VCPU is not a real physical CPU, and can run on any physical CPU from moment to moment. Setting the affinity within the domU will cause the interrupts to be handled by a specific VCPU, but that has no meaning to the hardware as it knows nothing about VCPUs. Since the VCPU can run on any PCPU, it makes sense that the hardware routing is for all PCPUs. In principle I guess you could pin the VCPU to a particular PCPU and then route only to that PCPU, but I''m not sure what that would achieve. J> from Konrad''s pcifront pvop domU: > lspci -vv -s 00:00.0 > 00:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08) > Interrupt: pin D routed to IRQ 127 > Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ > Queue=0/1 Enable+ > Address: 00000000fee00000 Data: 4043 > Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9 > Vector table: BAR=4 offset=00004100 > PBA: BAR=4 offset=00004000 > cat /proc/irq/127/smp_affinity > 0001 > uname -a > Linux kaan-40-dpm 2.6.36-rc7-pvops-kpcif-08-2-domu-5.8.dcinco-debug #1 > SMP Tue Nov 9 10:36:45 PST 2010 x86_64 GNU/Linux > > > from 2.6.32.25 pvops dom0: > lspci -vv -s 11:00.3 > 11:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08) > Interrupt: pin D routed to IRQ 4426 > Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ > Queue=0/1 Enable+ > Address: 00000000fee00000 Data: 4043 > Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9 > Vector table: BAR=4 offset=00004100 > PBA: BAR=4 offset=00004000 > uname -a > Linux kaan-40 2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug #1 SMP > PREEMPT Fri Nov 5 16:13:32 PDT 2010 x86_64 GNU/Linux > > > (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch > input to DOM0) > (XEN) Guest interrupt information: > (XEN) IRQ: 67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:43 > type=PCI-MSI status=00000010 in-flight=0 > domain-list=2:127(----), > > > xm list > Name ID Mem VCPUs State Time(s) > Domain-0 0 1024 1 r----- 43.6 > domU 2 2048 16 -b---- 63.0 > xm vcpu-list > Name ID VCPU CPU State Time(s) CPU Affinity > Domain-0 0 0 0 r-- 44.7 0 > domU 2 0 0 -b- 17.1 0 > domU 2 1 1 -b- 4.9 1 > domU 2 2 2 -b- 3.1 2 > domU 2 3 3 -b- 4.7 3 > domU 2 4 4 -b- 3.0 4 > domU 2 5 5 -b- 3.9 5 > domU 2 6 6 -b- 4.9 6 > domU 2 7 7 -b- 1.9 7 > domU 2 8 8 -b- 3.7 8 > domU 2 9 9 -b- 1.8 9 > domU 2 10 10 -b- 1.8 10 > domU 2 11 11 -b- 4.3 11 > domU 2 12 12 -b- 2.8 12 > domU 2 13 13 -b- 2.7 13 > domU 2 14 14 -b- 1.9 14 > domU 2 15 15 -b- 3.9 15 > xm info > release : 2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug > version : #1 SMP PREEMPT Fri Nov 5 16:13:32 PDT 2010 > machine : x86_64 > hw_caps : > bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 > max_node_id : 1 > xen_major : 4 > xen_minor : 0 > xen_extra : .2-rc1-pre > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_changeset : Fri Sep 17 17:06:57 2010 +0100 21350:6e0ffcd2d9e0 > xen_commandline : dummy=dummy dom0_mem=1024M dom0_max_vcpus=1 > dom0_vcpus_pin=true iommu=1,passthrough,no-intremap loglvl=all > loglvl_guest=all loglevl=10 debug apic=on apic_verbosity=verbose > extra_guest_irqs=80 com1=115200,8n1 console=com1 console_to_ring > noirqbalance xen-pciback.permissive acpi=force numa=on > > > - Dante > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Nov-10 08:13 UTC
Re: [Xen-devel] IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
>>> On 10.11.10 at 02:18, Dante Cinco <dantecinco@gmail.com> wrote: > I have a Fibre Channel HBA that I''m passing through to the pvops domU. > In domU its IRQ 127 and I''ve affinitized it to VCPU0 (out of 16) by > setting /proc/irq/127/smp_affinity to 0001 (default setting was FFFF). > However, when I checked the interrupt bindings in Xen, it still shows > IRQ 127 as being affinitized to all CPUs (all F''s). I checked lspci in > both dom0 and domU and the address portion (00000000fee00000) of the > MSI address/data seem to match Xen''s interrupt binding report (I > expected the digits to the right of ''fee'' to be non-zero if it''s > affinitized to one specific CPU).See -unstable c/s 21625:0695a5cdcb42. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dante Cinco
2010-Nov-10 20:18 UTC
Re: [Xen-devel] IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
On Wed, Nov 10, 2010 at 12:13 AM, Jan Beulich <JBeulich@novell.com> wrote:>>>> On 10.11.10 at 02:18, Dante Cinco <dantecinco@gmail.com> wrote: >> I have a Fibre Channel HBA that I''m passing through to the pvops domU. >> In domU its IRQ 127 and I''ve affinitized it to VCPU0 (out of 16) by >> setting /proc/irq/127/smp_affinity to 0001 (default setting was FFFF). >> However, when I checked the interrupt bindings in Xen, it still shows >> IRQ 127 as being affinitized to all CPUs (all F''s). I checked lspci in >> both dom0 and domU and the address portion (00000000fee00000) of the >> MSI address/data seem to match Xen''s interrupt binding report (I >> expected the digits to the right of ''fee'' to be non-zero if it''s >> affinitized to one specific CPU). > > See -unstable c/s 21625:0695a5cdcb42. > > Jan > >Jan, After seeing your response (cs 21625, x86: IRQ affinity should track vCPU affinity), I installed the latest xen-unstable-4.1 (cs 22382) and the result is the same --- IRQ affinity in Xen is set to all F''s after explicitly setting the IRQ smp_affinity in domU to a specific CPU. I do have the VCPUs assigned to domU pinned to PCPUs on a one-to-one basis (VCPU0 -> PCPU0, VCPU1 -> PCPU1, etc.). BTW, I actually have 16 of these devices PCI-passed through to domU which has16 VCPUs and I''m affinitizing each device to its own dedicated CPU. Before explicitly setting the IRQ smp_affinity, all 16 devices are set to 0001. See the explicit IRQ smp_affinity setting below. If I look in the domU''s /proc/interrupts, it shows the interrupts for a given IRQ going only to the CPU it has been affinitized to (no interrupts going to the other CPUs) which is expected. My system has 24 PCPUs (dual socket X5650 Xeon/6-core Westmere) but I''m only assigning the first 16 CPUs to domU and the VCPUs are pinned to their respective PCPU. The part I don''t understand is if the IRQ affinity per Xen (all F''s) is correct, how does the interrupt get handled if it is directed to a PCPU that is not assigned/pinned to a VCPU in domU? Jeremy, to address your question, pinning the VCPUs to specific PCPUs and affinitizing the IRQ to specific VCPUs in domU has worked very well for us on the HVM kernel. The I/O performance is significantly better compared with no affinitization. We''re trying to transition to the pvops domU with the expectation that our affinitization strategy will still be applicable and maintain or even possibly improve I/O performance. These are the IRQ smp_affinity settings in pvops domU: cat /proc/irq/112/smp_affinity 8000 cat /proc/irq/113/smp_affinity 4000 cat /proc/irq/114/smp_affinity 2000 cat /proc/irq/115/smp_affinity 1000 cat /proc/irq/116/smp_affinity 0800 cat /proc/irq/117/smp_affinity 0400 cat /proc/irq/118/smp_affinity 0200 cat /proc/irq/119/smp_affinity 0100 cat /proc/irq/120/smp_affinity 0080 cat /proc/irq/121/smp_affinity 0040 cat /proc/irq/122/smp_affinity 0020 cat /proc/irq/123/smp_affinity 0010 cat /proc/irq/124/smp_affinity 0008 cat /proc/irq/125/smp_affinity 0004 cat /proc/irq/126/smp_affinity 0002 cat /proc/irq/127/smp_affinity 0001 (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch input to DOM0) (XEN) Guest interrupt information: (XEN) IRQ: 67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:7a type=PCI-MSI status=00000010 in-flight=0 domain-list=1:127(----), (XEN) IRQ: 68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:8a type=PCI-MSI status=00000010 in-flight=0 domain-list=1:126(----), (XEN) IRQ: 69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:92 type=PCI-MSI status=00000010 in-flight=0 domain-list=1:125(----), (XEN) IRQ: 70 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:9a type=PCI-MSI status=00000010 in-flight=0 domain-list=1:124(----), (XEN) IRQ: 71 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:a2 type=PCI-MSI status=00000010 in-flight=0 domain-list=1:123(----), (XEN) IRQ: 72 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:aa type=PCI-MSI status=00000010 in-flight=0 domain-list=1:122(----), (XEN) IRQ: 73 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b2 type=PCI-MSI status=00000010 in-flight=0 domain-list=1:121(----), (XEN) IRQ: 74 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ba type=PCI-MSI status=00000010 in-flight=0 domain-list=1:120(----), (XEN) IRQ: 75 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c2 type=PCI-MSI status=00000010 in-flight=0 domain-list=1:119(----), (XEN) IRQ: 76 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ca type=PCI-MSI status=00000010 in-flight=0 domain-list=1:118(----), (XEN) IRQ: 77 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d2 type=PCI-MSI status=00000010 in-flight=0 domain-list=1:117(----), (XEN) IRQ: 78 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:da type=PCI-MSI status=00000010 in-flight=0 domain-list=1:116(----), (XEN) IRQ: 79 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:23 type=PCI-MSI status=00000010 in-flight=0 domain-list=1:115(----), (XEN) IRQ: 80 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2b type=PCI-MSI status=00000010 in-flight=0 domain-list=1:114(----), (XEN) IRQ: 81 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:33 type=PCI-MSI status=00000010 in-flight=0 domain-list=1:113(----), (XEN) IRQ: 82 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3b type=PCI-MSI status=00000010 in-flight=0 domain-list=1:112(----), xm info xen_changeset : Tue Nov 09 20:37:46 2010 +0000 22382:a15b0a2dc276 xm list Name ID Mem VCPUs State Time(s) Domain-0 0 1024 1 r----- 50.3 domU 1 2048 16 -b---- 220.9 xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 r-- 50.6 0 domU 1 0 0 -b- 32.2 0 domU 1 1 1 -b- 19.8 1 domU 1 2 2 -b- 19.0 2 domU 1 3 3 -b- 13.9 3 domU 1 4 4 -b- 8.4 4 domU 1 5 5 -b- 16.6 5 domU 1 6 6 -b- 26.6 6 domU 1 7 7 -b- 8.0 7 domU 1 8 8 -b- 9.6 8 domU 1 9 9 -b- 16.5 9 domU 1 10 10 -b- 9.0 10 domU 1 11 11 -b- 8.2 11 domU 1 12 12 -b- 12.4 12 domU 1 13 13 -b- 11.6 13 domU 1 14 14 -b- 4.9 14 domU 1 15 15 -b- 4.5 15 - Dante _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Nov-11 13:22 UTC
Re: [Xen-devel] IRQ affinity: Xen view different from pvops (konrad-pcifront) domU
>>> On 10.11.10 at 21:18, Dante Cinco <dantecinco@gmail.com> wrote: > After seeing your response (cs 21625, x86: IRQ affinity should track > vCPU affinity), I installed the latest xen-unstable-4.1 (cs 22382) and > the result is the same --- IRQ affinity in Xen is set to all F''s after > explicitly setting the IRQ smp_affinity in domU to a specific CPU. I > do have the VCPUs assigned to domU pinned to PCPUs on a one-to-one > basis (VCPU0 -> PCPU0, VCPU1 -> PCPU1, etc.).You may need to do some debugging then to find out what''s not working in your case.> My system has 24 PCPUs (dual socket X5650 Xeon/6-core Westmere) but > I''m only assigning the first 16 CPUs to domU and the VCPUs are pinned > to their respective PCPU. The part I don''t understand is if the IRQ > affinity per Xen (all F''s) is correct, how does the interrupt get > handled if it is directed to a PCPU that is not assigned/pinned to a > VCPU in domU?Since the guest handles event channel signals only, all it takes is binding the event channel to the correct vCPU. When the channel gets signaled, that vCPU will call the handler. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel