This worked for me for a fairly recent version of xen-unstable (cs:
19429:6dcccd2dded6). I used the corresponding -xen kernel for domU. I
used a freshly compiled version of the ixgbe driver (1.3.56.5-NAPI).
MSI-X worked fine, though I''ve only tried ping, not bulk data
transfers.
packets looked fine when I looked at it with tcpdump.
there were quite a few changes to the MSI code recently.
Rolf
Fischer, Anna wrote:> Is it possible to use MSI-X in a PV Xen DomU (2.6.18.8) for a PCI
pass-through device? I am running an Intel ixgbe NIC in DomU, but I do not seem
to be able to run it with MSI-X enabled. In fact, I do not get it to work in any
interrupt mode. The DomU/Dom0 kernel is compiled with PCI_MSI=y and I have msi=1
set in Xen''s boot options.
>
> I do seem to get an interrupt clash when running up the ixgbe device driver
in DomU. Also, packets seem to be corrupted when I capture with tcpdump on the
interfaces.
>
> tcpdump -i eth1 -xx gives me this:
>
> 01:46:56.770226 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00
(oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length
46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:56.973564 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00
(oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length
46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:57.176902 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00
(oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length
46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
> 01:46:57.380238 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00
(oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length
46
> 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
> 0x0030: 0000 0000 0000 0000 0000 0000
>
>
> I have tried to run DomU/DomU with the irqpoll boot parameter but when I do
that then the DomU fails to boot and hangs with a kernel panic:
>
> Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5
> Copyright (c) 1999-2008 Intel Corporation.
> PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
> ixgbe: Interrupt Type set to 1
> ixgbe: 0000:02:00.0: ixgbe_check_options: Multiple queues are not supported
while MSI-X is disabled. Disabling Multiple Queues.
> ixgbe: 0000:02:00.0: ixgbe_check_options: RSS is not supported while
multiple queues are disabled. Disabling RSS.
> ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx
Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8)
00:21:f7:58:01:6d
> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
> PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
> BUG: unable to handle kernel NULL pointer dereference at virtual address
0000002d
> printing eip:
> c01fccaa
> 2c8fa000 -> *pde = 00000000:60b62001
> 2cfc1000 -> *pme = 00000000:00000000
> Oops: 0002 [#1]
> SMP
> Modules linked in: ixgbe aacraid usb_storage libusual ata_piix ahci libata
sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c01fccaa>] Not tainted VLI
> EFLAGS: 00210246 (2.6.18.8-xen #5)
> EIP is at pci_enable_msix+0x1da/0x510
> eax: ed717800 ebx: 00000011 ecx: 00000000 edx: 00000004
> esi: 00000000 edi: ed717800 ebp: ec951f80 esp: ecab9d54
> ds: 007b es: 007b ss: 0069
> Process modprobe (pid: 1566, ti=ecab8000 task=eccd2e10 task.ti=ecab8000)
> Stack: 00000098 ecfd4400 c036d220 c015f905 ecfd4400 ecfd4400 00000004
ee0e639a
> 00000000 00000000 00000008 ec951f60 000000b2 00000002 00030003
0000007b
> 0000007b 00000000 c017682e ec951f80 00000004 00000010 ecfd4400
00000000
> Call Trace:
> [<c015f905>] __kzalloc+0x15/0x50
> [<ee0e639a>] ixgbe_alloc_queues+0x8a/0x580 [ixgbe]
> [<c017682e>] __kmalloc+0xbe/0xf0
> [<ee0e6a13>] ixgbe_init_interrupt_scheme+0x183/0x340 [ixgbe]
> [<ee0e3b81>] ixgbe_read_pci_cfg_word+0x21/0x30 [ixgbe]
> [<ee0e8c17>] ixgbe_probe+0x6e7/0x9e0 [ixgbe]
> [<c017682e>] __kmalloc+0xbe/0xf0
> [<c01f5ef6>] pci_device_probe+0x56/0x80
> [<c023ecd4>] driver_probe_device+0x44/0xc0
> [<c023ee52>] __driver_attach+0x82/0x90
> [<c023e64a>] bus_for_each_dev+0x3a/0x60
> [<c023ec16>] driver_attach+0x16/0x20
> [<c023edd0>] __driver_attach+0x0/0x90
> [<c023e2bc>] bus_add_driver+0x8c/0x140
> [<c01f6097>] __pci_register_driver+0x47/0x70
> [<c0143b08>] sys_init_module+0x148/0x1b40
> [<c01ea55f>] prio_tree_insert+0x1f/0x260
> [<c01187fc>] do_page_fault+0x10c/0xc6f
> [<c010acc0>] sys_mmap2+0xd0/0xe0
> [<c01059bf>] syscall_call+0x7/0xb
> Code: 00 89 54 24 38 ba ed ff ff ff 89 4c 24 3c 0b 4c 24 38 0f 85 2d 01 00
00 8b 4c 24 28 89 d6 89 8f b0 01 00 00 e9 62 fe ff ff 89 f8 <89> 5b 1c e8
3e f2 ff ff 8b 7c 24 18 89 44 24 54 85 ff 0f 8e 48
> EIP: [<c01fccaa>] pci_enable_msix+0x1da/0x510 SS:ESP 0069:ecab9d54
> udevd-event[1564]: run_program: ''/sbin/modprobe''
abnormal exit
>
>
> I have also tried to use legacy interrupts by setting InterruptType to 0:
>
> Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5-lro
> Copyright (c) 1999-2008 Intel Corporation.
> PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
> PCI: Setting latency timer of device 0000:02:00.0 to 64
> ixgbe: Interrupt Type set to 0
> ixgbe: Multiple Queue Support Disabled
> ixgbe: Receive-Side Scaling (RSS) set to 0
> ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
> ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx
Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8)
00:21:f7:58:01:6d
> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
> PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
> PCI: Setting latency timer of device 0000:02:00.1 to 64
> ixgbe: Interrupt Type set to 0
> ixgbe: Multiple Queue Support Disabled
> ixgbe: Receive-Side Scaling (RSS) set to 0
> ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
> ixgbe: 0000:02:00.1: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx
Queue count = 1, Tx Queue count = 1
> ixgbe: 0000:02:00.1: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8)
00:21:f7:58:01:6c
> ixgbe: eth1: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
>
> Without MSI and MSI-X the driver loads properly. However, I still end up
with corrupted packets on tcpdump, and additionally I am getting a kernel crash
when the guest is under high network load:
>
> BUG: unable to handle kernel NULL pointer dereference at virtual address
00000034
> printing eip:
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#1]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod
ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0158df7>] Not tainted VLI
> EFLAGS: 00010082 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#2]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod
ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
> <0>Kernel panic - not syncing: Fatal exception in interrupt
> BUG: warning at
/home/user/Download/linux-2.6.18-xen.hg/arch/i386/kernel/smp-xen.c:511/smp_call_function()
> [<c0112541>] <1>BUG: unable to handle kernel paging request at
virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#3]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod
ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Oops: 0002 [#4]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod
ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 0
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2c934000 -> *pde = 00000000:60f1a001
> 2c8e7000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
> <0>Kernel panic - not syncing: Fatal exception in interrupt
> <1>BUG: unable to handle kernel NULL pointer dereference at virtual
address 00000064
> printing eip:
> c0158df7
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Oops: 0002 [#5]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod
ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 1
> EIP: 0061:[<c0158df7>] Not tainted VLI
> EFLAGS: 00010282 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Oops: 0002 [#6]
> SMP
> Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod
ixgbe aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
> CPU: 1
> EIP: 0061:[<c0145df6>] Not tainted VLI
> EFLAGS: 00010002 (2.6.18.8-xen #5)
> BUG: unable to handle kernel paging request at virtual address 245c8bf1
> printing eip:
> c0145df6
> 2d2e1000 -> *pde = 00000000:6051f001
> 2d2e2000 -> *pme = 00000000:00000000
> Recursive die() failure, output suppressed
>
> What are the correct interrupt settings for using ixgbe in PV pass-through
mode? I have tried all three possible modes but I do not seem to get the device
working properly under high network load.
>
> I am using Xen 3.3.1.
>
> release : 2.6.18.8-xen
> version : #5 SMP Thu Mar 5 04:14:30 PST 2009
> machine : i686
> nr_cpus : 2
> nr_nodes : 1
> cores_per_socket : 2
> threads_per_core : 1
> cpu_mhz : 2200
> virt_caps : hvm
> total_memory : 2039
> free_memory : 6
> node_to_cpu : node0:0-1
> node_to_memory : node0:6
> xen_major : 3
> xen_minor : 3
> xen_extra : .1
> xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xf5800000
> xen_changeset : unavailable
> cc_compiler : gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)
> cc_compile_by : root
> cc_compile_domain : localdomain
> cc_compile_date : Sun Mar 8 05:40:51 PDT 2009
> xend_config_format : 4
>
> I am using PCI pass-through to assign the NIC to my PV guest, so this is
the part of my config file:
>
> pci = [ ''02:00.0'',''02:00.1'' ]
>
> I am also hiding the PCI devices from Dom0, so my Dom0 GRUB config looks
like this:
>
> title Xen 3.3
> root (hd0,1)
> kernel /boot/xen-3.3.1.gz console=vga msi=1
> module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro console=tty0 rhgb
quiet pciback.permissive pciback.hide=(02:00.0)(02:00.1)
reassigndev=0000:02:00.0,0000:02:00.1
> module /boot/initrd-2.6.18.8-xen.img
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel