thr3ads.net - Xen devel - Issue with MSI in a HVM domU with several passed through PCI devices [Jun 2012]

If this information is useful, please help other people find it:
Share via:

Rolu

2012-Jun-18 20:57 UTC

Issue with MSI in a HVM domU with several passed through PCI devices

Hi,

I have a HVM domU running Ubuntu 12.04. It has several PCI devices
passed through to it (USB controllers, onboard sound, ATI video card
with gfx_passthru 0). It has some issues, the most apparent one with
the sound, which doesn''t play normally but repeats the same short
fragment over and over for a while until moving on to the next
fragment. The Xen dmesg prints lots of lines like:

(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3

The number behind 108:d varies occasionally.

Since vmsi.c seems to deal with passing on MSI interrupts, I tried
booting the domU with pci=nomsi. This made things work and sound now
plays normally. Not using MSI at all doesn''t seem like a very good
solution though.

What makes this happen, and can I fix it in a better way?

Data:
Xen: 4.2 unstable rev. 25483
Dom0 Ubuntu 12.04 3.4.0-030400-generic #201205210521

----- Xen dmesg when starting the domU without pci=nomsi:
(XEN) [2012-06-18 20:13:35] memory.c:131:d0 Could not allocate
order=18 extent: id=3 memflags=0 (0 of 1)
(XEN) [2012-06-18 20:13:37] physdev.c:174: dom3: 16:-1 already mapped to 16
(XEN) [2012-06-18 20:13:37] physdev.c:174: dom3: 16:-1 already mapped to 16
(XEN) [2012-06-18 20:13:37] HVM3: HVM Loader
(XEN) [2012-06-18 20:13:37] HVM3: Detected Xen v4.2-unstable
(XEN) [2012-06-18 20:13:37] HVM3: Xenbus rings @0xfeffc000, event channel 3
(XEN) [2012-06-18 20:13:37] HVM3: System requested ROMBIOS
(XEN) [2012-06-18 20:13:37] HVM3: CPU speed is 3300 MHz
(XEN) [2012-06-18 20:13:37] irq.c:270: Dom3 PCI link 0 changed 0 -> 5
(XEN) [2012-06-18 20:13:37] HVM3: PCI-ISA link 0 routed to IRQ5
(XEN) [2012-06-18 20:13:37] irq.c:270: Dom3 PCI link 1 changed 0 -> 10
(XEN) [2012-06-18 20:13:37] HVM3: PCI-ISA link 1 routed to IRQ10
(XEN) [2012-06-18 20:13:37] irq.c:270: Dom3 PCI link 2 changed 0 -> 11
(XEN) [2012-06-18 20:13:37] HVM3: PCI-ISA link 2 routed to IRQ11
(XEN) [2012-06-18 20:13:37] irq.c:270: Dom3 PCI link 3 changed 0 -> 5
(XEN) [2012-06-18 20:13:37] HVM3: PCI-ISA link 3 routed to IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 01:2 INTD->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 01:3 INTA->IRQ10
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 03:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 04:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 05:0 INTA->IRQ10
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 06:0 INTA->IRQ11
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 07:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 08:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 INTA->IRQ5
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 bar 10 size 10000000: e000000c
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 02:0 bar 10 size 02000000: f0000008
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 03:0 bar 14 size 01000000: f2000008
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 bar 18 size 00020000: f3000004
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 bar 30 size 00020000: f3020000
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 05:0 bar 10 size 00010000: f3040004
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 08:0 bar 10 size 00004000: f3050004
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 02:0 bar 14 size 00001000: f3054000
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 06:0 bar 10 size 00001000: f3055000
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 07:0 bar 10 size 00001000: f3056000
(XEN) [2012-06-18 20:13:37] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 03:0 bar 10 size 00000100: 0000c001
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 04:0 bar 10 size 00000100: 0000c101
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 04:0 bar 14 size 00000100: f3057000
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 10:0 bar 20 size 00000100: 0000c201
(XEN) [2012-06-18 20:13:37] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 01:2 bar 20 size 00000020: 0000c301
(XEN) [2012-06-18 20:13:37] HVM3: pci dev 01:1 bar 20 size 00000010: 0000c321
(XEN) [2012-06-18 20:13:37] HVM3: Multiprocessor initialisation:
(XEN) [2012-06-18 20:13:37] HVM3:  - CPU0 ... 36-bit phys ... fixed
MTRRs ... var MTRRs [3/8] ... done.
(XEN) [2012-06-18 20:13:37] HVM3: Testing HVM environment:
(XEN) [2012-06-18 20:13:37] HVM3:  - REP INSB across page boundaries ... passed
(XEN) [2012-06-18 20:13:37] HVM3:  - GS base MSRs and SWAPGS ... passed
(XEN) [2012-06-18 20:13:37] HVM3: Passed 2 of 2 tests
(XEN) [2012-06-18 20:13:37] HVM3: Writing SMBIOS tables ...
(XEN) [2012-06-18 20:13:37] HVM3: Loading ROMBIOS ...
(XEN) [2012-06-18 20:13:37] HVM3: 12636 bytes of ROMBIOS high-memory extensions:
(XEN) [2012-06-18 20:13:37] HVM3:   Relocating to 0xfc001000-0xfc00415c ... done
(XEN) [2012-06-18 20:13:37] HVM3: Creating MP tables ...
(XEN) [2012-06-18 20:13:37] HVM3: Loading Cirrus VGABIOS ...
(XEN) [2012-06-18 20:13:37] HVM3: Loading PCI Option ROM ...
(XEN) [2012-06-18 20:13:37] HVM3:  - Manufacturer: http://ipxe.org
(XEN) [2012-06-18 20:13:37] HVM3:  - Product name: iPXE
(XEN) [2012-06-18 20:13:37] HVM3: Option ROMs:
(XEN) [2012-06-18 20:13:37] HVM3:  c0000-c8fff: VGA BIOS
(XEN) [2012-06-18 20:13:37] HVM3:  c9000-d8fff: Etherboot ROM
(XEN) [2012-06-18 20:13:37] HVM3: Loading ACPI ...
(XEN) [2012-06-18 20:13:37] HVM3: vm86 TSS at fc010280
(XEN) [2012-06-18 20:13:37] HVM3: BIOS map:
(XEN) [2012-06-18 20:13:37] HVM3:  f0000-fffff: Main BIOS
(XEN) [2012-06-18 20:13:37] HVM3: E820 table:
(XEN) [2012-06-18 20:13:37] HVM3:  [00]: 00000000:00000000 -
00000000:0009e000: RAM
(XEN) [2012-06-18 20:13:37] HVM3:  [01]: 00000000:0009e000 -
00000000:000a0000: RESERVED
(XEN) [2012-06-18 20:13:37] HVM3:  HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) [2012-06-18 20:13:37] HVM3:  [02]: 00000000:000e0000 -
00000000:00100000: RESERVED
(XEN) [2012-06-18 20:13:37] HVM3:  [03]: 00000000:00100000 -
00000000:e0000000: RAM
(XEN) [2012-06-18 20:13:37] HVM3:  HOLE: 00000000:e0000000 - 00000000:fc000000
(XEN) [2012-06-18 20:13:37] HVM3:  [04]: 00000000:fc000000 -
00000001:00000000: RESERVED
(XEN) [2012-06-18 20:13:37] HVM3:  [05]: 00000001:00000000 -
00000002:9f800000: RAM
(XEN) [2012-06-18 20:13:37] HVM3: Invoking ROMBIOS ...
(XEN) [2012-06-18 20:13:37] HVM3: $Revision: 1.221 $ $Date: 2008/12/07
17:32:29 $
(XEN) [2012-06-18 20:13:38] stdvga.c:147:d3 entering stdvga and caching modes
(XEN) [2012-06-18 20:13:38] HVM3: VGABios $Id: vgabios.c,v 1.67
2008/01/27 09:44:12 vruppert Exp $
(XEN) [2012-06-18 20:13:38] HVM3: Bochs BIOS - build: 06/23/99
(XEN) [2012-06-18 20:13:38] HVM3: $Revision: 1.221 $ $Date: 2008/12/07
17:32:29 $
(XEN) [2012-06-18 20:13:38] HVM3: Options: apmbios pcibios eltorito PMM
(XEN) [2012-06-18 20:13:38] HVM3:
(XEN) [2012-06-18 20:13:38] HVM3: ata0-0: PCHS=16383/16/63
translation=lba LCHS=1024/255/63
(XEN) [2012-06-18 20:13:38] HVM3: ata0 master: QEMU HARDDISK ATA-7
Hard-Disk (30720 MBytes)
(XEN) [2012-06-18 20:13:40] HVM3: IDE time out
(XEN) [2012-06-18 20:13:40] HVM3: ata1 master: QEMU DVD-ROM ATAPI-4
CD-Rom/DVD-Rom
(XEN) [2012-06-18 20:13:41] HVM3: IDE time out
(XEN) [2012-06-18 20:13:41] HVM3:
(XEN) [2012-06-18 20:13:41] HVM3:
(XEN) [2012-06-18 20:13:41] HVM3:
(XEN) [2012-06-18 20:13:41] HVM3: Press F12 for boot menu.
(XEN) [2012-06-18 20:13:41] HVM3:
(XEN) [2012-06-18 20:13:41] HVM3: Booting from Hard Disk...
(XEN) [2012-06-18 20:13:41] HVM3: Booting from 0000:7c00
(XEN) [2012-06-18 20:13:43] stdvga.c:151:d3 leaving stdvga
(XEN) [2012-06-18 20:13:55] irq.c:350: Dom3 callback via changed to
Direct Vector 0xf3
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3040 mfn=f7d00 nr=10
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3055 mfn=f7d18 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3056 mfn=f7d17 nr=1
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3050 mfn=f7d10 nr=4
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:remove: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:remove: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=e0000 mfn=e0000 nr=10000
(XEN) [2012-06-18 20:13:55] memory_map:add: dom3 gfn=f3000 mfn=f7c20 nr=20
(XEN) [2012-06-18 20:13:55] ioport_map:add: dom3 gport=c200 mport=e000 nr=100
(XEN) [2012-06-18 20:13:55] irq.c:270: Dom3 PCI link 0 changed 5 -> 0
(XEN) [2012-06-18 20:13:55] irq.c:270: Dom3 PCI link 1 changed 10 -> 0
(XEN) [2012-06-18 20:13:55] irq.c:270: Dom3 PCI link 2 changed 11 -> 0
(XEN) [2012-06-18 20:13:55] irq.c:270: Dom3 PCI link 3 changed 5 -> 0
(XEN) [2012-06-18 20:13:56] irq.c:2234: dom3: pirq 49 or emuirq 23
already mapped
(XEN) [2012-06-18 20:13:56] vmsi.c:108:d3 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:02] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d0 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d0 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:21] vmsi.c:108:d0 Unsupported delivery mode 3
(XEN) [2012-06-18 20:14:51] vmsi.c:108:d32767 Unsupported delivery mode 3
(XEN) [2012-06-18 20:24:36] vmsi.c:108:d32767 Unsupported delivery mode 3

----- Xen dmesg when starting dom0:

 __  __            _  _    ____                     _        _     _
 \ \/ /___ _ __   | || |  |___ \    _   _ _ __  ___| |_ __ _| |__ | | ___
  \  // _ \ ''_ \  | || |_   __) |__| | | | ''_ \/ __| __/ _` |
''_ \| |/ _ \
  /  \  __/ | | | |__   _| / __/|__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_|    |_|(_)_____|   \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

(XEN) Xen version 4.2-unstable (root@roce.org) (gcc version 4.6.3
(Ubuntu/Linaro 4.6.3-1ubuntu5) ) Fri Jun 15 05:32:29 CEST 2012
(XEN) Latest ChangeSet: Thu Jun 14 16:05:42 2012 +0100 25483:9dffb1864f2c
(XEN) Bootloader: GRUB 1.99-21ubuntu3.1
(XEN) Command line: placeholder xsave=0 loglvl=all guest_loglvl=all
console_timestamps=1 noreboot com1=115200,8n1,0x3f8,4 console=com1,vga
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 3 MBR signatures
(XEN)  Found 3 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040004000 (usable)
(XEN)  0000000040004000 - 0000000040005000 (reserved)
(XEN)  0000000040005000 - 00000000bd8a7000 (usable)
(XEN)  00000000bd8a7000 - 00000000bde1f000 (reserved)
(XEN)  00000000bde1f000 - 00000000be09f000 (ACPI NVS)
(XEN)  00000000be09f000 - 00000000be0a4000 (ACPI data)
(XEN)  00000000be0a4000 - 00000000be0e7000 (ACPI NVS)
(XEN)  00000000be0e7000 - 00000000beb81000 (usable)
(XEN)  00000000beb81000 - 00000000beff2000 (reserved)
(XEN)  00000000beff2000 - 00000000bf000000 (usable)
(XEN)  00000000bf800000 - 00000000cfa00000 (reserved)
(XEN)  00000000f8000000 - 00000000fc000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000042f600000 (usable)
(XEN) ACPI: RSDP 000F0450, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT BE084080, 0084 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP BE08E218, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DSDT BE0841A0, A077 (r2 ALASKA    A M I       14 INTL 20051117)
(XEN) ACPI: FACS BE09DF80, 0040
(XEN) ACPI: APIC BE08E310, 0072 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: ASF! BE08E388, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) ACPI: MCFG BE08E430, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: SSDT BE08E470, 04A6 (r1 Intel_ AoacTabl     1000 INTL 20091112)
(XEN) ACPI: AAFT BE08E918, 00C2 (r1 ALASKA OEMAAFT   1072009 MSFT       97)
(XEN) ACPI: HPET BE08E9E0, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT BE08EA18, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) ACPI: SSDT BE08ED88, 09AA (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT BE08F738, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) ACPI: DMAR BE0901D0, 00B8 (r1 INTEL      SNB         1 INTL        1)
(XEN) ACPI: BGRT BE090288, 003C (r0 ALASKA    A M I  1072009 AMI     10013)
(XEN) System RAM: 16086MB (16473004kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-000000042f600000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fceb0
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[404,0], pm1x_evt[400,0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT -
be09df80/0000000000000000, using 32
(XEN) ACPI:                  wakeup_vec[be09df8c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) Processor #4 7:10 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) Processor #6 7:10 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) Table is not found!
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 4 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 760 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3300.119 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:1239: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank
0 extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC deadline timer enabled
(XEN) [2012-06-18 20:50:08] Platform timer is 14.318MHz HPET
(XEN) [2012-06-18 20:50:08] Allocated console ring of 32 KiB.
(XEN) [2012-06-18 20:50:08] VMX: Supported advanced features:
(XEN) [2012-06-18 20:50:08]  - APIC MMIO access virtualisation
(XEN) [2012-06-18 20:50:08]  - APIC TPR shadow
(XEN) [2012-06-18 20:50:08]  - Extended Page Tables (EPT)
(XEN) [2012-06-18 20:50:08]  - Virtual-Processor Identifiers (VPID)
(XEN) [2012-06-18 20:50:08]  - Virtual NMI
(XEN) [2012-06-18 20:50:08]  - MSR direct-access bitmap
(XEN) [2012-06-18 20:50:08]  - Unrestricted Guest
(XEN) [2012-06-18 20:50:08] HVM: ASIDs enabled.
(XEN) [2012-06-18 20:50:08] HVM: VMX enabled
(XEN) [2012-06-18 20:50:08] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [2012-06-18 20:50:08] HVM: HAP page sizes: 4kB, 2MB
(XEN) [2012-06-18 20:50:08] Brought up 4 CPUs
(XEN) [2012-06-18 20:50:08] ACPI sleep modes: S3
(XEN) [2012-06-18 20:50:08] mcheck_poll: Machine check polling timer started.
(XEN) [2012-06-18 20:50:08] *** LOADING DOMAIN 0 ***
(XEN) [2012-06-18 20:50:08] elf_parse_binary: phdr: paddr=0x1000000
memsz=0xad7000
(XEN) [2012-06-18 20:50:08] elf_parse_binary: phdr: paddr=0x1c00000
memsz=0xd90e8
(XEN) [2012-06-18 20:50:08] elf_parse_binary: phdr: paddr=0x1cda000
memsz=0x14480
(XEN) [2012-06-18 20:50:08] elf_parse_binary: phdr: paddr=0x1cef000
memsz=0x656000
(XEN) [2012-06-18 20:50:08] elf_parse_binary: memory: 0x1000000 -> 0x2345000
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: GUEST_OS = "linux"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: XEN_VERSION =
"xen-3.0"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: ENTRY = 0xffffffff81cef200
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: HYPERCALL_PAGE
0xffffffff81001000
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: FEATURES
"!writable_page_tables|pae_pgdir_above_4gb"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: PAE_MODE = "yes"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: LOADER = "generic"
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: HV_START_LOW 0xffff800000000000
(XEN) [2012-06-18 20:50:08] elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) [2012-06-18 20:50:08] elf_xen_addr_calc_check: addresses:
(XEN) [2012-06-18 20:50:08]     virt_base        = 0xffffffff80000000
(XEN) [2012-06-18 20:50:08]     elf_paddr_offset = 0x0
(XEN) [2012-06-18 20:50:08]     virt_offset      = 0xffffffff80000000
(XEN) [2012-06-18 20:50:08]     virt_kstart      = 0xffffffff81000000
(XEN) [2012-06-18 20:50:08]     virt_kend        = 0xffffffff82345000
(XEN) [2012-06-18 20:50:08]     virt_entry       = 0xffffffff81cef200
(XEN) [2012-06-18 20:50:08]     p2m_base         = 0xffffffffffffffff
(XEN) [2012-06-18 20:50:08]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [2012-06-18 20:50:08]  Dom0 kernel: 64-bit, PAE, lsb, paddr
0x1000000 -> 0x2345000
(XEN) [2012-06-18 20:50:08] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [2012-06-18 20:50:08]  Dom0 alloc.:
0000000418000000->0000000420000000 (3987939 pages to be allocated)
(XEN) [2012-06-18 20:50:08]  Init. ramdisk:
000000042cff7000->000000042f5ffa00
(XEN) [2012-06-18 20:50:08] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [2012-06-18 20:50:08]  Loaded kernel:
ffffffff81000000->ffffffff82345000
(XEN) [2012-06-18 20:50:08]  Init. ramdisk:
ffffffff82345000->ffffffff8494da00
(XEN) [2012-06-18 20:50:08]  Phys-Mach map:
ffffffff8494e000->ffffffff8680df60
(XEN) [2012-06-18 20:50:08]  Start info:   
ffffffff8680e000->ffffffff8680e4b4
(XEN) [2012-06-18 20:50:08]  Page tables:  
ffffffff8680f000->ffffffff86848000
(XEN) [2012-06-18 20:50:08]  Boot stack:   
ffffffff86848000->ffffffff86849000
(XEN) [2012-06-18 20:50:08]  TOTAL:        
ffffffff80000000->ffffffff86c00000
(XEN) [2012-06-18 20:50:08]  ENTRY ADDRESS: ffffffff81cef200
(XEN) [2012-06-18 20:50:08] Dom0 has maximum 4 VCPUs
(XEN) [2012-06-18 20:50:08] elf_load_binary: phdr 0 at
0xffffffff81000000 -> 0xffffffff81ad7000
(XEN) [2012-06-18 20:50:08] elf_load_binary: phdr 1 at
0xffffffff81c00000 -> 0xffffffff81cd90e8
(XEN) [2012-06-18 20:50:08] elf_load_binary: phdr 2 at
0xffffffff81cda000 -> 0xffffffff81cee480
(XEN) [2012-06-18 20:50:08] elf_load_binary: phdr 3 at
0xffffffff81cef000 -> 0xffffffff81dc6000
(XEN) [2012-06-18 20:50:11] Scrubbing Free RAM: .done.
(XEN) [2012-06-18 20:50:11] Initial low memory virq threshold set at
0x4000 pages.
(XEN) [2012-06-18 20:50:11] Std. Loglevel: All
(XEN) [2012-06-18 20:50:11] Guest Loglevel: All
(XEN) [2012-06-18 20:50:11] Xen is relinquishing VGA console.
(XEN) [2012-06-18 20:50:11] *** Serial input -> DOM0 (type
''CTRL-a''
three times to switch input to Xen)
(XEN) [2012-06-18 20:50:11] Freed 244kB init memory.
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:01.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:02.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:14.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:16.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1a.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1b.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.4
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.5
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.6
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1c.7
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1d.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1f.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1f.2
(XEN) [2012-06-18 20:50:12] PCI add device 0000:00:1f.3
(XEN) [2012-06-18 20:50:12] PCI add device 0000:01:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:01:00.1
(XEN) [2012-06-18 20:50:12] PCI add device 0000:03:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:04:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:05:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:06:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:07:01.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:07:02.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:07:03.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:0a:00.0
(XEN) [2012-06-18 20:50:12] PCI add device 0000:0b:02.0
(XEN) [2012-06-18 20:50:41] no cpu_id for acpi_id 5
(XEN) [2012-06-18 20:50:41] no cpu_id for acpi_id 6
(XEN) [2012-06-18 20:50:41] no cpu_id for acpi_id 7
(XEN) [2012-06-18 20:50:41] no cpu_id for acpi_id 8

Jan Beulich

2012-Jun-19 09:45 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

>>> On 18.06.12 at 22:57, Rolu <rolu@roce.org> wrote:
> I have a HVM domU running Ubuntu 12.04. It has several PCI devices
> passed through to it (USB controllers, onboard sound, ATI video card
> with gfx_passthru 0). It has some issues, the most apparent one with
> the sound, which doesn''t play normally but repeats the same short
> fragment over and over for a while until moving on to the next
> fragment. The Xen dmesg prints lots of lines like:
> 
> (XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode 3
Which indicates something wrong in the guest kernel or in the
translation layers (qemu, less likely the hypervisor itself). Valid
MSIs can''t have a delivery mode of 3 (a reserved encoding).
> The number behind 108:d varies occasionally.
Yes, that number is bogus (has no guaranteed relation to the
target domain).
> Since vmsi.c seems to deal with passing on MSI interrupts, I tried
> booting the domU with pci=nomsi. This made things work and sound now
> plays normally. Not using MSI at all doesn''t seem like a very good
> solution though.
> 
> What makes this happen, and can I fix it in a better way?
Unless this is in the guest kernel, we''ll likely need a code fix here,
but for determining what and where, we''d need you to provide
the qemu log for the domain as well (there ought to be entries
starting with "Update msi with pirq", and the gflags value is of
particular interest). Depending on what we see, we may then
need you to do some further debugging.

Jan

Rolu

2012-Jun-20 13:58 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Tue, Jun 19, 2012 at 11:45 AM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 18.06.12 at 22:57, Rolu <rolu@roce.org> wrote:
>> I have a HVM domU running Ubuntu 12.04. It has several PCI devices
>> passed through to it (USB controllers, onboard sound, ATI video card
>> with gfx_passthru 0). It has some issues, the most apparent one with
>> the sound, which doesn''t play normally but repeats the same
short
>> fragment over and over for a while until moving on to the next
>> fragment. The Xen dmesg prints lots of lines like:
>>
>> (XEN) [2012-06-18 20:14:16] vmsi.c:108:d32767 Unsupported delivery mode
3
>
> Which indicates something wrong in the guest kernel or in the
> translation layers (qemu, less likely the hypervisor itself). Valid
> MSIs can''t have a delivery mode of 3 (a reserved encoding).
>
>> The number behind 108:d varies occasionally.
>
> Yes, that number is bogus (has no guaranteed relation to the
> target domain).
>
>> Since vmsi.c seems to deal with passing on MSI interrupts, I tried
>> booting the domU with pci=nomsi. This made things work and sound now
>> plays normally. Not using MSI at all doesn''t seem like a very
good
>> solution though.
>>
>> What makes this happen, and can I fix it in a better way?
>
> Unless this is in the guest kernel, we''ll likely need a code fix
here,
> but for determining what and where, we''d need you to provide
> the qemu log for the domain as well (there ought to be entries
> starting with "Update msi with pirq", and the gflags value is of
> particular interest). Depending on what we see, we may then
> need you to do some further debugging.
>
There are, these: (I''ve copied the full logs below)

pt_msi_update: Update msi with pirq 37 gvec 0 gflags 3031
pt_msi_update: Update msi with pirq 36 gvec 0 gflags 3030
pt_msi_update: Update msi with pirq 35 gvec 0 gflags 302f

To produce the logs I started the domU and let it run until it
displayed the graphical login screen, which plays a short sound.

The qemu logs give several errors and warnings, such as (there are
multiple of each of these):

pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_pci_write_config: [00:05:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]

Are these related, and/or cause for worry? I''ve looked around some but
apart from the fact that they have to do with PCI it doesn''t tell me
much. They occur no matter whether I use pci=nomsi or not.

The qemu log, without pci=nomsi set, sound stutters:

domid: 3
-videoram option does not work with cirrus vga device model. Videoram set to 4M.
Strip off blktap sub-type prefix to /panda/ubuntuhv2.img (drv
''aio'')
Using file /panda/ubuntuhv2.img in read-write mode
Strip off blktap sub-type prefix to
/panda/ubuntu-12.04-desktop-amd64.iso (drv ''aio'')
Using file /panda/ubuntu-12.04-desktop-amd64.iso in read-only mode
Watching /local/domain/0/device-model/3/logdirty/cmd
Watching /local/domain/0/device-model/3/command
Watching /local/domain/3/cpu
char device redirected to /dev/pts/2
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = bfd25022-34fc-43e9-82ed-70fb4fdbd7bd
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/3/xen_extended_power_mgmt): read error
medium change watch on `hdc'' (index: 1):
aio:/panda/ubuntu-12.04-desktop-amd64.iso
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
Log-dirty: no command yet.
vcpu-set: watch node error.
xs_read(/local/domain/3/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/3/log-throttling''
medium change watch on `/local/domain/3/log-throttling'' - unknown
device, ignored
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:14.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x14.0x0
pt_register_regions: IO region registered (size=0x00010000 base_addr=0xf7d00004)
pt_msi_setup: msi mapped with pirq 37
pci_intx: intx=1
register_real_device: Real physical device 00:14.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1a.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1a.0x0
pt_register_regions: IO region registered (size=0x00000400 base_addr=0xf7d18000)
pci_intx: intx=1
register_real_device: Real physical device 00:1a.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1d.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1d.0x0
pt_register_regions: IO region registered (size=0x00000400 base_addr=0xf7d17000)
pci_intx: intx=1
register_real_device: Real physical device 00:1d.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1b.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1b.0x0
pt_register_regions: IO region registered (size=0x00004000 base_addr=0xf7d10004)
pt_msi_setup: msi mapped with pirq 36
pci_intx: intx=1
register_real_device: Real physical device 00:1b.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x10000000 base_addr=0xe000000c)
pt_register_regions: IO region registered (size=0x00020000 base_addr=0xf7c20004)
pt_register_regions: IO region registered (size=0x00000100 base_addr=0x0000e001)
pt_register_regions: Expansion ROM registered (size=0x00020000
base_addr=0xf7c00000)
pt_msi_setup: msi mapped with pirq 35
pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!
IRQ type = MSI-INTx
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=1
cirrus vga map change while on lfb mode
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=1
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=1
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=1
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=1
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=1
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=1
mapping vram to f0000000 - f0400000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
Unknown PV product 3 loaded in guest
PV driver build 1
region type 1 at [c100,c200).
region type 0 at [f3057000,f3057100).
squash iomem [f3057000, f3057100).
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_pci_write_config: [00:05:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_pci_write_config: [00:06:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_pci_write_config: [00:07:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_pci_write_config: [00:08:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 37 gvec 0 gflags 3031
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 36 gvec 0 gflags 3030
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_update: Update msi with pirq 35 gvec 0 gflags 302f



The qemu log, with pci=nomsi set, sound is okay:

domid: 2
-videoram option does not work with cirrus vga device model. Videoram set to 4M.
Strip off blktap sub-type prefix to /panda/ubuntuhv2.img (drv
''aio'')
Using file /panda/ubuntuhv2.img in read-write mode
Strip off blktap sub-type prefix to
/panda/ubuntu-12.04-desktop-amd64.iso (drv ''aio'')
Using file /panda/ubuntu-12.04-desktop-amd64.iso in read-only mode
Watching /local/domain/0/device-model/2/logdirty/cmd
Watching /local/domain/0/device-model/2/command
Watching /local/domain/2/cpu
char device redirected to /dev/pts/2
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = 376e7089-c20e-40d9-8e93-c7e4bec2e72d
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/2/xen_extended_power_mgmt): read error
medium change watch on `hdc'' (index: 1):
aio:/panda/ubuntu-12.04-desktop-amd64.iso
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
Log-dirty: no command yet.
vcpu-set: watch node error.
xs_read(/local/domain/2/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/2/log-throttling''
medium change watch on `/local/domain/2/log-throttling'' - unknown
device, ignored
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:14.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x14.0x0
pt_register_regions: IO region registered (size=0x00010000 base_addr=0xf7d00004)
pt_msi_setup: msi mapped with pirq 37
pci_intx: intx=1
register_real_device: Real physical device 00:14.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1a.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1a.0x0
pt_register_regions: IO region registered (size=0x00000400 base_addr=0xf7d18000)
pci_intx: intx=1
register_real_device: Real physical device 00:1a.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1d.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1d.0x0
pt_register_regions: IO region registered (size=0x00000400 base_addr=0xf7d17000)
pci_intx: intx=1
register_real_device: Real physical device 00:1d.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 00:1b.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x0:0x1b.0x0
pt_register_regions: IO region registered (size=0x00004000 base_addr=0xf7d10004)
pt_msi_setup: msi mapped with pirq 36
pci_intx: intx=1
register_real_device: Real physical device 00:1b.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 01:00.0 ...
pt_iomul_init: Error: pt_iomul_init can''t open file
/dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x10000000 base_addr=0xe000000c)
pt_register_regions: IO region registered (size=0x00020000 base_addr=0xf7c20004)
pt_register_regions: IO region registered (size=0x00000100 base_addr=0x0000e001)
pt_register_regions: Expansion ROM registered (size=0x00020000
base_addr=0xf7c00000)
pt_msi_setup: msi mapped with pirq 35
pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!
IRQ type = MSI-INTx
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=1
cirrus vga map change while on lfb mode
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=1
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=1
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=1
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=1
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=1
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=1
mapping vram to f0000000 - f0400000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
Unknown PV product 3 loaded in guest
PV driver build 1
region type 1 at [c100,c200).
region type 0 at [f3057000,f3057100).
squash iomem [f3057000, f3057100).
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_pci_write_config: [00:05:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3040000 maddr=f7d00000 type=0 len=65536 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_pci_write_config: [00:06:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3055000 maddr=f7d18000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_pci_write_config: [00:07:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3056000 maddr=f7d17000 type=0 len=4096 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_pci_write_config: [00:08:0] Warning: Guest attempt to set address
to unused Base Address Register. [Offset:30h][Length:4]
pt_iomem_map: e_phys=f3050000 maddr=f7d10000 type=0 len=16384 index=0
first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=ffff pio_base=e000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456
index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f7c20000 type=0 len=131072 index=2
first_map=0
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=0
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]
pt_pci_read_config: [00:10:0] Error: Failed to read register with
offset exceeding FFh. [Offset:ffh][Length:1]

Jan Beulich

2012-Jun-20 16:03 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

>>> On 20.06.12 at 15:58, Rolu <rolu@roce.org> wrote:
> On Tue, Jun 19, 2012 at 11:45 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>> Unless this is in the guest kernel, we''ll likely need a code
fix here,
>> but for determining what and where, we''d need you to provide
>> the qemu log for the domain as well (there ought to be entries
>> starting with "Update msi with pirq", and the gflags value is
of
>> particular interest). Depending on what we see, we may then
>> need you to do some further debugging.
>>
> 
> There are, these: (I''ve copied the full logs below)
> 
> pt_msi_update: Update msi with pirq 37 gvec 0 gflags 3031
> pt_msi_update: Update msi with pirq 36 gvec 0 gflags 3030
> pt_msi_update: Update msi with pirq 35 gvec 0 gflags 302f
Okay, so at that point the bad value is already there. I''d
suggest taking it up the usage chain, so adding some logging
in pt_msgdata_reg_write() (where the original value -
ptdev->msi->data - is being computed) would likely be a
good first step.

At the same time, adding logging to the guest kernel would
be nice, to see what value it actually writes (in a current
kernel this would be in __write_msi_msg()).
> The qemu logs give several errors and warnings, such as (there are
> multiple of each of these):
> 
> pt_iomul_init: Error: pt_iomul_init can''t open file
> /dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
> pt_pci_write_config: [00:05:0] Warning: Guest attempt to set address
> to unused Base Address Register. [Offset:30h][Length:4]
> pt_pci_read_config: [00:10:0] Error: Failed to read register with
> offset exceeding FFh. [Offset:ffh][Length:1]
> 
> Are these related, and/or cause for worry? I''ve looked around some
but
> apart from the fact that they have to do with PCI it doesn''t tell
me
> much. They occur no matter whether I use pci=nomsi or not.
I don''t know.

Jan

Rolu

2012-Jun-24 02:21 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com>
wrote:>>>> On 20.06.12 at 15:58, Rolu <rolu@roce.org> wrote:
>> On Tue, Jun 19, 2012 at 11:45 AM, Jan Beulich <JBeulich@suse.com>
wrote:
>>> Unless this is in the guest kernel, we''ll likely need a
code fix here,
>>> but for determining what and where, we''d need you to
provide
>>> the qemu log for the domain as well (there ought to be entries
>>> starting with "Update msi with pirq", and the gflags
value is of
>>> particular interest). Depending on what we see, we may then
>>> need you to do some further debugging.
>>>
>>
>> There are, these: (I''ve copied the full logs below)
>>
>> pt_msi_update: Update msi with pirq 37 gvec 0 gflags 3031
>> pt_msi_update: Update msi with pirq 36 gvec 0 gflags 3030
>> pt_msi_update: Update msi with pirq 35 gvec 0 gflags 302f
>
> Okay, so at that point the bad value is already there. I''d
> suggest taking it up the usage chain, so adding some logging
> in pt_msgdata_reg_write() (where the original value -
> ptdev->msi->data - is being computed) would likely be a
> good first step.
>
This took a while as I''m not very familiar with it all yet.

The write function gets passed a *value of 0x4300 and that is also
what gets assigned to ptdev->msi->data. The delivery mode is bit 8 to
10 of that value, which is indeed 3. lspci -vvv on the domU also shows
that the MSI data field is 4300, but I suppose it just reads that back
from qemu. Anyway, this seems in order, at least as far as I can see.
> At the same time, adding logging to the guest kernel would
> be nice, to see what value it actually writes (in a current
> kernel this would be in __write_msi_msg()).
>
Turns out that msg->data here is also 0x4300, so it seems the guest
kernel is producing these values. I caused it to make a stack trace
and this pointed back to xen_hvm_setup_msi_irqs. This function uses
the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
current data field and if it isn''t equal to the macro it uses
xen_msi_compose_msg to make a new message, but that function just sets
the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
then gets passed to __write_msi_msg and that''s that. There are no
other writes through __write_msi_msg (except for the same thing for
other devices).

The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
decoded as the delivery mode, so it seems the kernel is intentionally
setting it to 3.

Jan Beulich

2012-Jun-25 07:49 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

>>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
> On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com>
wrote:
>> At the same time, adding logging to the guest kernel would
>> be nice, to see what value it actually writes (in a current
>> kernel this would be in __write_msi_msg()).
>>
> 
> Turns out that msg->data here is also 0x4300, so it seems the guest
> kernel is producing these values. I caused it to make a stack trace
> and this pointed back to xen_hvm_setup_msi_irqs. This function uses
> the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
> current data field and if it isn''t equal to the macro it uses
> xen_msi_compose_msg to make a new message, but that function just sets
> the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
> then gets passed to __write_msi_msg and that''s that. There are no
> other writes through __write_msi_msg (except for the same thing for
> other devices).
> 
> The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends up
> decoded as the delivery mode, so it seems the kernel is intentionally
> setting it to 3.
So that can never have worked properly afaict. Stefano, the
code as it is currently - using literal (3 << 8) - is clearly bogus.
Your original commit at least had a comment saying that the
reserved delivery mode encoding is intentional here, but that
comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
In any case - the cooperation with qemu apparently doesn''t
work, as the reserved encoding should never make it through
to the hypervisor. Could you explain what the intention here
was?

And regardless of anything, can the literal numbers please be
replaced by proper manifest constants - the "8" here already
has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
proper symbolic would permit locating where this is being (or
really, as it doesn''t appear to work supposed to be) consumed
in qemu, provided it uses the same definition (i.e. that one
should go into one of the public headers).

Jan

Stefano Stabellini

2012-Jun-25 11:38 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Mon, 25 Jun 2012, Jan Beulich wrote:> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich <JBeulich@suse.com>
wrote:
> >> At the same time, adding logging to the guest kernel would
> >> be nice, to see what value it actually writes (in a current
> >> kernel this would be in __write_msi_msg()).
> >>
> > 
> > Turns out that msg->data here is also 0x4300, so it seems the guest
> > kernel is producing these values. I caused it to make a stack trace
> > and this pointed back to xen_hvm_setup_msi_irqs. This function uses
> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks the
> > current data field and if it isn''t equal to the macro it uses
> > xen_msi_compose_msg to make a new message, but that function just sets
> > the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300. This
> > then gets passed to __write_msi_msg and that''s that. There
are no
> > other writes through __write_msi_msg (except for the same thing for
> > other devices).
> > 
> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which ends
up
> > decoded as the delivery mode, so it seems the kernel is intentionally
> > setting it to 3.
> 
> So that can never have worked properly afaict. Stefano, the
> code as it is currently - using literal (3 << 8) - is clearly bogus.
> Your original commit at least had a comment saying that the
> reserved delivery mode encoding is intentional here, but that
> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
> In any case - the cooperation with qemu apparently doesn''t
> work, as the reserved encoding should never make it through
> to the hypervisor. Could you explain what the intention here
> was?
> 
> And regardless of anything, can the literal numbers please be
> replaced by proper manifest constants - the "8" here already
> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
> proper symbolic would permit locating where this is being (or
> really, as it doesn''t appear to work supposed to be) consumed
> in qemu, provided it uses the same definition (i.e. that one
> should go into one of the public headers).
The (3 << 8) is unimportant. The delivery mode chosen is
"reserved"
because notifications are not supposed to be delivered as MSI anymore.

This is what should happen:

1) Linux configures the device with a 0 vector number and the pirq number
in the address field;

2) QEMU notices a vector number of 0 and reads the pirq number from the
address field, passing it to xc_domain_update_msi_irq;

3) Xen assignes the given pirq to the physical MSI;

4) The guest issues a EVTCHNOP_bind_pirq hypercall;

5) Xen sets the pirq as "IRQ_PT";

6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
returns true so Xen calls send_guest_pirq instead.


Obviously 6) is not happening. hvm_domain_use_pirq is:

is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq !=
IRQ_UNBOUND

My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
above).
So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
(__startup_pirq doesn''t get called), or Xen is erroring out in
map_domain_emuirq_pirq.

Rolu

2012-Jun-26 02:31 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:> On Mon, 25 Jun 2012, Jan Beulich wrote:
>> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org> wrote:
>> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich
<JBeulich@suse.com> wrote:
>> >> At the same time, adding logging to the guest kernel would
>> >> be nice, to see what value it actually writes (in a current
>> >> kernel this would be in __write_msi_msg()).
>> >>
>> >
>> > Turns out that msg->data here is also 0x4300, so it seems the
guest
>> > kernel is producing these values. I caused it to make a stack
trace
>> > and this pointed back to xen_hvm_setup_msi_irqs. This function
uses
>> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It checks
the
>> > current data field and if it isn''t equal to the macro it
uses
>> > xen_msi_compose_msg to make a new message, but that function just
sets
>> > the data field of the message to XEN_PIRQ_MSI_DATA - so, 0x4300.
This
>> > then gets passed to __write_msi_msg and that''s that.
There are no
>> > other writes through __write_msi_msg (except for the same thing
for
>> > other devices).
>> >
>> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8) which
ends up
>> > decoded as the delivery mode, so it seems the kernel is
intentionally
>> > setting it to 3.
>>
>> So that can never have worked properly afaict. Stefano, the
>> code as it is currently - using literal (3 << 8) - is clearly
bogus.
>> Your original commit at least had a comment saying that the
>> reserved delivery mode encoding is intentional here, but that
>> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
>> In any case - the cooperation with qemu apparently doesn''t
>> work, as the reserved encoding should never make it through
>> to the hypervisor. Could you explain what the intention here
>> was?
>>
>> And regardless of anything, can the literal numbers please be
>> replaced by proper manifest constants - the "8" here already
>> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
>> proper symbolic would permit locating where this is being (or
>> really, as it doesn''t appear to work supposed to be) consumed
>> in qemu, provided it uses the same definition (i.e. that one
>> should go into one of the public headers).
>
> The (3 << 8) is unimportant. The delivery mode chosen is
"reserved"
> because notifications are not supposed to be delivered as MSI anymore.
>
> This is what should happen:
>
> 1) Linux configures the device with a 0 vector number and the pirq number
> in the address field;
>
> 2) QEMU notices a vector number of 0 and reads the pirq number from the
> address field, passing it to xc_domain_update_msi_irq;
>
> 3) Xen assignes the given pirq to the physical MSI;
>
> 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
>
> 5) Xen sets the pirq as "IRQ_PT";
>
> 6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
> returns true so Xen calls send_guest_pirq instead.
>
>
> Obviously 6) is not happening. hvm_domain_use_pirq is:
>
> is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq !=
IRQ_UNBOUND
>
> My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
> above).
This appears to be true. I added logging to hvm_pci_msi_assert in
xen/drivers/passthrough/io.c and it indicates that
pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
before an unsupported delivery mode message.

I also log pirq->pirq but I found that most of the time I can''t find
this value anywhere else (I''m not sure how to interpret the value,
though). For example, in my last try:

* I get an unsupported delivery mode error for pirq->pirq 55, 54 and
53. The vast majority are for 54.
* I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
but complains it''s already mapped.
* I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
22, 52, 48, 47. Also never 54 or 53.
* map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17, 55.
* The qemu log mentions pirq 35, 36 and 37

It seems pirq values don''t always mean the same? Is it a coincidence
that 55 occurs almost everywhere, or is something going wrong with the
other two values (53 and 54 versus 16 and 17)?

I have three MSI capable devices passed through to the domU, and I do
see groups of three distinct pirqs in the data above - just not the
same ones in every place I look.
> So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
> (__startup_pirq doesn''t get called), or Xen is erroring out in
> map_domain_emuirq_pirq.
evtchn_bind_pirq gets called, though I''m not sure if it is with the
right data.

map_domain_emuirq_pirq always gets past the checks in the top half
(i.e. up to the line /* do not store emuirq mappings for pt devices
*/), except for one time with pirq=49,emuirq=23 where it finds they
are already mapped.
It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
This implies their info->arch.hvm.emuirq is also set to -2 (haven''t
directly logged that but it''s the only assignment there).

Interestingly, I get an unsupported delivery mode error for pirq 55
where my logging says pirq->arch.hvm.emuirq is -1, *after*
map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.

Jan Beulich

2012-Jun-26 07:42 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

>>> On 26.06.12 at 04:31, Rolu <rolu@roce.org> wrote:
> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
>> This is what should happen:
>>
>> 1) Linux configures the device with a 0 vector number and the pirq
number
>> in the address field;
>>
>> 2) QEMU notices a vector number of 0 and reads the pirq number from the
>> address field, passing it to xc_domain_update_msi_irq;
>>
>> 3) Xen assignes the given pirq to the physical MSI;
>>
>> 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
>>
>> 5) Xen sets the pirq as "IRQ_PT";
>>
>> 6) When Xen tries to inject the MSI into the guest, hvm_domain_use_pirq
>> returns true so Xen calls send_guest_pirq instead.
>>
>>
>> Obviously 6) is not happening. hvm_domain_use_pirq is:
>>
>> is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq !=
IRQ_UNBOUND
>>
>> My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
>> above).
> 
> This appears to be true. I added logging to hvm_pci_msi_assert in
> xen/drivers/passthrough/io.c and it indicates that
> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
> before an unsupported delivery mode message.
> 
> I also log pirq->pirq but I found that most of the time I can''t
find
> this value anywhere else (I''m not sure how to interpret the value,
> though). For example, in my last try:
> 
> * I get an unsupported delivery mode error for pirq->pirq 55, 54 and
> 53. The vast majority are for 54.
> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
> Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
> but complains it''s already mapped.
> * I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
> 22, 52, 48, 47. Also never 54 or 53.
> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17, 
> 55.
> * The qemu log mentions pirq 35, 36 and 37
> 
> It seems pirq values don''t always mean the same? Is it a
coincidence
> that 55 occurs almost everywhere, or is something going wrong with the
> other two values (53 and 54 versus 16 and 17)?
> 
> I have three MSI capable devices passed through to the domU, and I do
> see groups of three distinct pirqs in the data above - just not the
> same ones in every place I look.
> 
>> So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
>> (__startup_pirq doesn''t get called), or Xen is erroring out in
>> map_domain_emuirq_pirq.
> 
> evtchn_bind_pirq gets called, though I''m not sure if it is with
the right
> data.
> 
> map_domain_emuirq_pirq always gets past the checks in the top half
> (i.e. up to the line /* do not store emuirq mappings for pt devices
> */), except for one time with pirq=49,emuirq=23 where it finds they
> are already mapped.
> It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
> This implies their info->arch.hvm.emuirq is also set to -2
(haven''t
> directly logged that but it''s the only assignment there).
> 
> Interestingly, I get an unsupported delivery mode error for pirq 55
> where my logging says pirq->arch.hvm.emuirq is -1, *after*
> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
Adding some logging at the kernel side would seem much more
promising to me based on Stefano''s analysis yesterday.

Jan

Stefano Stabellini

2012-Jun-26 12:59 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Tue, 26 Jun 2012, Rolu wrote:> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Mon, 25 Jun 2012, Jan Beulich wrote:
> >> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org>
wrote:
> >> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich
<JBeulich@suse.com> wrote:
> >> >> At the same time, adding logging to the guest kernel
would
> >> >> be nice, to see what value it actually writes (in a
current
> >> >> kernel this would be in __write_msi_msg()).
> >> >>
> >> >
> >> > Turns out that msg->data here is also 0x4300, so it seems
the guest
> >> > kernel is producing these values. I caused it to make a stack
trace
> >> > and this pointed back to xen_hvm_setup_msi_irqs. This
function uses
> >> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300. It
checks the
> >> > current data field and if it isn''t equal to the
macro it uses
> >> > xen_msi_compose_msg to make a new message, but that function
just sets
> >> > the data field of the message to XEN_PIRQ_MSI_DATA - so,
0x4300. This
> >> > then gets passed to __write_msi_msg and that''s that.
There are no
> >> > other writes through __write_msi_msg (except for the same
thing for
> >> > other devices).
> >> >
> >> > The macro XEN_PIRQ_MSI_DATA contains a part (3 << 8)
which ends up
> >> > decoded as the delivery mode, so it seems the kernel is
intentionally
> >> > setting it to 3.
> >>
> >> So that can never have worked properly afaict. Stefano, the
> >> code as it is currently - using literal (3 << 8) - is
clearly bogus.
> >> Your original commit at least had a comment saying that the
> >> reserved delivery mode encoding is intentional here, but that
> >> comment got lost with the later introduction of XEN_PIRQ_MSI_DATA.
> >> In any case - the cooperation with qemu apparently
doesn''t
> >> work, as the reserved encoding should never make it through
> >> to the hypervisor. Could you explain what the intention here
> >> was?
> >>
> >> And regardless of anything, can the literal numbers please be
> >> replaced by proper manifest constants - the "8" here
already
> >> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
> >> proper symbolic would permit locating where this is being (or
> >> really, as it doesn''t appear to work supposed to be)
consumed
> >> in qemu, provided it uses the same definition (i.e. that one
> >> should go into one of the public headers).
> >
> > The (3 << 8) is unimportant. The delivery mode chosen is
"reserved"
> > because notifications are not supposed to be delivered as MSI anymore.
> >
> > This is what should happen:
> >
> > 1) Linux configures the device with a 0 vector number and the pirq
number
> > in the address field;
> >
> > 2) QEMU notices a vector number of 0 and reads the pirq number from
the
> > address field, passing it to xc_domain_update_msi_irq;
> >
> > 3) Xen assignes the given pirq to the physical MSI;
> >
> > 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
> >
> > 5) Xen sets the pirq as "IRQ_PT";
> >
> > 6) When Xen tries to inject the MSI into the guest,
hvm_domain_use_pirq
> > returns true so Xen calls send_guest_pirq instead.
> >
> >
> > Obviously 6) is not happening. hvm_domain_use_pirq is:
> >
> > is_hvm_domain(d) && pirq && pirq->arch.hvm.emuirq
!= IRQ_UNBOUND
> >
> > My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT (see
> > above).
> 
> This appears to be true. I added logging to hvm_pci_msi_assert in
> xen/drivers/passthrough/io.c and it indicates that
> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
> before an unsupported delivery mode message.
> 
> I also log pirq->pirq but I found that most of the time I can''t
find
> this value anywhere else (I''m not sure how to interpret the value,
> though). For example, in my last try:
> 
> * I get an unsupported delivery mode error for pirq->pirq 55, 54 and
> 53. The vast majority are for 54.
> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
> Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
> but complains it''s already mapped.
> * I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
> 22, 52, 48, 47. Also never 54 or 53.
> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16, 17,
55.
> * The qemu log mentions pirq 35, 36 and 37
> 
> It seems pirq values don''t always mean the same? Is it a
coincidence
> that 55 occurs almost everywhere, or is something going wrong with the
> other two values (53 and 54 versus 16 and 17)?
> 
> I have three MSI capable devices passed through to the domU, and I do
> see groups of three distinct pirqs in the data above - just not the
> same ones in every place I look.
> 
> > So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
> > (__startup_pirq doesn''t get called), or Xen is erroring out
in
> > map_domain_emuirq_pirq.
> 
> evtchn_bind_pirq gets called, though I''m not sure if it is with
the right data.
> 
> map_domain_emuirq_pirq always gets past the checks in the top half
> (i.e. up to the line /* do not store emuirq mappings for pt devices
> */), except for one time with pirq=49,emuirq=23 where it finds they
> are already mapped.
> It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
> This implies their info->arch.hvm.emuirq is also set to -2
(haven''t
> directly logged that but it''s the only assignment there).
> 
> Interestingly, I get an unsupported delivery mode error for pirq 55
> where my logging says pirq->arch.hvm.emuirq is -1, *after*
> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
Looking back at your QEMU logs, it seems that pt_msi_setup is not
called (or it is not called at the right time), otherwise you should
get:

pt_msi_setup requested pirq = %d

in your logs.
Could you try disabling msitranslate? You can do that adding

pci_msitranslate=0

to your VM config file.
If that works, probably this (untested) QEMU patch could fix your problem:



diff --git a/hw/pt-msi.c b/hw/pt-msi.c
index 70c4023..09b1391 100644
--- a/hw/pt-msi.c
+++ b/hw/pt-msi.c
@@ -59,6 +59,26 @@ static void msix_set_enable(struct pt_dev *dev, int en)
 
 /* MSI virtuailization functions */
 
+int pt_msi_pirq(struct pt_dev *d, int *pirq)
+{
+    int gvec = d->msi->data & 0xFF;
+    if (gvec != 0) {
+        return -1;
+    }
+
+    /* if gvec is 0, the guest is asking for a particular pirq that
+     * is passed as dest_id */
+    *pirq = (dev->msi->addr_hi & 0xffffff00) |
+        ((dev->msi->addr_lo >> MSI_TARGET_CPU_SHIFT) & 0xff);
+    if (!pirq) {
+        /* this probably identifies an misconfiguration of the guest,
+         * try the emulated path */
+        *pirq = -1;
+        return -1;
+    } else {
+        PT_LOG("%s requested pirq = %d\n", __func__, *pirq);
+    }
+}
 /*
  * setup physical msi, but didn''t enable it
  */
@@ -74,18 +94,7 @@ int pt_msi_setup(struct pt_dev *dev)
     }
 
     gvec = dev->msi->data & 0xFF;
-    if (!gvec) {
-        /* if gvec is 0, the guest is asking for a particular pirq that
-         * is passed as dest_id */
-        pirq = (dev->msi->addr_hi & 0xffffff00) |
-               ((dev->msi->addr_lo >> MSI_TARGET_CPU_SHIFT) &
0xff);
-        if (!pirq)
-            /* this probably identifies an misconfiguration of the guest,
-             * try the emulated path */
-            pirq = -1;
-        else
-            PT_LOG("pt_msi_setup requested pirq = %d\n", pirq);
-    }
+    pt_msi_pirq(d, &pirq);
 
     if ( xc_physdev_map_pirq_msi(xc_handle, domid, AUTO_ASSIGN, &pirq,
                                  PCI_DEVFN(dev->pci_dev->dev,
@@ -138,6 +147,8 @@ int pt_msi_update(struct pt_dev *d)
     addr = (uint64_t)d->msi->addr_hi << 32 | d->msi->addr_lo;
     gflags = __get_msi_gflags(d->msi->data, addr);
 
+    pt_msi_pirq(d, &d->msi->pirq);
+
     PT_LOG("Update msi with pirq %x gvec %x gflags %x\n",
            d->msi->pirq, gvec, gflags);

Rolu

2012-Jun-26 21:01 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Tue, Jun 26, 2012 at 2:59 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:> On Tue, 26 Jun 2012, Rolu wrote:
>> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Mon, 25 Jun 2012, Jan Beulich wrote:
>> >> >>> On 24.06.12 at 04:21, Rolu <rolu@roce.org>
wrote:
>> >> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich
<JBeulich@suse.com> wrote:
>> >> >> At the same time, adding logging to the guest kernel
would
>> >> >> be nice, to see what value it actually writes (in a
current
>> >> >> kernel this would be in __write_msi_msg()).
>> >> >>
>> >> >
>> >> > Turns out that msg->data here is also 0x4300, so it
seems the guest
>> >> > kernel is producing these values. I caused it to make a
stack trace
>> >> > and this pointed back to xen_hvm_setup_msi_irqs. This
function uses
>> >> > the macro XEN_PIRQ_MSI_DATA, which evaluates to 0x4300.
It checks the
>> >> > current data field and if it isn''t equal to the
macro it uses
>> >> > xen_msi_compose_msg to make a new message, but that
function just sets
>> >> > the data field of the message to XEN_PIRQ_MSI_DATA - so,
0x4300. This
>> >> > then gets passed to __write_msi_msg and that''s
that. There are no
>> >> > other writes through __write_msi_msg (except for the same
thing for
>> >> > other devices).
>> >> >
>> >> > The macro XEN_PIRQ_MSI_DATA contains a part (3 <<
8) which ends up
>> >> > decoded as the delivery mode, so it seems the kernel is
intentionally
>> >> > setting it to 3.
>> >>
>> >> So that can never have worked properly afaict. Stefano, the
>> >> code as it is currently - using literal (3 << 8) - is
clearly bogus.
>> >> Your original commit at least had a comment saying that the
>> >> reserved delivery mode encoding is intentional here, but that
>> >> comment got lost with the later introduction of
XEN_PIRQ_MSI_DATA.
>> >> In any case - the cooperation with qemu apparently
doesn''t
>> >> work, as the reserved encoding should never make it through
>> >> to the hypervisor. Could you explain what the intention here
>> >> was?
>> >>
>> >> And regardless of anything, can the literal numbers please be
>> >> replaced by proper manifest constants - the "8" here
already
>> >> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
>> >> proper symbolic would permit locating where this is being (or
>> >> really, as it doesn''t appear to work supposed to be)
consumed
>> >> in qemu, provided it uses the same definition (i.e. that one
>> >> should go into one of the public headers).
>> >
>> > The (3 << 8) is unimportant. The delivery mode chosen is
"reserved"
>> > because notifications are not supposed to be delivered as MSI
anymore.
>> >
>> > This is what should happen:
>> >
>> > 1) Linux configures the device with a 0 vector number and the pirq
number
>> > in the address field;
>> >
>> > 2) QEMU notices a vector number of 0 and reads the pirq number
from the
>> > address field, passing it to xc_domain_update_msi_irq;
>> >
>> > 3) Xen assignes the given pirq to the physical MSI;
>> >
>> > 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
>> >
>> > 5) Xen sets the pirq as "IRQ_PT";
>> >
>> > 6) When Xen tries to inject the MSI into the guest,
hvm_domain_use_pirq
>> > returns true so Xen calls send_guest_pirq instead.
>> >
>> >
>> > Obviously 6) is not happening. hvm_domain_use_pirq is:
>> >
>> > is_hvm_domain(d) && pirq &&
pirq->arch.hvm.emuirq != IRQ_UNBOUND
>> >
>> > My guess is that emuirq is IRQ_UNBOUND when it should be IRQ_PT
(see
>> > above).
>>
>> This appears to be true. I added logging to hvm_pci_msi_assert in
>> xen/drivers/passthrough/io.c and it indicates that
>> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time right
>> before an unsupported delivery mode message.
>>
>> I also log pirq->pirq but I found that most of the time I
can''t find
>> this value anywhere else (I''m not sure how to interpret the
value,
>> though). For example, in my last try:
>>
>> * I get an unsupported delivery mode error for pirq->pirq 55, 54 and
>> 53. The vast majority are for 54.
>> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c. It
>> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
>> Never for 54 or 53. It also gets called with pirq=49,emuirq=23 once
>> but complains it''s already mapped.
>> * I have logging in evtchn_bind_pirq in xen/common/event_channel.c. It
>> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21, 19,
>> 22, 52, 48, 47. Also never 54 or 53.
>> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq 16,
17, 55.
>> * The qemu log mentions pirq 35, 36 and 37
>>
>> It seems pirq values don''t always mean the same? Is it a
coincidence
>> that 55 occurs almost everywhere, or is something going wrong with the
>> other two values (53 and 54 versus 16 and 17)?
>>
>> I have three MSI capable devices passed through to the domU, and I do
>> see groups of three distinct pirqs in the data above - just not the
>> same ones in every place I look.
>>
>> > So maybe the guest is not issuing a EVTCHNOP_bind_pirq hypercall
>> > (__startup_pirq doesn''t get called), or Xen is erroring
out in
>> > map_domain_emuirq_pirq.
>>
>> evtchn_bind_pirq gets called, though I''m not sure if it is
with the right data.
>>
>> map_domain_emuirq_pirq always gets past the checks in the top half
>> (i.e. up to the line /* do not store emuirq mappings for pt devices
>> */), except for one time with pirq=49,emuirq=23 where it finds they
>> are already mapped.
>> It is called three times with an emuirq of -2, for pirq 16, 17 and 55.
>> This implies their info->arch.hvm.emuirq is also set to -2
(haven''t
>> directly logged that but it''s the only assignment there).
>>
>> Interestingly, I get an unsupported delivery mode error for pirq 55
>> where my logging says pirq->arch.hvm.emuirq is -1, *after*
>> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
>
> Looking back at your QEMU logs, it seems that pt_msi_setup is not
> called (or it is not called at the right time), otherwise you should
> get:
>
> pt_msi_setup requested pirq = %d
>
> in your logs.
> Could you try disabling msitranslate? You can do that adding
>
> pci_msitranslate=0
>
> to your VM config file.
I tried that, but it didn''t work.
> If that works, probably this (untested) QEMU patch could fix your problem:
>
I appreciate the help.

I applied the patch anyway just to see what would happen (had to edit
a few dev versus d variable names) but it didn''t help. It also breaks
pt_msi_update, as I get in the qemu log:

pt_msi_update: Update msi with pirq 2f gvec 0 gflags 302f
pt_msi_update: Error: Binding of MSI failed.
pt_msi_update: Error: Unmapping of MSI failed.
pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 80

I added some logging to pt_msi_setup (without the patch). It does get
called, and it does so rather early in the boot process, each time
right before lines as these:

pci_intx: intx=1
register_real_device: Real physical device 00:1b.0 registered successfuly!
IRQ type = MSI-INTx

At this point dev->msi->data, addr_hi and addr_lo are all 0, which
doesn''t seem right. Is it being called prematurely?

Stefano Stabellini

2012-Jun-27 13:36 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Tue, 26 Jun 2012, Rolu wrote:> On Tue, Jun 26, 2012 at 2:59 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Tue, 26 Jun 2012, Rolu wrote:
> >> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
> >> <stefano.stabellini@eu.citrix.com> wrote:
> >> > On Mon, 25 Jun 2012, Jan Beulich wrote:
> >> >> >>> On 24.06.12 at 04:21, Rolu
<rolu@roce.org> wrote:
> >> >> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich
<JBeulich@suse.com> wrote:
> >> >> >> At the same time, adding logging to the guest
kernel would
> >> >> >> be nice, to see what value it actually writes
(in a current
> >> >> >> kernel this would be in __write_msi_msg()).
> >> >> >>
> >> >> >
> >> >> > Turns out that msg->data here is also 0x4300, so
it seems the guest
> >> >> > kernel is producing these values. I caused it to
make a stack trace
> >> >> > and this pointed back to xen_hvm_setup_msi_irqs.
This function uses
> >> >> > the macro XEN_PIRQ_MSI_DATA, which evaluates to
0x4300. It checks the
> >> >> > current data field and if it isn''t equal to
the macro it uses
> >> >> > xen_msi_compose_msg to make a new message, but that
function just sets
> >> >> > the data field of the message to XEN_PIRQ_MSI_DATA -
so, 0x4300. This
> >> >> > then gets passed to __write_msi_msg and
that''s that. There are no
> >> >> > other writes through __write_msi_msg (except for the
same thing for
> >> >> > other devices).
> >> >> >
> >> >> > The macro XEN_PIRQ_MSI_DATA contains a part (3
<< 8) which ends up
> >> >> > decoded as the delivery mode, so it seems the kernel
is intentionally
> >> >> > setting it to 3.
> >> >>
> >> >> So that can never have worked properly afaict. Stefano,
the
> >> >> code as it is currently - using literal (3 << 8) -
is clearly bogus.
> >> >> Your original commit at least had a comment saying that
the
> >> >> reserved delivery mode encoding is intentional here, but
that
> >> >> comment got lost with the later introduction of
XEN_PIRQ_MSI_DATA.
> >> >> In any case - the cooperation with qemu apparently
doesn''t
> >> >> work, as the reserved encoding should never make it
through
> >> >> to the hypervisor. Could you explain what the intention
here
> >> >> was?
> >> >>
> >> >> And regardless of anything, can the literal numbers
please be
> >> >> replaced by proper manifest constants - the "8"
here already
> >> >> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
> >> >> proper symbolic would permit locating where this is being
(or
> >> >> really, as it doesn''t appear to work supposed to
be) consumed
> >> >> in qemu, provided it uses the same definition (i.e. that
one
> >> >> should go into one of the public headers).
> >> >
> >> > The (3 << 8) is unimportant. The delivery mode chosen
is "reserved"
> >> > because notifications are not supposed to be delivered as MSI
anymore.
> >> >
> >> > This is what should happen:
> >> >
> >> > 1) Linux configures the device with a 0 vector number and the
pirq number
> >> > in the address field;
> >> >
> >> > 2) QEMU notices a vector number of 0 and reads the pirq
number from the
> >> > address field, passing it to xc_domain_update_msi_irq;
> >> >
> >> > 3) Xen assignes the given pirq to the physical MSI;
> >> >
> >> > 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
> >> >
> >> > 5) Xen sets the pirq as "IRQ_PT";
> >> >
> >> > 6) When Xen tries to inject the MSI into the guest,
hvm_domain_use_pirq
> >> > returns true so Xen calls send_guest_pirq instead.
> >> >
> >> >
> >> > Obviously 6) is not happening. hvm_domain_use_pirq is:
> >> >
> >> > is_hvm_domain(d) && pirq &&
pirq->arch.hvm.emuirq != IRQ_UNBOUND
> >> >
> >> > My guess is that emuirq is IRQ_UNBOUND when it should be
IRQ_PT (see
> >> > above).
> >>
> >> This appears to be true. I added logging to hvm_pci_msi_assert in
> >> xen/drivers/passthrough/io.c and it indicates that
> >> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time
right
> >> before an unsupported delivery mode message.
> >>
> >> I also log pirq->pirq but I found that most of the time I
can''t find
> >> this value anywhere else (I''m not sure how to interpret
the value,
> >> though). For example, in my last try:
> >>
> >> * I get an unsupported delivery mode error for pirq->pirq 55,
54 and
> >> 53. The vast majority are for 54.
> >> * I have logging in map_domain_emuirq_pirq in xen/arch/x86/irq.c.
It
> >> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17, 55.
> >> Never for 54 or 53. It also gets called with pirq=49,emuirq=23
once
> >> but complains it''s already mapped.
> >> * I have logging in evtchn_bind_pirq in
xen/common/event_channel.c. It
> >> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice), 21,
19,
> >> 22, 52, 48, 47. Also never 54 or 53.
> >> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for pirq
16, 17, 55.
> >> * The qemu log mentions pirq 35, 36 and 37
> >>
> >> It seems pirq values don''t always mean the same? Is it a
coincidence
> >> that 55 occurs almost everywhere, or is something going wrong with
the
> >> other two values (53 and 54 versus 16 and 17)?
> >>
> >> I have three MSI capable devices passed through to the domU, and I
do
> >> see groups of three distinct pirqs in the data above - just not
the
> >> same ones in every place I look.
> >>
> >> > So maybe the guest is not issuing a EVTCHNOP_bind_pirq
hypercall
> >> > (__startup_pirq doesn''t get called), or Xen is
erroring out in
> >> > map_domain_emuirq_pirq.
> >>
> >> evtchn_bind_pirq gets called, though I''m not sure if it
is with the right data.
> >>
> >> map_domain_emuirq_pirq always gets past the checks in the top half
> >> (i.e. up to the line /* do not store emuirq mappings for pt
devices
> >> */), except for one time with pirq=49,emuirq=23 where it finds
they
> >> are already mapped.
> >> It is called three times with an emuirq of -2, for pirq 16, 17 and
55.
> >> This implies their info->arch.hvm.emuirq is also set to -2
(haven''t
> >> directly logged that but it''s the only assignment there).
> >>
> >> Interestingly, I get an unsupported delivery mode error for pirq
55
> >> where my logging says pirq->arch.hvm.emuirq is -1, *after*
> >> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
> >
> > Looking back at your QEMU logs, it seems that pt_msi_setup is not
> > called (or it is not called at the right time), otherwise you should
> > get:
> >
> > pt_msi_setup requested pirq = %d
> >
> > in your logs.
> > Could you try disabling msitranslate? You can do that adding
> >
> > pci_msitranslate=0
> >
> > to your VM config file.
> 
> I tried that, but it didn''t work.
> 
> > If that works, probably this (untested) QEMU patch could fix your
problem:
> >
> 
> I appreciate the help.
> 
> I applied the patch anyway just to see what would happen (had to edit
> a few dev versus d variable names) but it didn''t help. It also
breaks
> pt_msi_update, as I get in the qemu log:
> 
> pt_msi_update: Update msi with pirq 2f gvec 0 gflags 302f
> pt_msi_update: Error: Binding of MSI failed.
> pt_msi_update: Error: Unmapping of MSI failed.
> pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 80
> 
> I added some logging to pt_msi_setup (without the patch). It does get
> called, and it does so rather early in the boot process, each time
> right before lines as these:
> 
> pci_intx: intx=1
> register_real_device: Real physical device 00:1b.0 registered successfuly!
> IRQ type = MSI-INTx
> 
> At this point dev->msi->data, addr_hi and addr_lo are all 0, which
> doesn''t seem right. Is it being called prematurely?
That''s because msitranslate is still enabled somehow, that is a
toolstack bug.
While we fix that bug, could you try this QEMU patch to forcefully disable
msitranslate?

diff --git a/xenstore.c b/xenstore.c
index ac90366..8af280e 100644
--- a/xenstore.c
+++ b/xenstore.c
@@ -427,7 +427,7 @@ uint32_t xenstore_read_target(void)
     return target_domid;
 }
 
-#define PT_PCI_MSITRANSLATE_DEFAULT 1
+#define PT_PCI_MSITRANSLATE_DEFAULT 0
 #define PT_PCI_POWER_MANAGEMENT_DEFAULT 0
 int direct_pci_msitranslate;
 int direct_pci_power_mgmt;

Rolu

2012-Jun-27 18:42 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Wed, Jun 27, 2012 at 3:36 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:> On Tue, 26 Jun 2012, Rolu wrote:
>> On Tue, Jun 26, 2012 at 2:59 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Tue, 26 Jun 2012, Rolu wrote:
>> >> On Mon, Jun 25, 2012 at 1:38 PM, Stefano Stabellini
>> >> <stefano.stabellini@eu.citrix.com> wrote:
>> >> > On Mon, 25 Jun 2012, Jan Beulich wrote:
>> >> >> >>> On 24.06.12 at 04:21, Rolu
<rolu@roce.org> wrote:
>> >> >> > On Wed, Jun 20, 2012 at 6:03 PM, Jan Beulich
<JBeulich@suse.com> wrote:
>> >> >> >> At the same time, adding logging to the
guest kernel would
>> >> >> >> be nice, to see what value it actually
writes (in a current
>> >> >> >> kernel this would be in __write_msi_msg()).
>> >> >> >>
>> >> >> >
>> >> >> > Turns out that msg->data here is also 0x4300,
so it seems the guest
>> >> >> > kernel is producing these values. I caused it to
make a stack trace
>> >> >> > and this pointed back to xen_hvm_setup_msi_irqs.
This function uses
>> >> >> > the macro XEN_PIRQ_MSI_DATA, which evaluates to
0x4300. It checks the
>> >> >> > current data field and if it isn''t
equal to the macro it uses
>> >> >> > xen_msi_compose_msg to make a new message, but
that function just sets
>> >> >> > the data field of the message to
XEN_PIRQ_MSI_DATA - so, 0x4300. This
>> >> >> > then gets passed to __write_msi_msg and
that''s that. There are no
>> >> >> > other writes through __write_msi_msg (except for
the same thing for
>> >> >> > other devices).
>> >> >> >
>> >> >> > The macro XEN_PIRQ_MSI_DATA contains a part (3
<< 8) which ends up
>> >> >> > decoded as the delivery mode, so it seems the
kernel is intentionally
>> >> >> > setting it to 3.
>> >> >>
>> >> >> So that can never have worked properly afaict.
Stefano, the
>> >> >> code as it is currently - using literal (3 <<
8) - is clearly bogus.
>> >> >> Your original commit at least had a comment saying
that the
>> >> >> reserved delivery mode encoding is intentional here,
but that
>> >> >> comment got lost with the later introduction of
XEN_PIRQ_MSI_DATA.
>> >> >> In any case - the cooperation with qemu apparently
doesn''t
>> >> >> work, as the reserved encoding should never make it
through
>> >> >> to the hypervisor. Could you explain what the
intention here
>> >> >> was?
>> >> >>
>> >> >> And regardless of anything, can the literal numbers
please be
>> >> >> replaced by proper manifest constants - the
"8" here already
>> >> >> has MSI_DATA_DELIVERY_MODE_SHIFT, and giving the 3 a
>> >> >> proper symbolic would permit locating where this is
being (or
>> >> >> really, as it doesn''t appear to work
supposed to be) consumed
>> >> >> in qemu, provided it uses the same definition (i.e.
that one
>> >> >> should go into one of the public headers).
>> >> >
>> >> > The (3 << 8) is unimportant. The delivery mode
chosen is "reserved"
>> >> > because notifications are not supposed to be delivered as
MSI anymore.
>> >> >
>> >> > This is what should happen:
>> >> >
>> >> > 1) Linux configures the device with a 0 vector number and
the pirq number
>> >> > in the address field;
>> >> >
>> >> > 2) QEMU notices a vector number of 0 and reads the pirq
number from the
>> >> > address field, passing it to xc_domain_update_msi_irq;
>> >> >
>> >> > 3) Xen assignes the given pirq to the physical MSI;
>> >> >
>> >> > 4) The guest issues a EVTCHNOP_bind_pirq hypercall;
>> >> >
>> >> > 5) Xen sets the pirq as "IRQ_PT";
>> >> >
>> >> > 6) When Xen tries to inject the MSI into the guest,
hvm_domain_use_pirq
>> >> > returns true so Xen calls send_guest_pirq instead.
>> >> >
>> >> >
>> >> > Obviously 6) is not happening. hvm_domain_use_pirq is:
>> >> >
>> >> > is_hvm_domain(d) && pirq &&
pirq->arch.hvm.emuirq != IRQ_UNBOUND
>> >> >
>> >> > My guess is that emuirq is IRQ_UNBOUND when it should be
IRQ_PT (see
>> >> > above).
>> >>
>> >> This appears to be true. I added logging to hvm_pci_msi_assert
in
>> >> xen/drivers/passthrough/io.c and it indicates that
>> >> pirq->arch.hvm.emuirq is -1 (while IRQ_PT is -2) every time
right
>> >> before an unsupported delivery mode message.
>> >>
>> >> I also log pirq->pirq but I found that most of the time I
can''t find
>> >> this value anywhere else (I''m not sure how to
interpret the value,
>> >> though). For example, in my last try:
>> >>
>> >> * I get an unsupported delivery mode error for pirq->pirq
55, 54 and
>> >> 53. The vast majority are for 54.
>> >> * I have logging in map_domain_emuirq_pirq in
xen/arch/x86/irq.c. It
>> >> gets called with pirq 19, 20, 21, 22, 23, 52, 51, 50, 16, 17,
55.
>> >> Never for 54 or 53. It also gets called with pirq=49,emuirq=23
once
>> >> but complains it''s already mapped.
>> >> * I have logging in evtchn_bind_pirq in
xen/common/event_channel.c. It
>> >> gets called with bind->pirq 16, 17, 51, 55, 49, 29 (twice),
21, 19,
>> >> 22, 52, 48, 47. Also never 54 or 53.
>> >> * map_domain_emuirq_pirq is called from evtchn_bind_pirq for
pirq 16, 17, 55.
>> >> * The qemu log mentions pirq 35, 36 and 37
>> >>
>> >> It seems pirq values don''t always mean the same? Is
it a coincidence
>> >> that 55 occurs almost everywhere, or is something going wrong
with the
>> >> other two values (53 and 54 versus 16 and 17)?
>> >>
>> >> I have three MSI capable devices passed through to the domU,
and I do
>> >> see groups of three distinct pirqs in the data above - just
not the
>> >> same ones in every place I look.
>> >>
>> >> > So maybe the guest is not issuing a EVTCHNOP_bind_pirq
hypercall
>> >> > (__startup_pirq doesn''t get called), or Xen is
erroring out in
>> >> > map_domain_emuirq_pirq.
>> >>
>> >> evtchn_bind_pirq gets called, though I''m not sure if
it is with the right data.
>> >>
>> >> map_domain_emuirq_pirq always gets past the checks in the top
half
>> >> (i.e. up to the line /* do not store emuirq mappings for pt
devices
>> >> */), except for one time with pirq=49,emuirq=23 where it finds
they
>> >> are already mapped.
>> >> It is called three times with an emuirq of -2, for pirq 16, 17
and 55.
>> >> This implies their info->arch.hvm.emuirq is also set to -2
(haven''t
>> >> directly logged that but it''s the only assignment
there).
>> >>
>> >> Interestingly, I get an unsupported delivery mode error for
pirq 55
>> >> where my logging says pirq->arch.hvm.emuirq is -1, *after*
>> >> map_domain_emuirq_pirq was called for pirq 55 and emuirq -2.
>> >
>> > Looking back at your QEMU logs, it seems that pt_msi_setup is not
>> > called (or it is not called at the right time), otherwise you
should
>> > get:
>> >
>> > pt_msi_setup requested pirq = %d
>> >
>> > in your logs.
>> > Could you try disabling msitranslate? You can do that adding
>> >
>> > pci_msitranslate=0
>> >
>> > to your VM config file.
>>
>> I tried that, but it didn''t work.
>>
>> > If that works, probably this (untested) QEMU patch could fix your
problem:
>> >
>>
>> I appreciate the help.
>>
>> I applied the patch anyway just to see what would happen (had to edit
>> a few dev versus d variable names) but it didn''t help. It also
breaks
>> pt_msi_update, as I get in the qemu log:
>>
>> pt_msi_update: Update msi with pirq 2f gvec 0 gflags 302f
>> pt_msi_update: Error: Binding of MSI failed.
>> pt_msi_update: Error: Unmapping of MSI failed.
>> pt_msgctrl_reg_write: Warning: Can not bind MSI for dev 80
>>
>> I added some logging to pt_msi_setup (without the patch). It does get
>> called, and it does so rather early in the boot process, each time
>> right before lines as these:
>>
>> pci_intx: intx=1
>> register_real_device: Real physical device 00:1b.0 registered
successfuly!
>> IRQ type = MSI-INTx
>>
>> At this point dev->msi->data, addr_hi and addr_lo are all 0,
which
>> doesn''t seem right. Is it being called prematurely?
>
> That''s because msitranslate is still enabled somehow, that is a
> toolstack bug.
> While we fix that bug, could you try this QEMU patch to forcefully disable
> msitranslate?
>
This worked!

The "unsupported delivery mode" message is gone. Sound works, although
there is still occasionally a very short stutter, but I expect that''s
a different issue. I''ve been testing with a KDE desktop with 3D
effects (cube, expo, that sort of stuff) and performance there has
gone up noticeably, from around 30-40 fps in most cases to near 60.
> diff --git a/xenstore.c b/xenstore.c
> index ac90366..8af280e 100644
> --- a/xenstore.c
> +++ b/xenstore.c
> @@ -427,7 +427,7 @@ uint32_t xenstore_read_target(void)
>     return target_domid;
>  }
>
> -#define PT_PCI_MSITRANSLATE_DEFAULT 1
> +#define PT_PCI_MSITRANSLATE_DEFAULT 0
>  #define PT_PCI_POWER_MANAGEMENT_DEFAULT 0
>  int direct_pci_msitranslate;
>  int direct_pci_power_mgmt;

Stefano Stabellini

2012-Jun-28 10:49 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Wed, 27 Jun 2012, Rolu wrote:> > That''s because msitranslate is still enabled somehow, that is
a
> > toolstack bug.
> > While we fix that bug, could you try this QEMU patch to forcefully
disable
> > msitranslate?
> >
> 
> This worked!
> 
> The "unsupported delivery mode" message is gone. Sound works,
although
> there is still occasionally a very short stutter, but I expect
that''s
> a different issue. I''ve been testing with a KDE desktop with 3D
> effects (cube, expo, that sort of stuff) and performance there has
> gone up noticeably, from around 30-40 fps in most cases to near 60.
Great, good to hear!
Now that we have narrowed down the issue to msitranslate (that should be
disabled by default anyway), I would like to fix it entirely if possible.
Could you please try the appended patch, without any other patches
(therefore msitranslate would still be enabled)?

Thanks for testing!!

---


diff --git a/hw/pass-through.c b/hw/pass-through.c
index 8581253..fc8c49f 100644
--- a/hw/pass-through.c
+++ b/hw/pass-through.c
@@ -3841,21 +3841,18 @@ static int pt_msgctrl_reg_write(struct pt_dev *ptdev,
                 PT_LOG("guest enabling MSI, disable MSI-INTx
translation\n");
                 pt_disable_msi_translate(ptdev);
             }
-            else
+            /* Init physical one */
+            PT_LOG("setup msi for dev %x\n", pd->devfn);
+            if (pt_msi_setup(ptdev))
             {
-                /* Init physical one */
-                PT_LOG("setup msi for dev %x\n", pd->devfn);
-                if (pt_msi_setup(ptdev))
-                {
-		    /* We do not broadcast the error to the framework code, so
-		     * that MSI errors are contained in MSI emulation code and
-		     * QEMU can go on running.
-		     * Guest MSI would be actually not working.
-		     */
-		    *value &= ~PCI_MSI_FLAGS_ENABLE;
-		    PT_LOG("Warning: Can not map MSI for dev %x\n", pd->devfn);
-		    return 0;
-                }
+                /* We do not broadcast the error to the framework code, so
+                 * that MSI errors are contained in MSI emulation code and
+                 * QEMU can go on running.
+                 * Guest MSI would be actually not working.
+                 */
+                *value &= ~PCI_MSI_FLAGS_ENABLE;
+                PT_LOG("Warning: Can not map MSI for dev %x\n",
pd->devfn);
+                return 0;
             }
             if (pt_msi_update(ptdev))
             {
diff --git a/hw/pt-msi.c b/hw/pt-msi.c
index 70c4023..73f737d 100644
--- a/hw/pt-msi.c
+++ b/hw/pt-msi.c
@@ -263,16 +263,8 @@ void pt_disable_msi_translate(struct pt_dev *dev)
     uint8_t e_device = 0;
     uint8_t e_intx = 0;
 
-    /* MSI_ENABLE bit should be disabed until the new handler is set */
-    msi_set_enable(dev, 0);
-
-    e_device = PCI_SLOT(dev->dev.devfn);
-    e_intx = pci_intx(dev);
-
-    if (xc_domain_unbind_pt_irq(xc_handle, domid, dev->msi->pirq,
-                                 PT_IRQ_TYPE_MSI_TRANSLATE, 0,
-                                 e_device, e_intx, 0))
-        PT_LOG("Error: Unbinding pt irq for MSI-INTx failed!\n");
+    pt_msi_disable(dev);
+    dev->msi->flags |= MSI_FLAG_UNINIT;
 
     if (dev->machine_irq)
     {
@@ -280,8 +272,6 @@ void pt_disable_msi_translate(struct pt_dev *dev)
                                        0, e_device, e_intx))
             PT_LOG("Error: Rebinding of interrupt failed!\n");
     }
-
-    dev->msi_trans_en = 0;
 }
 
 static int pt_msix_update_one(struct pt_dev *dev, int entry_nr)

Rolu

2012-Jun-29 01:42 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Thu, Jun 28, 2012 at 12:49 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:> On Wed, 27 Jun 2012, Rolu wrote:
>> > That''s because msitranslate is still enabled somehow,
that is a
>> > toolstack bug.
>> > While we fix that bug, could you try this QEMU patch to forcefully
disable
>> > msitranslate?
>> >
>>
>> This worked!
>>
>> The "unsupported delivery mode" message is gone. Sound works,
although
>> there is still occasionally a very short stutter, but I expect
that''s
>> a different issue. I''ve been testing with a KDE desktop with
3D
>> effects (cube, expo, that sort of stuff) and performance there has
>> gone up noticeably, from around 30-40 fps in most cases to near 60.
>
> Great, good to hear!
> Now that we have narrowed down the issue to msitranslate (that should be
> disabled by default anyway), I would like to fix it entirely if possible.
> Could you please try the appended patch, without any other patches
> (therefore msitranslate would still be enabled)?
>
> Thanks for testing!!
>
I''ve reverted the previous patch and applied this one, and I''m
pleased
to report it''s still working. Thanks for your help!

Stefano Stabellini

2012-Jun-29 11:11 UTC

head link

Re: Issue with MSI in a HVM domU with several passed through PCI devices

On Fri, 29 Jun 2012, Rolu wrote:> On Thu, Jun 28, 2012 at 12:49 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Wed, 27 Jun 2012, Rolu wrote:
> >> > That''s because msitranslate is still enabled
somehow, that is a
> >> > toolstack bug.
> >> > While we fix that bug, could you try this QEMU patch to
forcefully disable
> >> > msitranslate?
> >> >
> >>
> >> This worked!
> >>
> >> The "unsupported delivery mode" message is gone. Sound
works, although
> >> there is still occasionally a very short stutter, but I expect
that''s
> >> a different issue. I''ve been testing with a KDE desktop
with 3D
> >> effects (cube, expo, that sort of stuff) and performance there has
> >> gone up noticeably, from around 30-40 fps in most cases to near
60.
> >
> > Great, good to hear!
> > Now that we have narrowed down the issue to msitranslate (that should
be
> > disabled by default anyway), I would like to fix it entirely if
possible.
> > Could you please try the appended patch, without any other patches
> > (therefore msitranslate would still be enabled)?
> >
> > Thanks for testing!!
> >
> 
> I''ve reverted the previous patch and applied this one, and
I''m pleased
> to report it''s still working. Thanks for your help!
> 
Thanks for testing! I am going to send the patch and add your tested-by.

Stefano Stabellini

2012-Jun-29 11:21 UTC

head link

[PATCH] qemu-xen-trad: fix msi_translate with PV event delivery

When switching from msitranslate to straight msi we need to make sure
that we respect PV event delivery for the msi if the guest asked for it:

- completely disable MSI on the device in pt_disable_msi_translate;
- then enable MSI again (pt_msi_setup), mapping the correct pirq to it.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Tested-by: Rolu <rolu@roce.org>

---

diff --git a/hw/pass-through.c b/hw/pass-through.c
index 8581253..fc8c49f 100644
--- a/hw/pass-through.c
+++ b/hw/pass-through.c
@@ -3841,21 +3841,18 @@ static int pt_msgctrl_reg_write(struct pt_dev *ptdev,
                 PT_LOG("guest enabling MSI, disable MSI-INTx
translation\n");
                 pt_disable_msi_translate(ptdev);
             }
-            else
+            /* Init physical one */
+            PT_LOG("setup msi for dev %x\n", pd->devfn);
+            if (pt_msi_setup(ptdev))
             {
-                /* Init physical one */
-                PT_LOG("setup msi for dev %x\n", pd->devfn);
-                if (pt_msi_setup(ptdev))
-                {
-		    /* We do not broadcast the error to the framework code, so
-		     * that MSI errors are contained in MSI emulation code and
-		     * QEMU can go on running.
-		     * Guest MSI would be actually not working.
-		     */
-		    *value &= ~PCI_MSI_FLAGS_ENABLE;
-		    PT_LOG("Warning: Can not map MSI for dev %x\n", pd->devfn);
-		    return 0;
-                }
+                /* We do not broadcast the error to the framework code, so
+                 * that MSI errors are contained in MSI emulation code and
+                 * QEMU can go on running.
+                 * Guest MSI would be actually not working.
+                 */
+                *value &= ~PCI_MSI_FLAGS_ENABLE;
+                PT_LOG("Warning: Can not map MSI for dev %x\n",
pd->devfn);
+                return 0;
             }
             if (pt_msi_update(ptdev))
             {
diff --git a/hw/pt-msi.c b/hw/pt-msi.c
index 70c4023..73f737d 100644
--- a/hw/pt-msi.c
+++ b/hw/pt-msi.c
@@ -263,16 +263,8 @@ void pt_disable_msi_translate(struct pt_dev *dev)
     uint8_t e_device = 0;
     uint8_t e_intx = 0;
 
-    /* MSI_ENABLE bit should be disabed until the new handler is set */
-    msi_set_enable(dev, 0);
-
-    e_device = PCI_SLOT(dev->dev.devfn);
-    e_intx = pci_intx(dev);
-
-    if (xc_domain_unbind_pt_irq(xc_handle, domid, dev->msi->pirq,
-                                 PT_IRQ_TYPE_MSI_TRANSLATE, 0,
-                                 e_device, e_intx, 0))
-        PT_LOG("Error: Unbinding pt irq for MSI-INTx failed!\n");
+    pt_msi_disable(dev);
+    dev->msi->flags |= MSI_FLAG_UNINIT;
 
     if (dev->machine_irq)
     {
@@ -280,8 +272,6 @@ void pt_disable_msi_translate(struct pt_dev *dev)
                                        0, e_device, e_intx))
             PT_LOG("Error: Rebinding of interrupt failed!\n");
     }
-
-    dev->msi_trans_en = 0;
 }
 
 static int pt_msix_update_one(struct pt_dev *dev, int entry_nr)

Stefano Stabellini

2012-Aug-22 11:10 UTC

head link

Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery

On Fri, 29 Jun 2012, Stefano Stabellini wrote:> When switching from msitranslate to straight msi we need to make sure
> that we respect PV event delivery for the msi if the guest asked for it:
> 
> - completely disable MSI on the device in pt_disable_msi_translate;
> - then enable MSI again (pt_msi_setup), mapping the correct pirq to it.
> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> Tested-by: Rolu <rolu@roce.org>
ping
> ---
> 
> diff --git a/hw/pass-through.c b/hw/pass-through.c
> index 8581253..fc8c49f 100644
> --- a/hw/pass-through.c
> +++ b/hw/pass-through.c
> @@ -3841,21 +3841,18 @@ static int pt_msgctrl_reg_write(struct pt_dev
*ptdev,
>                  PT_LOG("guest enabling MSI, disable MSI-INTx
translation\n");
>                  pt_disable_msi_translate(ptdev);
>              }
> -            else
> +            /* Init physical one */
> +            PT_LOG("setup msi for dev %x\n", pd->devfn);
> +            if (pt_msi_setup(ptdev))
>              {
> -                /* Init physical one */
> -                PT_LOG("setup msi for dev %x\n", pd->devfn);
> -                if (pt_msi_setup(ptdev))
> -                {
> -		    /* We do not broadcast the error to the framework code, so
> -		     * that MSI errors are contained in MSI emulation code and
> -		     * QEMU can go on running.
> -		     * Guest MSI would be actually not working.
> -		     */
> -		    *value &= ~PCI_MSI_FLAGS_ENABLE;
> -		    PT_LOG("Warning: Can not map MSI for dev %x\n",
pd->devfn);
> -		    return 0;
> -                }
> +                /* We do not broadcast the error to the framework code, so
> +                 * that MSI errors are contained in MSI emulation code and
> +                 * QEMU can go on running.
> +                 * Guest MSI would be actually not working.
> +                 */
> +                *value &= ~PCI_MSI_FLAGS_ENABLE;
> +                PT_LOG("Warning: Can not map MSI for dev %x\n",
pd->devfn);
> +                return 0;
>              }
>              if (pt_msi_update(ptdev))
>              {
> diff --git a/hw/pt-msi.c b/hw/pt-msi.c
> index 70c4023..73f737d 100644
> --- a/hw/pt-msi.c
> +++ b/hw/pt-msi.c
> @@ -263,16 +263,8 @@ void pt_disable_msi_translate(struct pt_dev *dev)
>      uint8_t e_device = 0;
>      uint8_t e_intx = 0;
>  
> -    /* MSI_ENABLE bit should be disabed until the new handler is set */
> -    msi_set_enable(dev, 0);
> -
> -    e_device = PCI_SLOT(dev->dev.devfn);
> -    e_intx = pci_intx(dev);
> -
> -    if (xc_domain_unbind_pt_irq(xc_handle, domid, dev->msi->pirq,
> -                                 PT_IRQ_TYPE_MSI_TRANSLATE, 0,
> -                                 e_device, e_intx, 0))
> -        PT_LOG("Error: Unbinding pt irq for MSI-INTx
failed!\n");
> +    pt_msi_disable(dev);
> +    dev->msi->flags |= MSI_FLAG_UNINIT;
>  
>      if (dev->machine_irq)
>      {
> @@ -280,8 +272,6 @@ void pt_disable_msi_translate(struct pt_dev *dev)
>                                         0, e_device, e_intx))
>              PT_LOG("Error: Rebinding of interrupt failed!\n");
>      }
> -
> -    dev->msi_trans_en = 0;
>  }
>  
>  static int pt_msix_update_one(struct pt_dev *dev, int entry_nr)
>

Ian Jackson

2012-Aug-30 15:12 UTC

head link

Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery

Stefano Stabellini writes ("[Xen-devel] [PATCH] qemu-xen-trad: fix
msi_translate with PV event delivery"):> When switching from msitranslate to straight msi we need to make sure
> that we respect PV event delivery for the msi if the guest asked for it:
> 
> - completely disable MSI on the device in pt_disable_msi_translate;
> - then enable MSI again (pt_msi_setup), mapping the correct pirq to it.
I have applied this to 4.2.

Thanks,
Ian.

Ian Campbell

2012-Sep-03 09:44 UTC

head link

Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery

On Thu, 2012-08-30 at 16:12 +0100, Ian Jackson wrote:> Stefano Stabellini writes ("[Xen-devel] [PATCH] qemu-xen-trad: fix
msi_translate with PV event delivery"):
> > When switching from msitranslate to straight msi we need to make sure
> > that we respect PV event delivery for the msi if the guest asked for
it:
> > 
> > - completely disable MSI on the device in pt_disable_msi_translate;
> > - then enable MSI again (pt_msi_setup), mapping the correct pirq to
it.
> 
> I have applied this to 4.2.
You''ve not updated QEMU_TAG.

Ian.

Xen devel - Jun 2012 - Issue with MSI in a HVM domU with several passed through PCI devices

Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

Re: Issue with MSI in a HVM domU with several passed through PCI devices

[PATCH] qemu-xen-trad: fix msi_translate with PV event delivery

Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery

Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery

Re: [PATCH] qemu-xen-trad: fix msi_translate with PV event delivery