Pkg xen devel - May 2013 - Bug#706543: xen-hypervisor-4.1-amd64: HVM PCI Passthrough not working any more after last upgrade

Package: xen-hypervisor-4.1-amd64
Version: 4.1.4-3
Severity: grave
Justification: causes non-serious data loss



-- System Information:
Debian Release: 7.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

xen-hypervisor-4.1-amd64 depends on no packages.

Versions of packages xen-hypervisor-4.1-amd64 recommends:
ii  xen-utils-4.1  4.1.4-3

Versions of packages xen-hypervisor-4.1-amd64 suggests:
pn  xen-docs-4.1  <none>

-- no debconf information

After last upgrade, its not possible any more to start Dom-Us based on HVM with
PCI passthrough. Before that, it worked for months.
Here is what happens:

------
# xm create lubuntu.cfg
Using config file "/etc/xen/lubuntu.cfg".
Error: (22, 'Invalid argument')
------

This is the config file:
------
kernel = "/usr/lib/xen-4.1/boot/hvmloader"
builder = "hvm"
name = "lubuntu"
memory = "4096"
vcpus = 4
vif=[ 'bridge=xenbr0,model=e1000,mac=00:11:3e:11:11:11' ]
disk = ['file:/vm/lubuntu/lubuntu-lts.img,ioemu:hda,w']
gfx_passthru=1
pci=['00:02.0','00:1d.0','00:1b.0']

pci_power_mgmt=1
pci_msitranslate=0
acpi=1
apic=1
xen_extended_power_mgmt=0
viridian=1
kernel = "/usr/lib/xen-4.1/boot/hvmloader"
builder = "hvm"
name = "lubuntu"
memory = "4096"
vcpus = 4
vif=[ 'bridge=xenbr0,model=e1000,mac=00:11:3e:11:11:11' ]
disk = ['file:/vm/lubuntu/lubuntu-lts.img,ioemu:hda,w']
gfx_passthru=1
pci=['00:02.0','00:1d.0','00:1b.0']

pci_power_mgmt=1
pci_msitranslate=0
acpi=1
apic=1
xen_extended_power_mgmt=0
viridian=1
xen_platform_pci=1
monitor=1

shadow_memory=8
on_poweroff = 'destroy'
on_reboot = 'destroy'
on_crash = 'destroy'
---------

Of course the devices are assignable:
---------
# xm pci-list-assignable-devices
0000:00:02.0
0000:00:1b.0
0000:00:1d.0
---------

In the xend.log it says:
---------
[2013-05-01 11:10:29 4897] DEBUG (pciif:320) pci: enabling ioport 0xf000/0x40
[2013-05-01 11:10:29 4897] DEBUG (pciif:334) pci: enabling iomem
0xf7800000/0x400000 pfn 0xf7800/0x400
[2013-05-01 11:10:29 4897] DEBUG (pciif:334) pci: enabling iomem
0xe0000000/0x10000000 pfn 0xe0000/0x10000
[2013-05-01 11:10:29 4897] DEBUG (pciif:351) pci: enabling irq 16
[2013-05-01 11:10:29 4897] INFO (pciquirk:92) NO quirks found for PCI device
[8086:1e26:1849:1e26]
[2013-05-01 11:10:29 4897] DEBUG (pciquirk:135) Permissive mode NOT enabled for
PCI device [8086:1e26:1849:1e26]
[2013-05-01 11:10:29 4897] DEBUG (pciif:334) pci: enabling iomem
0xf7c07000/0x400 pfn 0xf7c07/0x1
[2013-05-01 11:10:29 4897] DEBUG (pciif:351) pci: enabling irq 23
[2013-05-01 11:10:29 4897] ERROR (XendDomainInfo:2927)
XendDomainInfo.initDomain: exception occurred
Traceback (most recent call last):
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
2914, in _initDomain
    self._createDevices()
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
2395, in _createDevices
    self.pci_device_configure_boot()
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
627, in pci_device_configure_boot
    self.pci_device_configure(dev_sxp, first_dev = first)
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
970, in pci_device_configure
    devid = self._createDevice('pci', existing_pci_conf)
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
2326, in _createDevice
    return self.getDeviceController(deviceClass).createDevice(devConfig)
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/DevController.py",
line 67, in createDevice
    self.setupDevice(config)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/pciif.py",
line 453, in setupDevice
    self.setupOneDevice(d)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/pciif.py",
line 353, in setupOneDevice
    allow_access = True)
Error: (22, 'Invalid argument')
[2013-05-01 11:10:29 4897] ERROR (XendDomainInfo:488) VM start failed
Traceback (most recent call last):
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
474, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendTask.py", line
209, in log_progress
    retval = func(*args, **kwds)
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
2930, in _initDomain
    raise VmError(str(exn))
VmError: (22, 'Invalid argument')
[2013-05-01 11:10:29 4897] DEBUG (XendDomainInfo:3071) XendDomainInfo.destroy:
domid=3
[2013-05-01 11:10:29 4897] DEBUG (XendDomainInfo:2401) Destroying device model
[2013-05-01 11:10:30 4897] INFO (image:615) lubuntu device model terminated
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2408) Releasing devices
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2414) Removing vif/0
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:1276)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2414) Removing vbd/768
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:1276)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2414) Removing vbd/5632
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:1276)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/5632
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2406) No device model
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2408) Releasing devices
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2414) Removing vif/0
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:1276)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2414) Removing vbd/768
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:1276)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:2414) Removing vbd/5632
[2013-05-01 11:10:30 4897] DEBUG (XendDomainInfo:1276)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/5632
[2013-05-01 11:10:30 4897] ERROR (XendDomainInfo:108) Domain construction failed
Traceback (most recent call last):
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
106, in create
    vm.start()
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
474, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendTask.py", line
209, in log_progress
    retval = func(*args, **kwds)
  File
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line
2930, in _initDomain
    raise VmError(str(exn))
VmError: (22, 'Invalid argument')
------------

Hardware Info:
Processor:  Intel(R) Core(TM) i5-3470T
GPU: Onboard HD2500
No secondary GPU

Debugging:
- First I thought it is GPU related, because the unbind/bind to Dom0 stopped
working about 4 weeks ago.
  When I did a
  echo -n 0000:00:02.0 > /sys/bus/pci/devices/0000:00:02.0/driver/unbind
  echo "0000:00:02.0" > /sys/bus/pci/drivers/i915/bind
  I got formaly my graphics card back to dom0. This stopped working about 4
weeks ago. I only got a wired screen showing some noise.
  So I thought it is related to i915 driver and I remove the graphics passthough
from the config and tried to only have the USB port passed through
- Only USB also does not work so it seems very likely to be hypervisor related.
- Then I was pointed to this issue:
http://www.novell.com/support/kb/doc.php?id=7012337
  But unfortunately "iommu=no-intremap" did not help

So the last hypervisor/kernel update broke my system completely :(

Would be thankful for any hints!

Package: xen-hypervisor-4.1-amd64
Version: 4.1.4-3
Followup-For: Bug #706543

Other people having the same problem here:
http://lists.xen.org/archives/html/xen-users/2013-05/msg00004.html

Seems to be a problem with a path applied to Xen 4.1 and 4.2:
http://xen.crc.id.au/bugs/view.php?id=5

Control: severity -1 important

On Wed, May 01, 2013 at 02:20:44PM +0200, Markus wrote:
> Justification: causes non-serious data loss
You did not show that.
> Would be thankful for any hints!
Please show the hypervisor log by running "xen dmesg".

Bastian

-- 
You're dead, Jim.
		-- McCoy, "The Tholian Web", stardate unknown

Processing control commands:
> severity -1 important
Bug #706543 [xen-hypervisor-4.1-amd64] xen-hypervisor-4.1-amd64: HVM PCI
Passthrough not working any more after last upgrade
Severity set to 'important' from 'grave'

-- 
706543: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=706543
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

Package: xen-hypervisor-4.1-amd64
Version: 4.1.4-3
Followup-For: Bug #706543

Here you go:
# xen dmesg
(XEN) Xen version 4.1.4 (Debian 4.1.4-3) (waldi at debian.org) (gcc version
4.7.2 (Debian 4.7.2-5) ) Fri Apr 19 11:34:20 UTC 2013
(XEN) Bootloader: GRUB 1.99-27+deb7u1
(XEN) Command line: placeholder iommu=no-intremap i915.blacklist=1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040004000 (usable)
(XEN)  0000000040004000 - 0000000040005000 (reserved)
(XEN)  0000000040005000 - 00000000cda15000 (usable)
(XEN)  00000000cda15000 - 00000000cdd4b000 (reserved)
(XEN)  00000000cdd4b000 - 00000000cddd6000 (usable)
(XEN)  00000000cddd6000 - 00000000cde73000 (ACPI NVS)
(XEN)  00000000cde73000 - 00000000ce4de000 (reserved)
(XEN)  00000000ce4de000 - 00000000ce4df000 (usable)
(XEN)  00000000ce4df000 - 00000000ce522000 (ACPI NVS)
(XEN)  00000000ce522000 - 00000000ced40000 (usable)
(XEN)  00000000ced40000 - 00000000ceff2000 (reserved)
(XEN)  00000000ceff2000 - 00000000cf000000 (usable)
(XEN)  00000000cf800000 - 00000000dfa00000 (reserved)
(XEN)  00000000f8000000 - 00000000fc000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000021e600000 (usable)
(XEN) ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT CDE64088, 008C (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP CDE6D8C8, 010C (r5 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI Warning (tbfadt-0232): FADT (revision 5) is longer than ACPI 2.0
version, truncating length 0x10C to 0xF4 [20070126]
(XEN) ACPI: DSDT CDE641A8, 971F (r2 ALASKA    A M I       22 INTL 20051117)
(XEN) ACPI: FACS CDE71080, 0040
(XEN) ACPI: APIC CDE6D9D8, 0072 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FPDT CDE6DA50, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: MCFG CDE6DA98, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: SSDT CDE6DAD8, 07E1 (r1 Intel_ AoacTabl     1000 INTL 20091112)
(XEN) ACPI: AAFT CDE6E2C0, 00EA (r1 ALASKA OEMAAFT   1072009 MSFT       97)
(XEN) ACPI: HPET CDE6E3B0, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT CDE6E3E8, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) ACPI: SSDT CDE6E758, 0968 (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT CDE6F0C0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) ACPI: DMAR CDE6FB58, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) ACPI: ASF! CDE6FC08, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) ACPI: BGRT CDE6FCB0, 0038 (r0 ALASKA    A M I  1072009 AMI     10013)
(XEN) System RAM: 7878MB (8067492kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - cde71080/0000000000000000,
using 32
(XEN) Processor #0 7:10 APIC version 21
(XEN) Processor #2 7:10 APIC version 21
(XEN) Processor #1 7:10 APIC version 21
(XEN) Processor #3 7:10 APIC version 21
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Table is not found!
(XEN) Not enabling x2APIC: depends on iommu_supports_eim.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2893.498 MHz processor.
(XEN) Initing memory sharing.
(XEN) Intel VT-d supported page sizes: 4kB.
(XEN) Intel VT-d supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 4 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x193d000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000210000000->0000000214000000 (1931341 pages to be
allocated)
(XEN)  Init. ramdisk: 000000021c5e8000->000000021e5ffe00
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff8193d000
(XEN)  Init. ramdisk: ffffffff8193d000->ffffffff83954e00
(XEN)  Phys-Mach map: ffffffff83955000->ffffffff84841328
(XEN)  Start info:    ffffffff84842000->ffffffff848424b4
(XEN)  Page tables:   ffffffff84843000->ffffffff8486c000
(XEN)  Boot stack:    ffffffff8486c000->ffffffff8486d000
(XEN)  TOTAL:         ffffffff80000000->ffffffff84c00000
(XEN)  ENTRY ADDRESS: ffffffff816ab200
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen)
(XEN) Freed 244kB init memory.
(XEN) physdev.c:168: dom0: wrong map_pirq type 3
(XEN) [VT-D]iommu.c:879: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:854: DMAR:[DMA Write] Request device [00:02.0] fault addr
21e0ab000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83021b42c5f0 bdf = 0:2.0 gmfn = 21e0ab
(XEN)     root_entry = ffff830215dd3000
(XEN)     root_entry[0] = 80f9001
(XEN)     context = ffff8300080f9000
(XEN)     context[10] = 1_1e2c1001
(XEN)     l3 = ffff83001e2c1000
(XEN)     l3_index = 8
(XEN)     l3[8] = 0
(XEN)     l3[8] not present



I did more tests inpried by the bug description on the xen-users mailing list
and I do have the same results here:
Passthrough VGA only: works(!)
Passthrough USB,Sound,VGA: not working
Passthrough USB only: not working
Passthrough Sound only: not working
Passthrough VGA, Sound: not working
Passthrough VGA, USB: not working

4.1.4-2 worked in any case.

Package: xen-hypervisor-4.1-amd64
Version: 4.1.4-3
Followup-For: Bug #706543

Actually, it turned out that this bug is a problem of xen-utils (see here:
http://lists.xen.org/archives/html/xen-devel/2013-05/msg00145.html)
In particular: /usr/lib/xen-4.1/lib/python/xen/xend/server/pciif.py

I can confirm when removing the lines
---
rc = xc.domain_irq_permission(domid =  fe_domid, pirq = dev.irq,
                    allow_access = True)
            if rc<0:
                raise VmError(('pci: failed to configure irq on device
'+
                            '%s - errno=%d')%(dev.name,rc))
---

everything works as usual.
Since this is most likely not a good patch we should wait for some official one
;)

Could you please move this bug report to package xen-utils-4.1  4.1.4-3?

Thanks!

Your message dated Fri, 06 Sep 2013 00:00:08 +0000
with message-id <E1VHjT6-0002yk-UI at franck.debian.org>
and subject line Bug#706543: fixed in xen 4.3.0-1
has caused the Debian Bug report #706543,
regarding xen-hypervisor-4.1-amd64: HVM PCI Passthrough not working any more
after last upgrade
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
706543: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=706543
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Markus <markus at trash-mail.com>
Subject: xen-hypervisor-4.1-amd64: HVM PCI Passthrough not working any more
after last upgrade
Date: Wed, 01 May 2013 14:20:44 +0200
Size: 9520
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20130906/5d322b75/attachment-0002.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Bastian Blank <waldi at debian.org>
Subject: Bug#706543: fixed in xen 4.3.0-1
Date: Fri, 06 Sep 2013 00:00:08 +0000
Size: 8043
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20130906/5d322b75/attachment-0003.mht>