Xen users - Sep 2014 - [PCI passthrough] My experience with PCI memory stomp/corruption issues when passing through multiple PCI devices in Xen

Hi!

I saw a lot of posts on xen-devel dealing with PCI stomp/memory
corruption issues when passing through VGA graphics adapter and other
PCI devices to domU and I thought I'd share my (hopefully a
"success")
story. ;-)

System:
- 8 core Intel i7-3770 CPU @ 3.40GHz
- Intel Z75 Express chipset
- integrated graphics Intel HD 4000
- Xen 4.4.1 stock Arch Linux

I've been trying to setup a small mean&lean Arch-based dom0 domain and
run day-to-day operations in a fully fledged Mint domU domain to keep
my other domUs from rebooting when I mess things up. After
successfully getting around performance degradation issues (thanks
again, Ian!) due to misbehaved HAP, I stumbled upon another
deal-breaker: I could easily map integrated Intel VGA card as primary
graphics adapter to domU or pass PCI USB controller (mouse & keyboard)
through, but I could never get them both at once to work with the
PCI/VGA passthrough system.

The symptoms were always similar: USB controller simply stopped
responding soon after boot or "xl pci-assign" and the only message in
logs was either "ehci-pci 0000:00:05.0: HC died; cleaning up" or some
error code from hid drivers trying to access USB keyboard/mouse.

After a lot of Googling, I found a potential culprit - PCI hole and
all the magic behind remapping the devices and/or memory when the
default hole (roughly 200 MB) is too small for all the pass-though PCI
devices in the system (I need roughly 300 MB for both VGA and USB
controller to work). By manually fiddling with hvmloader and
qemu-xen-traditional and setting various remapping parameters, I was
able to confirm that both memory and device remapping logic is not to
blame. I also found a potential workaround in due process: by
overriding hvmloader's tendency to remap memory to 64-bit space
instead of remapping PCI devices to 64-bit space. This was the default
behavior for quite some time but was "improved" recently to deal
with
>2 GB PCI holes.
By remapping VGA adapter to 64-bit space, I was able to boot domU and
use both VGA and USB controller at the same time. The only thing that
bothered me was the inability to see any boot messages in domU until X
environment was loaded and the graphics card reset.

Not good enough, so I fiddled some more and due to some unexplained
stroke of luck, I was able to find a working combination:
1. I set the low memory bar in HVM guest's E820 map to 1 GB below the
end of PCI hole (set HVM_BELOW_4G_RAM_END to 0xBC000000 in
xen/include/public/hvm/e820.h)
2. I set the PCI hole start address in hvmloader to 0xe0000000
(PCI_MEM_START in tools/firmware/hvmloader/config.h), making it
"aligned" with qemu-xen-traditional's default value. This possibly
breaks upstream qemu until recent patches for configurable MMIO size
are applied. I'll be able to test this later when I try to get Windows
8 to work inside this configuration.

The key point is - if I set low memory bar in HVM guest's E820 map to
0xe0000000 (right up to the beginning of PCI hole size), something
messes up with my USB PCI controller and the thing simply doesn't
work. Setting the low memory bar lower apparently leaves some buffer
space for whomever or whatever is poking at that addresses and my USB
controller now works as charm. :-D It might be a hardware bug (saw a
lot of reports in xen-devel) or a software bug - I don't know.

Other (possibily unrelated) notes:
- forcing intel_iommu=on and iommu=on in Linux PV-on-HVM domU kernel
parameters is mandatory in this configuration; using iommu=soft
garbles the screen on Mint (Ubuntu) kernel 3.14.1-031401-generic
- maxmem= and memory= in domU configuration must be set to the same
value; Xen refuses to pass PCI devices through when the domain is
created with less-than-maximum memory. Shrinking the domain's memory
with "xl mem-set" appears to work, but is painfully slow. Setting
maxmem after the domain is created is not possible (xl mem-max has no
effect and neither does xenstore-write) so I have to start the domain
with maximum amount of memory and shrink it afterwards. Some memory is
"lost" due to max-mem situation overhead but I don't care.
- I was never able to get Intel HD 4000 to work as a secondary
graphics adapter (VGA passthrough without gfx_passthru=1) in HVM Linux
domU although supposedly this is possible for Windows domU. Others
blame it on Linux i915 drivers - I didn't try the latest and the
greatest version from git so I cannot confirm this.

Anyway [feeling victorious], maybe someone will find this info useful
when stumbling upon PCI devices mysteriously not working in one
pass-through configuration and working in others.

Best regards,
Andrej