Jo Mills
2014-Apr-27 11:23 UTC
Create domU with pciback fails, then my system re-boots! xen-hypervisor-4.3-amd64
Hi,
I am having difficulty in getting my domU with pciback to start and
would be very grateful for any help or advice on where I am going
wrong. My system is a GenuineIntel Intel(R) Core(TM)2 Quad CPU,
motherboard in Intel S32000SHV, 8GB ram. BIOS is the latest released
version.
dom0 is Debian Jessie:
Linux version 3.13-1-amd64 (debian-kernel@lists.debian.org) (gcc
version 4.8.2 (Debian 4.8.2-16) ) #1 SMP Debian 3.13.10-1
(2014-04-15)
xen-hypervisor-4.3-amd64 4.3.0-3+b1
xen-system-amd64 4.3.0-3+b1
xen-tools 4.4-1
xen-utils-4.3 4.3.0-3+b1
xen-utils-common 4.3.0-3
xenstore-utils 4.3.0-3+b1
There are four ethernet devices fitted, and after the various udev
renaming of Ethernet interfaces these come out as being:
eth0 via-rhine 0000:04:00.0 assigned for zone LOC xenbr0
eth1 via-rhine 0000:04:01.0 assigned for zone DMZ (pci-passthrough)
eth2 e1000 0000:04:02.0 used for DBRB
eth3 e1000e 0000:01:00.0 planned for Windows client domU
Aside: In "xl dmesg" I get:
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation not enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
so I am hopeful of at some stage, getting a windows client to run.
On power up, dom0 starts OK, but I have never managed to get my file
/etc/modprobe.d/xen-pciback.conf
to work. It contains the following:
install via-rhine /sbin/modprobe pciback ; /sbin/modprobe --first-time
--ignore-install via_rhine
options xen-pciback hide=(0000:04:01.0)
install e1000 /sbin/modprobe pciback ; /sbin/modprobe --first-time
--ignore-install e1000
options xen-pciback hide=(0000:01:00.0)
However, if I issue the commands:
~# xl pci-assignable-add 04:01.0
~# xl pci-assignable-add 01:00.0
~# xl pci-assignable-list
I get the following:
0000:01:00.0
0000:04:01.0
In syslog I can also see:
pciback 0000:04:01.0: seizing device
xen: registering gsi 17 triggering 0 polarity 1
Already setup the GSI :17
e1000e 0000:01:00.0 eth3: removed PHC
pciback 0000:01:00.0: seizing device
xen: registering gsi 16 triggering 0 polarity 1
Already setup the GSI :16
so this would seem to be OK.
~# ifconfig
then shows:
eth0 Link encap:Ethernet HWaddr 00:24:01:a3:b9:ef
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2813 errors:0 dropped:0 overruns:0 frame:0
TX packets:1323 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:256913 (250.8 KiB) TX bytes:331402 (323.6 KiB)
eth2 Link encap:Ethernet HWaddr 00:15:17:ca:37:d5
inet addr:192.168.10.202 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::215:17ff:feca:37d5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:103247 errors:0 dropped:0 overruns:0 frame:0
TX packets:24142 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:128195865 (122.2 MiB) TX bytes:2626900 (2.5 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:88 errors:0 dropped:0 overruns:0 frame:0
TX packets:88 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6468 (6.3 KiB) TX bytes:6468 (6.3 KiB)
xenbr0 Link encap:Ethernet HWaddr 00:24:01:a3:b9:ef
inet addr:192.168.1.202 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::224:1ff:fea3:b9ef/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2813 errors:0 dropped:0 overruns:0 frame:0
TX packets:1323 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:217381 (212.2 KiB) TX bytes:331402 (323.6 KiB)
DRBD and xenbr0 are working so it all seems OK so far. My config file
for the domU is:
~# cat /etc/xen/vm-server-21.cfg
#
# Configuration file for the Xen instance vm-server-21, created
# by xen-tools 3.9 on Thu Apr 8 12:37:42 2010.
#
# Modified according to xl.cfg on 20-APR-2104
#
#
# Hostname
#
name = 'vm-server-21'
#
# Builder
#
##builder = 'generic'
#
# cpus
#
cpus = '1-3'
#
# Memory in MB
#
memory = '1024'
#
# Bootloader (Not the previous path via /etc/xen/default/...)
#
### bootloader = 'pygrub'
#
# Kernel
(https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1210293.html)
#
kernel = '/boot/vmlinuz-2.6.26-2-xen-amd64'
ramdisk = '/boot/initrd.img-2.6.26-2-xen-amd64'
#
# Disk device(s).
#
root = '/dev/xvdb ro'
#
# Pygrub needs the system partition to appear first
#
disk = [
'drbd:vm-21-disk,xvdb,w',
'phy:/dev/green/vm-server-21-swap,xvda,w',
]
#
# Networking
#
## vif = [ 'ip=192.168.2.221,mac=00:16:3e:dd:02:00' ]
#
# Add support for looped through pci NIC
# (Same device number on both blue and green
#
pci = [ '04:01.0' ]
#
# Behaviour
#
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
extra = 'xencons=tty swiotlb=force'
When I issue the command
~# xl -vvv create -c /etc/xen/vm-server-21.cfg
all hell breaks lose and after a bit the server re-boots.
The output from the xl command is:
Parsing config from /etc/xen/vm-server-21.cfg
libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x9d71f0: create:
how=(nil) callback=(nil) poller=0x9d6ba0
libxl: verbose: libxl_create.c:130:libxl__domain_build_info_setdefault: qemu-xen
is unavailable, use qemu-xen-traditional instead: No such file or directory
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvdb
spec.backend=unknown
libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvdb, uses
script=... assuming phy backend
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvdb,
using backend phy
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda
spec.backend=unknown
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda,
using backend phy
libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no bootloader
configured, using user supplied kernel
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x9d7608:
deregister unregistered
libxl: debug: libxl_x86.c:82:e820_sanitize: Memory: 1048576kB End of RAM:
0x40000 (PFN) Delta: 0kB, PCI start: 3665832kB (0xdfbea PFN), Balloon 0kB
libxl: debug: libxl_x86.c:201:e820_sanitize: : [0 -> 40000] RAM
libxl: debug: libxl_x86.c:201:e820_sanitize: : [40000 -> dfbea] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfbea -> dfc96] ACPI NVS
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfc96 -> dfcfa] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfcfa -> dfd5f] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfd5f -> dfd69] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfd69 -> dfddf] ACPI NVS
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfddf -> dfde5] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfde5 -> dfdff] ACPI
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfdff -> dfe00] Unusable
libxl: debug: libxl_x86.c:201:e820_sanitize: : [dfe00 -> dff00] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: : [f0000 -> f4000] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: : [fee00 -> fee01] Reserved
libxl: debug: libxl_x86.c:201:e820_sanitize: : [fff80 -> fff8c] Reserved
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvdb ro
xencons=tty swiotlb=force", features="(null)"
libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 0 path
/boot/vmlinuz-2.6.26-2-xen-amd64
domainbuilder: detail: xc_dom_kernel_file:
filename="/boot/vmlinuz-2.6.26-2-xen-amd64"
domainbuilder: detail: xc_dom_malloc_filemap : 1666 kB
domainbuilder: detail: xc_dom_malloc : 7801 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x1a0b72 -> 0x79e530
domainbuilder: detail: xc_dom_ramdisk_file:
filename="/boot/initrd.img-2.6.26-2-xen-amd64"
domainbuilder: detail: xc_dom_malloc_filemap : 7926 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.3, caps xen-3.0-x86_64
xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ...
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x200000 memsz=0x2fe000
xc: detail: elf_parse_binary: phdr: paddr=0x4fe000 memsz=0x529a8
xc: detail: elf_parse_binary: phdr: paddr=0x551000 memsz=0x888
xc: detail: elf_parse_binary: phdr: paddr=0x552000 memsz=0xdf918
xc: detail: elf_parse_binary: memory: 0x200000 -> 0x631918
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff80200000
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80208000
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: FEATURES =
"writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail: virt_base = 0xffffffff80000000
xc: detail: elf_paddr_offset = 0x0
xc: detail: virt_offset = 0xffffffff80000000
xc: detail: virt_kstart = 0xffffffff80200000
xc: detail: virt_kend = 0xffffffff80631918
xc: detail: virt_entry = 0xffffffff80200000
xc: detail: p2m_base = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64:
0xffffffff80200000 -> 0xffffffff80631918
domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k
each
domainbuilder: detail: xc_dom_mem_init: 0x40000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc : 2048 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment: kernel : 0xffffffff80200000
-> 0xffffffff80632000 (pfn 0x200 + 0x432 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x200+0x432
at 0x7fea4385c000
xc: detail: elf_load_binary: phdr 0 at 0x7fea4385c000 -> 0x7fea43b5a000
xc: detail: elf_load_binary: phdr 1 at 0x7fea43b5a000 -> 0x7fea43bac9a8
xc: detail: elf_load_binary: phdr 2 at 0x7fea43bad000 -> 0x7fea43bad888
xc: detail: elf_load_binary: phdr 3 at 0x7fea43bae000 -> 0x7fea43bec6b0
domainbuilder: detail: xc_dom_alloc_segment: ramdisk : 0xffffffff80632000
-> 0xffffffff81d14000 (pfn 0x632 + 0x16e2 pages)
domainbuilder: detail: xc_dom_malloc : 137 kB
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x632+0x16e2 at 0x7fea4217a000
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x7bd8b0 -> 0x16e1610
domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0xffffffff81d14000
-> 0xffffffff81f14000 (pfn 0x1d14 + 0x200 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x1d14+0x200 at 0x7fea41f7a000
domainbuilder: detail: xc_dom_alloc_page : start info : 0xffffffff81f14000
(pfn 0x1f14)
domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xffffffff81f15000
(pfn 0x1f15)
domainbuilder: detail: xc_dom_alloc_page : console : 0xffffffff81f16000
(pfn 0x1f16)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000
-> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000
-> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000
-> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000
-> 0xffffffff81ffffff, 16 table(s)
domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xffffffff81f17000
-> 0xffffffff81f2a000 (pfn 0x1f17 + 0x13 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1f17+0x13
at 0x7fea46b00000
domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xffffffff81f2a000
(pfn 0x1f2a)
domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xffffffff81f2b000
domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0xffffffff82000000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64
<= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x40000
domainbuilder: detail: clear_page: pfn 0x1f16, mfn 0x1ddb0e
domainbuilder: detail: clear_page: pfn 0x1f15, mfn 0x1ddb0f
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1f14+0x1
at 0x7fea46cdf000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff80208000 pfn=0x208
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail: allocated
domainbuilder: detail: malloc : 10027 kB
domainbuilder: detail: anon mmap : 0 bytes
domainbuilder: detail: mapped
domainbuilder: detail: file mmap : 9593 kB
domainbuilder: detail: domU mmap : 29856 kB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xdfc99
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x1f17 mfn 0x1ddb0d
domainbuilder: detail: launch_vm: called, ctxt=0x7fffdf91bd50
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvdb
spec.backend=phy
libxl: debug: libxl_device.c:188:disk_try_backend: Disk vdev=xvdb, uses
script=... assuming phy backend
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x9d8548
wpath=/local/domain/0/backend/vbd/2/51728/state token=3/0: register slotnum=3
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda
spec.backend=phy
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x9dfb98
wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: register slotnum=2
libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x9d71f0: inprogress:
poller=0x9d6ba0, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x9d8548
wpath=/local/domain/0/backend/vbd/2/51728/state token=3/0: event
epath=/local/domain/0/backend/vbd/2/51728/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend
/local/domain/0/backend/vbd/2/51728/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x9d8548
wpath=/local/domain/0/backend/vbd/2/51728/state token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x9d8548:
deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script:
/etc/xen/scripts/block-drbd add
libxl: debug: libxl_event.c:472:watchfd_callback: watch
epath=/local/domain/0/backend/vbd/2/51728/state token=3/0: empty slot
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x9dfb98
wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: event
epath=/local/domain/0/backend/vbd/2/51712/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend
/local/domain/0/backend/vbd/2/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x9dfb98
wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: deregister slotnum=2
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x9dfb98:
deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script:
/etc/xen/scripts/block add
libxl: debug: libxl_event.c:472:watchfd_callback: watch
epath=/local/domain/0/backend/vbd/2/51712/state token=2/1: empty slot
(It just hangs at this point)
On the server's console I get:
e1000 0000:04:02.0 eth2: Detected Tx Unit Hang
Tx Queue <0>
TDH <0>
TDT <2>
next_to_use <2>
next_to_clean <0>
buffer_info[next_to_clean]
time_stamp <100002ef6>
next_to_watch <0>
jiffies <1000034d3>
next_to_watch.status <0>
looping over and over again, and then as mentioned above, the server
hangs. I have modified /etc/xen/scripts/block-drbd as per a xen bug
report (I cannot recall at the moment which one), but the change was:
case $t in
drbd)
to become
case $t in
drbd|phy)
Any help will be greatly appreciated.
Regards,
Jo.