Hello,
I''m trying to run PV machines using the new libxenlight toolstack on
NetBSD. So far, I''ve been able to connect to the domu using the
console (I was unable to do so before). I''m attaching a little patch
that removes setting the consback to IOEMU when detecting a qdisk
(that was preventing the domu from even booting). With the
introduction of libxenlight, Xen doesn''t use vbd for disk backend
anymore, it uses qdisk, which I assume Qemu automatically attaches to
running guests. It works fine with HVM guests, but it seems to fail
with PV guests. The config file I''m using is:
bootloader = ''/usr/xen42/bin/pygrub''
vcpus = ''1''
memory = ''30720''
root = ''/dev/xvda2 ro''
disk = [
''file:/home/xen/debian/disk.img,xvda2,w'',
''file:/home/xen/debian/swap.img,xvda1,w'',
]
name = ''debian''
on_poweroff = ''destroy''
on_reboot = ''restart''
on_crash = ''restart''
And the output from booting:
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.32-5-xen-amd64 (Debian
2.6.32-34squeeze1) (dannf@debian.org) (gcc version 4.3.5 (Debian
4.3.5-4) ) #1 SMP Thu May 19 01:16:47 UTC 2011
[ 0.000000] Command line: root=/dev/xvda2 ro root=/dev/xvda2 ro
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] ACPI in unprivileged domain disabled
[ 0.000000] released 0 pages of unused memory
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
[ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
[ 0.000000] Xen: 0000000000100000 - 0000000780000000 (usable)
[ 0.000000] DMI not present or invalid.
[ 0.000000] last_pfn = 0x780000 max_arch_pfn = 0x400000000
[ 0.000000] last_pfn = 0x100000 max_arch_pfn = 0x400000000
[ 0.000000] init_memory_mapping: 0000000000000000-0000000100000000
[ 0.000000] init_memory_mapping: 0000000100000000-0000000780000000
[ 0.000000] RAMDISK: 016ba000 - 02feb000
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at 0000000000000000-0000000780000000
[ 0.000000] Bootmem setup node 0 0000000000000000-0000000780000000
[ 0.000000] NODE_DATA [0000000000008000 - 000000000000ffff]
[ 0.000000] bootmap [00000000008c7000 - 00000000009b6fff] pages f0
[ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 0780000000]
[ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==>
[0000000000 - 0000001000]
[ 0.000000] #1 [0006bee000 - 0006c29000] XEN PAGETABLES ==>
[0006bee000 - 0006c29000]
[ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==>
[0000006000 - 0000008000]
[ 0.000000] #3 [0001000000 - 00016999d4] TEXT DATA BSS ==>
[0001000000 - 00016999d4]
[ 0.000000] #4 [00016ba000 - 0002feb000] RAMDISK ==>
[00016ba000 - 0002feb000]
[ 0.000000] #5 [0002feb000 - 0006bee000] XEN START INFO ==>
[0002feb000 - 0006bee000]
[ 0.000000] #6 [0000100000 - 00008c7000] PGTABLE ==>
[0000100000 - 00008c7000]
[ 0.000000] #7 [0006c29000 - 000a043000] PGTABLE ==>
[0006c29000 - 000a043000]
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000000 -> 0x00001000
[ 0.000000] DMA32 0x00001000 -> 0x00100000
[ 0.000000] Normal 0x00100000 -> 0x00780000
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000000 -> 0x000000a0
[ 0.000000] 0: 0x00000100 -> 0x00780000
[ 0.000000] SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
[ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.000000] No local APIC present
[ 0.000000] APIC: disable apic facility
[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[ 0.000000] PCI: Warning: Cannot find a gap in the 32bit address range
[ 0.000000] PCI: Unassigned devices with 32bit resource registers may break!
[ 0.000000] Allocating PCI resources starting at 780100000 (gap:
780100000:400000)
[ 0.000000] Booting paravirtualized kernel on Xen
[ 0.000000] Xen version: 4.2-unstable (preserve-AD)
[ 0.000000] NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:1 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 30 pages/cpu @ffff88002804f000 s90328
r8192 d24360 u122880
[ 0.000000] pcpu-alloc: s90328 r8192 d24360 u122880 alloc=30*4096
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Xen: using vcpu_info placement
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on.
Total pages: 7754710
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: root=/dev/xvda2 ro root=/dev/xvda2 ro
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Initializing CPU#0
[ 0.000000] Checking aperture...
[ 0.000000] No AGP bridge found
[ 0.000000] Memory: 30803904k/31457280k available (3149k kernel
code, 384k absent, 652992k reserved, 1907k data, 604k init)
[ 0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0,
CPUs=1, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] NR_IRQS:4352 nr_irqs:512
[ 0.000000] Console: colour dummy device 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] console [hvc0] enabled
[ 0.000000] installing Xen timer for CPU 0
[ 0.000000] Detected 2394.046 MHz processor.
[ 0.004000] Calibrating delay loop (skipped), value calculated
using timer frequency.. 4788.09 BogoMIPS (lpj=9576184)
[ 0.004000] Security Framework initialized
[ 0.004000] SELinux: Disabled at boot.
[ 0.005680] Dentry cache hash table entries: 4194304 (order: 13,
33554432 bytes)
[ 0.014625] Inode-cache hash table entries: 2097152 (order: 12,
16777216 bytes)
[ 0.017597] Mount-cache hash table entries: 256
[ 0.017761] Initializing cgroup subsys ns
[ 0.017769] Initializing cgroup subsys cpuacct
[ 0.017776] Initializing cgroup subsys devices
[ 0.017782] Initializing cgroup subsys freezer
[ 0.017786] Initializing cgroup subsys net_cls
[ 0.017827] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.017835] CPU: L2 cache: 256K
[ 0.017838] CPU: L3 cache: 12288K
[ 0.017844] CPU 0/0x20 -> Node 0
[ 0.017848] CPU: Unsupported number of siblings 32
[ 0.017854] Performance Events: unsupported p6 CPU model 44 no PMU
driver, software events only.
[ 0.017874] SMP alternatives: switching to UP code
[ 0.020000] Freeing SMP alternatives: 28k freed
[ 0.020192] Brought up 1 CPUs
[ 0.020309] devtmpfs: initialized
[ 0.024342] Grant table initialized
[ 0.024351] regulator: core version 0.5
[ 0.024403] NET: Registered protocol family 16
[ 0.025064] PCI: setting up Xen PCI frontend stub
[ 0.026729] bio: create slab <bio-0> at 0
[ 0.026798] ACPI: Interpreter disabled.
[ 0.026833] xen_balloon: Initialising balloon driver with page order 0.
[ 0.026878] vgaarb: loaded
[ 0.026939] PCI: System does not support PCI
[ 0.026944] PCI: System does not support PCI
[ 0.027019] Switching to clocksource xen
[ 0.028001] pnp: PnP ACPI: disabled
[ 0.028001] NET: Registered protocol family 2
[ 0.028001] IP route cache hash table entries: 524288 (order: 10,
4194304 bytes)
[ 0.028460] TCP established hash table entries: 524288 (order: 11,
8388608 bytes)
[ 0.030137] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 0.030331] TCP: Hash tables configured (established 524288 bind 65536)
[ 0.030337] TCP reno registered
[ 0.030405] NET: Registered protocol family 1
[ 0.030463] Unpacking initramfs...
[ 0.051340] Freeing initrd memory: 25796k freed
[ 0.058949] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.058964] DMA: Placing 64MB software IO TLB between
ffff880020000000 - ffff880024000000
[ 0.058970] DMA: software IO TLB at phys 0x20000000 - 0x24000000
[ 0.059058] platform rtc_cmos: registered platform RTC device (no
PNP device found)
[ 0.059272] audit: initializing netlink socket (disabled)
[ 0.059288] type=2000 audit(1311165823.857:1): initialized
[ 0.064506] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.065744] VFS: Disk quotas dquot_6.5.2
[ 0.065796] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.065877] msgmni has been set to 32768
[ 0.066057] alg: No test for stdrng (krng)
[ 0.066112] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 253)
[ 0.066119] io scheduler noop registered
[ 0.066122] io scheduler anticipatory registered
[ 0.066127] io scheduler deadline registered
[ 0.066157] io scheduler cfq registered (default)
[ 0.072642] registering netback
[ 0.074001] Linux agpgart interface v0.103
[ 0.074032] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.173825] input: Macintosh mouse button emulation as
/devices/virtual/input/input0
[ 0.173872] PNP: No PS/2 controller found. Probing ports directly.
[ 0.174689] i8042.c: No controller found.
[ 0.174750] mice: PS/2 mouse device common for all mice
[ 0.174829] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[ 0.174883] cpuidle: using governor ladder
[ 0.174890] cpuidle: using governor menu
[ 0.174897] No iBFT detected.
[ 0.175118] TCP cubic registered
[ 0.175221] NET: Registered protocol family 10
[ 0.175788] Mobile IPv6
[ 0.175797] NET: Registered protocol family 17
[ 0.175889] registered taskstats version 1
[ 5.272069] XENBUS: Waiting for devices to initialise:
295s...290s...285s...280s...275s...270s...265s...260s...255s...250s...245s...240s...235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...130s...125s....
udevd[46]: worker [53] unexpectedly returned with status 0x0100
udevd[46]: worker [53] failed while handling
''/devices/vbd-51713''
udevd[46]: worker [54] unexpectedly returned with status 0x0100
udevd[46]: worker [54] failed while handling
''/devices/vbd-51714''
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Waiting for root file system ...
120s...115s...110s...105s...100s...95s...90s...done.
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/xvda2 does not exist. Dropping to a shell!
BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash)
Enter ''help'' for a list of built-in commands.
/bin/sh: can''t access tty; job control turned off
(initramfs)
[ 600.488236] XENBUS: Timeout connecting to device: device/vbd/51714
(local state 3, remote state 1)
[ 600.488328] XENBUS: Timeout connecting to device: device/vbd/51713
(local state 3, remote state 1)
>From what I see, the guest is unable to find the disk, which is
strange, because HVM guests work fine, and the disk is attached
correctly. Does this also happen when using file:/ backend (qdisk)
under Linux also? Is it a problem with Qemu or is it related to libxl?
I''m also attaching the output from xenstore, although I think it is ok
(or at least is the same as the one I have when running a HVM
machine):
/local/domain/0/backend = "" (n0)
/local/domain/0/backend/qdisk = "" (n0)
/local/domain/0/backend/qdisk/1 = "" (n0)
/local/domain/0/backend/qdisk/1/51714 = "" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/frontend
"/local/domain/1/device/vbd/51714" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/params
"aio:/home/xen/debian/disk.img" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/frontend-id = "1" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/online = "1" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/removable = "0" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/bootable = "1" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/state = "1" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/dev = "xvda2" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/type = "qdisk" (n0,r1)
/local/domain/0/backend/qdisk/1/51714/mode = "w" (n0,r1)
/local/domain/0/backend/qdisk/1/51713 = "" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/frontend
"/local/domain/1/device/vbd/51713" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/params
"aio:/home/xen/debian/swap.img" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/frontend-id = "1" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/online = "1" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/removable = "0" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/bootable = "1" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/state = "1" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/dev = "xvda1" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/type = "qdisk" (n0,r1)
/local/domain/0/backend/qdisk/1/51713/mode = "w" (n0,r1)
/local/domain/0/backend/console = "" (n0)
/local/domain/0/backend/console/1 = "" (n0)
/local/domain/0/backend/console/1/0 = "" (n0,r1)
/local/domain/0/backend/console/1/0/frontend "/local/domain/1/console"
(n0,r1)
/local/domain/0/backend/console/1/0/frontend-id = "1" (n0,r1)
/local/domain/0/backend/console/1/0/online = "1" (n0,r1)
/local/domain/0/backend/console/1/0/state = "1" (n0,r1)
/local/domain/0/backend/console/1/0/domain = "debian" (n0,r1)
/local/domain/0/backend/console/1/0/protocol = "vt100" (n0,r1)
/local/domain/0/backend/pci = "" (n0)
/local/domain/0/backend/pci/1 = "" (n0)
/local/domain/0/backend/pci/1/0 = "" (n0,r1)
/local/domain/0/backend/pci/1/0/frontend
"/local/domain/1/device/pci/0" (n0,r1)
/local/domain/0/backend/pci/1/0/frontend-id = "1" (n0,r1)
/local/domain/0/backend/pci/1/0/online = "1" (n0,r1)
/local/domain/0/backend/pci/1/0/state = "1" (n0,r1)
/local/domain/0/backend/pci/1/0/domain = "debian" (n0,r1)
/local/domain/0/backend/pci/1/0/num_devs = "0" (n0,r1)
/local/domain/0/device-model = "" (n0)
/local/domain/0/device-model/1 = "" (n0)
/local/domain/0/device-model/1/disable_pf = "1" (n0)
/local/domain/0/device-model/1/state = "running" (n0)
/local/domain/1 = "" (n0,r1)
/local/domain/1/vm = "/vm/4bfc3adf-cdb2-e011-a177-1803730a9a51"
(n0,r1)
/local/domain/1/name = "debian" (n0,r1)
/local/domain/1/control = "" (n0,r1)
/local/domain/1/control/shutdown = "" (n1)
/local/domain/1/control/platform-feature-multiprocessor-suspend = "1"
(n0,r1)
/local/domain/1/device = "" (n0,r1)
/local/domain/1/device/suspend = "" (n1)
/local/domain/1/device/suspend/event-channel = "" (n1)
/local/domain/1/device/vbd = "" (n0,r1)
/local/domain/1/device/vbd/51714 = "" (n1,r0)
/local/domain/1/device/vbd/51714/backend
"/local/domain/0/backend/qdisk/1/51714" (n1,r0)
/local/domain/1/device/vbd/51714/backend-id = "0" (n1,r0)
/local/domain/1/device/vbd/51714/state = "3" (n1,r0)
/local/domain/1/device/vbd/51714/virtual-device = "51714" (n1,r0)
/local/domain/1/device/vbd/51714/device-type = "disk" (n1,r0)
/local/domain/1/device/vbd/51714/ring-ref = "9" (n1,r0)
/local/domain/1/device/vbd/51714/event-channel = "9" (n1,r0)
/local/domain/1/device/vbd/51714/protocol = "x86_64-abi" (n1,r0)
/local/domain/1/device/vbd/51713 = "" (n1,r0)
/local/domain/1/device/vbd/51713/backend
"/local/domain/0/backend/qdisk/1/51713" (n1,r0)
/local/domain/1/device/vbd/51713/backend-id = "0" (n1,r0)
/local/domain/1/device/vbd/51713/state = "3" (n1,r0)
/local/domain/1/device/vbd/51713/virtual-device = "51713" (n1,r0)
/local/domain/1/device/vbd/51713/device-type = "disk" (n1,r0)
/local/domain/1/device/vbd/51713/ring-ref = "10" (n1,r0)
/local/domain/1/device/vbd/51713/event-channel = "10" (n1,r0)
/local/domain/1/device/vbd/51713/protocol = "x86_64-abi" (n1,r0)
/local/domain/1/device/pci = "" (n0,r1)
/local/domain/1/device/pci/0 = "" (n1,r0)
/local/domain/1/device/pci/0/backend "/local/domain/0/backend/pci/1/0"
(n1,r0)
/local/domain/1/device/pci/0/backend-id = "0" (n1,r0)
/local/domain/1/device/pci/0/state = "3" (n1,r0)
/local/domain/1/device/pci/0/pci-op-ref = "8" (n1,r0)
/local/domain/1/device/pci/0/event-channel = "8" (n1,r0)
/local/domain/1/device/pci/0/magic = "7" (n1,r0)
/local/domain/1/data = "" (n1)
/local/domain/1/cpu = "" (n0,r1)
/local/domain/1/cpu/0 = "" (n0,r1)
/local/domain/1/cpu/0/availability = "online" (n0,r1)
/local/domain/1/memory = "" (n0,r1)
/local/domain/1/memory/static-max = "31457280" (n0,r1)
/local/domain/1/memory/target = "31457280" (n0,r1)
/local/domain/1/memory/videoram = "0" (n0,r1)
/local/domain/1/error = "" (n0,r1)
/local/domain/1/drivers = "" (n0,r1)
/local/domain/1/attr = "" (n0,r1)
/local/domain/1/messages = "" (n0,r1)
/local/domain/1/domid = "1" (n0,r1)
/local/domain/1/store = "" (n0,r1)
/local/domain/1/store/port = "1" (n0,r1)
/local/domain/1/store/ring-ref = "6454766" (n0,r1)
/local/domain/1/console = "" (n1,r0)
/local/domain/1/console/backend "/local/domain/0/backend/console/1/0"
(n1,r0)
/local/domain/1/console/backend-id = "0" (n1,r0)
/local/domain/1/console/limit = "1048576" (n1,r0)
/local/domain/1/console/type = "xenconsoled" (n1,r0)
/local/domain/1/console/output = "pty" (n1,r0)
/local/domain/1/console/port = "2" (n1,r0)
/local/domain/1/console/ring-ref = "8569076" (n1,r0)
/local/domain/1/console/tty = "/dev/pts/2" (n1,r0)
/local/domain/1/hvmloader = "" (n0,r1)
/local/domain/1/hvmloader/bios = "rombios" (n0,r1)
/local/domain/1/image = "" (n0,r1)
/local/domain/1/image/device-model-pid = "230" (n0,r1)
Let''s see if someone can shed some light over this.
Thanks again, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jul-20 11:37 UTC
Re: [Xen-devel] libxl: problem with devices in PV
On Wed, 20 Jul 2011, Roger Pau Monné wrote:> Hello, > > I''m trying to run PV machines using the new libxenlight toolstack on > NetBSD. So far, I''ve been able to connect to the domu using the > console (I was unable to do so before). I''m attaching a little patch > that removes setting the consback to IOEMU when detecting a qdisk > (that was preventing the domu from even booting). With the > introduction of libxenlight, Xen doesn''t use vbd for disk backend > anymore, it uses qdisk, which I assume Qemu automatically attaches to > running guests. It works fine with HVM guests, but it seems to fail > with PV guests. The config file I''m using is: >Xen uses qdisk only when blktap is not available. I suggest you install blktap if you can because it is significantly faster than qdisk at the moment. Otherwise if you use a physical partition and specify phy: in the config file you should get the kernel based blkback that is the fastest option available.> bootloader = ''/usr/xen42/bin/pygrub'' > vcpus = ''1'' > memory = ''30720'' > > root = ''/dev/xvda2 ro'' > disk = [ > ''file:/home/xen/debian/disk.img,xvda2,w'', > ''file:/home/xen/debian/swap.img,xvda1,w'', > ] > > name = ''debian'' > > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > > And the output from booting: > > [ 0.000000] Initializing cgroup subsys cpuset > [ 0.000000] Initializing cgroup subsys cpu > [ 0.000000] Linux version 2.6.32-5-xen-amd64 (Debian > 2.6.32-34squeeze1) (dannf@debian.org) (gcc version 4.3.5 (Debian > 4.3.5-4) ) #1 SMP Thu May 19 01:16:47 UTC 2011 > [ 0.000000] Command line: root=/dev/xvda2 ro root=/dev/xvda2 ro > [ 0.000000] KERNEL supported cpus: > [ 0.000000] Intel GenuineIntel > [ 0.000000] AMD AuthenticAMD > [ 0.000000] Centaur CentaurHauls > [ 0.000000] ACPI in unprivileged domain disabled > [ 0.000000] released 0 pages of unused memory > [ 0.000000] BIOS-provided physical RAM map: > [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) > [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) > [ 0.000000] Xen: 0000000000100000 - 0000000780000000 (usable) > [ 0.000000] DMI not present or invalid. > [ 0.000000] last_pfn = 0x780000 max_arch_pfn = 0x400000000 > [ 0.000000] last_pfn = 0x100000 max_arch_pfn = 0x400000000 > [ 0.000000] init_memory_mapping: 0000000000000000-0000000100000000 > [ 0.000000] init_memory_mapping: 0000000100000000-0000000780000000 > [ 0.000000] RAMDISK: 016ba000 - 02feb000 > [ 0.000000] No NUMA configuration found > [ 0.000000] Faking a node at 0000000000000000-0000000780000000 > [ 0.000000] Bootmem setup node 0 0000000000000000-0000000780000000 > [ 0.000000] NODE_DATA [0000000000008000 - 000000000000ffff] > [ 0.000000] bootmap [00000000008c7000 - 00000000009b6fff] pages f0 > [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 0780000000] > [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> > [0000000000 - 0000001000] > [ 0.000000] #1 [0006bee000 - 0006c29000] XEN PAGETABLES ==> > [0006bee000 - 0006c29000] > [ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> > [0000006000 - 0000008000] > [ 0.000000] #3 [0001000000 - 00016999d4] TEXT DATA BSS ==> > [0001000000 - 00016999d4] > [ 0.000000] #4 [00016ba000 - 0002feb000] RAMDISK ==> > [00016ba000 - 0002feb000] > [ 0.000000] #5 [0002feb000 - 0006bee000] XEN START INFO ==> > [0002feb000 - 0006bee000] > [ 0.000000] #6 [0000100000 - 00008c7000] PGTABLE ==> > [0000100000 - 00008c7000] > [ 0.000000] #7 [0006c29000 - 000a043000] PGTABLE ==> > [0006c29000 - 000a043000] > [ 0.000000] Zone PFN ranges: > [ 0.000000] DMA 0x00000000 -> 0x00001000 > [ 0.000000] DMA32 0x00001000 -> 0x00100000 > [ 0.000000] Normal 0x00100000 -> 0x00780000 > [ 0.000000] Movable zone start PFN for each node > [ 0.000000] early_node_map[2] active PFN ranges > [ 0.000000] 0: 0x00000000 -> 0x000000a0 > [ 0.000000] 0: 0x00000100 -> 0x00780000 > [ 0.000000] SFI: Simple Firmware Interface v0.7 http://simplefirmware.org > [ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs > [ 0.000000] No local APIC present > [ 0.000000] APIC: disable apic facility > [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000 > [ 0.000000] PCI: Warning: Cannot find a gap in the 32bit address range > [ 0.000000] PCI: Unassigned devices with 32bit resource registers may break! > [ 0.000000] Allocating PCI resources starting at 780100000 (gap: > 780100000:400000) > [ 0.000000] Booting paravirtualized kernel on Xen > [ 0.000000] Xen version: 4.2-unstable (preserve-AD) > [ 0.000000] NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:1 nr_node_ids:1 > [ 0.000000] PERCPU: Embedded 30 pages/cpu @ffff88002804f000 s90328 > r8192 d24360 u122880 > [ 0.000000] pcpu-alloc: s90328 r8192 d24360 u122880 alloc=30*4096 > [ 0.000000] pcpu-alloc: [0] 0 > [ 0.000000] Xen: using vcpu_info placement > [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. > Total pages: 7754710 > [ 0.000000] Policy zone: Normal > [ 0.000000] Kernel command line: root=/dev/xvda2 ro root=/dev/xvda2 ro > [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) > [ 0.000000] Initializing CPU#0 > [ 0.000000] Checking aperture... > [ 0.000000] No AGP bridge found > [ 0.000000] Memory: 30803904k/31457280k available (3149k kernel > code, 384k absent, 652992k reserved, 1907k data, 604k init) > [ 0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, > CPUs=1, Nodes=1 > [ 0.000000] Hierarchical RCU implementation. > [ 0.000000] NR_IRQS:4352 nr_irqs:512 > [ 0.000000] Console: colour dummy device 80x25 > [ 0.000000] console [tty0] enabled > [ 0.000000] console [hvc0] enabled > [ 0.000000] installing Xen timer for CPU 0 > [ 0.000000] Detected 2394.046 MHz processor. > [ 0.004000] Calibrating delay loop (skipped), value calculated > using timer frequency.. 4788.09 BogoMIPS (lpj=9576184) > [ 0.004000] Security Framework initialized > [ 0.004000] SELinux: Disabled at boot. > [ 0.005680] Dentry cache hash table entries: 4194304 (order: 13, > 33554432 bytes) > [ 0.014625] Inode-cache hash table entries: 2097152 (order: 12, > 16777216 bytes) > [ 0.017597] Mount-cache hash table entries: 256 > [ 0.017761] Initializing cgroup subsys ns > [ 0.017769] Initializing cgroup subsys cpuacct > [ 0.017776] Initializing cgroup subsys devices > [ 0.017782] Initializing cgroup subsys freezer > [ 0.017786] Initializing cgroup subsys net_cls > [ 0.017827] CPU: L1 I cache: 32K, L1 D cache: 32K > [ 0.017835] CPU: L2 cache: 256K > [ 0.017838] CPU: L3 cache: 12288K > [ 0.017844] CPU 0/0x20 -> Node 0 > [ 0.017848] CPU: Unsupported number of siblings 32 > [ 0.017854] Performance Events: unsupported p6 CPU model 44 no PMU > driver, software events only. > [ 0.017874] SMP alternatives: switching to UP code > [ 0.020000] Freeing SMP alternatives: 28k freed > [ 0.020192] Brought up 1 CPUs > [ 0.020309] devtmpfs: initialized > [ 0.024342] Grant table initialized > [ 0.024351] regulator: core version 0.5 > [ 0.024403] NET: Registered protocol family 16 > [ 0.025064] PCI: setting up Xen PCI frontend stub > [ 0.026729] bio: create slab <bio-0> at 0 > [ 0.026798] ACPI: Interpreter disabled. > [ 0.026833] xen_balloon: Initialising balloon driver with page order 0. > [ 0.026878] vgaarb: loaded > [ 0.026939] PCI: System does not support PCI > [ 0.026944] PCI: System does not support PCI > [ 0.027019] Switching to clocksource xen > [ 0.028001] pnp: PnP ACPI: disabled > [ 0.028001] NET: Registered protocol family 2 > [ 0.028001] IP route cache hash table entries: 524288 (order: 10, > 4194304 bytes) > [ 0.028460] TCP established hash table entries: 524288 (order: 11, > 8388608 bytes) > [ 0.030137] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) > [ 0.030331] TCP: Hash tables configured (established 524288 bind 65536) > [ 0.030337] TCP reno registered > [ 0.030405] NET: Registered protocol family 1 > [ 0.030463] Unpacking initramfs... > [ 0.051340] Freeing initrd memory: 25796k freed > [ 0.058949] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) > [ 0.058964] DMA: Placing 64MB software IO TLB between > ffff880020000000 - ffff880024000000 > [ 0.058970] DMA: software IO TLB at phys 0x20000000 - 0x24000000 > [ 0.059058] platform rtc_cmos: registered platform RTC device (no > PNP device found) > [ 0.059272] audit: initializing netlink socket (disabled) > [ 0.059288] type=2000 audit(1311165823.857:1): initialized > [ 0.064506] HugeTLB registered 2 MB page size, pre-allocated 0 pages > [ 0.065744] VFS: Disk quotas dquot_6.5.2 > [ 0.065796] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) > [ 0.065877] msgmni has been set to 32768 > [ 0.066057] alg: No test for stdrng (krng) > [ 0.066112] Block layer SCSI generic (bsg) driver version 0.4 > loaded (major 253) > [ 0.066119] io scheduler noop registered > [ 0.066122] io scheduler anticipatory registered > [ 0.066127] io scheduler deadline registered > [ 0.066157] io scheduler cfq registered (default) > [ 0.072642] registering netback > [ 0.074001] Linux agpgart interface v0.103 > [ 0.074032] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled > [ 0.173825] input: Macintosh mouse button emulation as > /devices/virtual/input/input0 > [ 0.173872] PNP: No PS/2 controller found. Probing ports directly. > [ 0.174689] i8042.c: No controller found. > [ 0.174750] mice: PS/2 mouse device common for all mice > [ 0.174829] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0 > [ 0.174883] cpuidle: using governor ladder > [ 0.174890] cpuidle: using governor menu > [ 0.174897] No iBFT detected. > [ 0.175118] TCP cubic registered > [ 0.175221] NET: Registered protocol family 10 > [ 0.175788] Mobile IPv6 > [ 0.175797] NET: Registered protocol family 17 > [ 0.175889] registered taskstats version 1 > [ 5.272069] XENBUS: Waiting for devices to initialise: > 295s...290s...285s...280s...275s...270s...265s...260s...255s...250s...245s...240s...235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...130s...125s.... > udevd[46]: worker [53] unexpectedly returned with status 0x0100 > udevd[46]: worker [53] failed while handling ''/devices/vbd-51713'' > udevd[46]: worker [54] unexpectedly returned with status 0x0100 > udevd[46]: worker [54] failed while handling ''/devices/vbd-51714'' > Begin: Loading essential drivers ... done. > Begin: Running /scripts/init-premount ... done. > Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. > Begin: Waiting for root file system ... > 120s...115s...110s...105s...100s...95s...90s...done. > Gave up waiting for root device. Common problems: > - Boot args (cat /proc/cmdline) > - Check rootdelay= (did the system wait long enough?) > - Check root= (did the system wait for the right device?) > - Missing modules (cat /proc/modules; ls /dev) > ALERT! /dev/xvda2 does not exist. Dropping to a shell! > > > BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash) > Enter ''help'' for a list of built-in commands. > > /bin/sh: can''t access tty; job control turned off > (initramfs) > [ 600.488236] XENBUS: Timeout connecting to device: device/vbd/51714 > (local state 3, remote state 1) > [ 600.488328] XENBUS: Timeout connecting to device: device/vbd/51713 > (local state 3, remote state 1) > > >From what I see, the guest is unable to find the disk, which is > strange, because HVM guests work fine, and the disk is attached > correctly. Does this also happen when using file:/ backend (qdisk) > under Linux also? Is it a problem with Qemu or is it related to libxl?It doesn''t happen under Linux; it is probably a problem with Qemu. Qemu uses the gntdev device (/dev/xen/gntdev) to open the grant table (necessary to implement xen backends in userspace). There doesn''t seem anything equivalent on NetBSD, hence it fails... --8323329-1950898534-1311161885=:12963 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --8323329-1950898534-1311161885=:12963--
On Wed, 2011-07-20 at 12:37 +0100, Stefano Stabellini wrote:> On Wed, 20 Jul 2011, Roger Pau Monné wrote: > > Hello, > > > > I''m trying to run PV machines using the new libxenlight toolstack on > > NetBSD. So far, I''ve been able to connect to the domu using the > > console (I was unable to do so before). I''m attaching a little patch > > that removes setting the consback to IOEMU when detecting a qdisk > > (that was preventing the domu from even booting). With the > > introduction of libxenlight, Xen doesn''t use vbd for disk backend > > anymore, it uses qdisk, which I assume Qemu automatically attaches to > > running guests. It works fine with HVM guests, but it seems to fail > > with PV guests. The config file I''m using is: > > > > Xen uses qdisk only when blktap is not available. > I suggest you install blktap if you can because it is significantly > faster than qdisk at the moment.This is a NetBSD host so I don''t think blktap is an option.> Otherwise if you use a physical partition and specify phy: in the config > file you should get the kernel based blkback that is the fastest option > available.phy: would be worth a go since it should tie into NetBSD''s equivalent of blkback, whereas file: presumably goes to qdisk in the absence of blktap. [...]> > >From what I see, the guest is unable to find the disk, which is > > strange, because HVM guests work fine, and the disk is attached > > correctly. Does this also happen when using file:/ backend (qdisk) > > under Linux also? Is it a problem with Qemu or is it related to libxl? > > It doesn''t happen under Linux; it is probably a problem with Qemu. Qemu > uses the gntdev device (/dev/xen/gntdev) to open the grant table > (necessary to implement xen backends in userspace). > There doesn''t seem anything equivalent on NetBSD, hence it fails...That makes some sense. libxl should really detect that qdisk isn''t an option under NetBSD (like it does for blktap) and abort if that is the case. This could be a case of a simple check in tools/libxl/libxl_device.c:disk_try_backend. (could be as simple as an #ifdef __NetBSD__ or we could e.g. add -DHAVE_QEMU_BACKENDS to the build infrastructure where appropriate). Presumably if libxl isn''t trying to start a qdisk it won''t try and use a qemu based console either (which also won''t work under NetBSD) and the original patch from this thread becomes unnecessary. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
2011/7/20 Ian Campbell <Ian.Campbell@citrix.com>:> On Wed, 2011-07-20 at 12:37 +0100, Stefano Stabellini wrote: >> On Wed, 20 Jul 2011, Roger Pau Monné wrote: >> > Hello, >> > >> > I''m trying to run PV machines using the new libxenlight toolstack on >> > NetBSD. So far, I''ve been able to connect to the domu using the >> > console (I was unable to do so before). I''m attaching a little patch >> > that removes setting the consback to IOEMU when detecting a qdisk >> > (that was preventing the domu from even booting). With the >> > introduction of libxenlight, Xen doesn''t use vbd for disk backend >> > anymore, it uses qdisk, which I assume Qemu automatically attaches to >> > running guests. It works fine with HVM guests, but it seems to fail >> > with PV guests. The config file I''m using is: >> > >> >> Xen uses qdisk only when blktap is not available. >> I suggest you install blktap if you can because it is significantly >> faster than qdisk at the moment. > > This is a NetBSD host so I don''t think blktap is an option.NetBSD has the vnd driver, which provides a disk interface to a file (it''s basically a loop device): http://netbsd.gw.com/cgi-bin/man-cgi?vnd+4+NetBSD-current It is used with xend and xenbackendd to mount raw disks. It is not the fancy blktap2, altough something similar to blktap2 *could* be implemented *easily* using the RUMP Anykernel I think: http://www.netbsd.org/docs/rump/index.html Don''t know for sure, but I think it should be possible to port the blktap2 driver or something very similar using this.> >> Otherwise if you use a physical partition and specify phy: in the config >> file you should get the kernel based blkback that is the fastest option >> available. > > phy: would be worth a go since it should tie into NetBSD''s equivalent of > blkback, whereas file: presumably goes to qdisk in the absence of > blktap.Haven''t tried phy:/ with unstable, but I supose it works, since it worked in previous versions.> > [...] >> > >From what I see, the guest is unable to find the disk, which is >> > strange, because HVM guests work fine, and the disk is attached >> > correctly. Does this also happen when using file:/ backend (qdisk) >> > under Linux also? Is it a problem with Qemu or is it related to libxl? >> >> It doesn''t happen under Linux; it is probably a problem with Qemu. Qemu >> uses the gntdev device (/dev/xen/gntdev) to open the grant table >> (necessary to implement xen backends in userspace). >> There doesn''t seem anything equivalent on NetBSD, hence it fails... > > That makes some sense. libxl should really detect that qdisk isn''t an > option under NetBSD (like it does for blktap) and abort if that is the > case. This could be a case of a simple check in > tools/libxl/libxl_device.c:disk_try_backend. (could be as simple as an > #ifdef __NetBSD__ or we could e.g. add -DHAVE_QEMU_BACKENDS to the build > infrastructure where appropriate). > > Presumably if libxl isn''t trying to start a qdisk it won''t try and use a > qemu based console either (which also won''t work under NetBSD) and the > original patch from this thread becomes unnecessary.What I don''t get is why qdisk work with HVM machines but not with PV machines, shouldn''t it be the same?> > Ian. > >Also, I saw that on Linux the loop device was used to mount images in the past, but with libxl the loop device isn''t used anymore right? I think that using the vnd device is the only way to get file images to work under NetBSD. Thanks, Roger. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2011-07-20 at 14:10 +0100, Roger Pau Monné wrote:> 2011/7/20 Ian Campbell <Ian.Campbell@citrix.com>: > > On Wed, 2011-07-20 at 12:37 +0100, Stefano Stabellini wrote: > >> On Wed, 20 Jul 2011, Roger Pau Monné wrote: > >> > Hello, > >> > > >> > I''m trying to run PV machines using the new libxenlight toolstack on > >> > NetBSD. So far, I''ve been able to connect to the domu using the > >> > console (I was unable to do so before). I''m attaching a little patch > >> > that removes setting the consback to IOEMU when detecting a qdisk > >> > (that was preventing the domu from even booting). With the > >> > introduction of libxenlight, Xen doesn''t use vbd for disk backend > >> > anymore, it uses qdisk, which I assume Qemu automatically attaches to > >> > running guests. It works fine with HVM guests, but it seems to fail > >> > with PV guests. The config file I''m using is: > >> > > >> > >> Xen uses qdisk only when blktap is not available. > >> I suggest you install blktap if you can because it is significantly > >> faster than qdisk at the moment. > > > > This is a NetBSD host so I don''t think blktap is an option. > > NetBSD has the vnd driver, which provides a disk interface to a file > (it''s basically a loop device): > > http://netbsd.gw.com/cgi-bin/man-cgi?vnd+4+NetBSD-current > > It is used with xend and xenbackendd to mount raw disks.If I understand correctly (which is remotish possibility): xenbackendd watches the backend/vbd directory and creates disks based on it. A "vbd" directory roughly corresponds to a "phy:" type device configuration in libxl. xend, at least on Linux, internally handles "file:" by setting up a loop device and translating the config into a "phy:"==vbd refering to the /dev/loopN device. I suspect that on NetBSD xend used to just create the vbd referring to the file directly and let xenbackendd take care of any necessary loop back stuff. libxl on the other hand does not do this loop device magic but instead palms "file:" off onto either qdisk or blktap and furthermore libxl requires that a "phy:" disk configuration refers to a block device and not a file. The upshot is that the backend selection logic in libxl is apparently not correct for NetBSD which _can_ handle files passed as "phy:" due to xenbackendd doing the right thing via the vnd driver. Fortunately Ian Jackson recently cleaned up the backend selection logic in libxl and it should be much cleaner and easier to express these sorts of system specific things, presumably by patching disk_try_backend to allow files as well as block devices for LIBXL_DISK_BACKEND_PHY on NetBSD.> >> Otherwise if you use a physical partition and specify phy: in the config > >> file you should get the kernel based blkback that is the fastest option > >> available. > > > > phy: would be worth a go since it should tie into NetBSD''s equivalent of > > blkback, whereas file: presumably goes to qdisk in the absence of > > blktap. > > Haven''t tried phy:/ with unstable, but I supose it works, since it > worked in previous versions.Since you are passing a file name a block device it is possible that libxl today might reject it when used with phy:/... [...]> What I don''t get is why qdisk work with HVM machines but not with PV > machines, shouldn''t it be the same?I suspect that in the HVM case you aren''t really getting a qdisk (a PV backend) but actually an emulated disk device. Or rather you are probably getting both but the guest only tries to use the emulated one and hence the PV backend never tries to initialise and so never falls over due to lack of gntdev. Can you confirm if your HVM guest uses emulated or PV frontends? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 07/20/11 15:30, Ian Campbell wrote:> On Wed, 2011-07-20 at 14:10 +0100, Roger Pau Monné wrote: >> 2011/7/20 Ian Campbell<Ian.Campbell@citrix.com>: >>> On Wed, 2011-07-20 at 12:37 +0100, Stefano Stabellini wrote: >>>> On Wed, 20 Jul 2011, Roger Pau Monné wrote: >>>>> Hello, >>>>> >>>>> I''m trying to run PV machines using the new libxenlight toolstack on >>>>> NetBSD. So far, I''ve been able to connect to the domu using the >>>>> console (I was unable to do so before). I''m attaching a little patch >>>>> that removes setting the consback to IOEMU when detecting a qdisk >>>>> (that was preventing the domu from even booting). With the >>>>> introduction of libxenlight, Xen doesn''t use vbd for disk backend >>>>> anymore, it uses qdisk, which I assume Qemu automatically attaches to >>>>> running guests. It works fine with HVM guests, but it seems to fail >>>>> with PV guests. The config file I''m using is: >>>>> >>>> >>>> Xen uses qdisk only when blktap is not available. >>>> I suggest you install blktap if you can because it is significantly >>>> faster than qdisk at the moment. >>> >>> This is a NetBSD host so I don''t think blktap is an option. >> >> NetBSD has the vnd driver, which provides a disk interface to a file >> (it''s basically a loop device): >> >> http://netbsd.gw.com/cgi-bin/man-cgi?vnd+4+NetBSD-current >> >> It is used with xend and xenbackendd to mount raw disks. > > If I understand correctly (which is remotish possibility): > > xenbackendd watches the backend/vbd directory and creates disks based on > it. A "vbd" directory roughly corresponds to a "phy:" type device > configuration in libxl.Correct.> xend, at least on Linux, internally handles "file:" by setting up a loop > device and translating the config into a "phy:"==vbd refering to > the /dev/loopN device. I suspect that on NetBSD xend used to just create > the vbd referring to the file directly and let xenbackendd take care of > any necessary loop back stuff.More precisely, xenbackendd invokes the hotplug scripts and those take care of choosing a spare vnd device.> libxl on the other hand does not do this loop device magic but instead > palms "file:" off onto either qdisk or blktap and furthermore libxl > requires that a "phy:" disk configuration refers to a block device and > not a file.>> The upshot is that the backend selection logic in libxl is apparently > not correct for NetBSD which _can_ handle files passed as "phy:" due to > xenbackendd doing the right thing via the vnd driver.Right.> Fortunately Ian Jackson recently cleaned up the backend selection logic > in libxl and it should be much cleaner and easier to express these sorts > of system specific things, presumably by patching disk_try_backend to > allow files as well as block devices for LIBXL_DISK_BACKEND_PHY on > NetBSD.Roger: In libxl/libxl_device.c:disk_try_backend() can you try disabling the second check in the LIBXL_DISK_BACKEND_PHY case, please? Or surrounding it with #ifdef __Linux__ ... #endif should do it. Christoph>>>> Otherwise if you use a physical partition and specify phy: in the config >>>> file you should get the kernel based blkback that is the fastest option >>>> available. >>> >>> phy: would be worth a go since it should tie into NetBSD''s equivalent of >>> blkback, whereas file: presumably goes to qdisk in the absence of >>> blktap. >> >> Haven''t tried phy:/ with unstable, but I supose it works, since it >> worked in previous versions. > > Since you are passing a file name a block device it is possible that > libxl today might reject it when used with phy:/... > > [...] >> What I don''t get is why qdisk work with HVM machines but not with PV >> machines, shouldn''t it be the same? > > I suspect that in the HVM case you aren''t really getting a qdisk (a PV > backend) but actually an emulated disk device. Or rather you are > probably getting both but the guest only tries to use the emulated one > and hence the PV backend never tries to initialise and so never falls > over due to lack of gntdev. > > Can you confirm if your HVM guest uses emulated or PV frontends? > > Ian.-- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85689 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Thanks for the help, I have a preliminary patch that allows booting PV machines with xl on NetBSD, but there seems to be some issues with PCI devices, the guest takes a very long time to start, and I think it''s because it''s waiting for PCI devices to initialize. [ 5.280068] XENBUS: Waiting for devices to initialise: 295s...290s...285s...280s...275s...270s...265s...260s...255s...250s...245s...240s...235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...90s...85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s...25s...20s...15s...10s...5s...0s... [ 300.280078] XENBUS: Device with no driver: device/vbd/51714 [ 300.280086] XENBUS: Device with no driver: device/vbd/51713 [ 300.280495] XENBUS: Timeout connecting to device: device/pci/0 (local state 3, remote state 1) xm didn''t add pci entries to xenstore, and I don''t know what should be the status of those entries, or if it''s best to disable the adding of pci entries to xenstore in libxl if no devices are configured? I''ve looked at http://wiki.xensource.com/xenwiki/XenStoreReference, but the wiki doesn''t have any information regarding PCI xenstore entries. The status of those entries in xenstore is the following: /local/domain/0/backend/pci = "" (n0) /local/domain/0/backend/pci/1 = "" (n0) /local/domain/0/backend/pci/1/0 = "" (n0,r1) /local/domain/0/backend/pci/1/0/frontend "/local/domain/1/device/pci/0" (n0,r1) /local/domain/0/backend/pci/1/0/frontend-id = "1" (n0,r1) /local/domain/0/backend/pci/1/0/online = "1" (n0,r1) /local/domain/0/backend/pci/1/0/state = "1" (n0,r1) /local/domain/0/backend/pci/1/0/domain = "debian" (n0,r1) /local/domain/0/backend/pci/1/0/num_devs = "0" (n0,r1) /local/domain/1/device/pci = "" (n0,r1) /local/domain/1/device/pci/0 = "" (n1,r0) /local/domain/1/device/pci/0/backend "/local/domain/0/backend/pci/1/0" (n1,r0) /local/domain/1/device/pci/0/backend-id = "0" (n1,r0) /local/domain/1/device/pci/0/state = "3" (n1,r0) /local/domain/1/device/pci/0/pci-op-ref = "8" (n1,r0) /local/domain/1/device/pci/0/event-channel = "8" (n1,r0) /local/domain/1/device/pci/0/magic = "7" (n1,r0) The patch attached breaks linux support, it is only attached for informative purposes. Regards, Roger. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-07-21 at 09:39 +0100, Roger Pau Monné wrote:> Hello, > > Thanks for the help, I have a preliminary patch that allows booting PV > machines with xl on NetBSD, but there seems to be some issues with PCI > devices, the guest takes a very long time to start, and I think it''s > because it''s waiting for PCI devices to initialize. > > [ 5.280068] XENBUS: Waiting for devices to initialise: > 295s...290s...285s...280s...275s...270s...265s...260s...255s...250s...245s...240s...235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...90s...85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s...25s...20s...15s...10s...5s...0s... > [ 300.280078] XENBUS: Device with no driver: device/vbd/51714 > [ 300.280086] XENBUS: Device with no driver: device/vbd/51713 > [ 300.280495] XENBUS: Timeout connecting to device: device/pci/0 > (local state 3, remote state 1) > > xm didn''t add pci entries to xenstore, and I don''t know what should be > the status of those entries, or if it''s best to disable the adding of > pci entries to xenstore in libxl if no devices are configured? > > I''ve looked at http://wiki.xensource.com/xenwiki/XenStoreReference, > but the wiki doesn''t have any information regarding PCI xenstore > entries. > > The status of those entries in xenstore is the following: > > /local/domain/0/backend/pci = "" (n0) > /local/domain/0/backend/pci/1 = "" (n0) > /local/domain/0/backend/pci/1/0 = "" (n0,r1) > /local/domain/0/backend/pci/1/0/frontend > "/local/domain/1/device/pci/0" (n0,r1) > /local/domain/0/backend/pci/1/0/frontend-id = "1" (n0,r1) > /local/domain/0/backend/pci/1/0/online = "1" (n0,r1) > /local/domain/0/backend/pci/1/0/state = "1" (n0,r1) > /local/domain/0/backend/pci/1/0/domain = "debian" (n0,r1) > /local/domain/0/backend/pci/1/0/num_devs = "0" (n0,r1)num_devs = 0, is it the case that you don''t actually have any pci devices in your configuration? I wonder if perhaps the netbsd pciback doesn''t like seeing a backend directory with no actual devices in it. Could you try experimentally commenting out the call to libxl__create_pci_backend in do_domain_create, or gating it on "if (d_config->num_pcidevs)". That call was added in 23565:72eafe80ebc1 but I don''t think adding the if would invalidate it.> /local/domain/1/device/pci = "" (n0,r1) > /local/domain/1/device/pci/0 = "" (n1,r0) > /local/domain/1/device/pci/0/backend > "/local/domain/0/backend/pci/1/0" (n1,r0) > /local/domain/1/device/pci/0/backend-id = "0" (n1,r0) > /local/domain/1/device/pci/0/state = "3" (n1,r0) > /local/domain/1/device/pci/0/pci-op-ref = "8" (n1,r0) > /local/domain/1/device/pci/0/event-channel = "8" (n1,r0) > /local/domain/1/device/pci/0/magic = "7" (n1,r0) > > The patch attached breaks linux support, it is only attached for > informative purposes. > > Regards, Roger._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
2011/7/21 Ian Campbell <Ian.Campbell@eu.citrix.com>:> On Thu, 2011-07-21 at 09:39 +0100, Roger Pau Monné wrote: >> Hello, >> >> Thanks for the help, I have a preliminary patch that allows booting PV >> machines with xl on NetBSD, but there seems to be some issues with PCI >> devices, the guest takes a very long time to start, and I think it''s >> because it''s waiting for PCI devices to initialize. >> >> [ 5.280068] XENBUS: Waiting for devices to initialise: >> 295s...290s...285s...280s...275s...270s...265s...260s...255s...250s...245s...240s...235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...90s...85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s...25s...20s...15s...10s...5s...0s... >> [ 300.280078] XENBUS: Device with no driver: device/vbd/51714 >> [ 300.280086] XENBUS: Device with no driver: device/vbd/51713 >> [ 300.280495] XENBUS: Timeout connecting to device: device/pci/0 >> (local state 3, remote state 1) >> >> xm didn''t add pci entries to xenstore, and I don''t know what should be >> the status of those entries, or if it''s best to disable the adding of >> pci entries to xenstore in libxl if no devices are configured? >> >> I''ve looked at http://wiki.xensource.com/xenwiki/XenStoreReference, >> but the wiki doesn''t have any information regarding PCI xenstore >> entries. >> >> The status of those entries in xenstore is the following: >> >> /local/domain/0/backend/pci = "" (n0) >> /local/domain/0/backend/pci/1 = "" (n0) >> /local/domain/0/backend/pci/1/0 = "" (n0,r1) >> /local/domain/0/backend/pci/1/0/frontend >> "/local/domain/1/device/pci/0" (n0,r1) >> /local/domain/0/backend/pci/1/0/frontend-id = "1" (n0,r1) >> /local/domain/0/backend/pci/1/0/online = "1" (n0,r1) >> /local/domain/0/backend/pci/1/0/state = "1" (n0,r1) >> /local/domain/0/backend/pci/1/0/domain = "debian" (n0,r1) >> /local/domain/0/backend/pci/1/0/num_devs = "0" (n0,r1) > > num_devs = 0, is it the case that you don''t actually have any pci > devices in your configuration?No, I didn''t have any PCI devices set.> > I wonder if perhaps the netbsd pciback doesn''t like seeing a backend > directory with no actual devices in it. > > Could you try experimentally commenting out the call to > libxl__create_pci_backend in do_domain_create, or gating it on "if > (d_config->num_pcidevs)".Disabling or setting it with a if condition makes the guest boot, I through that setting the appropriate values in the status of xenstore should also work, but it''s useless to have references to PCI devices in xenstore if no devices are actually configured.> > That call was added in 23565:72eafe80ebc1 but I don''t think adding the > if would invalidate it. > >> /local/domain/1/device/pci = "" (n0,r1) >> /local/domain/1/device/pci/0 = "" (n1,r0) >> /local/domain/1/device/pci/0/backend >> "/local/domain/0/backend/pci/1/0" (n1,r0) >> /local/domain/1/device/pci/0/backend-id = "0" (n1,r0) >> /local/domain/1/device/pci/0/state = "3" (n1,r0) >> /local/domain/1/device/pci/0/pci-op-ref = "8" (n1,r0) >> /local/domain/1/device/pci/0/event-channel = "8" (n1,r0) >> /local/domain/1/device/pci/0/magic = "7" (n1,r0) >> >> The patch attached breaks linux support, it is only attached for >> informative purposes. >> >> Regards, Roger.Thanks again for all the help, I would prepare a new proper patch to be added to mainline Xen. Regards, Roger. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
BTW, would it be ok to create a new disk backend, like LOOP or VND (LIBXL_DISK_BACKEND_VND)? Now it''s a bit hackish, since type is always set to phy, I have to guess the real type by checking if ''params'' starts with ''/dev'' on xenbackend. I think it would be clearer to set the type to LIBXL_DISK_BACKEND_VND if passed file is not a block device and OS is NetBSD in disk_try_backend. Regards, Roger. 2011/7/21 Roger Pau Monné <roger.pau@entel.upc.edu>:> 2011/7/21 Ian Campbell <Ian.Campbell@eu.citrix.com>: >> On Thu, 2011-07-21 at 09:39 +0100, Roger Pau Monné wrote: >>> Hello, >>> >>> Thanks for the help, I have a preliminary patch that allows booting PV >>> machines with xl on NetBSD, but there seems to be some issues with PCI >>> devices, the guest takes a very long time to start, and I think it''s >>> because it''s waiting for PCI devices to initialize. >>> >>> [ 5.280068] XENBUS: Waiting for devices to initialise: >>> 295s...290s...285s...280s...275s...270s...265s...260s...255s...250s...245s...240s...235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...90s...85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s...25s...20s...15s...10s...5s...0s... >>> [ 300.280078] XENBUS: Device with no driver: device/vbd/51714 >>> [ 300.280086] XENBUS: Device with no driver: device/vbd/51713 >>> [ 300.280495] XENBUS: Timeout connecting to device: device/pci/0 >>> (local state 3, remote state 1) >>> >>> xm didn''t add pci entries to xenstore, and I don''t know what should be >>> the status of those entries, or if it''s best to disable the adding of >>> pci entries to xenstore in libxl if no devices are configured? >>> >>> I''ve looked at http://wiki.xensource.com/xenwiki/XenStoreReference, >>> but the wiki doesn''t have any information regarding PCI xenstore >>> entries. >>> >>> The status of those entries in xenstore is the following: >>> >>> /local/domain/0/backend/pci = "" (n0) >>> /local/domain/0/backend/pci/1 = "" (n0) >>> /local/domain/0/backend/pci/1/0 = "" (n0,r1) >>> /local/domain/0/backend/pci/1/0/frontend >>> "/local/domain/1/device/pci/0" (n0,r1) >>> /local/domain/0/backend/pci/1/0/frontend-id = "1" (n0,r1) >>> /local/domain/0/backend/pci/1/0/online = "1" (n0,r1) >>> /local/domain/0/backend/pci/1/0/state = "1" (n0,r1) >>> /local/domain/0/backend/pci/1/0/domain = "debian" (n0,r1) >>> /local/domain/0/backend/pci/1/0/num_devs = "0" (n0,r1) >> >> num_devs = 0, is it the case that you don''t actually have any pci >> devices in your configuration? > > No, I didn''t have any PCI devices set. > >> >> I wonder if perhaps the netbsd pciback doesn''t like seeing a backend >> directory with no actual devices in it. >> >> Could you try experimentally commenting out the call to >> libxl__create_pci_backend in do_domain_create, or gating it on "if >> (d_config->num_pcidevs)". > > Disabling or setting it with a if condition makes the guest boot, I > through that setting the appropriate values in the status of xenstore > should also work, but it''s useless to have references to PCI devices > in xenstore if no devices are actually configured. > >> >> That call was added in 23565:72eafe80ebc1 but I don''t think adding the >> if would invalidate it. >> >>> /local/domain/1/device/pci = "" (n0,r1) >>> /local/domain/1/device/pci/0 = "" (n1,r0) >>> /local/domain/1/device/pci/0/backend >>> "/local/domain/0/backend/pci/1/0" (n1,r0) >>> /local/domain/1/device/pci/0/backend-id = "0" (n1,r0) >>> /local/domain/1/device/pci/0/state = "3" (n1,r0) >>> /local/domain/1/device/pci/0/pci-op-ref = "8" (n1,r0) >>> /local/domain/1/device/pci/0/event-channel = "8" (n1,r0) >>> /local/domain/1/device/pci/0/magic = "7" (n1,r0) >>> >>> The patch attached breaks linux support, it is only attached for >>> informative purposes. >>> >>> Regards, Roger. > > Thanks again for all the help, I would prepare a new proper patch to > be added to mainline Xen. > > Regards, Roger. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-07-21 at 10:19 +0100, Roger Pau Monné wrote:> 2011/7/21 Ian Campbell <Ian.Campbell@eu.citrix.com>: > > On Thu, 2011-07-21 at 09:39 +0100, Roger Pau Monné wrote: > > > > I wonder if perhaps the netbsd pciback doesn''t like seeing a backend > > directory with no actual devices in it. > > > > Could you try experimentally commenting out the call to > > libxl__create_pci_backend in do_domain_create, or gating it on "if > > (d_config->num_pcidevs)". > > Disabling or setting it with a if condition makes the guest boot, I > through that setting the appropriate values in the status of xenstore > should also work, but it''s useless to have references to PCI devices > in xenstore if no devices are actually configured.There was a tricky interaction between the support for hotplugging PCI devices and adding them one-by-one to a non-running guest during domain build (the latter confuses the Linux pciback because it waits for the frontend to respond to the reconfiguration requests for the second and subsequent devices added and the guest ain''t running). At the time we made the creation of the pcibackend xenstore directory unconditional because it simplified things However I think enough fixes have accumulated since then that we can switch to only creating the backend if the guest has any pci devices configured and hotplug will still work once the guest is unpaused. (IOW I think adding the if (...) is correct and safe but we should be mindful of breaking hotplug)> > > > That call was added in 23565:72eafe80ebc1 but I don''t think adding the > > if would invalidate it. > > > >> /local/domain/1/device/pci = "" (n0,r1) > >> /local/domain/1/device/pci/0 = "" (n1,r0) > >> /local/domain/1/device/pci/0/backend > >> "/local/domain/0/backend/pci/1/0" (n1,r0) > >> /local/domain/1/device/pci/0/backend-id = "0" (n1,r0) > >> /local/domain/1/device/pci/0/state = "3" (n1,r0) > >> /local/domain/1/device/pci/0/pci-op-ref = "8" (n1,r0) > >> /local/domain/1/device/pci/0/event-channel = "8" (n1,r0) > >> /local/domain/1/device/pci/0/magic = "7" (n1,r0) > >> > >> The patch attached breaks linux support, it is only attached for > >> informative purposes. > >> > >> Regards, Roger. > > Thanks again for all the help, I would prepare a new proper patch to > be added to mainline Xen.That would be great, thanks. WRT the proper fix I think rather than sprinkling ifdef __NetBSD__ and __Linux__ about it would preferable to collect that logic together in one place (maybe in libxl_osdeps.h?) and define meaningful tags (HAVE_DISK_PHY_FILE_SUPPORT and HAVE_DISK_BLKTAP_SUPPORT or something better) to use in the actual code. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2011-07-21 at 10:39 +0100, Roger Pau Monné wrote:> BTW, would it be ok to create a new disk backend, like LOOP or VND > (LIBXL_DISK_BACKEND_VND)? > > Now it''s a bit hackish, since type is always set to phy, I have to > guess the real type by checking if ''params'' starts with ''/dev'' on > xenbackend. I think it would be clearer to set the type to > LIBXL_DISK_BACKEND_VND if passed file is not a block device and OS is > NetBSD in disk_try_backend.The VND stuff is transparent to the toolstack though, it''s internal to xenbackendd and the hotplug scripts, all the toolstack sees is a vbd directory in xenstore. So I think it would be better to stick with the phy/vbd types within the toolstack. Can the code in disk_try_backend become: [in some suitable central location:] #if __NetBSD__ #define HAVE_PHY_BACKNED_FILE_SUPPORT #endif [...] case LIBXL_DISK_BACKEND_PHY: if (!(a->disk->format == LIBXL_DISK_FORMAT_RAW || a->disk->format == LIBXL_DISK_FORMAT_EMPTY)) { goto bad_format; } if (a->disk->format == LIBXL_DISK_FORMAT_EMPTY) return backend; #ifdef HAVE_PHY_BACKEND_FILE_SUPPORT if (S_ISREG(a->stab.st_mode)) /* S_ISLNK? */ return backend; #endif if (S_ISBLK(a->stab.st_mode)) return backend; LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Disk vdev=%s, backend phy" " unsuitable as phys path not a block device" #ifdef HAVE_PHY_... " nor a regular file" #endif , a->disk->vdev); return 0; ? (perhaps the log message construction could be done cleaner) Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel