thr3ads.net - Xen users - Xen 4.1.2 - stub domains for IO. Not working ? [Feb 2013]

If this information is useful, please help other people find it:
Share via:

Sylvain Munaut

2013-Feb-19 14:58 UTC

Xen 4.1.2 - stub domains for IO. Not working ?

Hi,

I''m trying to use stub domains for IO for HVM domains, under Xen 4.1.2
with xl toolstack

The domain works fine with either qemu-xen-traditional or qemu
upstream, but I''d like to try stub domains.

If I understand the doc correctly just adding
"device_model_stubdomain_override = 1" to the config should do the
trick.

However doing that doesn''t really work.

 - qemu-dm is still started in the dom0 which AFAIU shouldn''t be the
case.
 - There is a xxx-dm domain that starts so that''s a good sign, however
they both stay stuck like :

wxp-00                                       7  1531     1     ------       0.0
wxp-00-dm                                    8    32     1     ---sc-       0.0

Here''s the xl -vvv log :

root@smallbox:/etc/xen # xl -vvv create vm-wxp-00
Parsing config from vm-wxp-00
libxl: debug: libxl_create.c:1174:do_domain_create: ao 0x2578f40:
create: how=(nil) callback=(nil) poller=0x2578820
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:265:libxl__device_disk_set_backend: Disk
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:677:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:321:libxl__bootloader_run: not a PV
domain, skipping bootloader
libxl: debug: libxl_event.c:561:libxl__ev_xswatch_deregister: watch
w=0x2579310: deregister unregistered
libxl: debug: libxl_numa.c:435:libxl__get_numa_candidate: New best
NUMA placement candidate found: nr_nodes=1, nr_cpus=2, nr_vcpus=4,
free_memkb=7321
libxl: detail: libxl_dom.c:192:numa_place_domain: NUMA placement
candidate with 1 nodes, 2 cpus and 7321 KB free selected
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x9dda8
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x19dda8
xc: info: VIRTUAL MEMORY ARRANGEMENT:
  Loader:        0000000000100000->000000000019dda8
  TOTAL:         0000000000000000->000000005f800000
  ENTRY ADDRESS: 0000000000100608
xc: info: PHYSICAL MEMORY ALLOCATION:
  4KB PAGES: 0x0000000000000200
  2MB PAGES: 0x00000000000002fb
  1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x0x7ffc12f5d000 -> 0x0x7ffc12ff1c35
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=phy
libxl: debug: libxl_event.c:512:libxl__ev_xswatch_register: watch
w=0x257a918 wpath=/local/domain/0/backend/vbd/7/51712/state token=3/0:
register slotnum=3
libxl: debug: libxl_create.c:1187:do_domain_create: ao 0x2578f40:
inprogress: poller=0x2578820, flags=i
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x257a918
wpath=/local/domain/0/backend/vbd/7/51712/state token=3/0: event
epath=/local/domain/0/backend/vbd/7/51712/state
libxl: debug: libxl_event.c:600:devstate_watch_callback: backend
/local/domain/0/backend/vbd/7/51712/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x257a918
wpath=/local/domain/0/backend/vbd/7/51712/state token=3/0: event
epath=/local/domain/0/backend/vbd/7/51712/state
libxl: debug: libxl_event.c:596:devstate_watch_callback: backend
/local/domain/0/backend/vbd/7/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:549:libxl__ev_xswatch_deregister: watch
w=0x257a918 wpath=/local/domain/0/backend/vbd/7/51712/state token=3/0:
deregister slotnum=3
libxl: debug: libxl_event.c:561:libxl__ev_xswatch_deregister: watch
w=0x257a918: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug
script: /etc/xen/scripts/block add
libxl: debug: libxl_numa.c:435:libxl__get_numa_candidate: New best
NUMA placement candidate found: nr_nodes=1, nr_cpus=2, nr_vcpus=5,
free_memkb=5783
libxl: detail: libxl_dom.c:192:numa_place_domain: NUMA placement
candidate with 1 nodes, 2 cpus and 5783 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline=" -d 7",
features=""
libxl: debug: libxl_dom.c:380:libxl__build_pv: pv kernel mapped 0 path
/usr/lib/xen-4.2/boot/ioemu-stubdom.gz

domainbuilder: detail: xc_dom_kernel_file:
filename="/usr/lib/xen-4.2/boot/ioemu-stubdom.gz"
domainbuilder: detail: xc_dom_malloc_filemap    : 826 kB
domainbuilder: detail: xc_dom_malloc            : 8487 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0xce974 -> 0x849cc3
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.2, caps
xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ...
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x0 memsz=0x566d18
xc: detail: elf_parse_binary: memory: 0x0 -> 0x566d18
xc: detail: elf_xen_parse: __xen_guest:
"GUEST_OS=Mini-OS,XEN_VER=xen-3.0,VIRT_BASE=0x0,ELF_PADDR_OFFSET=0x0,HYPERCALL_PAGE=0x2,LOADER=generic"
xc: detail: elf_xen_parse_guest_info: GUEST_OS="Mini-OS"
xc: detail: elf_xen_parse_guest_info: XEN_VER="xen-3.0"
xc: detail: elf_xen_parse_guest_info: VIRT_BASE="0x0"
xc: detail: elf_xen_parse_guest_info: ELF_PADDR_OFFSET="0x0"
xc: detail: elf_xen_parse_guest_info: HYPERCALL_PAGE="0x2"
xc: detail: elf_xen_parse_guest_info: LOADER="generic"
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0x0
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0x0
xc: detail:     virt_kstart      = 0x0
xc: detail:     virt_kend        = 0x566d18
xc: detail:     virt_entry       = 0x0
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0x0 ->
0x566d18
domainbuilder: detail: xc_dom_mem_init: mem 32 MB, pages 0x2000 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x2000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x0 ->
0x567000  (pfn 0x0 + 0x567 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x0+0x567
at 0x7ffc10181000
xc: detail: elf_load_binary: phdr 0 at 0x0x7ffc10181000 -> 0x0x7ffc106e7d18
domainbuilder: detail: xc_dom_alloc_segment:   phys2mach    : 0x567000
-> 0x577000  (pfn 0x567 + 0x10 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x567+0x10
at 0x7ffc10171000
domainbuilder: detail: xc_dom_alloc_page   :   start info   : 0x577000
(pfn 0x577)
domainbuilder: detail: xc_dom_alloc_page   :   xenstore     : 0x578000
(pfn 0x578)
domainbuilder: detail: xc_dom_alloc_page   :   console      : 0x579000
(pfn 0x579)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48:
0x0000000000000000 -> 0x0000ffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39:
0x0000000000000000 -> 0x0000007fffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30:
0x0000000000000000 -> 0x000000003fffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21:
0x0000000000000000 -> 0x00000000007fffff, 4 table(s)
domainbuilder: detail: xc_dom_alloc_segment:   page tables  : 0x57a000
-> 0x581000  (pfn 0x57a + 0x7 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x57a+0x7
at 0x7ffc1016a000
domainbuilder: detail: xc_dom_alloc_page   :   boot stack   : 0x581000
(pfn 0x581)
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x582000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x800000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x2000
domainbuilder: detail: clear_page: pfn 0x579, mfn 0x1f58fd
domainbuilder: detail: clear_page: pfn 0x578, mfn 0x1f58fe
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x577+0x1
at 0x7ffc10167000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0x2000 pfn=0x2
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 8586 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 826 kB
domainbuilder: detail:       domU mmap          : 5628 kB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xd7d0e
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x57a mfn 0x1f58fc
domainbuilder: detail: launch_vm: called, ctxt=0x7fff20e44ba0
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=phy
libxl: debug: libxl_event.c:512:libxl__ev_xswatch_register: watch
w=0x257da68 wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1:
register slotnum=3
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x257da68
wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: event
epath=/local/domain/0/backend/vbd/8/51712/state
libxl: debug: libxl_event.c:600:devstate_watch_callback: backend
/local/domain/0/backend/vbd/8/51712/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x257da68
wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: event
epath=/local/domain/0/backend/vbd/8/51712/state
libxl: debug: libxl_event.c:596:devstate_watch_callback: backend
/local/domain/0/backend/vbd/8/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:549:libxl__ev_xswatch_deregister: watch
w=0x257da68 wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1:
deregister slotnum=3
libxl: debug: libxl_event.c:561:libxl__ev_xswatch_deregister: watch
w=0x257da68: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug
script: /etc/xen/scripts/block add
libxl: debug: libxl_dm.c:1142:libxl__spawn_local_dm: Spawning
device-model /usr/lib/xen-4.2/bin/qemu-dm with arguments:
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:
/usr/lib/xen-4.2/bin/qemu-dm
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:   -d
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:   8
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:   -domain-name
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:   wxp-00-dm
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:   -vnc
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:   0.0.0.0:0,password
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:   -M
libxl: debug: libxl_dm.c:1144:libxl__spawn_local_dm:   xenpv
libxl: debug: libxl_event.c:512:libxl__ev_xswatch_register: watch
w=0x2579908 wpath=/local/domain/0/device-model/8/state token=3/2:
register slotnum=3
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x2579908
wpath=/local/domain/0/device-model/8/state token=3/2: event
epath=/local/domain/0/device-model/8/state
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x2579908
wpath=/local/domain/0/device-model/8/state token=3/2: event
epath=/local/domain/0/device-model/8/state
libxl: debug: libxl_event.c:549:libxl__ev_xswatch_deregister: watch
w=0x2579908 wpath=/local/domain/0/device-model/8/state token=3/2:
deregister slotnum=3
libxl: debug: libxl_event.c:561:libxl__ev_xswatch_deregister: watch
w=0x2579908: deregister unregistered
libxl: debug: libxl_event.c:512:libxl__ev_xswatch_register: watch
w=0x25812e8 wpath=/local/domain/0/backend/vif/8/0/state token=3/3:
register slotnum=3
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x25812e8
wpath=/local/domain/0/backend/vif/8/0/state token=3/3: event
epath=/local/domain/0/backend/vif/8/0/state
libxl: debug: libxl_event.c:600:devstate_watch_callback: backend
/local/domain/0/backend/vif/8/0/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x25812e8
wpath=/local/domain/0/backend/vif/8/0/state token=3/3: event
epath=/local/domain/0/backend/vif/8/0/state
libxl: debug: libxl_event.c:596:devstate_watch_callback: backend
/local/domain/0/backend/vif/8/0/state wanted state 2 ok
libxl: debug: libxl_event.c:549:libxl__ev_xswatch_deregister: watch
w=0x25812e8 wpath=/local/domain/0/backend/vif/8/0/state token=3/3:
deregister slotnum=3
libxl: debug: libxl_event.c:561:libxl__ev_xswatch_deregister: watch
w=0x25812e8: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug
script: /etc/xen/scripts/vif-bridge online
libxl: debug: libxl_event.c:512:libxl__ev_xswatch_register: watch
w=0x2581e78 wpath=/local/domain/0/backend/vif/7/0/state token=3/4:
register slotnum=3
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x2581e78
wpath=/local/domain/0/backend/vif/7/0/state token=3/4: event
epath=/local/domain/0/backend/vif/7/0/state
libxl: debug: libxl_event.c:600:devstate_watch_callback: backend
/local/domain/0/backend/vif/7/0/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:457:watchfd_callback: watch w=0x2581e78
wpath=/local/domain/0/backend/vif/7/0/state token=3/4: event
epath=/local/domain/0/backend/vif/7/0/state
libxl: debug: libxl_event.c:596:devstate_watch_callback: backend
/local/domain/0/backend/vif/7/0/state wanted state 2 ok
libxl: debug: libxl_event.c:549:libxl__ev_xswatch_deregister: watch
w=0x2581e78 wpath=/local/domain/0/backend/vif/7/0/state token=3/4:
deregister slotnum=3
libxl: debug: libxl_event.c:561:libxl__ev_xswatch_deregister: watch
w=0x2581e78: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug
script: /etc/xen/scripts/vif-bridge online
libxl: debug: libxl_event.c:1667:libxl__ao_progress_report: ao
0x2578f40: progress report: ignored
libxl: debug: libxl_event.c:1499:libxl__ao_complete: ao 0x2578f40:
complete, rc=0
libxl: debug: libxl_event.c:1471:libxl__ao__destroy: ao 0x2578f40: destroy
Daemon running with PID 4294
xc: debug: hypercall buffer: total allocations:813 total releases:813
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:799 misses:4 toobig:10


Cheers,

    Sylvain

Ian Campbell

2013-Feb-19 15:01 UTC

head link

Re: Xen 4.1.2 - stub domains for IO. Not working ?

On Tue, 2013-02-19 at 14:58 +0000, Sylvain Munaut wrote:> Hi,
> 
> I''m trying to use stub domains for IO for HVM domains, under Xen
4.1.2
> with xl toolstack
> 
> The domain works fine with either qemu-xen-traditional or qemu
> upstream, but I''d like to try stub domains.
> 
> If I understand the doc correctly just adding
> "device_model_stubdomain_override = 1" to the config should do
the
> trick.
NB this only currently works with qemu-xen-trad, not the upstream
version.
> 
> However doing that doesn''t really work.
> 
>  - qemu-dm is still started in the dom0 which AFAIU shouldn''t be
the case.
This is normal, since the qemu-dm in dom0 provides some PV services
(e.g. frame buffer backend) to the stubdomain.
>  - There is a xxx-dm domain that starts so that''s a good sign,
however
> they both stay stuck like :
> 
> wxp-00                                       7  1531     1     ------      
0.0
> wxp-00-dm                                    8    32     1     ---sc-      
0.0
The ''c'' in the status indicates that the stub domain has
crashed.

There may be a clue as to why in the Xen console log or you may find
something use on the stubdom console, I think you can "xl console
wxp-00-dm" to see that, or you can enable console logging as described
in
http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen#Guest_console_logs

Ian.

Sylvain Munaut

2013-Feb-19 15:29 UTC

head link

Re: Xen 4.1.2 - stub domains for IO. Not working ?

Hi Ian,

> NB this only currently works with qemu-xen-trad, not the upstream
> version.
Ok, not a problem. I have both installed and working in this setup.

>>  - qemu-dm is still started in the dom0 which AFAIU shouldn''t
be the case.
>
> This is normal, since the qemu-dm in dom0 provides some PV services
> (e.g. frame buffer backend) to the stubdomain.
Ah ok, true I didn''t think of that.

> The ''c'' in the status indicates that the stub domain has
crashed.
>
> There may be a clue as to why in the Xen console log or you may find
> something use on the stubdom console, I think you can "xl console
> wxp-00-dm" to see that, or you can enable console logging as described
> in
> http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen#Guest_console_logs
I enabled console logging and checked that it worked using a PV domain.

Unfortunately the main domain only creates and empty log file and the
-dm domain doesn''t create anything at all.
I tried attaching to console, but that didn''t help

There is also no errors reported in the ''xl dmesg'' output or
the linux
''dmesg'' output either.


Cheers,

     Sylvain

Ian Campbell

2013-Feb-20 09:16 UTC

head link

Re: Xen 4.1.2 - stub domains for IO. Not working ?

On Tue, 2013-02-19 at 15:29 +0000, Sylvain Munaut wrote:> > The ''c'' in the status indicates that the stub domain
has crashed.
> >
> > There may be a clue as to why in the Xen console log or you may find
> > something use on the stubdom console, I think you can "xl console
> > wxp-00-dm" to see that, or you can enable console logging as
described
> > in
> > http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen#Guest_console_logs
> 
> I enabled console logging and checked that it worked using a PV domain.
> 
> Unfortunately the main domain only creates and empty log file and the
> -dm domain doesn''t create anything at all.
> I tried attaching to console, but that didn''t help
> 
> There is also no errors reported in the ''xl dmesg'' output
or the linux
> ''dmesg'' output either.
Odd for the crash to be totally silent in all the logs.

You may need "loglvl=all guest_loglvl=all" on your hypervisor command
line. Failing that rebuilding the hypervisor with "debug=y" (passed on
the make command line) might expose something useful.

Lastly you could try running /usr/lib*/xen/bin/xenctx on the dm domain,
which might give you an IP address which can be related to the kernel
binary image with gdb or addr2line etc.

Ian.

Sylvain Munaut

2013-Feb-20 15:32 UTC

head link

Re: Xen 4.1.2 - stub domains for IO. Not working ?

Hi,
> Odd for the crash to be totally silent in all the logs.
When I tried to replace the binary xen.gz and ioemu-stubdom.gz with
debug=y version, it started working.

It seems it might be linked to the debian build system that produces a
ioemu-stubdom.gz which is much smaller than the one produced by a
''normal'' ( ./configure && make ) complilation.
I''m looking into that.

In any case, thanks for the help, now I have a lead to follow.

Cheers,

     Sylvain

Xen users - Feb 2013 - Xen 4.1.2 - stub domains for IO. Not working ?

Xen 4.1.2 - stub domains for IO. Not working ?

Re: Xen 4.1.2 - stub domains for IO. Not working ?

Re: Xen 4.1.2 - stub domains for IO. Not working ?

Re: Xen 4.1.2 - stub domains for IO. Not working ?

Re: Xen 4.1.2 - stub domains for IO. Not working ?