Xen devel - May 2006 - Unable to bring up Mini-OS on x86_64

I am getting the following error when I try to bring up Mini-OS on x86_64 Xen
(unstable: 9994). It is dying in xc_linux_build() causing xend to fault and
restart. Any clue why this happening? Has something changed in the builder that
Mini-OS needs to be aware of?

PS: I tried reverting changeset 9993 (Allow linking of mini-os with
application-specific object code), but still got the same error

Thanks,

Aravindh Puthiyaparambil
Xen Development Team
Unisys, Tredyffrin, PA
E-Mail:  aravindh.puthiyaparambil@unisys.com 

xenbox:~/xen/xen-unstable/extras/mini-os # xm create domain_config Using config
file "domain_config".
Error: <ProtocolError for /RPC2: -1 >

Dmesg
-----
xenbox kernel: python[12171] general protection rip:2b14fba3776e rsp:427ff7d0
error:0

Xend.log
--------
[2006-05-16 13:30:55 xend.XendDomainInfo] DEBUG (XendDomainInfo:186)
XendDomainInfo.create([''vm'', [''name'',
''Mini-OS''], [''memory'', 32],
[''on_crash'', ''destroy''],
[''vcpus'', 1], [''image'',
[''linux'', [''kernel'',
''/root/xen/xen-unstable.hg/extras/mini-os/mini-os.elf'']]]])
[2006-05-16 13:30:55 xend.XendDomainInfo] DEBUG (XendDomainInfo:292)
parseConfig: config is [''vm'', [''name'',
''Mini-OS''], [''memory'', 32],
[''on_crash'', ''destroy''],
[''vcpus'', 1], [''image'',
[''linux'', [''kernel'',
''/root/xen/xen-unstable.hg/extras/mini-os/mini-os.elf'']]]]
[2006-05-16 13:30:55 xend.XendDomainInfo] DEBUG (XendDomainInfo:391)
parseConfig: result is {''uuid'': None,
''on_crash'': ''destroy'',
''on_reboot'': None, ''image'':
[''linux'', [''kernel'',
''/root/xen/xen-unstable.hg/extras/mini-os/mini-os.elf'']],
''on_poweroff'': None, ''bootloader_args'':
None, ''cpus'': None, ''name'':
''Mini-OS'', ''backend'': [],
''vcpus'': 1, ''cpu_weight'': None,
''features'': None, ''vcpu_avail'': None,
''memory'': 32, ''device'': [],
''bootloader'': None, ''cpu'': None,
''maxmem'': None}
[2006-05-16 13:30:55 xend.XendDomainInfo] DEBUG (XendDomainInfo:1229)
XendDomainInfo.construct: None
[2006-05-16 13:30:55 xend.XendDomainInfo] DEBUG (XendDomainInfo:1261)
XendDomainInfo.initDomain: 1 1.0
[2006-05-16 13:30:55 xend] DEBUG (balloon:126) Balloon: free 3705; need 33;
done.
[2006-05-16 13:30:55 xend] INFO (image:133) buildDomain os=linux dom=1 vcpus=1
[2006-05-16 13:30:55 xend] DEBUG (image:171) dom            = 1
[2006-05-16 13:30:55 xend] DEBUG (image:172) image          =
/root/xen/xen-unstable.hg/extras/mini-os/mini-os.elf
[2006-05-16 13:30:55 xend] DEBUG (image:173) store_evtchn   = 1
[2006-05-16 13:30:55 xend] DEBUG (image:174) console_evtchn = 2
[2006-05-16 13:30:55 xend] DEBUG (image:175) cmdline        [2006-05-16 13:30:55
xend] DEBUG (image:176) ramdisk        [2006-05-16 13:30:55 xend] DEBUG
(image:177) vcpus          = 1
[2006-05-16 13:30:55 xend] DEBUG (image:178) features       [2006-05-16 13:30:55
xend] CRITICAL (SrvDaemon:195) Xend died due to signal 11!  Restarting it.
[2006-05-16 13:30:55 xend] INFO (SrvDaemon:283) Xend Daemon started
[2006-05-16 13:30:55 xend] INFO (SrvDaemon:287) Xend changeset: Tue May 16
09:08:06 2006 +0100 9994:4fd6ba0e3336.
[2006-05-16 13:30:55 xend.XendDomainInfo] DEBUG (XendDomainInfo:208)
XendDomainInfo.recreate({''paused'': 0,
''cpu_time'': 128897604096L, ''ssidref'': 0,
''handle'': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
''shutdown_reason'': 0, ''dying'': 0,
''dom'': 0, ''mem_kb'': 257024,
''maxmem_kb'': 17179869180, ''max_vcpu_id'': 3,
''crashed'': 0, ''running'': 1,
''shutdown'': 0, ''online_vcpus'': 1,
''blocked'': 0})
[2006-05-16 13:30:55 xend.XendDomainInfo] INFO (XendDomainInfo:220) Recreating
domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2006-05-16 13:30:56 xend] DEBUG (XendDomain:153) number of vcpus to use is 0
[2006-05-16 13:30:56 xend.XendDomainInfo] DEBUG (XendDomainInfo:208)
XendDomainInfo.recreate({''paused'': 1,
''cpu_time'': 0L, ''ssidref'': 0,
''handle'': [41, 26, 222, 114, 44, 108, 25, 70, 96, 32, 205,
220, 180, 114, 246, 3], ''shutdown_reason'': 0,
''dying'': 0, ''dom'': 1,
''mem_kb'': 32768, ''maxmem_kb'': 32768,
''max_vcpu_id'': 0, ''crashed'': 0,
''running'': 0, ''shutdown'': 0,
''online_vcpus'': 1, ''blocked'': 0})
[2006-05-16 13:30:56 xend.XendDomainInfo] INFO (XendDomainInfo:220) Recreating
domain 1, UUID 291ade72-2c6c-1946-6020-cddcb472f603.
[2006-05-16 13:30:56 xend.XendDomainInfo] DEBUG (XendDomainInfo:702) Storing VM
details: {''name'': ''Domain-1'',
''on_reboot'': ''restart'',
''image'': ''()'',
''on_poweroff'': ''destroy'',
''uuid'':
''291ade72-2c6c-1946-6020-cddcb472f603'',
''on_crash'': ''restart'',
''vcpus'': ''1'',
''vcpu_avail'': ''1'',
''memory'': ''32'', ''maxmem'':
''32''}
[2006-05-16 13:30:56 xend.XendDomainInfo] DEBUG (XendDomainInfo:727) Storing
domain details: {''cpu/0/availability'':
''online'', ''memory/target'':
''32768'', ''name'':
''Domain-1'', ''console/limit'':
''1048576'', ''vm'':
''/vm/291ade72-2c6c-1946-6020-cddcb472f603'',
''domid'': ''1''}
[2006-05-16 13:30:56 xend.XendDomainInfo] DEBUG (XendDomainInfo:910)
XendDomainInfo.handleShutdownWatch
[2006-05-16 13:30:56 xend.XendDomainInfo] DEBUG (XendDomainInfo:910)
XendDomainInfo.handleShutdownWatch
[2006-05-16 13:31:04 xend.XendDomainInfo] DEBUG (XendDomainInfo:1397)
XendDomainInfo.destroy: domid=1
[2006-05-16 13:31:04 xend.XendDomainInfo] DEBUG (XendDomainInfo:1405)
XendDomainInfo.destroyDomain(1)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 16 May 2006, at 20:57, Puthiyaparambil, Aravindh wrote:
> I am getting the following error when I try to bring up Mini-OS on 
> x86_64 Xen (unstable: 9994). It is dying in xc_linux_build() causing 
> xend to fault and restart. Any clue why this happening? Has something 
> changed in the builder that Mini-OS needs to be aware of?
>
> PS: I tried reverting changeset 9993 (Allow linking of mini-os with 
> application-specific object code), but still got the same error
Whatever the problem is, it''d be good to fix whatever bug this is 
triggering in the builder as well as fixing mini-os (although maybe the 
bug is solely in the builder). It shouldn''t be possible for bogus 
kernel images to kill xend, although I''m sure xc_linux_build() has 
various holes in this respect. It''d be nice to fix them as we find them
though.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> xenbox:~/xen/xen-unstable/extras/mini-os # xm create domain_config Using
config file "domain_config".
> Error: <ProtocolError for /RPC2: -1 >
I got this as well with Linux a few days ago after a pull and rebuild.
Fix for me was a clean slate, rebuild must''ve had a bad dependency
somehwere.  So a simple:
   make -C tools clean && make -C tools && sudo make -C tools
install
fixed it.  I didn''t actually have the time to track the dependency
breakage in tools/

thanks,
-chris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > xenbox:~/xen/xen-unstable/extras/mini-os # xm create domain_config
Using
> config file "domain_config".
> > Error: <ProtocolError for /RPC2: -1 >
> 
> I got this as well with Linux a few days ago after a pull and rebuild.
> Fix for me was a clean slate, rebuild must''ve had a bad dependency
> somehwere.  So a simple:
>    make -C tools clean && make -C tools && sudo make -C
tools install
> fixed it.
I did a fresh clone and a build and I am still running into the same
issue. So it looks like there is a bug in xc_linux_build() and/or
Mini-OS.

What is the best way to debug libxc? I usually do fprintf(stderr,...)
and the output shows up in xend-debug.log. But it is very flaky as
sometimes messages are overwritten. Is there a better way to do prints?

Thanks,
Aravindh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 5/17/06, Puthiyaparambil, Aravindh
<aravindh.puthiyaparambil@unisys.com>
> I did a fresh clone and a build and I am still running into the same
> issue. So it looks like there is a bug in xc_linux_build() and/or
> Mini-OS.
>
> What is the best way to debug libxc? I usually do fprintf(stderr,...)
> and the output shows up in xend-debug.log. But it is very flaky as
> sometimes messages are overwritten. Is there a better way to do prints?
I would recommend writing a small c program that calls libxc directly,
that will save you a bunch of time fighting the python/xend logging.

I have attached a patch with my own hand-rolled tools as an example.
The most interesting one is buscreate, which reads a config-file like:

kernel=/home/jacobg/xen-3.0-testing2.hg/linux-2.6.16-xen/vmlinux
mem=16
maxmem=16
console=1
initrd=/home/jacobg/domUinitrd
cmdline=mem=16M init=/linuxrc root=/dev/ram xencons=ttyS0
vbd=706,301,w

- and creates a domain without needing xend at all. xenstored and
xenconsoled need to be running though.

Jacob


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Whatever the problem is, it''d be good to fix whatever bug this is
> triggering in the builder as well as fixing mini-os (although maybe
the
> bug is solely in the builder). It shouldn''t be possible for bogus
> kernel images to kill xend, although I''m sure xc_linux_build() has
> various holes in this respect. It''d be nice to fix them as we find
them
> though.
Ok, I have found the bug. Mini-OS does not define VIRT_BASE in its
__xen_guest section. This causes dsi->elf_paddr_offset in
parseelfimage() [xc_load_elf.c] to be set to 0. This in turn causes
parray[pa>>PAGE_SHIFT] in loadelfimage[xc_load_elf.c] to go out of
bounds causing a segmentation fault.

This can also happen when an OS elfimage has VIRT_BASE or
ELF_PADDR_OFFSET set to 0. 

The case where VIRT_BASE is set correctly and ELF_PADDR_OFFSET is 0 has
been taken care of.

The fix here is to always ensure that VIRT_BASE is always defined and
when defined it is not set to 0 i.e. always ensure that virt_base
variable in parseelfimage() [xc_load_elf.c] is not 0. 

I will send in a patch for this.

Thanks,
Aravindh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I would recommend writing a small c program that calls libxc directly,
> that will save you a bunch of time fighting the python/xend logging.
> 
> I have attached a patch with my own hand-rolled tools as an example.
> The most interesting one is buscreate, which reads a config-file like:
Jacob,

Your tools were very helpful. I am interested in receiving updates :-) 

One thing I was never able to get "listdoms" to work. I think
do_dom0_op(xc_handle, &op) in xc_domain_getinfo()is failing.

Thanks,
Aravindh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 5/17/06, Puthiyaparambil, Aravindh
<aravindh.puthiyaparambil@unisys.com> wrote:
>
> > I would recommend writing a small c program that calls libxc directly,
> > that will save you a bunch of time fighting the python/xend logging.
> >
> > I have attached a patch with my own hand-rolled tools as an example.
> > The most interesting one is buscreate, which reads a config-file like:
>
> Jacob,
>
> Your tools were very helpful. I am interested in receiving updates :-)
hi,

my minimal domain management tools (for various reasons known as
''migrationtools'') are now available standalone from
http://www.diku.dk/~jacobg .

Jacob

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel