Xen devel - Sep 2013 - Problems in starting up domU with Xen-ARM (on Cortex-A15 Fast Models)

Hi everyone,

I''m trying to start up domU on xen for ARM (Cortex-A15 Fast models). I
follow the instructions to build xen-tools with this link:
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/CrossCompiling.
But there seems to be something wrong when I start up domU.

First, I get errors in create domU with xl:
root@(none):/home/xiaogw# xl create domU-config
xc: error: Could not obtain handle on privileged command interface (2 = No
such file or directory): Internal error
libxl: error: libxl.c:94:libxl_ctx_alloc: cannot open libxc handle: No such
file or directory
cannot init xl context

I solve this problem by:
root@(none):/# mount -t proc proc /proc
root@(none):/# mount -t xenfs xen /proc/xen/
root@(none):/# xl list
Name
              ID   Mem VCPUs    State    Time(s)
^C

But it will halt after executing "xl list", or "xl create
domU-config". I
don''t know why? Is there something wrong with my xen-tools? Or my
dom0/domU
kernel? I use linux kernel for domU as the same for dom0, but with the
following instruction:
cat arch/arm/boot/zImage arch/arm/boot/dts/xenvm-4.2.dtb > /mnt/root/image

----------------------------------------------------------------------------------------------------------------
There are some information with dom0:
root@(none):/home/xiaogw# ps -A |grep xen
   10 ?        00:00:00 xenwatch
   11 ?        00:00:00 xenbus

root@(none):/home/xiaogw# ls
domU-config  image
root@(none):/home/xiaogw# cat domU-config
kernel = "/home/xiaogw/image"
memory = 128
name = "win"
vcpus = 1
disk = [ ''phy:/dev/loop0,xvda,w'' ]

root@(none):/home/xiaogw# /etc/init.d/xencommons status
xenstore-read: xs_open: Connection refused
root@(none):/home/xiaogw# /etc/init.d/xencommons start
Starting C xenstored...FATAL: Failed to open evtchn device: No such file or
directory
..............................
Could not start xenstored

Can I get some help? Thanks!

Xiaoguang Wang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Sun, Sep 29, 2013 at 12:04 AM, XiaoGuang Wang <xjtuwxg@gmail.com>
wrote:
[...]
> First, I get errors in create domU with xl:
> root@(none):/home/xiaogw# xl create domU-config
> xc: error: Could not obtain handle on privileged command interface (2 = No
> such file or directory): Internal error
This.
> root@(none):/home/xiaogw# cat domU-config
> kernel = "/home/xiaogw/image"
> memory = 128
> name = "win"
> vcpus = 1
> disk = [ ''phy:/dev/loop0,xvda,w'' ]
>
(Ah, Windows on ARM? Just kidding... :-)))))))
> root@(none):/home/xiaogw# /etc/init.d/xencommons status
> xenstore-read: xs_open: Connection refused
> root@(none):/home/xiaogw# /etc/init.d/xencommons start
> Starting C xenstored...FATAL: Failed to open evtchn device: No such file or
> directory
And this.

Make sure you have all Xen drivers complied in / loaded.

Wei.

Hi Wei,

2013/9/28 Wei Liu <liuw@liuw.name>
> On Sun, Sep 29, 2013 at 12:04 AM, XiaoGuang Wang <xjtuwxg@gmail.com>
> wrote:
> [...]
> > First, I get errors in create domU with xl:
> > root@(none):/home/xiaogw# xl create domU-config
> > xc: error: Could not obtain handle on privileged command interface (2
> No
> > such file or directory): Internal error
>
> This.
>
> > root@(none):/home/xiaogw# cat domU-config
> > kernel = "/home/xiaogw/image"
> > memory = 128
> > name = "win"
> > vcpus = 1
> > disk = [ ''phy:/dev/loop0,xvda,w'' ]
> >
>
> (Ah, Windows on ARM? Just kidding... :-)))))))
>
> I think the name field is just a name, nothing else. And I use the config
sample in this URL:
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions#DomU_kernel_and_DTS

> > root@(none):/home/xiaogw# /etc/init.d/xencommons status
> > xenstore-read: xs_open: Connection refused
> > root@(none):/home/xiaogw# /etc/init.d/xencommons start
> > Starting C xenstored...FATAL: Failed to open evtchn device: No such
file
> or
> > directory
>
> And this.
>
> Make sure you have all Xen drivers complied in / loaded.
>
What do you mean "complied* in / loaded*"? And I follow the
instruction
listed in:
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/CrossCompiling.

And I copy the compiled xen-tools files to the dom0 image (root FS, or .img
file) with:
# rsync -avp dist/install/ /mnt/
>
> Wei.
>
I''m still not clear why the system will halt when I type "xl
list" or "xl
create domU_config"?

Thanks!

Sincerely,
Xiaoguang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Sun, Sep 29, 2013 at 3:53 PM, XiaoGuang Wang <xjtuwxg@gmail.com> wrote:
[...]
>> And this.
>>
>> Make sure you have all Xen drivers complied in / loaded.
>
> What do you mean "complied in / loaded"? And I follow the
instruction listed
> in:
>
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/CrossCompiling.
> And I copy the compiled xen-tools files to the dom0 image (root FS, or .img
> file) with:
> # rsync -avp dist/install/ /mnt/
>>
That page tells you how to build your kernel. You will need to enable
Xen drivers in you kernel configuration by yourself.
>>
>> Wei.
>
> I''m still not clear why the system will halt when I type "xl
list" or "xl
> create domU_config"?
>
If you don''t have Xen device drivers in Dom0 the toolstack will not
work. The error messages are clear enough.

Wei.
> Thanks!
>
> Sincerely,
> Xiaoguang

2013/9/29 Wei Liu <liuw@liuw.name>
> On Sun, Sep 29, 2013 at 3:53 PM, XiaoGuang Wang <xjtuwxg@gmail.com>
wrote:
> [...]
> >> And this.
> >>
> >> Make sure you have all Xen drivers complied in / loaded.
> >
> > What do you mean "complied in / loaded"? And I follow the
instruction
> listed
> > in:
> >
>
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/CrossCompiling
> .
> > And I copy the compiled xen-tools files to the dom0 image (root FS, or
> .img
> > file) with:
> > # rsync -avp dist/install/ /mnt/
> >>
>
> That page tells you how to build your kernel. You will need to enable
> Xen drivers in you kernel configuration by yourself.
>
> >>
> >> Wei.
> >
> > I''m still not clear why the system will halt when I type
"xl list" or "xl
> > create domU_config"?
> >
>
> If you don''t have Xen device drivers in Dom0 the toolstack will
not
> work. The error messages are clear enough.
>
Sorry Wei, I think I don''t catch what you mean. So is my Dom0 linux
kernel
config wrong, so that the dom0 kernel don''t have Xen device drivers? I
compile the dom0 kernel with this config file:
http://xenbits.xen.org/people/sstabellini/config

BTW, when I type "xl info" it works right. But when I type "xl
list" or "xl
create domU_config", it will halt. So can you tell me what should I do?
Thanks!

root@(none):/home/xiaogw# xl info
host                   : (none)
release                : 3.8.0-rc4+
version                : #2 Wed Sep 25 10:39:06 EDT 2013
machine                : armv7l
nr_cpus                : 1
max_cpu_id             : 127
nr_nodes               : 1
cores_per_socket       : 1
threads_per_core       : 1
cpu_mhz                : 100
hw_caps                :
00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000
virt_caps              :
total_memory           : 2048
free_memory            : 1629
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 4
xen_extra              : -unstable
xen_version            : 4.4-unstable
xen_caps               : xen-3.0-armv7l
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0x200000
xen_changeset          : Mon Aug 26 12:40:44 2013 +0200 git:8a7769b-dirty
xen_commandline        : dom0_mem=128M dom0_max_vcpus=1 console=com1,vga
cc_compiler            : arm-linux-gnueabi-gcc (Ubuntu/Linaro
4.6.3-1ubuntu5) 4.6.3
cc_compile_by          : xiaoguang
cc_compile_domain      :
cc_compile_date        : Wed Sep 25 14:55:40 EDT 2013
xend_config_format     : 4
root@(none):/home/xiaogw# xl list
Name                                        ID   Mem VCPUs    State
Time(s)

^C^C^C^C^C^C

> Wei.
>
> > Thanks!
> >
> > Sincerely,
> > Xiaoguang
>
Sincerely,
Xiaoguang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Sun, Sep 29, 2013 at 6:39 PM, XiaoGuang Wang <xjtuwxg@gmail.com> wrote:
[...]
>> If you don''t have Xen device drivers in Dom0 the toolstack
will not
>> work. The error messages are clear enough.
>
> Sorry Wei, I think I don''t catch what you mean. So is my Dom0
linux kernel
> config wrong, so that the dom0 kernel don''t have Xen device
drivers? I
> compile the dom0 kernel with this config file:
> http://xenbits.xen.org/people/sstabellini/config
>
If you''re using this config I don''t think you can see those
"device
not found" errors in your first mail.

Check what you have in /dev/xen/*. I think you need at least evtchn
and privcmd to make xenstored work.

Wei.

2013/9/29 Wei Liu <liuw@liuw.name>
> On Sun, Sep 29, 2013 at 6:39 PM, XiaoGuang Wang <xjtuwxg@gmail.com>
wrote:
> [...]
> >> If you don''t have Xen device drivers in Dom0 the
toolstack will not
> >> work. The error messages are clear enough.
> >
> > Sorry Wei, I think I don''t catch what you mean. So is my Dom0
linux
> kernel
> > config wrong, so that the dom0 kernel don''t have Xen device
drivers? I
> > compile the dom0 kernel with this config file:
> > http://xenbits.xen.org/people/sstabellini/config
> >
>
> If you''re using this config I don''t think you can see
those "device
> not found" errors in your first mail.
>
> Check what you have in /dev/xen/*. I think you need at least evtchn
> and privcmd to make xenstored work.
>
There is no /dev/xen/ directory. And I recompile dom0 with linux kernel
3.11, it still doesn''t work. I''m considering if there is sth.
wrong with my
xen-tools compilation. :(
>
> Wei.
>
Xiaoguang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Sun, 2013-09-29 at 22:55 -0400, XiaoGuang Wang wrote:
> There is no /dev/xen/ directory.
Are you running udev?

What is the contents of /proc/misc?

Does /proc/xen exist? What is its contents.
> I''m considering if there is sth. wrong with my xen-tools
> compilation. :(
This is nothing to do with the xen-tools, it''s either a kernel issue or
a system configuration issue.

Ian.

Hi,
2013/9/30 Ian Campbell <Ian.Campbell@citrix.com>
> On Sun, 2013-09-29 at 22:55 -0400, XiaoGuang Wang wrote:
>
> > There is no /dev/xen/ directory.
>
> Are you running udev?
>
I think the udev is not running. Because when I start udevd, it comes up
error message and halts.
root@(none):/home/xiaogw# udevd
udevd[56]: error: runtime directory ''/run/udev'' not writable,
for now
falling back to ''/dev/.udev''
^C^C^C^C
>
> What is the contents of /proc/misc?
>
root@(none):/home/xiaogw# cat /proc/misc
 56 network_throughput
 57 network_latency
 58 cpu_dma_latency
  1 psaux
200 tun
237 loop-control
 59 xen/privcmd
 60 xen/gntdev
 61 xen/evtchn
 62 xen/xenbus_backend
 63 xen/xenbus

>
> Does /proc/xen exist? What is its contents.
>
root@(none):/home/xiaogw# ls /proc/xen/
capabilities  privcmd  xenbus  xsd_kva    xsd_port
>
> > I''m considering if there is sth. wrong with my xen-tools
> > compilation. :(
>
> This is nothing to do with the xen-tools, it''s either a kernel
issue or
> a system configuration issue.
>
> Ian.
>
> Thanks!
Xiaoguang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Mon, 2013-09-30 at 09:40 -0400, XiaoGuang Wang wrote:
> Hi,
> 2013/9/30 Ian Campbell <Ian.Campbell@citrix.com>
>         On Sun, 2013-09-29 at 22:55 -0400, XiaoGuang Wang wrote:
>         
>         > There is no /dev/xen/ directory.
>         
>         
>         Are you running udev?
> I think the udev is not running. Because when I start udevd, it comes
> up error message and halts.
> root@(none):/home/xiaogw# udevd 
> udevd[56]: error: runtime directory ''/run/udev'' not
writable, for now
> falling back to ''/dev/.udev''
This is your problem, which has nothing to do with Xen AFAICT.

Firstly you probably want to start udev via the initscripts not by hand,
secondly you probably need other (distro dependent) initscripts to run
first to ensure things like /run/ are mounted.

I guess you are booting with init=/bin/sh or something similar?

Ian.

2013/9/30 Ian Campbell <Ian.Campbell@citrix.com>
> On Mon, 2013-09-30 at 09:40 -0400, XiaoGuang Wang wrote:
> > Hi,
> > 2013/9/30 Ian Campbell <Ian.Campbell@citrix.com>
> >         On Sun, 2013-09-29 at 22:55 -0400, XiaoGuang Wang wrote:
> >
> >         > There is no /dev/xen/ directory.
> >
> >
> >         Are you running udev?
> > I think the udev is not running. Because when I start udevd, it comes
> > up error message and halts.
> > root@(none):/home/xiaogw# udevd
> > udevd[56]: error: runtime directory ''/run/udev'' not
writable, for now
> > falling back to ''/dev/.udev''
>
> This is your problem, which has nothing to do with Xen AFAICT.
>
> Firstly you probably want to start udev via the initscripts not by hand,
> secondly you probably need other (distro dependent) initscripts to run
> first to ensure things like /run/ are mounted.
>
> I guess you are booting with init=/bin/sh or something similar?
>
Sorry Ian, I''m a newcomer on Xen and system works. I have no idea about
start udev via initscripts. I modify .config and set cmdline (remove
init=/bin/sh) with the following:
CONFIG_CMDLINE="earlyprintk=xenboot console=ttyAMA1 root=/dev/mmcblk0 debug
rw"
And it still doesn''t work.

Actually, when the dom0 starts up, the /proc is empty. And I have to
manually the /proc and /proc/xen with:
mount -t proc proc /proc
mount -t xenfs xen /proc/xen
Which are the results I found from google.
>
> Ian.
>
>
> Thanks.
Xiaoguang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Mon, 2013-09-30 at 10:46 -0400, XiaoGuang Wang wrote:
> 2013/9/30 Ian Campbell <Ian.Campbell@citrix.com>
>         On Mon, 2013-09-30 at 09:40 -0400, XiaoGuang Wang wrote:
>         > Hi,
>         > 2013/9/30 Ian Campbell <Ian.Campbell@citrix.com>
>         >         On Sun, 2013-09-29 at 22:55 -0400, XiaoGuang Wang
>         wrote:
>         >
>         >         > There is no /dev/xen/ directory.
>         >
>         >
>         >         Are you running udev?
>         > I think the udev is not running. Because when I start udevd,
>         it comes
>         > up error message and halts.
>         > root@(none):/home/xiaogw# udevd
>         > udevd[56]: error: runtime directory
''/run/udev'' not
>         writable, for now
>         > falling back to ''/dev/.udev''
>         
>         
>         This is your problem, which has nothing to do with Xen AFAICT.
>         
>         Firstly you probably want to start udev via the initscripts
>         not by hand,
>         secondly you probably need other (distro dependent)
>         initscripts to run
>         first to ensure things like /run/ are mounted.
>         
>         I guess you are booting with init=/bin/sh or something
>         similar?
> Sorry Ian, I''m a newcomer on Xen and system works. I have no idea
> about start udev via initscripts.
This is nothing to do with Xen, it''s just normal (and expected) Linux
knowledge I''m afraid. The /root/init.sh which I use is:
        #!/bin/sh
        set -x
        mount -o remount,rw /
        mount -t proc none /proc
        mount -t sysfs none /sys
        mount -t tmpfs none /run
        mkdir /run/lock
        /etc/init.d/udev start
        mount -t devpts none /dev/pts
        hostname -F /etc/hostname
        /etc/init.d/xencommons start
        echo 9 > /proc/sysrq-trigger 
        cd /root
        exec /bin/bash

But TBH if you don''t know what an initscript is then it might be best
if you try
and find someone locally (e.g. in your department) who knows about Linux
and ask them to help bring you up to speed.

Ian.

2013/10/1 Ian Campbell <Ian.Campbell@citrix.com>
> On Mon, 2013-09-30 at 10:46 -0400, XiaoGuang Wang wrote:
> > 2013/9/30 Ian Campbell <Ian.Campbell@citrix.com>
> >         On Mon, 2013-09-30 at 09:40 -0400, XiaoGuang Wang wrote:
> >         > Hi,
> >         > 2013/9/30 Ian Campbell <Ian.Campbell@citrix.com>
> >         >         On Sun, 2013-09-29 at 22:55 -0400, XiaoGuang Wang
> >         wrote:
> >         >
> >         >         > There is no /dev/xen/ directory.
> >         >
> >         >
> >         >         Are you running udev?
> >         > I think the udev is not running. Because when I start
udevd,
> >         it comes
> >         > up error message and halts.
> >         > root@(none):/home/xiaogw# udevd
> >         > udevd[56]: error: runtime directory
''/run/udev'' not
> >         writable, for now
> >         > falling back to ''/dev/.udev''
> >
> >
> >         This is your problem, which has nothing to do with Xen AFAICT.
> >
> >         Firstly you probably want to start udev via the initscripts
> >         not by hand,
> >         secondly you probably need other (distro dependent)
> >         initscripts to run
> >         first to ensure things like /run/ are mounted.
> >
> >         I guess you are booting with init=/bin/sh or something
> >         similar?
> > Sorry Ian, I''m a newcomer on Xen and system works. I have no
idea
> > about start udev via initscripts.
>
> This is nothing to do with Xen, it''s just normal (and expected)
Linux
> knowledge I''m afraid. The /root/init.sh which I use is:
>         #!/bin/sh
>         set -x
>         mount -o remount,rw /
>         mount -t proc none /proc
>         mount -t sysfs none /sys
>         mount -t tmpfs none /run
>         mkdir /run/lock
>         /etc/init.d/udev start
>         mount -t devpts none /dev/pts
>         hostname -F /etc/hostname
>         /etc/init.d/xencommons start
>         echo 9 > /proc/sysrq-trigger
>         cd /root
>         exec /bin/bash
>
> But TBH if you don''t know what an initscript is then it might be
best if
> you try
> and find someone locally (e.g. in your department) who knows about Linux
> and ask them to help bring you up to speed.
>
> Ian.
>
> Hi Ian/all,
When start udev, there comes error:  initctl: Unable to connect to Upstart:
Failed to connect to socket /com/ubuntu/upstart: Connection refused
I''m considering that it is because I didn''t configure the dom0
network. But
when I start the network, some tools are not found in arm chroot FS, e.g.
brctl.
Is it necessary to start network when start udev?

Thanks.

Sincerely,
Xiaoguang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, 2013-10-02 at 10:03 -0400, XiaoGuang Wang wrote:
> When start udev, there comes error:  initctl: Unable to connect to
> Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection
> refused
> 
> I''m considering that it is because I didn''t configure the
dom0
> network. But when I start the network, some tools are not found in arm
> chroot FS, e.g. brctl.
> 
> Is it necessary to start network when start udev?
upstart is the Ubuntu initsystem (i.e fills the same niche as sysvinit
or systemd). The "Connection refused" is likely to be a Unix domain
socket or something and not networking.

I doubt you need actual networking here, if nothing else udev is
responsible for loading the network drivers!

Ian.

2013/10/2 Ian Campbell <Ian.Campbell@citrix.com>
> On Wed, 2013-10-02 at 10:03 -0400, XiaoGuang Wang wrote:
>
> > When start udev, there comes error:  initctl: Unable to connect to
> > Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection
> > refused
> >
> > I''m considering that it is because I didn''t
configure the dom0
> > network. But when I start the network, some tools are not found in arm
> > chroot FS, e.g. brctl.
> >
> > Is it necessary to start network when start udev?
>
> upstart is the Ubuntu initsystem (i.e fills the same niche as sysvinit
> or systemd). The "Connection refused" is likely to be a Unix
domain
> socket or something and not networking.
>
> I doubt you need actual networking here, if nothing else udev is
> responsible for loading the network drivers!
>
Yes. I solve it by using the following two commands:
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

While actually, there is still something wrong when start xencommons:

root@dom0:/root# /etc/init.d/xencommons start
Starting C xenstored....FATAL: Failed to open evtchn device: No such file
or directory
.............................
Could not start xenstored

I think I have already set the XEN_DEV_EVTCHN in dom0 .config file:
1578 # Xen driver support
1579 #
1580
CONFIG_XEN_DEV_EVTCHN=y

1581 CONFIG_XEN_BACKEND=y
1582 CONFIG_XENFS=y
1583 CONFIG_XEN_COMPAT_XENFS=y
1584 # CONFIG_XEN_SYS_HYPERVISOR is not set
1585 CONFIG_XEN_XENBUS_FRONTEND=y
1586 CONFIG_XEN_GNTDEV=y
1587 # CONFIG_XEN_GRANT_DEV_ALLOC is not set
1588 CONFIG_XEN_PRIVCMD=y
1589 # CONFIG_STAGING is not set
1590 CONFIG_CLKDEV_LOOKUP=y
1591 CONFIG_HAVE_CLK_PREPARE=y
1592 CONFIG_COMMON_CLK=y

I don''t know why it still happens.

Sincerely,
Xiaoguang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, 2013-10-02 at 16:00 -0400, XiaoGuang Wang wrote:
> I don''t know why it still happens.
Returning to my earlier question -- what is the contents of /dev/xen/
now?

Ian.

2013/10/3 Ian Campbell <Ian.Campbell@citrix.com>
>
> > I don''t know why it still happens.
>
> Returning to my earlier question -- what is the contents of /dev/xen/
> now?
>
Still no /dev/xen directory.

Xiaoguang


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, 2013-10-03 at 09:12 -0400, XiaoGuang Wang wrote:
> 2013/10/3 Ian Campbell <Ian.Campbell@citrix.com>
>         
>         > I don''t know why it still happens.
>         
>         
>         Returning to my earlier question -- what is the contents
>         of /dev/xen/
>         now?
> 
> Still no /dev/xen directory.
It seems likely that udev is not actually working then.

You might have more luck dropping the init= line and letting the full
Ubuntu startup run its course.

But, please try and find someone nearby who can help you debug this
Linux related issue in a more interactive way -- it''s going to take
forever to do it via email like this. No specific Xen knowledge should
be needed.

Ian.