thr3ads.net - Xen devel - [Xen-devel] can''t start domU [Nov 2005]

If this information is useful, please help other people find it:
Share via:

Charles Coffing

2005-Nov-11 22:12 UTC

[Xen-devel] can''t start domU

I''ve been getting hangs while attempting to start domUs on recent
changesets.  I get an error such as:
Error: Device 0 (vif) could not be connected. Hotplug scripts not
working.

Yet if I run diagnose.py on the now- paused domU, I get this:
# python diagnose.py 2
Domain ID is 2.
Domain name is ExampleDomain.
Found 2 device classes in use.
Found 1 vbd devices.
Found device vbd, 769.
Backend is stuck waiting for frontend for device vbd, 769.
Device vbd, 769 hotplugging has completed successfully.
Found 0 vif devices.

Any idea why the front end would be stuck?

I''m running udev 068.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Nivedita Singhvi

2005-Nov-11 22:44 UTC

head link

Re: [Xen-devel] can''t start domU

Charles Coffing wrote:> I''ve been getting hangs while attempting to start domUs on recent
> changesets.  I get an error such as:
> Error: Device 0 (vif) could not be connected. Hotplug scripts not
> working.
> 
> Yet if I run diagnose.py on the now- paused domU, I get this:
> # python diagnose.py 2
> Domain ID is 2.
> Domain name is ExampleDomain.
> Found 2 device classes in use.
> Found 1 vbd devices.
> Found device vbd, 769.
> Backend is stuck waiting for frontend for device vbd, 769.
> Device vbd, 769 hotplugging has completed successfully.
> Found 0 vif devices.
> 
> Any idea why the front end would be stuck?
> 
> I''m running udev 068.
All of these are failures to create the vif devices.
Please check /var/log/messages, xend-debug.log, etc.
You can put a set -x in your /etc/xen/vif-bridge (or equivalent)
script for more verbose detail.

Also, check to see that your install went successfully.

 From previous debugging over the past week, most of
these cases were the lack of brctl (or execute perms
for root), or the lack of other utilities (make install
should catch these).

thanks,
Nivedita

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Charles Coffing

2005-Nov-11 23:13 UTC

head link

Re: [Xen-devel] can''t start domU

>>> On Fri, Nov 11, 2005 at  3:44 pm, in message<43751EDA.80304@us.ibm.com>,
Nivedita Singhvi <niv@us.ibm.com> wrote: > Charles Coffing wrote:
>> I''ve been getting hangs while attempting to start domUs on
recent
>> changesets.  I get an error such as:
>> Error: Device 0 (vif) could not be connected. Hotplug scripts not
>> working.
[snip]> 
> All of these are failures to create the vif devices.
> Please check /var/log/messages, xend- debug.log, etc.
> You can put a set - x in your /etc/xen/vif- bridge (or equivalent)
> script for more verbose detail.
> 
> Also, check to see that your install went successfully.
> 
>  From previous debugging over the past week, most of
> these cases were the lack of brctl (or execute perms
> for root), or the lack of other utilities (make install
> should catch these).
The obvious stuff looks good:  The install went okay; brctl exists; all
scripts are executable.

I''ll look more closely at the bridge script.

Thanks,
Charles


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Steve Dobbelstein

2005-Nov-15 19:59 UTC

head link

Re: [Xen-devel] can''t start domU

"Charles Coffing" <ccoffing@novell.com> wrote on 11/11/2005
05:13:25 PM:
> >>> On Fri, Nov 11, 2005 at  3:44 pm, in message
> <43751EDA.80304@us.ibm.com>,
> Nivedita Singhvi <niv@us.ibm.com> wrote:
> > Charles Coffing wrote:
> >> I''ve been getting hangs while attempting to start domUs
on recent
> >> changesets.  I get an error such as:
> >> Error: Device 0 (vif) could not be connected. Hotplug scripts not
> >> working.
> [snip]
> >
> > All of these are failures to create the vif devices.
> > Please check /var/log/messages, xend- debug.log, etc.
> > You can put a set - x in your /etc/xen/vif- bridge (or equivalent)
> > script for more verbose detail.
> >
> > Also, check to see that your install went successfully.
> >
> >  From previous debugging over the past week, most of
> > these cases were the lack of brctl (or execute perms
> > for root), or the lack of other utilities (make install
> > should catch these).
>
> The obvious stuff looks good:  The install went okay; brctl exists; all
> scripts are executable.
>
> I''ll look more closely at the bridge script.
Has anyone found a solution to this problem?  I''m seeing the same thing
but
didn''t want to just post a "me, too".

My domU config file does not have a "vif" entry; the defaults are fine
for
me.

/var/log/messages, /var/log/xend-debug.log, and /var/log/xend.log don''t
contain any additional information; they look the same as the ones on a
system that is working.

I added lines to /etc/xen/scripts/vif-bridge:
echo "Turning debugging on"
echo "turning debugging on">/tmp/vif-bridge.out
set -x

I don''t get any debug output upon running "xm create".  I
don''t get a file
/tmp/vif-bridge, so it appears that vif-bridge isn''t even called.

hotplug is at version 058.  I tried the "echo /sbin/hotplug >
/proc/sys/kernel/hotplug" trick, but that didn''t help.  Besides,
the system
that works is also running hotplug 058 and it has /proc/sys/kernel/hotplug
set to /sbin/udevsend.

Apparently something is not creating the vif for the domain.  I have been
crawling through the xend scripts to try and figure out where vifs get
created in hopes of finding some clues as to what is going wrong, but being
barely a python newbie I get lost in the code.  :(  (Time to add another
programming language to my skill set.)

Any solutions or pointers on how to proceed would be appreciated.

Steve D.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ewan Mellor

2005-Nov-15 23:34 UTC

head link

Re: [Xen-devel] can''t start domU

On Tue, Nov 15, 2005 at 01:59:25PM -0600, Steve Dobbelstein wrote:
> "Charles Coffing" <ccoffing@novell.com> wrote on 11/11/2005
05:13:25 PM:
> 
> > >>> On Fri, Nov 11, 2005 at  3:44 pm, in message
> > <43751EDA.80304@us.ibm.com>,
> > Nivedita Singhvi <niv@us.ibm.com> wrote:
> > > Charles Coffing wrote:
> > >> I''ve been getting hangs while attempting to start
domUs on recent
> > >> changesets.  I get an error such as:
> > >> Error: Device 0 (vif) could not be connected. Hotplug scripts
not
> > >> working.
> > [snip]
> > >
> > > All of these are failures to create the vif devices.
> > > Please check /var/log/messages, xend- debug.log, etc.
> > > You can put a set - x in your /etc/xen/vif- bridge (or
equivalent)
> > > script for more verbose detail.
> > >
> > > Also, check to see that your install went successfully.
> > >
> > >  From previous debugging over the past week, most of
> > > these cases were the lack of brctl (or execute perms
> > > for root), or the lack of other utilities (make install
> > > should catch these).
> >
> > The obvious stuff looks good:  The install went okay; brctl exists;
all
> > scripts are executable.
> >
> > I''ll look more closely at the bridge script.
> 
> Has anyone found a solution to this problem?  I''m seeing the same
thing but
> didn''t want to just post a "me, too".
> 
> My domU config file does not have a "vif" entry; the defaults are
fine for
> me.
> 
> /var/log/messages, /var/log/xend-debug.log, and /var/log/xend.log
don''t
> contain any additional information; they look the same as the ones on a
> system that is working.
> 
> I added lines to /etc/xen/scripts/vif-bridge:
> echo "Turning debugging on"
> echo "turning debugging on">/tmp/vif-bridge.out
> set -x
> 
> I don''t get any debug output upon running "xm create". 
I don''t get a file
> /tmp/vif-bridge, so it appears that vif-bridge isn''t even called.
> 
> hotplug is at version 058.  I tried the "echo /sbin/hotplug >
> /proc/sys/kernel/hotplug" trick, but that didn''t help. 
Besides, the system
> that works is also running hotplug 058 and it has /proc/sys/kernel/hotplug
> set to /sbin/udevsend.
> 
> Apparently something is not creating the vif for the domain.  I have been
> crawling through the xend scripts to try and figure out where vifs get
> created in hopes of finding some clues as to what is going wrong, but being
> barely a python newbie I get lost in the code.  :(  (Time to add another
> programming language to my skill set.)
Have you tried today''s changeset?  There have been plenty of changes in
this
area in the last week, and a few today.

What distro?  (Not that this will help, but it seems a good idea to keep track
of problematic distro versions.)

Does /sbin/hotplug or /sbin/udevsend run at all?

If so, does it make it as far as /etc/hotplug/xen-backend.agent?

If so, how about /etc/xen/scripts/vif-common.sh?

What do you get if you run

PYTHONPATH=/usr/lib/python python /usr/lib/python/xen/util/diagnose.py <dom
ID>

Thanks for your help,

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Steve Dobbelstein

2005-Nov-16 17:19 UTC

head link

Re: [Xen-devel] can''t start domU

Ewan Mellor <ewan@xensource.com> wrote on 11/15/2005 05:34:48 PM:

Thanks for your assitance, Ewan.
> On Tue, Nov 15, 2005 at 01:59:25PM -0600, Steve Dobbelstein wrote:
>
> > "Charles Coffing" <ccoffing@novell.com> wrote on
11/11/2005 05:13:25
PM:> >
> > > >>> On Fri, Nov 11, 2005 at  3:44 pm, in message
> > > <43751EDA.80304@us.ibm.com>,
> > > Nivedita Singhvi <niv@us.ibm.com> wrote:
> > > > Charles Coffing wrote:
> > > >> I''ve been getting hangs while attempting to
start domUs on recent
> > > >> changesets.  I get an error such as:
> > > >> Error: Device 0 (vif) could not be connected. Hotplug
scripts not
> > > >> working.
> > > [snip]
> > > >
> > > > All of these are failures to create the vif devices.
> > > > Please check /var/log/messages, xend- debug.log, etc.
> > > > You can put a set - x in your /etc/xen/vif- bridge (or
equivalent)
> > > > script for more verbose detail.
> > > >
> > > > Also, check to see that your install went successfully.
> > > >
> > > >  From previous debugging over the past week, most of
> > > > these cases were the lack of brctl (or execute perms
> > > > for root), or the lack of other utilities (make install
> > > > should catch these).
> > >
> > > The obvious stuff looks good:  The install went okay; brctl
exists;
all> > > scripts are executable.
> > >
> > > I''ll look more closely at the bridge script.
> >
> > Has anyone found a solution to this problem?  I''m seeing the
same thing
but> > didn''t want to just post a "me, too".
> >
> > My domU config file does not have a "vif" entry; the
defaults are fine
for> > me.
> >
> > /var/log/messages, /var/log/xend-debug.log, and /var/log/xend.log
don''t
> > contain any additional information; they look the same as the ones on
a
> > system that is working.
> >
> > I added lines to /etc/xen/scripts/vif-bridge:
> > echo "Turning debugging on"
> > echo "turning debugging on">/tmp/vif-bridge.out
> > set -x
> >
> > I don''t get any debug output upon running "xm
create".  I don''t get a
file> > /tmp/vif-bridge, so it appears that vif-bridge isn''t even
called.
> >
> > hotplug is at version 058.  I tried the "echo /sbin/hotplug >
> > /proc/sys/kernel/hotplug" trick, but that didn''t help. 
Besides, the
system> > that works is also running hotplug 058 and it has
/proc/sys/kernel/hotplug> > set to /sbin/udevsend.
> >
> > Apparently something is not creating the vif for the domain.  I have
been> > crawling through the xend scripts to try and figure out where vifs get
> > created in hopes of finding some clues as to what is going wrong, but
being> > barely a python newbie I get lost in the code.  :(  (Time to add
another> > programming language to my skill set.)
>
> Have you tried today''s changeset?  There have been plenty of
changes in
this> area in the last week, and a few today.
I grabbed the latest source, 7826, and still have the problem.
> What distro?  (Not that this will help, but it seems a good idea to keep
track> of problematic distro versions.)
Fedora Core 4 for both the dom0 and the domU
> Does /sbin/hotplug or /sbin/udevsend run at all?
>
> If so, does it make it as far as /etc/hotplug/xen-backend.agent?
I''m not sure how to tell if /sbin/udevsend binary gets run.  So I did
the
"echo /sbin/hotplug > /proc/sys/kernel/hotplug" trick so that I
could hack
debugging stuff into the /etc/hotplug/xen-backend.agent script.  It now
looks like:

-----------
#! /bin/sh

date=`date`
echo "$date $0 $*" >> /var/log/xen-backend.log
echo "XENBUS_TYPE = \"$XENBUS_TYPE\"  ACTION =
\"$ACTION\"" >>
/var/log/xen-backend.log
PATH=/etc/xen/scripts:$PATH

case "$XENBUS_TYPE" in
  vbd)
    echo "/etc/xen/scripts/block $ACTION" >>
/var/log/xen-backend.log
    /etc/xen/scripts/block "$ACTION"
    ;;
  vif)
    if [ -n "$script" ]; then
      echo "$script $ACTION" >> /var/log/xen-backend.log
    else
      echo "\$script is empty" >> /var/log/xen-backend.log
    fi
    [ -n "$script" ] && $script "$ACTION"
    ;;
esac

case "$ACTION" in
  add)
    ;;
  remove)
    # remove device backend store entries
    xenstore-rm -t "$XENBUS_PATH"       || true
    xenstore-rm -t "error/$XENBUS_PATH" || true
    ;;
  online)
    ;;
  offline)
    ;;
esac
-----------

After I run "xm create /tmp/dom1/cfg" /var/log/xen-backend.log has:

-----------
Wed Nov 16 10:54:37 CST 2005 /etc/hotplug/xen-backend.agent
XENBUS_TYPE = "vbd"  ACTION = "add"
/etc/xen/scripts/block add
Wed Nov 16 10:54:37 CST 2005 /etc/hotplug/xen-backend.agent
XENBUS_TYPE = "vbd"  ACTION = "add"
/etc/xen/scripts/block add
-----------

Looks like it doesn''t get called to create the vif.
> If so, how about /etc/xen/scripts/vif-common.sh?
I added similar debugging statements to /etc/xen/scripts/vif-bridge and
/etc/xen/scripts/vif-common.sh.  I don''t get any log file for either
vif-*
script.  Looks like they are not called.
> What do you get if you run
>
> PYTHONPATH=/usr/lib/python python /usr/lib/python/xen/util/diagnose.
> py <dom ID>
-----------
[root@gir scripts]# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      768     4 r-----   112.4
dom1                               4      768     4 -b----    12.3
[root@gir scripts]# PYTHONPATH=/usr/lib/python python
/usr/lib/python/xen/util/diagnose.py 4
Domain ID is 4.
Domain name is dom1.
Found 1 device classes in use.
Found 2 vbd devices.
Found device vbd, 2049.
Backend is in state Connected.
Frontend is in state Connected.
Device vbd, 2049 hotplugging has completed successfully, and is connected
to physical device 0xfd01.
Found device vbd, 2050.
Backend is in state Connected.
Frontend is in state Connected.
Device vbd, 2050 hotplugging has completed successfully, and is connected
to physical device 0xfd00.
-----------

FWIW, /tmp/dom1.cfg says:

-----------
kernel = "/boot/vmlinuz-xenU-smp"
memory = 768
name = "dom1"
vcpus = 4
nics = 1
disk = [''phy:/dev/virt-blkdev-backend/dom1,sda1,w'',
''phy:/dev/virt-blkdev-backend/usr,sda2,r'']
root = "/dev/sda1 ro"
extra = " profile=1 GATEWAY=10.0.1.254 NETMASK=255.255.0.0
IPADDR=10.0.134.1 HOSTNAME=dom1"
on_poweroff = ''destroy''
on_reboot   = ''destroy''
on_crash    = ''destroy''
-----------
> Thanks for your help,
Thanks for yours.
> Ewan.
Steve D.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ewan Mellor

2005-Nov-16 18:45 UTC

head link

Re: [Xen-devel] can''t start domU

On Wed, Nov 16, 2005 at 11:19:12AM -0600, Steve Dobbelstein wrote:
> I''m not sure how to tell if /sbin/udevsend binary gets run.
mv /sbin/udevsend{,.real}
cat >/sbin/udevsend
#!/bin/sh
echo "udevsend running" >>/var/log/udev.log
exec /sbin/udevsend.real
^D

ought to do it.  I think that it''s clear that the whole hotplug
subsystem isn''t
running in your case, so I wouldn''t worry about it.
> [root@gir scripts]# PYTHONPATH=/usr/lib/python python
> /usr/lib/python/xen/util/diagnose.py 4
> Domain ID is 4.
> Domain name is dom1.
> Found 1 device classes in use.
> Found 2 vbd devices.
> Found device vbd, 2049.
> Backend is in state Connected.
> Frontend is in state Connected.
> Device vbd, 2049 hotplugging has completed successfully, and is connected
> to physical device 0xfd01.
> Found device vbd, 2050.
> Backend is in state Connected.
> Frontend is in state Connected.
> Device vbd, 2050 hotplugging has completed successfully, and is connected
> to physical device 0xfd00.
This suggests that the tools have not even got as far as trying to create a
vif for you.  Are you getting the "Device 0 (vif) could not be connected.
Hotplug scripts not working." message, same as Charles Coffing?  That
really
would be very curious.  In any case, your config file ought to be causing Xend
to at least try to create a vif.  I will need to see your /var/log/xend.log to
pin this down, and if you can get a xenstore trace, that will be useful too.

Cheers,

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ewan Mellor

2005-Nov-16 18:49 UTC

head link

Re: [Xen-devel] can''t start domU

On Wed, Nov 16, 2005 at 06:45:39PM +0000, Ewan Mellor wrote:
> On Wed, Nov 16, 2005 at 11:19:12AM -0600, Steve Dobbelstein wrote:
> 
> > I''m not sure how to tell if /sbin/udevsend binary gets run.
> 
> mv /sbin/udevsend{,.real}
> cat >/sbin/udevsend
> #!/bin/sh
> echo "udevsend running" >>/var/log/udev.log
> exec /sbin/udevsend.real
> ^D
Followed by 

chmod a+x /sbin/udevsend

of course.

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

peter Bier

2005-Nov-16 20:14 UTC

head link

[Xen-devel] Re: can''t start domU

Ewan Mellor <ewan <at> xensource.com> writes:
> 
> On Wed, Nov 16, 2005 at 06:45:39PM +0000, Ewan Mellor wrote:
> 
> > On Wed, Nov 16, 2005 at 11:19:12AM -0600, Steve Dobbelstein wrote:
> > 
> > > I''m not sure how to tell if /sbin/udevsend binary gets
run.
> > 
> > mv /sbin/udevsend{,.real}
> > cat >/sbin/udevsend
> > #!/bin/sh
> > echo "udevsend running" >>/var/log/udev.log
> > exec /sbin/udevsend.real
> > ^D
> 
> Followed by 
> 
> chmod a+x /sbin/udevsend
> 
> of course.
> 
> Ewan.
> 

I get the same error message when starting up a domain and I found out, that 
xenrestore says ENOENT when it gets the 
request "read /local/domain/0/backend/vif/20/0/hotplug-status", which
I
found out by setting the environment variable XENSTORED_TRACE and looking
into the file /var/log/senstored-trace.log

the output of the diagnose script is:

Domain ID is 20.
Domain name is ExampleDomain.
Found 2 device classes in use.
Found 4 vbd devices.
Found device vbd, 769.
Backend is in state Initialising.
Frontend is in state Initialising.
Device vbd, 769 hotplugging failed.
Found device vbd, 833.
Backend is in state Initialising.
Frontend is in state Initialising.
Device vbd, 833 hotplugging failed.
Found device vbd, 8705.
Backend is in state Initialising.
Frontend is in state Initialising.
Device vbd, 8705 hotplugging failed.
Found device vbd, 8451.
Backend is in state Initialising.
Frontend is in state Initialising.
Device vbd, 8451 hotplugging failed.
Found 1 vif devices.
Found device vif, 0.
Backend is in state Closing.
Frontend is in state Initialising.
Device vif, 0 hotplugging has completed successfully, and is using handle 0.

Peter Bier

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hien Nguyen

2005-Nov-16 20:23 UTC

head link

Re: [Xen-devel] can''t start domU

The problem occurred when configured xend-config.sxp with route. See
detalies in the bug #392:
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=392

# tail -20 /var/log/xend.log
[2005-11-16 13:53:05 xend.XendDomainInfo] DEBUG (XendDomainInfo:574)
Storing domain details: {''console/ring-ref'':
''122397'', ''console/port'':
''2'',
''name'': ''x335-hien1-vm4'',
''console/limit'': ''1048576'',
''vm'':
''/vm/32184982-022a-fddd-cc97-6bf93f56afcf'',
''domid'': ''1'',
''cpu/0/availability'': ''online'',
''memory/target'': ''1048576'',
''store/ring-ref'': ''122398'',
''store/port'': ''1''}
[2005-11-16 13:53:05 xend] DEBUG (DevController:82) Waiting for devices
vif.
[2005-11-16 13:53:05 xend] DEBUG (DevController:88) Waiting for 0.
[2005-11-16 13:53:05 xend] DEBUG (DevController:309) hotplugStatus 0
[2005-11-16 13:53:10 xend] ERROR (SrvBase:87) Request wait_for_devices
failed.
Traceback (most recent call last):
  File "/usr/lib/python/xen/web/SrvBase.py", line 85, in perform
    return op_method(op, req)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 68, in
op_wait_for_devices
    return self.dom.waitForDevices()
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 1200, in
waitForDevices    self.waitForDevices_(c)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 856, in
waitForDevices_    return
self.getDeviceController(deviceClass).waitForDevices()
  File "/usr/lib/python/xen/xend/server/DevController.py", line 84, in
waitForDevices
    return map(self.waitForDevice, self.deviceIDs())
  File "/usr/lib/python/xen/xend/server/DevController.py", line 93, in
waitForDevice
    raise VmError( ("Device %s (%s) could not be connected. "
VmError: Device 0 (vif) could not be connected. Hotplug scripts not working
[2005-11-16 13:53:10 xend] DEBUG (DevController:309) hotplugStatus 0


changeset:   7793:090e44133d40
tag:         tip
user:        kaf24@firebug.cl.cam.ac.uk
date:        Mon Nov 14 11:13:38 2005 +0100
summary:     Use make_lowmem_page_readonly/writable() in preference to the

Regards,

 Hien Nguyen
 Linux Technology Center             AUSTIN
 Phone: (512) 838-4140            Tie Line: 678-4140
 e-mail: hien1@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Steve Dobbelstein

2005-Nov-16 23:21 UTC

head link

Re: [Xen-devel] can''t start domU

Ewan Mellor <ewan@xensource.com> wrote on 11/16/2005 12:45:39 PM:
> On Wed, Nov 16, 2005 at 11:19:12AM -0600, Steve Dobbelstein wrote:
>
> > I''m not sure how to tell if /sbin/udevsend binary gets run.
>
> mv /sbin/udevsend{,.real}
> cat >/sbin/udevsend
> #!/bin/sh
> echo "udevsend running" >>/var/log/udev.log
> exec /sbin/udevsend.real
> ^D
>
> ought to do it.
Of course.  My duh.  I have been scratching my head over this and
scrambling it with python code that I forgot the obvious.
> I think that it''s clear that the whole hotplug
> subsystem isn''t
> running in your case, so I wouldn''t worry about it.
/var/log/udesend.log says:
--------
Wed Nov 16 16:54:21 CST 2005 udevsend net
Wed Nov 16 16:57:22 CST 2005 udevsend xen-backend
Wed Nov 16 16:57:22 CST 2005 udevsend xen-backend
--------

Looks like hotplug is initiated, but perhaps not for the vif stuff.
> > [root@gir scripts]# PYTHONPATH=/usr/lib/python python
> > /usr/lib/python/xen/util/diagnose.py 4
> > Domain ID is 4.
> > Domain name is dom1.
> > Found 1 device classes in use.
> > Found 2 vbd devices.
> > Found device vbd, 2049.
> > Backend is in state Connected.
> > Frontend is in state Connected.
> > Device vbd, 2049 hotplugging has completed successfully, and is
connected> > to physical device 0xfd01.
> > Found device vbd, 2050.
> > Backend is in state Connected.
> > Frontend is in state Connected.
> > Device vbd, 2050 hotplugging has completed successfully, and is
connected> > to physical device 0xfd00.
>
> This suggests that the tools have not even got as far as trying to create
a> vif for you.  Are you getting the "Device 0 (vif) could not be
connected.
> Hotplug scripts not working." message, same as Charles Coffing?  That
really> would be very curious.  In any case, your config file ought to be causing
Xend> to at least try to create a vif.  I will need to see your
/var/log/xend.log to> pin this down, and if you can get a xenstore trace, that will be usefultoo.

I had seen the "Device 0 (vif) could not be connected. Hotplug scripts not
working." message on occasion in the past.  It''s not doing it now.

Here are some helpful debug files.

(See attached file: xend.log)(See attached file: xend.trace.gz)(See
attached file: xend-debug.log)(See attached file: xenstored-trace.log.gz)

I''m afraid this is going to end up being some user error because
something
got changed elsewhere on the system, but I can''t figure out what.  The
machine used to not exhibit the problem with build 7713.  I upgraded the
machine to a later build (can''t remember which offhand) and saw the
problem.  I then backed down to 7713 but still saw the problem.  It has
been doing it on every build since then.  I have tried cleaning everything
out (make uninstall, rm -rf /etc/xen, etc.) and rebuilding and
reinstalling.  I am beginning to suspect something outside of Xen is
causing it to trip up.

Thanks a lot for your help.

Steve D.




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ewan Mellor

2005-Nov-16 23:39 UTC

head link

Re: [Xen-devel] can''t start domU

On Wed, Nov 16, 2005 at 01:23:22PM -0700, Hien Nguyen wrote:
> The problem occurred when configured xend-config.sxp with route. See
> detalies in the bug #392:
> [1]http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=392
> 
> changeset: 7793:090e44133d40
> tag: tip
> user: kaf24@firebug.cl.cam.ac.uk
> date: Mon Nov 14 11:13:38 2005 +0100
> summary: Use make_lowmem_page_readonly/writable() in preference to the
This is fixed in changeset 7814:a064c5804eae.

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hien Nguyen

2005-Nov-17 16:39 UTC

head link

Re: [Xen-devel] can''t start domU

It still happend with 7826:

changeset:   7826:9bf6f907b3ff
tag:         tip
user:        kaf24@firebug.cl.cam.ac.uk
date:        Wed Nov 16 04:29:52 2005 +0100
summary:     Trivial coding style fix.

[root@x335-hien1 xen]# xm create -c x335-hien1-vm4.cfg.switch
Using config file "x335-hien1-vm4.cfg.switch".
Error: Device 0 (vif) could not be connected. Hotplug scripts not working
[root@x335-hien1 xen]# tail -30 /var/log/xend.log
[2005-11-17 10:25:32 xend] DEBUG (image:174) cmdline        = 
root=/dev/sde1 ro
ro selinux=0 3
[2005-11-17 10:25:32 xend] DEBUG (image:175) ramdisk        [2005-11-17 10:25:32
xend] DEBUG (image:176) vcpus          = 1
[2005-11-17 10:25:32 xend] DEBUG (blkif:24) exception looking up device 
number for /dev/sde1: [Errno 2] No such file or directory:
''/dev/sde1''
[2005-11-17 10:25:32 xend] DEBUG (DevController:302) DevController: 
writing {''virtual-device'': ''2113'',
''backend-id'': ''0'',
''state'': ''1'',
''backend'':
''/local/domain/0/backend/vbd/23/2113''} to
/local/domain/23/device/vbd/2113.
[2005-11-17 10:25:32 xend] DEBUG (DevController:304) DevController: 
writing {''domain'': ''x335-rhel4-vm3'',
''params'':
''/FC4-FILEVBD4/FC4-32-filevbd4'',
''frontend-id'': ''23'',
''type'': ''file'',
''state'': ''1'', ''dev'':
''/dev/sde1'', ''frontend'':
''/local/domain/23/device/vbd/2113''} to 
/local/domain/0/backend/vbd/23/2113.
[2005-11-17 10:25:32 xend] DEBUG (DevController:302) DevController: 
writing {''virtual-device'': ''2096'',
''backend-id'': ''0'',
''state'': ''1'',
''backend'':
''/local/domain/0/backend/vbd/23/2096''} to
/local/domain/23/device/vbd/2096.
[2005-11-17 10:25:32 xend] DEBUG (DevController:304) DevController: 
writing {''domain'': ''x335-rhel4-vm3'',
''params'': ''sdd'',
''frontend-id'': ''23'',
''type'': ''phy'', ''state'':
''1'', ''dev'': ''sdd'',
''frontend'':
''/local/domain/23/device/vbd/2096''} to 
/local/domain/0/backend/vbd/23/2096.
[2005-11-17 10:25:32 xend] DEBUG (DevController:302) DevController: 
writing {''backend-id'': ''0'',
''mac'': ''AA:00:00:17:12:6D'',
''handle'': ''0'',
''state'': ''1'', ''backend'':
''/local/domain/0/backend/vif/23/0''} to
/local/domain/23/device/vif/0.
[2005-11-17 10:25:32 xend] DEBUG (DevController:304) DevController: 
writing {''domain'': ''x335-rhel4-vm3'',
''handle'': ''0'', ''script'':
''/etc/xen/scripts/vif-route'', ''ip'':
''9.3.189.251'', ''state'':
''1'',
''frontend'': ''/local/domain/23/device/vif/0'',
''mac'': ''AA:00:00:17:12:6D'',
''frontend-id'': ''23''} to
/local/domain/0/backend/vif/23/0.
[2005-11-17 10:25:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:557) 
Storing VM details: {''ssidref'': ''0'',
''uuid'':
''358df4d7-27f6-cc69-21e1-11a97b441f46'',
''on_reboot'': ''restart'',
''image'':
"(linux (kernel /boot/vmlinuz-2.6-xenU) (root ''/dev/sde1
ro'') (args ''ro
selinux=0 3'') (vcpus 1) (vcpus 1) (boot c) (display
localhost:10.0))",
''on_poweroff'': ''destroy'',
''name'': ''x335-rhel4-vm3'',
''vcpus'': ''1'',
''vcpu_avail'': ''1'',
''memory'': ''512'',
''on_crash'': ''restart'',
''maxmem'':
''512''}
[2005-11-17 10:25:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:582) 
Storing domain details: {''console/ring-ref'':
''122612'', ''console/port'':
''2'', ''name'':
''x335-rhel4-vm3'', ''console/limit'':
''1048576'', ''vm'':
''/vm/358df4d7-27f6-cc69-21e1-11a97b441f46'',
''domid'': ''23'',
''cpu/0/availability'': ''online'',
''memory/target'': ''524288'',
''store/ring-ref'': ''122613'',
''store/port'': ''1''}
[2005-11-17 10:25:32 xend] DEBUG (DevController:88) Waiting for devices 
vif.
[2005-11-17 10:25:32 xend] DEBUG (DevController:94) Waiting for 0.
[2005-11-17 10:25:32 xend] DEBUG (DevController:361) hotplugStatusCallback 
/local/domain/0/backend/vif/23/0/hotplug-status.
[2005-11-17 10:25:37 xend] ERROR (SrvBase:87) Request wait_for_devices 
failed.
Traceback (most recent call last):
  File "/usr/lib/python/xen/web/SrvBase.py", line 85, in perform
    return op_method(op, req)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 68, in 
op_wait_for_devices
    return self.dom.waitForDevices()
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 1205, in 
waitForDevices    self.waitForDevices_(c)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 864, in 
waitForDevices_    return 
self.getDeviceController(deviceClass).waitForDevices()
  File "/usr/lib/python/xen/xend/server/DevController.py", line 90, in
waitForDevices
    return map(self.waitForDevice, self.deviceIDs())
  File "/usr/lib/python/xen/xend/server/DevController.py", line 100,
in
waitForDevice
    raise VmError("Device %s (%s) could not be connected. "
VmError: Device 0 (vif) could not be connected. Hotplug scripts not 
working



Regards,

 Hien Nguyen
 Linux Technology Center             AUSTIN
 Phone: (512) 838-4140            Tie Line: 678-4140
 e-mail: hien1@us.ibm.com




Ewan Mellor <ewan@xensource.com> 
11/16/2005 05:39 PM

To
Hien Nguyen/Austin/IBM@IBMUS
cc
xen-devel@lists.xensource.com
Subject
Re: [Xen-devel] can''t start domU






On Wed, Nov 16, 2005 at 01:23:22PM -0700, Hien Nguyen wrote:
> The problem occurred when configured xend-config.sxp with route. See
> detalies in the bug #392:
> [1]http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=392
> 
> changeset: 7793:090e44133d40
> tag: tip
> user: kaf24@firebug.cl.cam.ac.uk
> date: Mon Nov 14 11:13:38 2005 +0100
> summary: Use make_lowmem_page_readonly/writable() in preference to the
This is fixed in changeset 7814:a064c5804eae.

Ewan.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ewan Mellor

2005-Nov-17 16:55 UTC

head link

Re: [Xen-devel] can''t start domU

On Thu, Nov 17, 2005 at 10:39:50AM -0600, Hien Nguyen wrote:
> It still happend with 7826:
> 
> changeset:   7826:9bf6f907b3ff
> tag:         tip
> user:        kaf24@firebug.cl.cam.ac.uk
> date:        Wed Nov 16 04:29:52 2005 +0100
> summary:     Trivial coding style fix.
> 
> [root@x335-hien1 xen]# xm create -c x335-hien1-vm4.cfg.switch
> Using config file "x335-hien1-vm4.cfg.switch".
> Error: Device 0 (vif) could not be connected. Hotplug scripts not working
And what do you get in /var/log/{debug,messages,syslog} now?

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hien Nguyen

2005-Nov-17 21:15 UTC

head link

Re: [Xen-devel] can''t start domU

There are no new messages in /var/log/xend-debug.log and /var/log/messages
after trying to create domU. Only xend.log recorded errors.

Regards,

 Hien Nguyen
 Linux Technology Center             AUSTIN
 Phone: (512) 838-4140            Tie Line: 678-4140
 e-mail: hien1@us.ibm.com



                                                                           
             Ewan Mellor                                                   
             <ewan@xensource.c                                             
             om>                                                        To 
                                                                           
             11/17/2005 10:55          Hien Nguyen/Austin/IBM@IBMUS        
             AM                                                         cc 
                                       xen-devel@lists.xensource.com       
                                                                           
                                                                           
                                                                           
                                                                           
                                                                   Subject 
                                       Re: [Xen-devel] can''t start domU
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




On Thu, Nov 17, 2005 at 10:39:50AM -0600, Hien Nguyen wrote:
> It still happend with 7826:
>
> changeset:   7826:9bf6f907b3ff
> tag:         tip
> user:        kaf24@firebug.cl.cam.ac.uk
> date:        Wed Nov 16 04:29:52 2005 +0100
> summary:     Trivial coding style fix.
>
> [root@x335-hien1 xen]# xm create -c x335-hien1-vm4.cfg.switch
> Using config file "x335-hien1-vm4.cfg.switch".
> Error: Device 0 (vif) could not be connected. Hotplug scripts not working
And what do you get in /var/log/{debug,messages,syslog} now?

Ewan.






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ewan Mellor

2005-Nov-18 17:32 UTC

head link

Re: [Xen-devel] can''t start domU

On Wed, Nov 16, 2005 at 05:21:31PM -0600, Steve Dobbelstein wrote:
> [Re failure to create vifs]
>
> I''m afraid this is going to end up being some user error because
something
> got changed elsewhere on the system, but I can''t figure out what. 
The
> machine used to not exhibit the problem with build 7713.  I upgraded the
> machine to a later build (can''t remember which offhand) and saw
the
> problem.  I then backed down to 7713 but still saw the problem.  It has
> been doing it on every build since then.  I have tried cleaning everything
> out (make uninstall, rm -rf /etc/xen, etc.) and rebuilding and
> reinstalling.  I am beginning to suspect something outside of Xen is
> causing it to trip up.
I''ve found your problem, in your /var/log/xend.trace file:
> |    |    | < server/netif.py:getDeviceDetails
> |    |    |  79         if devid == 0:
> |    |    |  80             return devid
These two lines in DevController.py have never existed in the Xen repository.
This was a patch posted to the xen-devel mailing list, and it was never right
in the first place.

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Steve Dobbelstein

2005-Nov-18 19:07 UTC

head link

Re: [Xen-devel] can''t start domU

Ewan Mellor <ewan@xensource.com> wrote on 11/18/2005 11:32:48 AM:
> On Wed, Nov 16, 2005 at 05:21:31PM -0600, Steve Dobbelstein wrote:
> I''ve found your problem, in your /var/log/xend.trace file:
>
> > |    |    | < server/netif.py:getDeviceDetails
> > |    |    |  79         if devid == 0:
> > |    |    |  80             return devid
>
> These two lines in DevController.py have never existed in the Xen
repository.> This was a patch posted to the xen-devel mailing list, and it was never
right> in the first place.
>
> Ewan.
Ugh.  Too many patches going around that are supposed to *fix* problems.

Thank you very much for your time, Ewan.  Sorry to bother you with a user
error.

Steve D.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Nov 2005 - can''t start domU

[Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

[Xen-devel] Re: can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU

Re: [Xen-devel] can''t start domU