Xen users - May 2007 - network-bridge fails leaving tmpbridge around

''service xend start'' is failing in the network-bridge script. 
It fails
in such a way as to kill the network completely and leave tmpbridge around.

I modified network-bridge by adding at the top:

	#!/bin/bash
	
	exec 2>/tmp/log
	set -x

so that I could see what commands are failing.  The complete log is 
attached, but it turns out that the last thing logged is:

	+ ifdown eth0
	+ ip link set eth0 name peth0
	+ ip link set tmpbridge name eth0
	SIOCSIFNAME: Device or resource busy

so it would seem to be failing here:

	op_start () {
	#...
	#...
	    if ! ifdown ${netdev}; then
	        # If ifdown fails, remember the IP details.
	        get_ip_info ${netdev}
	        ip link set ${netdev} down
	        ip addr flush ${netdev}
	    fi
	    ip link set ${netdev} name ${pdev}
	    ip link set ${tdev} name ${bridge}    <--- fails

It''s strange because tmpbridge has just been created and eth0 is down, 
so how can either device be busy?

Fedora 7, xen-3.1.0-0.rc7.1.fc7

Rich.

-- 
Emerging Technologies, Red Hat  http://et.redhat.com/~rjones/
64 Baker Street, London, W1U 7DF     Mobile: +44 7866 314 421



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, May 11, 2007 at 12:12:52PM +0100, Richard W.M. Jones
wrote:
> ''service xend start'' is failing in the network-bridge
script.  It fails
> in such a way as to kill the network completely and leave tmpbridge around.
Can you capture

  ifconfig -a
  brctl show
  ip route show

Before & after doing the service xend start. 
> 
> I modified network-bridge by adding at the top:
> 
> 	#!/bin/bash
> 	
> 	exec 2>/tmp/log
> 	set -x
> 
> so that I could see what commands are failing.  The complete log is 
> attached, but it turns out that the last thing logged is:
> 
> 	+ ifdown eth0
> 	+ ip link set eth0 name peth0
> 	+ ip link set tmpbridge name eth0
> 	SIOCSIFNAME: Device or resource busy
I''m puzzelled how the eth0 name can be busy, when we just renamed
it (apparently without error) to be peth0 instead.
> so it would seem to be failing here:
> 
> 	op_start () {
> 	#...
> 	#...
> 	    if ! ifdown ${netdev}; then
> 	        # If ifdown fails, remember the IP details.
> 	        get_ip_info ${netdev}
> 	        ip link set ${netdev} down
> 	        ip addr flush ${netdev}
> 	    fi
> 	    ip link set ${netdev} name ${pdev}
> 	    ip link set ${tdev} name ${bridge}    <--- fails
> 
> It''s strange because tmpbridge has just been created and eth0 is
down,
> so how can either device be busy?
Indeed, very odd.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel P. Berrange wrote:
> On Fri, May 11, 2007 at 12:12:52PM +0100, Richard W.M. Jones wrote:
>> ''service xend start'' is failing in the network-bridge
script.  It fails
>> in such a way as to kill the network completely and leave tmpbridge
around.
> 
> Can you capture
> 
>   ifconfig -a
>   brctl show
>   ip route show
> 
> Before & after doing the service xend start. 
Attached are:

brctl.before.txt	brctl show (before)
ifconfig.before.txt	ifconfig -a (before)
ip-route.before.txt	ip route show (before)

Then after I did service xend start:

brctl.after.txt		brctl show (after)
ifconfig.after.txt	ifconfig -a (after)
ip-route.after.txt	ip route show (after)

and xend-config.sxp.

One thing that''s a bit strange is the sudden presence of virbr0.  I 
don''t understand where it comes from.  Note: there is no libvirtd or 
qemu running on this machine.

Rich.

-- 
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street,
Windsor, Berkshire, SL4 1TE, United Kingdom.  Registered in England and 
Wales
under Company Registration No. 03798903









_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Richard W.M. Jones wrote:
> One thing that''s a bit strange is the sudden presence of virbr0. 
I
> don''t understand where it comes from.  Note: there is no libvirtd
or
> qemu running on this machine.
Turns out that the mysterious presence of virbr0 was the clue.  Doing:

   rm -r /var/lib/xend/state

fixed the problem.  Thanks to Daniel Berrange.

Rich.

-- 
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street,
Windsor, Berkshire, SL4 1TE, United Kingdom.  Registered in England and 
Wales
under Company Registration No. 03798903


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Richard W.M. Jones wrote:
> Richard W.M. Jones wrote:
>> One thing that''s a bit strange is the sudden presence of
virbr0.  I
>> don''t understand where it comes from.  Note: there is no
libvirtd or
>> qemu running on this machine.
>
> Turns out that the mysterious presence of virbr0 was the clue.  Doing:
>
>   rm -r /var/lib/xend/state
I''ve got the same problem. Removing the state directory didn''t
solve my
problem. Some output of ifconfig, route, brctl is attached to the mail

Regards Jonas







_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, May 11, 2007 at 03:28:48PM +0200, Weismueller, Jonas
wrote:
> Richard W.M. Jones wrote:
> >Richard W.M. Jones wrote:
> >>One thing that''s a bit strange is the sudden presence of
virbr0.  I
> >>don''t understand where it comes from.  Note: there is no
libvirtd or
> >>qemu running on this machine.
> >
> >Turns out that the mysterious presence of virbr0 was the clue.  Doing:
> >
> >  rm -r /var/lib/xend/state
> I''ve got the same problem. Removing the state directory
didn''t solve my
> problem. Some output of ifconfig, route, brctl is attached to the mail
To clarify - you need to

  service xend stop
  rm -r /var/lib/xend/state/*
  <reboot>

To be sure that all trace of the bogus state & messed up networking 
is gone. This was a bug in one of the earlier 3.1-rc candidates that
is now fixed

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, May 11, 2007 at 03:28:48PM +0200, Weismueller, Jonas
wrote:
> Richard W.M. Jones wrote:
> >Richard W.M. Jones wrote:
> >>One thing that''s a bit strange is the sudden presence of
virbr0.  I
> >>don''t understand where it comes from.  Note: there is no
libvirtd or
> >>qemu running on this machine.
> >
> >Turns out that the mysterious presence of virbr0 was the clue.  Doing:
> >
> >  rm -r /var/lib/xend/state
> I''ve got the same problem. Removing the state directory
didn''t solve my
> problem. Some output of ifconfig, route, brctl is attached to the mail
Actually re-reading your logs attached the prescence of
''virbr0'' is OK
in your case - it was there before starting xend, so should be there
after.

Can you try something slightly different.

 - Turn off xend service so it doesn''t start upon boot
 - Reboot
 - Run ''/etc/xen/scripts/network-bridge start'' manually

If that works, then can you reboot again & manually run
''/etc/init.d/xend start'' instead. If that fails, I''d
be intersted in
the xend.log & xend-debug.log

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel P. Berrange wrote:
> On Fri, May 11, 2007 at 03:28:48PM +0200, Weismueller, Jonas wrote:
>   
>> Richard W.M. Jones wrote:
>>     
>>> Richard W.M. Jones wrote:
>>>       
>>>> One thing that''s a bit strange is the sudden presence
of virbr0.  I
>>>> don''t understand where it comes from.  Note: there is
no libvirtd or
>>>> qemu running on this machine.
>>>>         
>>> Turns out that the mysterious presence of virbr0 was the clue. 
Doing:
>>>
>>>  rm -r /var/lib/xend/state
>>>       
>> I''ve got the same problem. Removing the state directory
didn''t solve my
>> problem. Some output of ifconfig, route, brctl is attached to the mail
>>     
>
> Actually re-reading your logs attached the prescence of
''virbr0'' is OK
> in your case - it was there before starting xend, so should be there
> after.
>   
I already turned of xend in runlevel 5 for the ouput of before. Then I 
did /etc/init.d/xend start and created the after output. Where does the 
virbbr0 bridge comes from?
> Can you try something slightly different.
>
>  - Turn off xend service so it doesn''t start upon boot
>  - Reboot
>  - Run ''/etc/xen/scripts/network-bridge start'' manually
>   
This doesn''t work. It results in the attached
output.
> If that works, then can you reboot again & manually run
> ''/etc/init.d/xend start'' instead. If that fails,
I''d be intersted in
> the xend.log & xend-debug.log#
>   
Is also attached.
Regards Jonas







_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, May 11, 2007 at 03:53:17PM +0200, Weismueller, Jonas
wrote:
> Daniel P. Berrange wrote:
> >On Fri, May 11, 2007 at 03:28:48PM +0200, Weismueller, Jonas wrote:
> >  
> >>Richard W.M. Jones wrote:
> >>    
> >>>Richard W.M. Jones wrote:
> >>>      
> >>>>One thing that''s a bit strange is the sudden
presence of virbr0.  I
> >>>>don''t understand where it comes from.  Note: there
is no libvirtd or
> >>>>qemu running on this machine.
> >>>>        
> >>>Turns out that the mysterious presence of virbr0 was the clue. 
Doing:
> >>>
> >>> rm -r /var/lib/xend/state
> >>>      
> >>I''ve got the same problem. Removing the state directory
didn''t solve my
> >>problem. Some output of ifconfig, route, brctl is attached to the
mail
> >>    
> >
> >Actually re-reading your logs attached the prescence of
''virbr0'' is OK
> >in your case - it was there before starting xend, so should be there
> >after.
> >  
> I already turned of xend in runlevel 5 for the ouput of before. Then I 
> did /etc/init.d/xend start and created the after output. Where does the 
> virbbr0 bridge comes from?
That is a libvirt managed network device - created by libvirtd init script.
You can try turning off the libvirtd initscript temporarily, but I
don''t
think it will help in this case.
> >Can you try something slightly different.
> >
> > - Turn off xend service so it doesn''t start upon boot
> > - Reboot
> > - Run ''/etc/xen/scripts/network-bridge start''
manually
> >  
> This doesn''t work. It results in the attached output.
So the key error message is:

      SIOCSIFNAME: Device or resource busy

Looking at the kernel source -EBUSY is returned based on

        if (dev->flags & IFF_UP)
                return -EBUSY;

And looking at your ifconfig output, tmpbridge is indeed UP
> tmpbridge Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
>           inet addr:172.16.135.156  Bcast:172.16.255.255  Mask:255.255.0.0
>           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0 
>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
But given the place in the network-bridge scrpit where you say it is failing,
the tmpbridge device has not been brought up yet, so I''ve no clue how
it
got into that state.

As a nasty hack you can try changing op_start method from

    ip link set ${netdev} name ${pdev}
    ip link set ${tdev} name ${bridge}

To

    ip link set ${netdev} name ${pdev}
    ip link set ${tdev} link down
    ip link set ${tdev} name ${bridge}

But I''d really like to know why it was in the UP state at all. So
it''d
be useful if you could edit op_start and put in a call ''ifconfig
${tdev}''
and "echo doing stage ...blah..." in between every step in op_start()
to
see just where it gets into the UP state

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel P. Berrange wrote:
>
> But given the place in the network-bridge scrpit where you say it is
failing,
> the tmpbridge device has not been brought up yet, so I''ve no clue
how it
> got into that state.
>
> As a nasty hack you can try changing op_start method from
>
>     ip link set ${netdev} name ${pdev}
>     ip link set ${tdev} name ${bridge}
>
> To
>
>     ip link set ${netdev} name ${pdev}
>     ip link set ${tdev} link down
>     ip link set ${tdev} name ${bridge}
>
> But I''d really like to know why it was in the UP state at all. So
it''d
> be useful if you could edit op_start and put in a call ''ifconfig
${tdev}''
> and "echo doing stage ...blah..." in between every step in
op_start() to
> see just where it gets into the UP state
>
> Dan.
>   
I disabled libvirtd in the runlevel to get rid of the virbr0 device and 
rebooted. The result is shown in the *.before files.
I changed the lines in op_start as you said:
op_start () {
...
#    ip link set ${netdev} name ${pdev}
#    ip link set ${tdev} name ${bridge}
     ip link set ${netdev} name ${pdev}
     ip link set ${tdev} link down
     ip link set ${tdev} name ${bridge}
...

A /etc/init.d/xend start results in the *.after files.

Debugging the code results in:
[root@scp-156 ~]# /etc/xen/scripts/network-bridge start
stage 1
tmpbridge: error fetching interface information: Device not found

Attached you will find the code block of op_start{}

Regards Jonas








_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Weismueller, Jonas wrote:
> Daniel P. Berrange wrote:
>>
>> But given the place in the network-bridge scrpit where you say it is 
>> failing,
>> the tmpbridge device has not been brought up yet, so I''ve no
clue how it
>> got into that state.
>>
>> Dan.
>>   
> I disabled libvirtd in the runlevel to get rid of the virbr0 device and 
> rebooted. The result is shown in the *.before files.
> I changed the lines in op_start as you said:
> op_start () {
> ...
> #    ip link set ${netdev} name ${pdev}
> #    ip link set ${tdev} name ${bridge}
>     ip link set ${netdev} name ${pdev}
>     ip link set ${tdev} link down
>     ip link set ${tdev} name ${bridge}
> ...
> 
> A /etc/init.d/xend start results in the *.after files.
Sorry for the late note on this thread, I just wanted to add it to this thread 
in case anyone else was in the same situation.

I found my problem with the tmpbridge  - any interface aliases on eth0 i.e. 
eth0:1 eth0:2 etc.

Even though I changed these to ONBOOT=no, they came up and seemed to block the 
peth-ization of eth0. After hiding the ifcfg-eth0:* scripts in a sub directory 
and rebooting, the bridge and peth device came up.

Also, I had tried rm''ing the state files as suggested in a previous
email but
that did not work.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users