thr3ads.net - Xen devel - [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script [Jun 2008]

If this information is useful, please help other people find it:
Share via:

Ray Barnes

2008-Jun-02 06:37 UTC

[Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

Resending this to xen-devel since no one rendered a response on xen-users

---

I''m bumping into a problem which looks like a bug.  Perhaps
it''s for
lack of knowledge (and _documentation_ of HVM).  If I do the
following:

brctl addbr bmette31
ifconfig bmette31 1.2.3.249 netmask 255.255.255.248
xm create bmette31

Where "bmette31" has a config file that has a vif entry like: vif = [
''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31'']  this
works
*just fine*.  I''m able to route through the host to the HVM domain
which is set to 1.2.3.250.  However, whenever I use a script which is
loosely based on a vif-bridge script that I got from /etc/xen/scripts
running Xen 3.2.1 compiled from source, it does not work.
Specifically, I can reach 1.2.3.249 from the outside but not
1.2.3.250.  As far as I can tell, the problem is that whenever ''brctl
addbr bmette31'' is invoked from outside the script, networking between
the host and the HVM guest works fine, but when invoked from within
the script, networking between the host and the HVM guest does not
work.  Note that while the domain is running (and using the script
below), ''brctl show'' indicates that I have a bridge called
bmette31
and that ''ifconfig bmette31'' shows the right IP.  So just to
recap -
if I comment out ''brctl addbr bmette31'' from the script below
and run
it manually then start the HVM guest, networking is fine, but if I run
that from within the script it does not work.

One thing I notice immediately is that when I create the bridge from
the command-line, the bridge starts out with all zeros in the MAC
address, and once the custom script runs upon starting the domU, it
gets a MAC address like so:

bridge name     bridge id               STP enabled     interfaces
bmette31                8000.8a10ff068f3f       no              vif61.0

That''s the working scenario as seen by ''brctl show''. 
The non-working
scenario (invoking ''brctl addbr bmette31'' from within the
script) look
like this:

bridge name     bridge id               STP enabled     interfaces
bmette31                8000.feffffffffff       no              vif62.0

FYI, I only care about this because I''m trying to make my domUs
extremely portable (relying only on a config file and a vif script),
i.e. for live migration.  Staticly creating hundreds (thousands?) of
bridges across all nodes isn''t an option.  Any ideas?


-Ray


#!/bin/bash
#
#vif bridge script for HVMs
#invoke like this:
#
#vif = [ ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31,
script=vif-bmette31'' ]

/usr/sbin/brctl addbr bmette31

dir=$(dirname "$0")
. "$dir/vif-common.sh"

bridge=${bridge:-}
bridge=$(xenstore_read_default "$XENBUS_PATH/bridge"
"$bridge")


RET=0
ip link show $bridge 1>/dev/null 2>&1 || RET=1
if [ "$RET" -eq 1 ]
then
#       do_without_error brctl addbr "$bridge"
    fatal "Could not find bridge device $bridge"
fi

case "$command" in
    online)
        setup_bridge_port "$vif"
        add_to_bridge "$bridge" "$vif"
        sleep 2
        ifconfig "$bridge" 1.2.3.249 netmask 255.255.255.248
        ;;

    offline)
        do_without_error ifconfig "$bridge" down
        do_without_error brctl delbr "$bridge"
        ;;
esac


log debug "Successful vif-bridge $command for $vif, bridge $bridge."
if [ "$command" == "online" ]
then
  success
fi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2008-Jun-02 06:55 UTC

head link

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

I doubt your script is getting executed. HVM guests have a script run by
qemu-dm: the default location of that is /etc/xen/qemu-ifup and I think xend
never overrides that. You probably need to do some command-line splicing in
tools/python/xen/xend/image.py -- we already extract the
''bridge'' parameter
and pass that along to qemu-dm, but the same is not done for the
''script''
parameter.

 -- Keir

On 2/6/08 07:37, "Ray Barnes" <tical.net@gmail.com> wrote:
> Resending this to xen-devel since no one rendered a response on xen-users
> 
> ---
> 
> I''m bumping into a problem which looks like a bug.  Perhaps
it''s for
> lack of knowledge (and _documentation_ of HVM).  If I do the
> following:
> 
> brctl addbr bmette31
> ifconfig bmette31 1.2.3.249 netmask 255.255.255.248
> xm create bmette31
> 
> Where "bmette31" has a config file that has a vif entry like: vif
= [
> ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31''] 
this works
> *just fine*.  I''m able to route through the host to the HVM domain
> which is set to 1.2.3.250.  However, whenever I use a script which is
> loosely based on a vif-bridge script that I got from /etc/xen/scripts
> running Xen 3.2.1 compiled from source, it does not work.
> Specifically, I can reach 1.2.3.249 from the outside but not
> 1.2.3.250.  As far as I can tell, the problem is that whenever
''brctl
> addbr bmette31'' is invoked from outside the script, networking
between
> the host and the HVM guest works fine, but when invoked from within
> the script, networking between the host and the HVM guest does not
> work.  Note that while the domain is running (and using the script
> below), ''brctl show'' indicates that I have a bridge
called bmette31
> and that ''ifconfig bmette31'' shows the right IP.  So just
to recap -
> if I comment out ''brctl addbr bmette31'' from the script
below and run
> it manually then start the HVM guest, networking is fine, but if I run
> that from within the script it does not work.
> 
> One thing I notice immediately is that when I create the bridge from
> the command-line, the bridge starts out with all zeros in the MAC
> address, and once the custom script runs upon starting the domU, it
> gets a MAC address like so:
> 
> bridge name     bridge id               STP enabled     interfaces
> bmette31                8000.8a10ff068f3f       no              vif61.0
> 
> That''s the working scenario as seen by ''brctl
show''.  The non-working
> scenario (invoking ''brctl addbr bmette31'' from within the
script) look
> like this:
> 
> bridge name     bridge id               STP enabled     interfaces
> bmette31                8000.feffffffffff       no              vif62.0
> 
> FYI, I only care about this because I''m trying to make my domUs
> extremely portable (relying only on a config file and a vif script),
> i.e. for live migration.  Staticly creating hundreds (thousands?) of
> bridges across all nodes isn''t an option.  Any ideas?
> 
> 
> -Ray
> 
> 
> #!/bin/bash
> #
> #vif bridge script for HVMs
> #invoke like this:
> #
> #vif = [ ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31,
> script=vif-bmette31'' ]
> 
> /usr/sbin/brctl addbr bmette31
> 
> dir=$(dirname "$0")
> . "$dir/vif-common.sh"
> 
> bridge=${bridge:-}
> bridge=$(xenstore_read_default "$XENBUS_PATH/bridge"
"$bridge")
> 
> 
> RET=0
> ip link show $bridge 1>/dev/null 2>&1 || RET=1
> if [ "$RET" -eq 1 ]
> then
> #       do_without_error brctl addbr "$bridge"
>     fatal "Could not find bridge device $bridge"
> fi
> 
> case "$command" in
>     online)
>         setup_bridge_port "$vif"
>         add_to_bridge "$bridge" "$vif"
>         sleep 2
>         ifconfig "$bridge" 1.2.3.249 netmask 255.255.255.248
>         ;;
> 
>     offline)
>         do_without_error ifconfig "$bridge" down
>         do_without_error brctl delbr "$bridge"
>         ;;
> esac
> 
> 
> log debug "Successful vif-bridge $command for $vif, bridge
$bridge."
> if [ "$command" == "online" ]
> then
>   success
> fi
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ray Barnes

2008-Jun-02 07:26 UTC

head link

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

Thanks Kier.  My script does get executed but apparently it''s being
done too late in the game for the addition of the bridge to be
effective.  To solve for this, based on your comments, I borrowed some
code from one of the other scripts and stuck it in qemu-ifup:

    if [ ! -e "/sys/class/net/${bridge}/bridge" ]; then
        brctl addbr ${bridge}
        brctl stp ${bridge} off
        brctl setfd ${bridge} 0
    fi

This works perfectly, only with the added side-effect that tap0 ends
up on the bridge too.  I''d imagine this will cause problems once I get
more than one HVM up under the same scenario on the same node, so I
delete tap0 from the bridge at the bottom of qemu-ifup.  As an aside,
most people (according to what I''ve read) seem to take it for granted
that you should bring up the bridge outside of Xen (i.e. in the OS''s
network scripts).  Obviously that''s a flawed methodology that
doesn''t
scale well.  Surely there''s a cleaner way of doing this?  Any plan to
code something like this into a future release?

-Ray


On Mon, Jun 2, 2008 at 2:55 AM, Keir Fraser <keir.fraser@eu.citrix.com>
wrote:> I doubt your script is getting executed. HVM guests have a script run by
> qemu-dm: the default location of that is /etc/xen/qemu-ifup and I think
xend
> never overrides that. You probably need to do some command-line splicing in
> tools/python/xen/xend/image.py -- we already extract the
''bridge'' parameter
> and pass that along to qemu-dm, but the same is not done for the
''script''
> parameter.
>
>  -- Keir
>
> On 2/6/08 07:37, "Ray Barnes" <tical.net@gmail.com> wrote:
>
>> Resending this to xen-devel since no one rendered a response on
xen-users
>>
>> ---
>>
>> I''m bumping into a problem which looks like a bug.  Perhaps
it''s for
>> lack of knowledge (and _documentation_ of HVM).  If I do the
>> following:
>>
>> brctl addbr bmette31
>> ifconfig bmette31 1.2.3.249 netmask 255.255.255.248
>> xm create bmette31
>>
>> Where "bmette31" has a config file that has a vif entry like:
vif = [
>> ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31'']
this works
>> *just fine*.  I''m able to route through the host to the HVM
domain
>> which is set to 1.2.3.250.  However, whenever I use a script which is
>> loosely based on a vif-bridge script that I got from /etc/xen/scripts
>> running Xen 3.2.1 compiled from source, it does not work.
>> Specifically, I can reach 1.2.3.249 from the outside but not
>> 1.2.3.250.  As far as I can tell, the problem is that whenever
''brctl
>> addbr bmette31'' is invoked from outside the script, networking
between
>> the host and the HVM guest works fine, but when invoked from within
>> the script, networking between the host and the HVM guest does not
>> work.  Note that while the domain is running (and using the script
>> below), ''brctl show'' indicates that I have a bridge
called bmette31
>> and that ''ifconfig bmette31'' shows the right IP.  So
just to recap -
>> if I comment out ''brctl addbr bmette31'' from the
script below and run
>> it manually then start the HVM guest, networking is fine, but if I run
>> that from within the script it does not work.
>>
>> One thing I notice immediately is that when I create the bridge from
>> the command-line, the bridge starts out with all zeros in the MAC
>> address, and once the custom script runs upon starting the domU, it
>> gets a MAC address like so:
>>
>> bridge name     bridge id               STP enabled     interfaces
>> bmette31                8000.8a10ff068f3f       no              vif61.0
>>
>> That''s the working scenario as seen by ''brctl
show''.  The non-working
>> scenario (invoking ''brctl addbr bmette31'' from within
the script) look
>> like this:
>>
>> bridge name     bridge id               STP enabled     interfaces
>> bmette31                8000.feffffffffff       no              vif62.0
>>
>> FYI, I only care about this because I''m trying to make my
domUs
>> extremely portable (relying only on a config file and a vif script),
>> i.e. for live migration.  Staticly creating hundreds (thousands?) of
>> bridges across all nodes isn''t an option.  Any ideas?
>>
>>
>> -Ray
>>
>>
>> #!/bin/bash
>> #
>> #vif bridge script for HVMs
>> #invoke like this:
>> #
>> #vif = [ ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31,
>> script=vif-bmette31'' ]
>>
>> /usr/sbin/brctl addbr bmette31
>>
>> dir=$(dirname "$0")
>> . "$dir/vif-common.sh"
>>
>> bridge=${bridge:-}
>> bridge=$(xenstore_read_default "$XENBUS_PATH/bridge"
"$bridge")
>>
>>
>> RET=0
>> ip link show $bridge 1>/dev/null 2>&1 || RET=1
>> if [ "$RET" -eq 1 ]
>> then
>> #       do_without_error brctl addbr "$bridge"
>>     fatal "Could not find bridge device $bridge"
>> fi
>>
>> case "$command" in
>>     online)
>>         setup_bridge_port "$vif"
>>         add_to_bridge "$bridge" "$vif"
>>         sleep 2
>>         ifconfig "$bridge" 1.2.3.249 netmask 255.255.255.248
>>         ;;
>>
>>     offline)
>>         do_without_error ifconfig "$bridge" down
>>         do_without_error brctl delbr "$bridge"
>>         ;;
>> esac
>>
>>
>> log debug "Successful vif-bridge $command for $vif, bridge
$bridge."
>> if [ "$command" == "online" ]
>> then
>>   success
>> fi
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2008-Jun-02 07:31 UTC

head link

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

On 2/6/08 08:26, "Ray Barnes" <tical.net@gmail.com> wrote:
> As an aside,
> most people (according to what I''ve read) seem to take it for
granted
> that you should bring up the bridge outside of Xen (i.e. in the
OS''s
> network scripts).  Obviously that''s a flawed methodology that
doesn''t
> scale well.  Surely there''s a cleaner way of doing this?  Any plan
to
> code something like this into a future release?
It depends what network topology you want. Most people want all VIFs on the
same bridge shared with a physical network interface.

What does bridge-per-vif let you do that you couldn''t do by other
means?

 -- Keir




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ray Barnes

2008-Jun-02 07:46 UTC

head link

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

On Mon, Jun 2, 2008 at 3:31 AM, Keir Fraser <keir.fraser@eu.citrix.com>
wrote:
> It depends what network topology you want. Most people want all VIFs on the
> same bridge shared with a physical network interface.
>
> What does bridge-per-vif let you do that you couldn''t do by other
means?
I definitely can''t have all my VIFs on a bridge with the physical NIC.
 My environment calls for routing IP blocks with the nexthop set to
the IP of the dom0 - this meets operational requirements and allows me
to configure paravirt and HVM domUs similarly.  If I bridged all VIFs
to the physical interface, I''d have to have a Switched Virtual
Interface for each HVM (for portability and other reasons) - this is a
broken methodology.  Many switches (i.e. C3550) run out of gas when
you configure a bunch of SVIs.  And in some cases I don''t have
administrative control over the infrastructure in front of the dom0 so
adding SVIs becomes even more cumbersome than normal.  Much easier to
just aggregate and route big blocks of IPs and make the dom0 split
them off every which way.

That''s a general overview, but as to your question on why I need
bridge-per-vif, traffic accounting and Layer 2 isolation are good
reasons IMO.

-Ray

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2008-Jun-02 08:08 UTC

head link

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

On 2/6/08 08:46, "Ray Barnes" <tical.net@gmail.com> wrote:
> That''s a general overview, but as to your question on why I need
> bridge-per-vif, traffic accounting and Layer 2 isolation are good
> reasons IMO.
Couldn''t you route directly to the vif, and get rid of the bridges
entirely?

 K.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ray Barnes

2008-Jun-02 09:12 UTC

head link

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

On Mon, Jun 2, 2008 at 4:08 AM, Keir Fraser <keir.fraser@eu.citrix.com>
wrote:
> Couldn''t you route directly to the vif, and get rid of the bridges
entirely?
I could - but it would be a worse hack (IMO).  I just ran a few tests
and surprisingly, routed vifs work, however the IP for the VIF must be
set in qemu-ifup and not in the custom vif script.  Forgive my
ignorance of what goes on "behind the curtain" but it seems to me that
not being able to override qemu-ifup (i.e. specifying a script to be
run instead of qemu-ifup) is the hangup here.  So the only way I could
use straight VIFs with no bridge is if I put a bunch of ''case''
statements for each vif/IP in qemu-ifup which will break my
provisioning model a bit; it would be easier to use bridges with the
aforementioned hack.  Should this be revisited in development or am I
way off base here?

-Ray

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Jun 2008 - BUG: [?] host-only networking under HVM is broken with custom script

[Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script

Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script