Ray Barnes
2008-Jun-02 06:37 UTC
[Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script
Resending this to xen-devel since no one rendered a response on xen-users --- I''m bumping into a problem which looks like a bug. Perhaps it''s for lack of knowledge (and _documentation_ of HVM). If I do the following: brctl addbr bmette31 ifconfig bmette31 1.2.3.249 netmask 255.255.255.248 xm create bmette31 Where "bmette31" has a config file that has a vif entry like: vif = [ ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31''] this works *just fine*. I''m able to route through the host to the HVM domain which is set to 1.2.3.250. However, whenever I use a script which is loosely based on a vif-bridge script that I got from /etc/xen/scripts running Xen 3.2.1 compiled from source, it does not work. Specifically, I can reach 1.2.3.249 from the outside but not 1.2.3.250. As far as I can tell, the problem is that whenever ''brctl addbr bmette31'' is invoked from outside the script, networking between the host and the HVM guest works fine, but when invoked from within the script, networking between the host and the HVM guest does not work. Note that while the domain is running (and using the script below), ''brctl show'' indicates that I have a bridge called bmette31 and that ''ifconfig bmette31'' shows the right IP. So just to recap - if I comment out ''brctl addbr bmette31'' from the script below and run it manually then start the HVM guest, networking is fine, but if I run that from within the script it does not work. One thing I notice immediately is that when I create the bridge from the command-line, the bridge starts out with all zeros in the MAC address, and once the custom script runs upon starting the domU, it gets a MAC address like so: bridge name bridge id STP enabled interfaces bmette31 8000.8a10ff068f3f no vif61.0 That''s the working scenario as seen by ''brctl show''. The non-working scenario (invoking ''brctl addbr bmette31'' from within the script) look like this: bridge name bridge id STP enabled interfaces bmette31 8000.feffffffffff no vif62.0 FYI, I only care about this because I''m trying to make my domUs extremely portable (relying only on a config file and a vif script), i.e. for live migration. Staticly creating hundreds (thousands?) of bridges across all nodes isn''t an option. Any ideas? -Ray #!/bin/bash # #vif bridge script for HVMs #invoke like this: # #vif = [ ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31, script=vif-bmette31'' ] /usr/sbin/brctl addbr bmette31 dir=$(dirname "$0") . "$dir/vif-common.sh" bridge=${bridge:-} bridge=$(xenstore_read_default "$XENBUS_PATH/bridge" "$bridge") RET=0 ip link show $bridge 1>/dev/null 2>&1 || RET=1 if [ "$RET" -eq 1 ] then # do_without_error brctl addbr "$bridge" fatal "Could not find bridge device $bridge" fi case "$command" in online) setup_bridge_port "$vif" add_to_bridge "$bridge" "$vif" sleep 2 ifconfig "$bridge" 1.2.3.249 netmask 255.255.255.248 ;; offline) do_without_error ifconfig "$bridge" down do_without_error brctl delbr "$bridge" ;; esac log debug "Successful vif-bridge $command for $vif, bridge $bridge." if [ "$command" == "online" ] then success fi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jun-02 06:55 UTC
Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script
I doubt your script is getting executed. HVM guests have a script run by qemu-dm: the default location of that is /etc/xen/qemu-ifup and I think xend never overrides that. You probably need to do some command-line splicing in tools/python/xen/xend/image.py -- we already extract the ''bridge'' parameter and pass that along to qemu-dm, but the same is not done for the ''script'' parameter. -- Keir On 2/6/08 07:37, "Ray Barnes" <tical.net@gmail.com> wrote:> Resending this to xen-devel since no one rendered a response on xen-users > > --- > > I''m bumping into a problem which looks like a bug. Perhaps it''s for > lack of knowledge (and _documentation_ of HVM). If I do the > following: > > brctl addbr bmette31 > ifconfig bmette31 1.2.3.249 netmask 255.255.255.248 > xm create bmette31 > > Where "bmette31" has a config file that has a vif entry like: vif = [ > ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31''] this works > *just fine*. I''m able to route through the host to the HVM domain > which is set to 1.2.3.250. However, whenever I use a script which is > loosely based on a vif-bridge script that I got from /etc/xen/scripts > running Xen 3.2.1 compiled from source, it does not work. > Specifically, I can reach 1.2.3.249 from the outside but not > 1.2.3.250. As far as I can tell, the problem is that whenever ''brctl > addbr bmette31'' is invoked from outside the script, networking between > the host and the HVM guest works fine, but when invoked from within > the script, networking between the host and the HVM guest does not > work. Note that while the domain is running (and using the script > below), ''brctl show'' indicates that I have a bridge called bmette31 > and that ''ifconfig bmette31'' shows the right IP. So just to recap - > if I comment out ''brctl addbr bmette31'' from the script below and run > it manually then start the HVM guest, networking is fine, but if I run > that from within the script it does not work. > > One thing I notice immediately is that when I create the bridge from > the command-line, the bridge starts out with all zeros in the MAC > address, and once the custom script runs upon starting the domU, it > gets a MAC address like so: > > bridge name bridge id STP enabled interfaces > bmette31 8000.8a10ff068f3f no vif61.0 > > That''s the working scenario as seen by ''brctl show''. The non-working > scenario (invoking ''brctl addbr bmette31'' from within the script) look > like this: > > bridge name bridge id STP enabled interfaces > bmette31 8000.feffffffffff no vif62.0 > > FYI, I only care about this because I''m trying to make my domUs > extremely portable (relying only on a config file and a vif script), > i.e. for live migration. Staticly creating hundreds (thousands?) of > bridges across all nodes isn''t an option. Any ideas? > > > -Ray > > > #!/bin/bash > # > #vif bridge script for HVMs > #invoke like this: > # > #vif = [ ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31, > script=vif-bmette31'' ] > > /usr/sbin/brctl addbr bmette31 > > dir=$(dirname "$0") > . "$dir/vif-common.sh" > > bridge=${bridge:-} > bridge=$(xenstore_read_default "$XENBUS_PATH/bridge" "$bridge") > > > RET=0 > ip link show $bridge 1>/dev/null 2>&1 || RET=1 > if [ "$RET" -eq 1 ] > then > # do_without_error brctl addbr "$bridge" > fatal "Could not find bridge device $bridge" > fi > > case "$command" in > online) > setup_bridge_port "$vif" > add_to_bridge "$bridge" "$vif" > sleep 2 > ifconfig "$bridge" 1.2.3.249 netmask 255.255.255.248 > ;; > > offline) > do_without_error ifconfig "$bridge" down > do_without_error brctl delbr "$bridge" > ;; > esac > > > log debug "Successful vif-bridge $command for $vif, bridge $bridge." > if [ "$command" == "online" ] > then > success > fi > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Barnes
2008-Jun-02 07:26 UTC
Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script
Thanks Kier. My script does get executed but apparently it''s being done too late in the game for the addition of the bridge to be effective. To solve for this, based on your comments, I borrowed some code from one of the other scripts and stuck it in qemu-ifup: if [ ! -e "/sys/class/net/${bridge}/bridge" ]; then brctl addbr ${bridge} brctl stp ${bridge} off brctl setfd ${bridge} 0 fi This works perfectly, only with the added side-effect that tap0 ends up on the bridge too. I''d imagine this will cause problems once I get more than one HVM up under the same scenario on the same node, so I delete tap0 from the bridge at the bottom of qemu-ifup. As an aside, most people (according to what I''ve read) seem to take it for granted that you should bring up the bridge outside of Xen (i.e. in the OS''s network scripts). Obviously that''s a flawed methodology that doesn''t scale well. Surely there''s a cleaner way of doing this? Any plan to code something like this into a future release? -Ray On Mon, Jun 2, 2008 at 2:55 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> I doubt your script is getting executed. HVM guests have a script run by > qemu-dm: the default location of that is /etc/xen/qemu-ifup and I think xend > never overrides that. You probably need to do some command-line splicing in > tools/python/xen/xend/image.py -- we already extract the ''bridge'' parameter > and pass that along to qemu-dm, but the same is not done for the ''script'' > parameter. > > -- Keir > > On 2/6/08 07:37, "Ray Barnes" <tical.net@gmail.com> wrote: > >> Resending this to xen-devel since no one rendered a response on xen-users >> >> --- >> >> I''m bumping into a problem which looks like a bug. Perhaps it''s for >> lack of knowledge (and _documentation_ of HVM). If I do the >> following: >> >> brctl addbr bmette31 >> ifconfig bmette31 1.2.3.249 netmask 255.255.255.248 >> xm create bmette31 >> >> Where "bmette31" has a config file that has a vif entry like: vif = [ >> ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31''] this works >> *just fine*. I''m able to route through the host to the HVM domain >> which is set to 1.2.3.250. However, whenever I use a script which is >> loosely based on a vif-bridge script that I got from /etc/xen/scripts >> running Xen 3.2.1 compiled from source, it does not work. >> Specifically, I can reach 1.2.3.249 from the outside but not >> 1.2.3.250. As far as I can tell, the problem is that whenever ''brctl >> addbr bmette31'' is invoked from outside the script, networking between >> the host and the HVM guest works fine, but when invoked from within >> the script, networking between the host and the HVM guest does not >> work. Note that while the domain is running (and using the script >> below), ''brctl show'' indicates that I have a bridge called bmette31 >> and that ''ifconfig bmette31'' shows the right IP. So just to recap - >> if I comment out ''brctl addbr bmette31'' from the script below and run >> it manually then start the HVM guest, networking is fine, but if I run >> that from within the script it does not work. >> >> One thing I notice immediately is that when I create the bridge from >> the command-line, the bridge starts out with all zeros in the MAC >> address, and once the custom script runs upon starting the domU, it >> gets a MAC address like so: >> >> bridge name bridge id STP enabled interfaces >> bmette31 8000.8a10ff068f3f no vif61.0 >> >> That''s the working scenario as seen by ''brctl show''. The non-working >> scenario (invoking ''brctl addbr bmette31'' from within the script) look >> like this: >> >> bridge name bridge id STP enabled interfaces >> bmette31 8000.feffffffffff no vif62.0 >> >> FYI, I only care about this because I''m trying to make my domUs >> extremely portable (relying only on a config file and a vif script), >> i.e. for live migration. Staticly creating hundreds (thousands?) of >> bridges across all nodes isn''t an option. Any ideas? >> >> >> -Ray >> >> >> #!/bin/bash >> # >> #vif bridge script for HVMs >> #invoke like this: >> # >> #vif = [ ''type=ioemu, mac=00:16:3e:00:00:07, bridge=bmette31, >> script=vif-bmette31'' ] >> >> /usr/sbin/brctl addbr bmette31 >> >> dir=$(dirname "$0") >> . "$dir/vif-common.sh" >> >> bridge=${bridge:-} >> bridge=$(xenstore_read_default "$XENBUS_PATH/bridge" "$bridge") >> >> >> RET=0 >> ip link show $bridge 1>/dev/null 2>&1 || RET=1 >> if [ "$RET" -eq 1 ] >> then >> # do_without_error brctl addbr "$bridge" >> fatal "Could not find bridge device $bridge" >> fi >> >> case "$command" in >> online) >> setup_bridge_port "$vif" >> add_to_bridge "$bridge" "$vif" >> sleep 2 >> ifconfig "$bridge" 1.2.3.249 netmask 255.255.255.248 >> ;; >> >> offline) >> do_without_error ifconfig "$bridge" down >> do_without_error brctl delbr "$bridge" >> ;; >> esac >> >> >> log debug "Successful vif-bridge $command for $vif, bridge $bridge." >> if [ "$command" == "online" ] >> then >> success >> fi >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jun-02 07:31 UTC
Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script
On 2/6/08 08:26, "Ray Barnes" <tical.net@gmail.com> wrote:> As an aside, > most people (according to what I''ve read) seem to take it for granted > that you should bring up the bridge outside of Xen (i.e. in the OS''s > network scripts). Obviously that''s a flawed methodology that doesn''t > scale well. Surely there''s a cleaner way of doing this? Any plan to > code something like this into a future release?It depends what network topology you want. Most people want all VIFs on the same bridge shared with a physical network interface. What does bridge-per-vif let you do that you couldn''t do by other means? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Barnes
2008-Jun-02 07:46 UTC
Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script
On Mon, Jun 2, 2008 at 3:31 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> It depends what network topology you want. Most people want all VIFs on the > same bridge shared with a physical network interface. > > What does bridge-per-vif let you do that you couldn''t do by other means?I definitely can''t have all my VIFs on a bridge with the physical NIC. My environment calls for routing IP blocks with the nexthop set to the IP of the dom0 - this meets operational requirements and allows me to configure paravirt and HVM domUs similarly. If I bridged all VIFs to the physical interface, I''d have to have a Switched Virtual Interface for each HVM (for portability and other reasons) - this is a broken methodology. Many switches (i.e. C3550) run out of gas when you configure a bunch of SVIs. And in some cases I don''t have administrative control over the infrastructure in front of the dom0 so adding SVIs becomes even more cumbersome than normal. Much easier to just aggregate and route big blocks of IPs and make the dom0 split them off every which way. That''s a general overview, but as to your question on why I need bridge-per-vif, traffic accounting and Layer 2 isolation are good reasons IMO. -Ray _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jun-02 08:08 UTC
Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script
On 2/6/08 08:46, "Ray Barnes" <tical.net@gmail.com> wrote:> That''s a general overview, but as to your question on why I need > bridge-per-vif, traffic accounting and Layer 2 isolation are good > reasons IMO.Couldn''t you route directly to the vif, and get rid of the bridges entirely? K. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ray Barnes
2008-Jun-02 09:12 UTC
Re: [Xen-devel] BUG: [?] host-only networking under HVM is broken with custom script
On Mon, Jun 2, 2008 at 4:08 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> Couldn''t you route directly to the vif, and get rid of the bridges entirely?I could - but it would be a worse hack (IMO). I just ran a few tests and surprisingly, routed vifs work, however the IP for the VIF must be set in qemu-ifup and not in the custom vif script. Forgive my ignorance of what goes on "behind the curtain" but it seems to me that not being able to override qemu-ifup (i.e. specifying a script to be run instead of qemu-ifup) is the hangup here. So the only way I could use straight VIFs with no bridge is if I put a bunch of ''case'' statements for each vif/IP in qemu-ifup which will break my provisioning model a bit; it would be easier to use bridges with the aforementioned hack. Should this be revisited in development or am I way off base here? -Ray _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel