Our current setup(based on 1.0 and 1.2) uses completely diskless machines. They pxeboot, which then runs grub, which then loads xen. dom0 and domN all do nfsroot, and everything is happy. So, along comes 2.0. It now uses a normal bridge to connect dom0 and domN. However, the bridge has a hole, where the network does not exist, while it copies the addresses from eth0 to br0, and changes all the routes. In nfsroot mode, this fails, as suddenly the network is inaccessible, so brctl(and friends) can no longer be found. It would be more safe, to not use a bridge to configure access to the other domains; this would leave networking running at all times on dom0. Otherwise, if the bridge setup code breaks, you would end up with dom0 not accessible from the net, even in normal disk-bound situations. ------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Brian Wolfe
2004-Aug-10 23:59 UTC
Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
Interesting. If you are correct, then this will also break my machines. 8-( Are you sayign that the old method is no longer there? that only the domain that grabs the net card will have direct access, the rest must go through the bridge? Not good. BTW, I''ve redone the debian/rules file to build any number of cpu-arch for 2.4 and 2.6 as well as priv and unpriv modes. I''ll toss you the patch once I have it at least building the .deb packages. Probably tonight or tomorrow morning. I''ve also changed up a bunch of dependancies and added xen-dev as well as the individual XenLinux image headers, source, and standard debian kernel image/modules/etc files. As for others pulling my debs (if anyone does), I''ll be placing the new 2.0-pre debs at the standard terabox.com/debian location in a couple days for playing with. It''s getting kinda large during compile to create all of the .deb packages. :) Adam, also need to chat with you on some solutions to the modules in debian linux domains.... On Tue, 2004-08-10 at 17:29, Adam Heath wrote:> Our current setup(based on 1.0 and 1.2) uses completely diskless machines. > They pxeboot, which then runs grub, which then loads xen. dom0 and domN all > do nfsroot, and everything is happy. > > So, along comes 2.0. It now uses a normal bridge to connect dom0 and domN. > However, the bridge has a hole, where the network does not exist, while it > copies the addresses from eth0 to br0, and changes all the routes. > > In nfsroot mode, this fails, as suddenly the network is inaccessible, so > brctl(and friends) can no longer be found. > > It would be more safe, to not use a bridge to configure access to the other > domains; this would leave networking running at all times on dom0. Otherwise, > if the bridge setup code breaks, you would end up with dom0 not accessible > from the net, even in normal disk-bound situations. > > > > > > > ------------------------------------------------------- > SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media > 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 > Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. > http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Adam Heath
2004-Aug-11 00:13 UTC
Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
On Tue, 10 Aug 2004, Brian Wolfe wrote:> Interesting. If you are correct, then this will also break my machines. > 8-( Are you sayign that the old method is no longer there? that only > the domain that grabs the net card will have direct access, the rest > must go through the bridge? Not good.The ones going thru the bridge can still do dhcp, if that''s what you''re asking. The problem is that the domain with real access to the nic looses access to the nic momentarily while the bridge is being configured. This breaks nfsroot setups. ------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-Aug-11 00:43 UTC
Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
> So, along comes 2.0. It now uses a normal bridge to connect dom0 and domN. > However, the bridge has a hole, where the network does not exist, while it > copies the addresses from eth0 to br0, and changes all the routes. > > In nfsroot mode, this fails, as suddenly the network is inaccessible, so > brctl(and friends) can no longer be found.It''s an unfortunate mis-feature of the Linux bridge code that when adding an interface to the bridge it doesn''t inherit the IP addresses associated with the interface: As I recall, one of either tx or rx breaks, but the other direction is OK. I presume 2.6 exhibits the same behaviour? We''ve tried to work around this in the /etc/xen/network script, but it''s certainly a problem for nfsroot dom0 systems. One option is to come up with a patch to the linux bridge code to ''fix'' the current arguably broken behaviour. It would be interesting to take this up with the bridge code maintainer. Any volumteers? The other alternative is to route rather than bridge VIF''s onto the real network. We''ve supplied example scripts for bridging, but it would be good to include example scripts for a routed setup too. Just edit the network-script and vif-script parameters in /etc/xen/xend-config.sxp to point at a pair of new scripts. I''ve had routed setups working just fine. The only slight annoyance is that I had to configure a dummy IP address for the backend (vifX.Y) interfaces to point routes through. I was hoping to set them up as explicit point-to-point links and avoid this, but ifconfig wouldn''t let me. Perhaps there''s some device flag that our backend driver should be setting to allow this? If so, a patch for this would be great. Thanks, Ian ------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Sachin Goyal
2004-Aug-11 03:05 UTC
Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
I have never tried pxeboot or diskless linux, but I guess you can include bridge tools (e.g brctl) in initrd image and configure the network card initially itself in bridge mode before nfs mounting the real root. sachin On Tue, 10 Aug 2004, Adam Heath wrote:> Our current setup(based on 1.0 and 1.2) uses completely diskless machines. > They pxeboot, which then runs grub, which then loads xen. dom0 and domN all > do nfsroot, and everything is happy. > > So, along comes 2.0. It now uses a normal bridge to connect dom0 and domN. > However, the bridge has a hole, where the network does not exist, while it > copies the addresses from eth0 to br0, and changes all the routes. > > In nfsroot mode, this fails, as suddenly the network is inaccessible, so > brctl(and friends) can no longer be found. > > It would be more safe, to not use a bridge to configure access to the other > domains; this would leave networking running at all times on dom0. Otherwise, > if the bridge setup code breaks, you would end up with dom0 not accessible > from the net, even in normal disk-bound situations. > > > > > > > ------------------------------------------------------- > SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media > 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 > Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. > http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel >------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Brian Wolfe
2004-Aug-11 05:52 UTC
Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
Hmm, that is a good idea. I have been pursuing the idea of a stripped down linux OS just for running as the Domain0. I''ll be taking the base from the Debian install CD and adding the necessary libraries and whatnot that Domain0 requires to perform it''s duties as the privileged domain. I have a partially working stripped down copy of the standard Debian unstable base install on a test server. Still needs work to reduce it further so that it will fit into a small initrd. The end goal is to have Domain0 not need ANY network or physical disk access. The config files for xen domains would be pulled either from a http based config server, or loaded from a temporarily mounted source at boot time, or pre-exist int he initrd image. Don''t expect an image for several weeks. 8-P I''ve got my hands full already getting the latest Xen packaged for Debian unstable and iSCSI packaged since I could not find any packages pre-made. The good news is that I have a working debian/rules file for compiling all of the targets and the start of .config files for 2.6.7 priv and unpriv. I''ll probably toss up some very broken .deb files Wednesday night (CST). I''m certain that Adam will find PLENTY of packaging flaws for me to fix up. :) I''va mangled his build scripts quite well by now. *grin* On Tue, 2004-08-10 at 22:05, Sachin Goyal wrote:> I have never tried pxeboot or diskless linux, but I guess you can include > bridge tools (e.g brctl) in initrd image and configure the network card initially > itself in bridge mode before nfs mounting the real root. > > sachin > > > On Tue, 10 Aug 2004, Adam Heath wrote: > > > Our current setup(based on 1.0 and 1.2) uses completely diskless machines. > > They pxeboot, which then runs grub, which then loads xen. dom0 and domN all > > do nfsroot, and everything is happy. > > > > So, along comes 2.0. It now uses a normal bridge to connect dom0 and domN. > > However, the bridge has a hole, where the network does not exist, while it > > copies the addresses from eth0 to br0, and changes all the routes. > > > > In nfsroot mode, this fails, as suddenly the network is inaccessible, so > > brctl(and friends) can no longer be found. > > > > It would be more safe, to not use a bridge to configure access to the other > > domains; this would leave networking running at all times on dom0. Otherwise, > > if the bridge setup code breaks, you would end up with dom0 not accessible > > from the net, even in normal disk-bound situations. > > > > > > > > > > > > > > ------------------------------------------------------- > > SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media > > 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 > > Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. > > http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/xen-devel > > > > > ------------------------------------------------------- > SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media > 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 > Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. > http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Tue, Aug 10, 2004 at 05:29:30PM -0500, Adam Heath wrote:> Our current setup(based on 1.0 and 1.2) uses completely diskless machines. > They pxeboot, which then runs grub, which then loads xen. dom0 and domN all > do nfsroot, and everything is happy. > > So, along comes 2.0. It now uses a normal bridge to connect dom0 and domN. > However, the bridge has a hole, where the network does not exist, while it > copies the addresses from eth0 to br0, and changes all the routes.Can you use initramfs to configure the bridge/networking and mount the rootfs? My work arround was to add and configure the bridged interface at boot time, also i found the script for copying over the IP addresses was never reliable especailly when multiple addresses had been added to the old interface, and it made no attempt to deal with v6 addresses and routing completely breaks.> In nfsroot mode, this fails, as suddenly the network is inaccessible, so > brctl(and friends) can no longer be found.Can you run brctl before you mount the rootfs via nfs? -- Bill Boughton ------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
James Harper
2004-Aug-11 12:02 UTC
RE: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
Isn''t the misfeature you describe a function of the brctl utilities? Couldn''t they be modified to add an option to ''brctl addif'', eg ''inherit'' so to make br0 inherit eth0''s address you might use this syntax: brctl addif br0 eth0 inherit brctl could then internally do it this way: 1. create bridge br0 2. give br0 eth0''s ip address 3. up br0 4. add eth0 to br0 5. remove address from eth0 but maybe the problem is that 3+4+5 need to be done in an atomic operation and there''s no way to do that from userspace. another solution which would maybe solve the nfsroot problem is to have the kernel configure a bridge at boot time via kernel parameters, eg bridge=br0,eth0 and then do everything on br0 from the start rather than starting with eth0 and moving to br0 at some later stage. failing all of that, initrd might solve all the problems. I think it would be incorrect to just always automatically add the ip address of an interface being added though. James From: Ian Pratt Sent: Wed 11/08/2004 10:43 AM To: brianw@terrabox.com; Adam Heath; xen-devel@lists.sourceforge.net Cc: Ian.Pratt@cl.cam.ac.uk Subject: Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot> So, along comes 2.0. It now uses a normal bridge to connect dom0 and domN. > However, the bridge has a hole, where the network does not exist, while it > copies the addresses from eth0 to br0, and changes all the routes. > > In nfsroot mode, this fails, as suddenly the network is inaccessible, so > brctl(and friends) can no longer be found.It''s an unfortunate mis-feature of the Linux bridge code that when adding an interface to the bridge it doesn''t inherit the IP addresses associated with the interface: As I recall, one of either tx or rx breaks, but the other direction is OK. I presume 2.6 exhibits the same behaviour? We''ve tried to work around this in the /etc/xen/network script, but it''s certainly a problem for nfsroot dom0 systems. One option is to come up with a patch to the linux bridge code to ''fix'' the current arguably broken behaviour. It would be interesting to take this up with the bridge code maintainer. Any volumteers? The other alternative is to route rather than bridge VIF''s onto the real network. We''ve supplied example scripts for bridging, but it would be good to include example scripts for a routed setup too. Just edit the network-script and vif-script parameters in /etc/xen/xend-config.sxp to point at a pair of new scripts. I''ve had routed setups working just fine. The only slight annoyance is that I had to configure a dummy IP address for the backend (vifX.Y) interfaces to point routes through. I was hoping to set them up as explicit point-to-point links and avoid this, but ifconfig wouldn''t let me. Perhaps there''s some device flag that our backend driver should be setting to allow this? If so, a patch for this would be great. Thanks, Ian ------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-Aug-11 12:57 UTC
Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
> Isn''t the misfeature you describe a function of the brctl utilities? Couldn''t they be modified to add an option to ''brctl addif'', eg ''inherit'' so to make br0 inherit eth0''s address you might use this syntax:The current situation as implemented by the bridge code is inconsistent (at least in my view). If I create a bridge and add eth0 to it, I can still send packets via eth0 with the original IP address -- I just can''t receive them until I move the IP address to the bridge.> brctl could then internally do it this way: > 1. create bridge br0 > 2. give br0 eth0''s ip address > 3. up br0 > 4. add eth0 to br0 > 5. remove address from eth0 > > but maybe the problem is that 3+4+5 need to be done in an atomic operation and there''s no way to do that from userspace.I''m not sure how atomic it would need to be to keep nfsroot happy -- certainly all the pages of the brctl binary and libraries had better already be paged in. Also, you need to worry about transferring all the routes that were pointing through eth0...> another solution which would maybe solve the nfsroot problem is to have the kernel configure a bridge at boot time via kernel parameters, eg bridge=br0,eth0 and then do everything on br0 from the start rather than starting with eth0 and moving to br0 at some later stage.That could possibly work, though I believe such boot time options are becoming deprecated in favour of initrd''s in 2.6.> I think it would be incorrect to just always automatically add the ip address of an interface being added though.You haven''t totally convinced me ;-) The fact that you can still TX makes me think it might just be a bug. Has anyone got the time/inclination to take this up with <bridge@osdl.org> or search the archives ? http://lists.osdl.org/pipermail/bridge/ Ian ------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Derek Glidden
2004-Aug-11 16:56 UTC
Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
On Aug 10, 2004, at 6:29 PM, Adam Heath wrote:> Our current setup(based on 1.0 and 1.2) uses completely diskless > machines. > They pxeboot, which then runs grub, which then loads xen. dom0 and > domN all > do nfsroot, and everything is happy. > > So, along comes 2.0. It now uses a normal bridge to connect dom0 and > domN. > However, the bridge has a hole, where the network does not exist, > while it > copies the addresses from eth0 to br0, and changes all the routes. > > In nfsroot mode, this fails, as suddenly the network is inaccessible, > so > brctl(and friends) can no longer be found. > > It would be more safe, to not use a bridge to configure access to the > other > domains; this would leave networking running at all times on dom0. > Otherwise, > if the bridge setup code breaks, you would end up with dom0 not > accessible > from the net, even in normal disk-bound situations.The bridge setup is just a function of the network script that xend calls when you start it. You can easily change the network script to do "something else". There''s nothing inherent in the xen kernel or xend (AFAIK; I haven''t had a lot of time to play with it for a couple of weeks) that requires you use bridging to do your network stuff, or that you do it specifically the way it comes out of the box. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- "I think that''s what they mean by | "nickels a day can feed a child." | http://www.eff.org/ I thought, "How can food be so | http://www.anti-dmca.org/ cheap over there?" It''s not, they |-------------------------- just eat the nickels." -- Peter Nguyen -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- "I think that''s what they mean by | "nickels a day can feed a child." | http://www.eff.org/ I thought, "How can food be so | http://www.anti-dmca.org/ cheap over there?" It''s not, they |-------------------------- just eat the nickels." -- Peter Nguyen ------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2004-Aug-12 05:59 UTC
Re: [Xen-devel] xen 2.0, networking, bridging, and nfsroot
> On Tue, Aug 10, 2004 at 05:29:30PM -0500, Adam Heath wrote: > > Our current setup(based on 1.0 and 1.2) uses completely diskless machines. > > They pxeboot, which then runs grub, which then loads xen. dom0 and domN all > > do nfsroot, and everything is happy. > > > > So, along comes 2.0. It now uses a normal bridge to connect dom0 and domN. > > However, the bridge has a hole, where the network does not exist, while it > > copies the addresses from eth0 to br0, and changes all the routes. > > Can you use initramfs to configure the bridge/networking and mount the rootfs?This should work fine. I believe using an initramfs is now the preferred way to setup things like NFS root these days, though using the in-kernel functions is certainly more convenient.> My work arround was to add and configure the bridged interface at boot > time, also i found the script for copying over the IP addresses > was never reliable especailly when multiple addresses had been added > to the old interface, and it made no attempt to deal with v6 addresses > and routing completely breaks.The script manages to copy multiple IPv4 addresses for me -- I''d be grateful for a an example of it failing. Fixing it for IPv6 shouldn''t be hard, particularly if you could post the output of ''ip addr show ethX'' and ''ip route list''. Ian ------------------------------------------------------- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel