There is a lot of confusion understanding Xen networking. I''ve attached two diagrams that may help explain it better for some Xen beginners. I think I would have benefited from this information when I was just getting started. The first diagram shows basic xen networking when you use network-bridge and vif-bridge scripts. The second diagram shows xen networking when using network-route and vif-route. For beginners, I recommand starting with bridge. I won''t even try to explain the -nat scripts here. I hope someone finds this information useful. Feel free to send me any constructive comments, corrections or improvements. ------------------------------ When using network-bridge/vif-bridge: when xend starts up, it runs the network-bridge script, which: - creates a new bridge named "xenbr0" - "real" ethernet interface eth0 is brought down - the IP and MAC addresses of eth0 are copied to virtual network interface veth0 - real interface eth0 is renamed "peth0" - virtual interface veth0 is renamed "eth0" - peth0 and vif0.0 are attached to bridge xenbr0 - the bridge, peth0, eth0 and vif0.0 are brought up when a domU starts up, xend (running in dom0) runs the vif-bridge script, which: - attaches vif<id#>.0 to xenbr0 - vif<id#>.0 is brought up ------------------------------ when using network-route/vif-route: when xend starts up, it runs network-route which: - enabled ip forwarding within dom0 when domU starts up, xend runs (within dom0) vif-route which: - copies the ip address from eth0 to vif<id#>.0 - brings up vif<id#>.0 - adds host static route for domU''s ip address specified in domU.sxp config file, pointing at interface vif<id#>.0 ------------------------------ xen creates, by default, seven pair of "connected virtual ethernet interfaces" for use by dom0. Think of them as two ethernet interfaces connected by an internal crossover ethernet cable. veth0 is connected to vif0.0, veth1 is connected to vif0.1, etc, up to veth7 -> vif0.7. You can use them by configuring IP and MAC addresses on the veth# end, then attaching the vif0.# end to a bridge. Every time you create a running domU instance, it is assigned a new domain id number. You don''t get to pick the number, sorry. The first domU will be id #1. The second one started will be #2, even if #1 isn''t running anymore. For each new domu, xen creates new "connected virtual ethernet interfaces", with one end of each pair is within the domU and the other end exists within dom0. For linux domU''s, the device name it sees is named "eth0". The other end of that virtual ethernet interface pair exists within dom0 as interface "vif<id#>.0". For example, domU #5''s eth0 is attached to vif5.0. If you create multiple network interfaces for a domU, it''s ends will be eth0, eth1, etc, whereas the dom0 end will be vif<id#>.0, vif<id#>.1, etc. When a domU is shutdown, the virtual ethernet interfaces for it are deleted. ------------------------------ Additional Notes: - you can change the bridge name from xenbr0 using: (network-script ''network-bridge bridge=mybridge'') in xend-config.sxp and rebooting or restarting xend - remember to configure the bridge to attach to in the domU''s config file (domu.sxp) using: set vif=[ ''bridge=mybridge'' ] or perhaps something like: set vif=[ ''mac=00:16:3e:01:01:01,bridge=mybridge'' ] - you can create multiple network interfaces, and attach them to different bridges using: set vif=[ ''mac=00:16:3e:70:01:01,bridge=br0'', ''mac=00:16:3e:70:02:01,bridge=br1'' ] - if you want to use multiple bridges, you must create them yourself, either manually, or via your own startup script, or via a custom script to replace network-bridge. For example: $ cd /etc/xen/scripts $ cp network-bridge network-custom $ cp vif-bridge vif-custom $ vi /etc/xen/xend-config.sxp (network-script network-custom) (vif-script vif-custom) $ vi network-custom # whatever you want - before you connect a physical interface to a bridge, remember to reset it''s mac and turn arp off. For example: # ip link set eth1 down # ip link set eth1 mac fe:ff:ff:ff:ff:ff arp off # brctl addif br1 eth1 # ip link set eth1 up ------------------------------ Good Luck! -- Patrick Wolfe email: pwolfe@employease.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Molle Bestefich
2006-Feb-01 18:07 UTC
[Xen-users] Re: an attempt to explain xen networking
Patrick Wolfe wrote: [...] Cool! Might I add some tiny suggestions? network-xen-bridge.png: * Move domU outside of dom0? What''s it doing there? * Name the domU "Linux domU #4", to illustrate that the VIF number derives from the domU #. (The current double "VIF #4" vs. "vif4" makes the "VIF #4" superfluous) * Remove the "Debian" wording, it''s not important. network-xen-route.png * (again,) while domU living inside of dom0 may be correct for some technical reason (RAM?), it seems confusing on a network diagram.. * (again,) the wording "Linux domU #4" would be preferable to "Linux domU VIF #4", IMHO at least. Super cool work, thanks a lot. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Patrick, does this concern Xen 3 only or is it wrong? :-) I do have a standard xen-bridged setup (xen 2.0.7), but in my case eth0 is not connected to a vif0.0 which in turn is connected to the bridge. My eth0 is directly connected to the bridge, there is no peth.0 connected to the bridge. Dirk Patrick Wolfe schrieb:>There is a lot of confusion understanding Xen networking. I''ve attached >two diagrams that may help explain it better for some Xen beginners. I >think I would have benefited from this information when I was just >getting started. > >The first diagram shows basic xen networking when you use network-bridge >and vif-bridge scripts. The second diagram shows xen networking when >using network-route and vif-route. For beginners, I recommand starting >with bridge. I won''t even try to explain the -nat scripts here. > >I hope someone finds this information useful. Feel free to send me any >constructive comments, corrections or improvements. > > >------------------------------ > >When using network-bridge/vif-bridge: > >when xend starts up, it runs the network-bridge script, which: >- creates a new bridge named "xenbr0" >- "real" ethernet interface eth0 is brought down >- the IP and MAC addresses of eth0 are copied to > virtual network interface veth0 >- real interface eth0 is renamed "peth0" >- virtual interface veth0 is renamed "eth0" >- peth0 and vif0.0 are attached to bridge xenbr0 >- the bridge, peth0, eth0 and vif0.0 are brought up > >when a domU starts up, xend (running in dom0) runs the vif-bridge >script, which: >- attaches vif<id#>.0 to xenbr0 >- vif<id#>.0 is brought up > > >------------------------------ > >when using network-route/vif-route: > >when xend starts up, it runs network-route which: >- enabled ip forwarding within dom0 > >when domU starts up, xend runs (within dom0) vif-route which: >- copies the ip address from eth0 to vif<id#>.0 >- brings up vif<id#>.0 >- adds host static route for domU''s ip address specified in > domU.sxp config file, pointing at interface vif<id#>.0 > > >------------------------------ > >xen creates, by default, seven pair of "connected virtual ethernet >interfaces" for use by dom0. Think of them as two ethernet interfaces >connected by an internal crossover ethernet cable. veth0 is connected >to vif0.0, veth1 is connected to vif0.1, etc, up to veth7 -> vif0.7. >You can use them by configuring IP and MAC addresses on the veth# end, >then attaching the vif0.# end to a bridge. > >Every time you create a running domU instance, it is assigned a new >domain id number. You don''t get to pick the number, sorry. The first >domU will be id #1. The second one started will be #2, even if #1 isn''t >running anymore. > >For each new domu, xen creates new "connected virtual ethernet >interfaces", with one end of each pair is within the domU and the other >end exists within dom0. For linux domU''s, the device name it sees is >named "eth0". The other end of that virtual ethernet interface pair >exists within dom0 as interface "vif<id#>.0". For example, domU #5''s >eth0 is attached to vif5.0. If you create multiple network interfaces >for a domU, it''s ends will be eth0, eth1, etc, whereas the dom0 end will >be vif<id#>.0, vif<id#>.1, etc. > >When a domU is shutdown, the virtual ethernet interfaces for it are >deleted. > > >------------------------------ > >Additional Notes: > >- you can change the bridge name from xenbr0 using: > (network-script ''network-bridge bridge=mybridge'') > in xend-config.sxp and rebooting or restarting xend >- remember to configure the bridge to attach to in the > domU''s config file (domu.sxp) using: > set vif=[ ''bridge=mybridge'' ] > or perhaps something like: > set vif=[ ''mac=00:16:3e:01:01:01,bridge=mybridge'' ] > >- you can create multiple network interfaces, and attach them > to different bridges using: > set vif=[ ''mac=00:16:3e:70:01:01,bridge=br0'', > ''mac=00:16:3e:70:02:01,bridge=br1'' ] > >- if you want to use multiple bridges, you must create them > yourself, either manually, or via your own startup script, > or via a custom script to replace network-bridge. For example: > $ cd /etc/xen/scripts > $ cp network-bridge network-custom > $ cp vif-bridge vif-custom > $ vi /etc/xen/xend-config.sxp > (network-script network-custom) > (vif-script vif-custom) > $ vi network-custom > # whatever you want > >- before you connect a physical interface to a bridge, > remember to reset it''s mac and turn arp off. For example: > > # ip link set eth1 down > # ip link set eth1 mac fe:ff:ff:ff:ff:ff arp off > # brctl addif br1 eth1 > # ip link set eth1 up > > >------------------------------ > >Good Luck! > > > > > > ------------------------------------------------------------------------ > > > ------------------------------------------------------------------------ > >------------------------------------------------------------------------ > >_______________________________________________ >Xen-users mailing list >Xen-users@lists.xensource.com >http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sanjay Upadhyay
2006-Feb-03 12:01 UTC
Re: [Xen-users] an attempt to explain xen networking
Hi Patrick, The explanation was much needed I guess for new people. The diagrams are cool. I personally would have some questions a) Why is the ARP turned off on the real ethernet b) earlier scripts(2.0.6) did not seem to create a virtual veth0, and copy the mac address and IP from eth0 to veth0.. however they did work too. Was there an apparent issue there ? c) Just in case with two nics, I dont think the network-bridge script is adequate. Note: The natting part (I mean network-nat and vif-nat scripts do not work on debian), but a similar diagram would help :) cheers On 2/1/06, Patrick Wolfe <pwolfe@employease.com> wrote:> There is a lot of confusion understanding Xen networking. I've attached > two diagrams that may help explain it better for some Xen beginners. I > think I would have benefited from this information when I was just > getting started. > > The first diagram shows basic xen networking when you use network-bridge > and vif-bridge scripts. The second diagram shows xen networking when > using network-route and vif-route. For beginners, I recommand starting > with bridge. I won't even try to explain the -nat scripts here. > > I hope someone finds this information useful. Feel free to send me any > constructive comments, corrections or improvements. > > > ------------------------------ > > When using network-bridge/vif-bridge: > > when xend starts up, it runs the network-bridge script, which: > - creates a new bridge named "xenbr0" > - "real" ethernet interface eth0 is brought down > - the IP and MAC addresses of eth0 are copied to > virtual network interface veth0 > - real interface eth0 is renamed "peth0" > - virtual interface veth0 is renamed "eth0" > - peth0 and vif0.0 are attached to bridge xenbr0 > - the bridge, peth0, eth0 and vif0.0 are brought up > > when a domU starts up, xend (running in dom0) runs the vif-bridge > script, which: > - attaches vif<id#>.0 to xenbr0 > - vif<id#>.0 is brought up > > > ------------------------------ > > when using network-route/vif-route: > > when xend starts up, it runs network-route which: > - enabled ip forwarding within dom0 > > when domU starts up, xend runs (within dom0) vif-route which: > - copies the ip address from eth0 to vif<id#>.0 > - brings up vif<id#>.0 > - adds host static route for domU's ip address specified in > domU.sxp config file, pointing at interface vif<id#>.0 > > > ------------------------------ > > xen creates, by default, seven pair of "connected virtual ethernet > interfaces" for use by dom0. Think of them as two ethernet interfaces > connected by an internal crossover ethernet cable. veth0 is connected > to vif0.0, veth1 is connected to vif0.1, etc, up to veth7 -> vif0.7. > You can use them by configuring IP and MAC addresses on the veth# end, > then attaching the vif0.# end to a bridge. > > Every time you create a running domU instance, it is assigned a new > domain id number. You don't get to pick the number, sorry. The first > domU will be id #1. The second one started will be #2, even if #1 isn't > running anymore. > > For each new domu, xen creates new "connected virtual ethernet > interfaces", with one end of each pair is within the domU and the other > end exists within dom0. For linux domU's, the device name it sees is > named "eth0". The other end of that virtual ethernet interface pair > exists within dom0 as interface "vif<id#>.0". For example, domU #5's > eth0 is attached to vif5.0. If you create multiple network interfaces > for a domU, it's ends will be eth0, eth1, etc, whereas the dom0 end will > be vif<id#>.0, vif<id#>.1, etc. > > When a domU is shutdown, the virtual ethernet interfaces for it are > deleted. > > > ------------------------------ > > Additional Notes: > > - you can change the bridge name from xenbr0 using: > (network-script 'network-bridge bridge=mybridge') > in xend-config.sxp and rebooting or restarting xend > - remember to configure the bridge to attach to in the > domU's config file (domu.sxp) using: > set vif=[ 'bridge=mybridge' ] > or perhaps something like: > set vif=[ 'mac=00:16:3e:01:01:01,bridge=mybridge' ] > > - you can create multiple network interfaces, and attach them > to different bridges using: > set vif=[ 'mac=00:16:3e:70:01:01,bridge=br0', > 'mac=00:16:3e:70:02:01,bridge=br1' ] > > - if you want to use multiple bridges, you must create them > yourself, either manually, or via your own startup script, > or via a custom script to replace network-bridge. For example: > $ cd /etc/xen/scripts > $ cp network-bridge network-custom > $ cp vif-bridge vif-custom > $ vi /etc/xen/xend-config.sxp > (network-script network-custom) > (vif-script vif-custom) > $ vi network-custom > # whatever you want > > - before you connect a physical interface to a bridge, > remember to reset it's mac and turn arp off. For example: > > # ip link set eth1 down > # ip link set eth1 mac fe:ff:ff:ff:ff:ff arp off > # brctl addif br1 eth1 > # ip link set eth1 up > > > ------------------------------ > > Good Luck! > > > -- > > Patrick Wolfe > > email: pwolfe@employease.com > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.1 (GNU/Linux) > > iD8DBQBD4MxOIibbc6631koRAm/jAJ95QIPoPTopXljYtgzzZwoU8pKVUwCfSqAK > g9Rafz/AsSVH9OjMFBs0Le8> =aSkb > -----END PGP SIGNATURE----- > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
CJ van den Berg
2006-Feb-03 12:29 UTC
Re: [Xen-users] an attempt to explain xen networking
On Fri, Feb 03, 2006 at 12:21:33PM +0100, Dirk H. Schulz wrote:> Hi Patrick, > > does this concern Xen 3 only or is it wrong? :-) > > I do have a standard xen-bridged setup (xen 2.0.7), but in my case eth0 > is not connected to a vif0.0 which in turn is connected to the bridge. > My eth0 is directly connected to the bridge, there is no peth.0 > connected to the bridge.Yes, this is new in 3.0. I''m not sure I get the point though myself. Maybe someone in the know would like to explain what the point of having peth is as opposed to just adding eth0 to the bridge? I have a feeling that there''s a good reason for this setup, but I can''t quite see what it is. -- CJ van den Berg mailto:cj@vdbonline.com xmpp:cj@vdbonline.com sip:8918316@sipgate.de _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
CJ van den Berg schrieb:>On Fri, Feb 03, 2006 at 12:21:33PM +0100, Dirk H. Schulz wrote: > > >>Hi Patrick, >> >>does this concern Xen 3 only or is it wrong? :-) >> >>I do have a standard xen-bridged setup (xen 2.0.7), but in my case eth0 >>is not connected to a vif0.0 which in turn is connected to the bridge. >>My eth0 is directly connected to the bridge, there is no peth.0 >>connected to the bridge. >> >> > >Yes, this is new in 3.0. I''m not sure I get the point though myself. > >Maybe someone in the know would like to explain what the point of having >peth is as opposed to just adding eth0 to the bridge? I have a feeling that >there''s a good reason for this setup, but I can''t quite see what it is. > >It is good to have the physical interface and the dom0 interface separated; thus you can e.g. setup a firewall on dom0 that does not affect the traffic to the domUs (just for protecting dom0 alone). Dirk _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Molle Bestefich
2006-Feb-03 12:50 UTC
[Xen-users] Re: an attempt to explain xen networking
Dirk H. Schulz wrote:> It is good to have the physical interface and the dom0 interface > separated; thus you can e.g. setup a firewall on dom0 that does not > affect the traffic to the domUs (just for protecting dom0 alone).Oh.. In that case, why is it _not_ good to have a separate physical and dom0 virtual interface when you''re doing routed setups? Per Patrick''s images, there are no peth0 etc. when using the routed setup vs. the bridging solution. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Patrick, On Wed, 2006-02-01 at 09:57 -0500, Patrick Wolfe wrote:> [...] > I hope someone finds this information useful. Feel free to send me any > constructive comments, corrections or improvements.Cool! When put in the wiki this would greatly benefit the usefulness of the XenNetworking page. I have one question about the following paragraph:> ------------------------------ > > when using network-route/vif-route: > > when xend starts up, it runs network-route which: > - enabled ip forwarding within dom0 > > when domU starts up, xend runs (within dom0) vif-route which: > - copies the ip address from eth0 to vif<id#>.0 > - brings up vif<id#>.0 > - adds host static route for domU''s ip address specified in > domU.sxp config file, pointing at interface vif<id#>.0 >As of xen 3.0, there is no file named domU.sxp, and I don''t see documentation on how to create such one. There''s only the xend-config.sxp, which has nothing to do with it, I assume. As written in my mail before where I asked for help about that networking stuff, my efforts trying to specify the domU ip address in the domU''s config file as described in a mail to this list beginning of 2005 were not successful at all. Anyway, thanks! If you need help to get that info in the wiki, please let me know. Henning _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Patrick Wolfe wrote:> I hope someone finds this information useful. Feel free to send me any > constructive comments, corrections or improvements.I waited someone to put the text in the wiki but it didn''t happen so I put the text in <http://wiki.xensource.com/xenwiki/XenNetworking>. Is it acceptible to put links to <http://lists.xensource.com/archives/html/xen-users/2006-02/pngn2CXkgHLRj.png> and <http://lists.xensource.com/archives/html/xen-users/2006-02/pngwwCW9X01Zw.png> ? Regards, ogi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 2/6/06, Ognyan Kulev <ogi@fmi.uni-sofia.bg> wrote:> > I waited someone to put the text in the wiki but it didn''t happen so I > put the text in <http://wiki.xensource.com/xenwiki/XenNetworking>. > > Is it acceptible to put links to > < > http://lists.xensource.com/archives/html/xen-users/2006-02/pngn2CXkgHLRj.png > > > and > < > http://lists.xensource.com/archives/html/xen-users/2006-02/pngwwCW9X01Zw.png > > > ? >I don''t think there should be any problems on that. I have been trying to get route working for myself, however wasn''t able to do so. Made a post on this couple of hrs back, hoping someone will show me the correct path. -- regards, Anand Gupta _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Anand Gupta wrote:> On 2/6/06, *Ognyan Kulev* <ogi@fmi.uni-sofia.bg > Is it acceptible to put links to > <http://lists.xensource.com/archives/html/xen-users/2006-02/pngn2CXkgHLRj.png> > and > < > http://lists.xensource.com/archives/html/xen-users/2006-02/pngwwCW9X01Zw.png> > ? > > I don''t think there should be any problems on that.I''ve added links to these illustrations. It would be good if someone have corrections and is sure about it to directly edit the pages. After all, that''s what the wiki:s are for :-) Regards, ogi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users