Xen users - Oct 2012 - network setup with one bonded interface and one virtual interface

Hi there.  I was wondering if someone could help me with the following
issue.  I have a server with two NICs.  I used to have eth0 as the internet
facing device and I would attach eth1 to br1 and run a nat-ed private
network for the VMs which would use br1 as the gateway.



I have recently changed my configuration.  For the purposes of redundancy I
have now aggregated eth0 and eth1 into a bonded device, bond0.  I want to
keep the VMs on a private network and haven''t quite worked out the
right way
to do this.  If I attach bond0 to a bridge, it seems I have to assign public
IPs to the VMs (please correct me if I''m wrong).  I have tried a few
other
configurations:

- create dummy0 and attach that to xenbr0 which is intended to be the
private network gateway

- create an alias bond0:1 and attach that to xenbr0

- I''ve also tried various combinations with the xen routing and nat
scripts
(which I''d rather not use if possible given they are becoming
deprecated)

- for all of the above I''ve used IP tables for the forwarding and
masquerading between the ''attempted'' private network and the
external facing
NIC



In all cases, I either end up cutting off access to bond0 or the private
network.



Is what I''m trying to do - have a single external interface and a
single
virtual internal nat-ed interface which can reach the outside internet ­
possible?  If so, could someone tell me what the basic setup looks like.  I
am using ubuntu12.04 for Dom0 so it would be helpful to get some guidance on
how the ''interfaces'' file should be constructed but most
important is to
know whether this can be done in theory (or do I not have enough
interfaces?).  I have read the wiki quite extensively and I appreciate there
are some sections covering this subject but nothing I''ve tried in
practice
has worked so far.



For what''s it worth, here''s the latest version of my
interfaces file where
I''m trying out the dummy device.  Thanks in advance for any assistance.



auto lo

iface lo inet loopback



# The bonded network interface

auto bond0

iface bond0 inet static

address [*** ]

netmask 255.255.255.240

gateway [ ***]

bond-slaves none

bond-mode active-backup

bond-miimon 100



# Enslave all the physical interfaces

auto eth0

iface eth0 inet manual

bond-master bond0



auto eth1

iface eth1 inet manual

bond-master bond0



# dummy interface

auto dummy0

iface dummy0 inet manual



# Configure the bridging interface

auto xenbr0

iface xenbr0 inet static

address 10.1.1.65

netmask 255.255.255.192

gateway 10.1.1.65

bridge-ports dummy0

bridge-fd 9

bridge-hello 2

bridge-maxage 12

bridge-stp off




_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

Adam Gold wrote:
>- create dummy0 and attach that to xenbr0 which is intended to be 
>the private network gateway
Very close - just don''t attach anything to the bridge - you can use 
(IIRC) "bridge-ports none" to do this.

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

On Thu, Oct 4, 2012 at 11:30 PM, Adam Gold <awg1@gmx.com>
wrote:
> Is what I''m trying to do - have a single external interface
You seem to already have that - bond0. Don''t mess with it.
> and a single
> virtual internal nat-ed interface which can reach the outside internet –
> possible?  If so, could someone tell me what the basic setup looks like.
installing libvirt-bin will automatically create virbr0 which does
just that, complete with the necessary iptables and dnsmasq setup.
Same thing with lxc, which will create lxcbr0.

As an alternative: http://wiki.1tux.org/wiki/Ubuntu/Bridge

While Simon said a bridge doesn''t need any dummy ports attached to it
(which is true in most cases), I like to attach dummy interfaces to it
anyway. This is because virtualbox will crash the whole system if you
use bridge networking and set virtualbox to use the bridge, while it
works just fine if you tell it to use the dummy interface.

-- 
Fajar

Hi.

Simon''s response stays valid.

El 04/10/12 11:30, Adam Gold escribió:
>  I want to keep the VMs on a private network and haven''t quite
> worked out the right way to do this.  If I attach bond0 to a bridge, it
> seems I have to assign public IPs to the VMs (please correct me if
I''m
> wrong).
Correction: you don''t *have* to. You may assign whatever IP''s
you like,
but topologically they will be in the same Ethernet network as the 
public interface. Sometimes this setup is useful, but looks like it''s 
not your case.
> - create dummy0 and attach that to xenbr0 which is intended to be the
> private network gateway
Should work, but as Simon said, dummy0 is not needed there.
> - create an alias bond0:1 and attach that to xenbr0
That''s weired, I guess it behaved funny. Alias are made on IP level,
and
the members of the bridge are on network interface (device) level. Wrong 
mix.
> - I''ve also tried various combinations with the xen routing and
nat
> scripts (which I''d rather not use if possible given they are
becoming
> deprecated)
You''d do well. Native OS''s tools for networking setup are much
more
predictable, between other advantages.
> - for all of the above I''ve used IP tables for the forwarding and
> masquerading between the ''attempted'' private network and
the external
> facing NIC
Make 2 bridges:
- One of them will contain you bond0 device and any VM''s interface you 
want public (if any).
- Other will not have any physical device, instead is will be used to 
attach the virtual interfaces of your DomU''s.

Make your routing rules and iptables setup as on any NAT router. Treat 
the first bridge as external interface, the second one will be internal.
> In all cases, I either end up cutting off access to bond0 or the private
> network.
Just keep your DomU''s network out of the bridge, containing bond0.

About interface file. Keep in mind the abstraction order of things.
You have two physical network interfaces. They already exists (but might 
be down, that is not a problem), no need to create them.
On top of them, you have the bond0 device, enslaving the interfaces.
On top of it, you have a ethernet bridge, of which bond0 is member.
You could assign IP directly to bond0, rendering it unusable for VM''s.
I
would advise to use it with a bridge, it will make your configuration 
more homogeneous.

So, interfaces file might look similar to this. That is from my own 
working config, I use another type of bonding and not sure about the 
right options for your case (active-backup):

==============================auto lo
iface lo inet loopback

#underlying bond, have no IP
auto bond0
iface bond0 inet manual
     slaves eth0 eth1
     bond-mode 802.3ad
     bond-miimon 100
     bond-downdelay 200
     bond-updelay 200

#external bridge
auto xenbr00
iface xenbr00 inet static
   address 11.22.33.44
   netmask 255.255.255.0
   gateway 11.22.33.254
   bridge_ports bond0
   bridge_stp off
   bridge_maxwait 0

#internal bridge
auto xenbr01
iface xenbr01 inet static
   address 192.168.X.X
   netmask 255.255.255.0
   bridge_ports none
   bridge_stp off
   bridge_maxwait 0

#another internal bridge, just as example
#let''s attach it to bond0''s VLAN2
#no IP assigned on Dom0, bridge is useful only for DomUs
auto xenbr02
iface xenbr02 inet manual
   bridge_ports bond0.2
   bridge_stp off
   bridge_maxwait 0
   bridge_fd 5

=============================
Note that eth0 and eth1 are not mentioned. Debian''s ifupdown scripts
are
smart enough, no need to confuse them. The interfaces will be woke up as 
soon as they implicitly required.

Another advice, leave Dom0 to venerate the hypervisor, and set up a 
separated DomU to serve as firewall. Much cleaner setup, and your 
firewall will need not to worry about the forwarding within the Dom0''s 
bridges.

Good idea to leave Dom0 a (second) external IP, heavily firewalled, just 
in case of doing something that risks you DomU firewall not doing well 
(like reboot). Not strictly necessarily.

Greetings.

-- 
Alexandre Kouznetsov

>
>On 04/10/2012 21:59, "Simon Hobson" <linux@thehobsons.co.uk>
wrote:
>
>- create dummy0 and attach that to xenbr0 which is intended to be
>the private network gateway
>
> Very close - just don''t attach anything to the bridge - you can
use
> (IIRC) "bridge-ports none" to do this.
>
-- 
> Simon Hobson
> 
> Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
> author Gladys Hobson. Novels - poetry - short stories - ideal as
> Christmas stocking fillers. Some available as e-books.

Thanks Simon, that did the trick!

Fajar: thank you for your recommendations as well, I will try them out.

Hello.

El 11/10/12 12:48, Adam Gold escribió:
> I wanted to thank you for an extremely clear and helpful response.
Great it was useful!
It happened that I just have delivered a similar setup, so I had the 
examples at hand.
> I''m
> going to work through trying out your suggestions and I may – if you
> don''t mind – revert with some additional questions if I have any.
Sure thing. Why don''t you post them to the list?
You could get more feedback, and any answer will be indexed by search 
engines for future reference. In any case, I''ll watch for it.
> Out
> of interest, if redundancy is the priority, which bond mode would you
> recommend?  From what I read here it seems active-backup is the
> way to go:
> http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding
Can''t tell for sure. I work with few bonded interfaces, so my knowledge
is too theoretical. I picked up configurations based on than, made 
tests, they gave expected results, so I stayed there.
Check http://www.linuxhorizon.ro/bonding.html fro resumed reference.

My favorite is 802.3ad aka mode=4. It seems to be smart enough to 
balance the traffic and provides failover. If I unplug one cable, in the 
worst case I''ll loose a few packets (TCP will easily correct that).

There are switches that clam they support 802.3ad, but in practice it 
turns they have a nonsense called "static LACP" (the standard implies 
that there is no 802.3ad without LACP). In that case I have successfully 
used balance-xor aka mode=2.

Never needed a pure failover without traffic balance. In case I needed 
that, I would choose between active-backup aka mode=1 and broadcast aka 
mode=3. The second one is more stupid, in the best sense of the word, so 
I can think of cases when it''s preferable.

If you think of fault tolerance, consider that the bonding is frequently 
done with ports on the same switch. Unless you have a fine piece of 
equipment that supports fancy stacking, which could have different ports 
of the same bond on different switches. It your bond is on the same 
switch and the switch dies, so does the bond, forget your fault 
tolerance. So, you better imagine your scenarios (intentional unplug of 
cable while making movements, broken cable, dead Ethernet port, etc), 
test (simulate) them and decide if the behavior of your setup is 
acceptable or not.

Greetings.

-- 
Alexandre Kouznetsov