tinc - Mar 2013 - Bridged nodes sharing local IP to be used as gateway for LXC

Hello everyone,

I am running multiple nodes each having multiple LXC containers. On
those nodes I am using Tinc to share the subnet 10.20.0.0. The LXC
containers may migrate between nodes and should keep the same
configuration.
On all nodes traffic from the external interfaces (connected to the
internet) is forwarded to the bridge interfaces and masqueraded. The
LXC containers are using 10.20.0.1 as their default gateway which
should stay the same independent of the node they are running on.

How to do the following while running Tinc in switch mode,

on Node A:

  ip link set dev eth0 up
  ip addr add 192.168.200.30/24 dev eth0
  ip route add default via 192.168.200.1 dev eth0    # Gateway
connected to the internet

  brctl addbr br0
  brctl setfd br0 0
  ip link set dev br0 up
  ip addr add 10.20.0.1/16 dev br0   # using same IP on node as node B
so the LXC containers can use same gateway IP
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  ip route add 10.20.0.0/16 dev br0

  sysctl -w net.ipv4.ip_forward=1

on Node B:

  ip link set dev eth0 up
  ip addr add 192.168.160.10/24 dev eth0
  ip route add default via 192.168.160.1 dev eth0    # Gateway
connected to the internet

  brctl addbr br0
  brctl setfd br0 0
  ip link set dev br0 up
  ip addr add 10.20.0.1/16 dev br0   # using same IP on node as node A
so the LXC containers can use same gateway IP
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  ip route add 10.20.0.0/16 dev br0

  sysctl -w net.ipv4.ip_forward=1

meaning that both nodes will share the same IP which the LXC container
can use as their default gateway?

To be honest I already tried this configuration. It ran well for about
2 days till I noticed that sometimes LXC containers on Node B were
using the default gateway on Node A and the other way around.

Is it possible to force the 10.20.0.1 on the local interface to be
used instead of the traffic being bridged over to the other side? Like
setting some metrics?

Best regards,

Mike

On Tue, Mar 12, 2013 at 12:21:33AM +0100, Mike Machuidel wrote:
> I am running multiple nodes each having multiple LXC containers. On
> those nodes I am using Tinc to share the subnet 10.20.0.0. The LXC
> containers may migrate between nodes and should keep the same
> configuration.
> On all nodes traffic from the external interfaces (connected to the
> internet) is forwarded to the bridge interfaces and masqueraded. The
> LXC containers are using 10.20.0.1 as their default gateway which
> should stay the same independent of the node they are running on.
[...]
> on Node A:
> 
>   ip addr add 10.20.0.1/16 dev br0   # using same IP on node as node B
> so the LXC containers can use same gateway IP
> 
> on Node B:
> 
>   ip addr add 10.20.0.1/16 dev br0   # using same IP on node as node A
> so the LXC containers can use same gateway IP
[...]
> meaning that both nodes will share the same IP which the LXC container
> can use as their default gateway?
> 
> To be honest I already tried this configuration. It ran well for about
> 2 days till I noticed that sometimes LXC containers on Node B were
> using the default gateway on Node A and the other way around.
Well if you have two hosts with the same IP addresses on a network, the one
which responds first to an ARP request will win. The MAC address discovered
via ARP will be cached, so if a LXC container migrates to another node it will
not automatically change to the local gateway. And even if no containers
migrate, it could be that the local host has a higher load than the remote
node, and the remote node's ARP replies will arrive before the local
node's.

You could use ebtables to filter ARP requests/replies going to/coming from the
remote gateway. There might be other solutions, but I cannot think of any right
now.
> Is it possible to force the 10.20.0.1 on the local interface to be
> used instead of the traffic being bridged over to the other side? Like
> setting some metrics?
I think the problem happens on the MAC layer, not on the IP layer, in which
case metrics will not help much.

-- 
Met vriendelijke groet / with kind regards,
     Guus Sliepen <guus at tinc-vpn.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL:
<http://www.tinc-vpn.org/pipermail/tinc/attachments/20130314/0f9d322f/attachment.pgp>