Linux Ethernet Bridging - Jan 2009 - [Bridge] Bridging without forwarding? (not bonding)

Hello,

It's entirely possible that I'm going about this all the wrong way...
I'm new here. :-)

I'm trying to set up an environment that works a bit like a private LAN
within Amazon's EC2 cloud. EC2 allows me to provision a bunch of servers on
demand, but they don't have predictable internal IP addresses for
communicating with one another, which makes things a little complicated.

What I'm trying to do is use a combination of vtun tap tunnels and bridging,
to make my servers feel like they're on a LAN together.

The topology I've been experimenting with for now is one where I have a
"master" server, a "slave" server, and many
"client" servers. I'd like the "master" and
"slave" servers to have static IPs, and the "client" servers
to get their addresses via DHCP. I have it so that the master and slave have a
tap tunnel between them, and every client has a tap tunnel to both the master
and the slave (but not to the other clients). Each client should be able to ping
both the master and slave. I don't care if they can ping the other clients
or not.

I've got it set up so that the master and slave each have a bridge interface
that aggregates all of the tunnels from each client (and the one to each other).
Each client has a bridge interface that aggregates the two tunnels to the master
and slave. All are called br0.

I'm not adding any eth interfaces to the bridge, just the tap interfaces.

I'm running a DHCP server on the master and slave in failover mode and
listening on br0, and on each client I'm attempting to obtain an IP address
for the br0 interface using a DHCP client.

I have stp switched on everywhere. If I disable stp all hell breaks loose and I
get duplicate packets all over the place.

I basically want each server to appear to have one interface, despite the fact
that there are multiple tunnels. I also want the system to work if either the
master or the slave server goes away.

What I was hoping would happen is that packets would take the shortest route to
where they want to go. They should only ever have to go over one tunnel, except
for clients attempting to contact each other (this never happens) or broadcast
packets (which only really need to go to the master/slave for DHCP anyway)

In practice, it seems that packets sent from a client server to a master or
slave are always going over the same tap interface, even if it is not the
shortest route. (Pings sent to both the master and slave are going out over
tap0, all that I can see on tap1 is stp traffic).

Really what I want to do is forget about stp and just have each bridge interface
send out packets over the correct tap interface based on what mac address is at
the other end, but not bother to forward anything on, as it should never be
necessary.

What am I missing?

Thanks,
Simon

-- 
Simon Detheridge - CTO, Widgit Software
26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680

Leave the master-slave server tunnel separate from the bridges?  That
connection is different from the others anyway, and making it
independent of the bridges will break the loop.

Alternatively, you could use ebtables to drop all packets in the FORWARD chain.

On Mon, Jan 5, 2009 at 2:31 PM, Simon Detheridge <simon at widgit.com>
wrote:
> Hello,
>
> It's entirely possible that I'm going about this all the wrong
way... I'm new here. :-)
>
> I'm trying to set up an environment that works a bit like a private LAN
within Amazon's EC2 cloud. EC2 allows me to provision a bunch of servers on
demand, but they don't have predictable internal IP addresses for
communicating with one another, which makes things a little complicated.
>
> What I'm trying to do is use a combination of vtun tap tunnels and
bridging, to make my servers feel like they're on a LAN together.
>
> The topology I've been experimenting with for now is one where I have a
"master" server, a "slave" server, and many
"client" servers. I'd like the "master" and
"slave" servers to have static IPs, and the "client" servers
to get their addresses via DHCP. I have it so that the master and slave have a
tap tunnel between them, and every client has a tap tunnel to both the master
and the slave (but not to the other clients). Each client should be able to ping
both the master and slave. I don't care if they can ping the other clients
or not.
>
> I've got it set up so that the master and slave each have a bridge
interface that aggregates all of the tunnels from each client (and the one to
each other). Each client has a bridge interface that aggregates the two tunnels
to the master and slave. All are called br0.
>
> I'm not adding any eth interfaces to the bridge, just the tap
interfaces.
>
> I'm running a DHCP server on the master and slave in failover mode and
listening on br0, and on each client I'm attempting to obtain an IP address
for the br0 interface using a DHCP client.
>
> I have stp switched on everywhere. If I disable stp all hell breaks loose
and I get duplicate packets all over the place.
>
> I basically want each server to appear to have one interface, despite the
fact that there are multiple tunnels. I also want the system to work if either
the master or the slave server goes away.
>
> What I was hoping would happen is that packets would take the shortest
route to where they want to go. They should only ever have to go over one
tunnel, except for clients attempting to contact each other (this never happens)
or broadcast packets (which only really need to go to the master/slave for DHCP
anyway)
>
> In practice, it seems that packets sent from a client server to a master or
slave are always going over the same tap interface, even if it is not the
shortest route. (Pings sent to both the master and slave are going out over
tap0, all that I can see on tap1 is stp traffic).
>
> Really what I want to do is forget about stp and just have each bridge
interface send out packets over the correct tap interface based on what mac
address is at the other end, but not bother to forward anything on, as it should
never be necessary.
>
> What am I missing?
>
> Thanks,
> Simon
>
> --
> Simon Detheridge - CTO, Widgit Software
> 26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680
> _______________________________________________
> Bridge mailing list
> Bridge at lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/bridge
>

On Mon, Jan 05, 2009 at 02:31:59PM +0000, Simon Detheridge
wrote:
> It's entirely possible that I'm going about this all the wrong
> way... I'm new here. :-)
> 
> What I'm trying to do is use a combination of vtun tap tunnels and
> bridging, to make my servers feel like they're on a LAN together.
Based on your description, what you really want is broadcast GRE.
Check out http://linux-ip.net/gl/ip-tunnels/node9.html for a basic
description.

If Amazon doesn't route multicast for you (which I'd imagine they
don't, there's info in the above doc on getting this going without
multicast.  Since you've got some way to discover neighbors for vtun,
do the same things to build your tunnel mesh.

Ross


-- 
Ross Vandegrift
ross at kallisti.us

"If the fight gets hot, the songs get hotter.  If the going gets tough,
the songs get tougher."
	--Woody Guthrie

I wrote:
> What I'm trying to do is use a combination of vtun tap tunnels and
> bridging, to make my servers feel like they're on a LAN together.
... snip ...
> Really what I want to do is forget about stp and just have each
> bridge interface send out packets over the correct tap interface based
> on what mac address is at the other end, but not bother to forward
> anything on, as it should never be necessary.
"Ross Vandegrift" <ross at kallisti.us>
wrote:
> Based on your description, what you really want is broadcast GRE.
> Check out http://linux-ip.net/gl/ip-tunnels/node9.html for a basic
> description.
Unfortunately, Amazon will only route tcp/udp. Other protocols don't work,
and as such it's not even possible to set up an ipsec-based VPN... VTun and
OpenVPN seem to be the only solutions.

richardvoigt at gmail.com wrote:
> Leave the master-slave server tunnel separate from the bridges?  That
> connection is different from the others anyway, and making it
> independent of the bridges will break the loop.
That doesn't work. I set up a standard ptp tunnel between the master/slave,
leaving the bridges just for the connections to the clients. When connecting a
bunch of clients, I still wind up with a box that routes all packets over one
tunnel. showstp tells me that one tunnel is blocked and the other is forwarding.
Suprising. I thought stp would find the shortest route, but this takes three
hops as packets can go client->master->different client->slave.
> Alternatively, you could use ebtables to drop all packets in the
> FORWARD chain.
Bingo. Switching off stp on all nodes, and saying "ebtables -P FORWARD
DROP" makes everything work exactly how I want. I hadn't found ebtables
until now. :-)

Thanks for the help... :-)
Simon

-- 
Simon Detheridge - CTO, Widgit Software
26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680