CentOS - Jan 2016 - CentOS-6.7, kvm bridges, virtual interfaces, and routes

I have been looking at this problem on and off for a considerable
period.  Given my lack of knowledge I have been unable to resolve this
quickly and in consequence it has been constantly shoved to the
background as other issues arise.

Here is the situation:

I have two dual-homed kvm hosts both running CentOS-6.7 and
identically configured.  These are connected to the same LAN segment
via br0/eth0 and to each other via a cross-over cable on br1/eth1. 
The IPv4 assigned to br0 on both is a publicly routeable address.  The
IPv4 assigned to br1 on both is a private address in the
192.168.0.0/16 address space.

The guests on each host have their virtual eth0 bridged with their
host's br0 and eth1 bridged with their host's br1.  The addresses used
by the guests on eth0 are publicly routeable, the addresses used on
eth1 are private.

I would like to configure br1/eth1 on both kvm hosts such that each is
a gateway to the other.

I then also would like to configure each kvm guest of each host such
that their traffic to the private network segment on the opposite host
routes through the x-over cable via br0 whilst everything else goes
out through br1 to the LAN and gateway.

Has anyone here done anything like this? If so, can you point me to
any online resource that could more or less walk me through the
process without me having to complete the coursework for a network
engineer.  I just want to keep data transfer traffic between pairs of
kvm guests off of the public lan without having to install more
hardware. The existing cabinets are not going to support it either
space wise or power wise.


An ASCII art diagram might help, or might not.

<pre>

kvmh1g1           eth0/192.168.51.1
                  eth1/aaa.bbb.ccc.151 <-------------> |
                                                       |
kvmh1             br1/aaa.bbb.ccc.51                   |
            |---> br0/192.168.51.1                     |
            X                                          |
kvmh2       |---> br0/192.168.52.1                     |
                  br1/aaa.bbb.ccc.52                   |
                                                       |
kvmh2g1           eth0/192.168.52.1                    |
                  eth1/aaa.bbb.ccc.251 <-------------> |
                                                       |
gateway           eth1/aaa.bbb.ccc.1 <---------------> |

</pre>

I have tried multiple approaches without success and in so many
variations that I no long can clearly recall the details.

At the moment my thought is that if br0 was set to 192.168.51.1/24 on
kvmh1 and to 192.168.52.1/24 on kvmh2 and a routing table entry was
made on kvmh1 to send traffic addressed to 192.168.52.0/24 through
192.168.51.1/24. And on kvmh2 br0 was set to 192.168.52.1/24 and a
routing table entry was made on kvmh2 to route traffic to
192.168.51.0/24 through 192.168.52.1/24.

I thought that if the kvm virtual guests on kvmh1 were then configured
to use addresses from 192.168.51.0/24 while those on kvmh2 used
192.168.52.0/24. And everything was configured to use their respective
host's br1 address as their gateway then this should work.  But I am
evidently either fundamentally wrong or I have misconfigured things
somehow.

Should this set up work as I imaging?  What would be the correct
static routing table entries to make it work?

-- 
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3

In article <55ae6ce7fe2cbdba1514f1072281c006.squirrel at
webmail.harte-lyne.ca>,
James B. Byrne <byrnejb at harte-lyne.ca> wrote:
> I have been looking at this problem on and off for a considerable
> period.  Given my lack of knowledge I have been unable to resolve this
> quickly and in consequence it has been constantly shoved to the
> background as other issues arise.
> 
> Here is the situation:
> 
> An ASCII art diagram might help, or might not.
> 
> <pre>
> 
> kvmh1g1           eth0/192.168.51.1
>                   eth1/aaa.bbb.ccc.151 <-------------> |
>                                                        |
> kvmh1             br1/aaa.bbb.ccc.51                   |
>             |---> br0/192.168.51.1                     |
>             X                                          |
> kvmh2       |---> br0/192.168.52.1                     |
>                   br1/aaa.bbb.ccc.52                   |
>                                                        |
> kvmh2g1           eth0/192.168.52.1                    |
>                   eth1/aaa.bbb.ccc.251 <-------------> |
>                                                        |
> gateway           eth1/aaa.bbb.ccc.1 <---------------> |
> 
> </pre>
> 
Why are you using two separate subnets, 192.168.51.0/24 and 192.168.52.0/24?
That is the core of your problem. You can't use a crossover cable between
different subnets; you would need a router. There may be an esoteric way,
but it's not a normal configuration.

But they don't need to be different subnets at all. Logically speaking, they
are the same subnet.

So give kvmh1:br0 192.168.51.1 and kvmh2:br0 192.168.51.2. Then they can
talk to each other easily, without doing anything special.

On the guests, give them 192.168.51.11 and 192.168.12 (for example).
I don't think they should use the same IP addresses as their hosts.

Cheers
Tony
-- 
Tony Mountifield
Work: tony at softins.co.uk - http://www.softins.co.uk
Play: tony at mountifield.org - http://tony.mountifield.org

Search for policy routing. Surprisingly, I cannot find anything about it in the
RHEL6 docs.

You don't say how exactly you tried. It should be sufficient to edit
rt_tables (maybe that's a step you missed? The actual number used
doesn't matter) and add the route-eth1/rule-eth1 files on every host
involved. I.e. no separate route/rule file shold be needed for the default
interfaces.
> I then also would like to configure each kvm guest of each host such
> that their traffic to the private network segment on the opposite host
> routes through the x-over cable via br0 whilst everything else goes
> out through br1 to the LAN and gateway.
ITYM ... x-over cable via br1 ... out through br0 ...  ?

On 01/09/2016 03:30 PM, isdtor wrote:
> Search for policy routing.
Policy routing isn't relevant.

In order to communicate across a LAN, two hosts must be in the same 
broadcast domain.  Hosts in 192.168.51.0/24 cannot communicate with 
hosts in 192.168.52.0/24.