libvirt users - Apr 2013 - Issue with macvtap bridge and forwarding

I have three hosts running Ubuntu 12.04 (libvirt 0.9.8).  The
configuration is one host running on bare metal while the other two
are KVM guests.

The first guest is my network router.  It has a direct connection to a
physical nic going out to the internet, and a bridged connection to a
nic for the lan.  The host has ip forwarding enabled and forwards my
lan traffic back and forth to the internet.

The second host is a "lan" machine, which is also on the bridged lan
nic.  There are several other physical hosts also on the switched lan
network this nic connects to.

Both hosts are configured on the lan tap as follows (different mac):

<interface type='direct'>
  <mac address='13:54:21:1f:f3:42'/>
  <source dev='eth1' mode='bridge'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>

Everything on the firewall host seems to work OK.  Hosts elsewhere on
the LAN can connect to it, and they have no problem routing through it
and out to the internet and back.  That is, traffic passes through
both nics and this host successfully.

The other guest can also be reached successfully from machines on the
LAN.  I can ping it and I can ssh to it.  I also note that it seems to
be able to be able to talk to the "router" host over the vtap bridge:
it can perform dns lookups against the router host, and they can seem
to reach each other's ports.

Where I'm getting stuck is that for whatever reason, the second guest
apparently cannot reach the internet via my router host.  It's the
only host anywhere on the lan that apparently can't pass forwarding
traffic via the router guest, and the only common feature appears to
be the macvtap bridge.  This issue *does not* happen when using a
common linux bridge in the otherwise same configuration and the same
hosts.  In the linux bridge scenario, the lan guest forwards traffic
via the router guest fine.  But when switching to the macvtap
configuration, suddenly the lan guest no longer forwards traffic via
the router guest and out to the internet.

What appears to be happening is that the traffic is crossing the
bridge, but the router host does not classify it or masq it properly,
and it never makes the internet trip as expected.  Further extending
my suspicions, initiating an outbound http connection does not raise
an entry in the conntrack table on the router for the problem host.

So my question is, what is it about macvtap bridge that would cause
traffic coming over via the tap bridge to be routed differently than
traffic coming *up* the bridge from the physical interface, or via a
traditional linux bridge?

On 04/04/2013 10:48 PM, Matt LaPlante wrote:
> I have three hosts running Ubuntu 12.04 (libvirt 0.9.8).  The
> configuration is one host running on bare metal while the other two
> are KVM guests.
>
> The first guest is my network router.  It has a direct connection to a
> physical nic going out to the internet, and a bridged connection to a
> nic for the lan.  The host has ip forwarding enabled and forwards my
> lan traffic back and forth to the internet.
>
> The second host is a "lan" machine, which is also on the bridged
lan
> nic.  There are several other physical hosts also on the switched lan
> network this nic connects to.
>
> Both hosts are configured on the lan tap as follows (different mac):
>
> <interface type='direct'>
>   <mac address='13:54:21:1f:f3:42'/>
>   <source dev='eth1' mode='bridge'/>
>   <model type='virtio'/>
>   <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
> </interface>
>
> Everything on the firewall host seems to work OK.  Hosts elsewhere on
> the LAN can connect to it, and they have no problem routing through it
> and out to the internet and back.  That is, traffic passes through
> both nics and this host successfully.
>
> The other guest can also be reached successfully from machines on the
> LAN.  I can ping it and I can ssh to it.  I also note that it seems to
> be able to be able to talk to the "router" host over the vtap
bridge:
> it can perform dns lookups against the router host, and they can seem
> to reach each other's ports.
>
> Where I'm getting stuck is that for whatever reason, the second guest
> apparently cannot reach the internet via my router host.  It's the
> only host anywhere on the lan that apparently can't pass forwarding
> traffic via the router guest, and the only common feature appears to
> be the macvtap bridge.  This issue *does not* happen when using a
> common linux bridge in the otherwise same configuration and the same
> hosts.  In the linux bridge scenario, the lan guest forwards traffic
> via the router guest fine.  But when switching to the macvtap
> configuration, suddenly the lan guest no longer forwards traffic via
> the router guest and out to the internet.
>
> What appears to be happening is that the traffic is crossing the
> bridge, but the router host does not classify it or masq it properly,
> and it never makes the internet trip as expected.  Further extending
> my suspicions, initiating an outbound http connection does not raise
> an entry in the conntrack table on the router for the problem host.
>
> So my question is, what is it about macvtap bridge that would cause
> traffic coming over via the tap bridge to be routed differently than
> traffic coming *up* the bridge from the physical interface, or via a
> traditional linux bridge?
The macvtap data path is architected differently than, for example, the
data path for a tap + linux host bridge. One big difference is that,
even in macvtap "bridge" mode, you can't use that interface to
communicate between a guest and its host - packets from the guest are
injected into the packet stream at a point too low to have the
possibility of reflecting them back up the stack to the host's IP.

Because it seems like you've mixed use of the words "host" and
"guest"
in your description, I was unable to follow exactly what the setup is
and which paths don't function, but when a host bridge setup works and a
macvtap setup doesn't, the point that I said above is almost always the
reason.

There's a description of the problem and one possible workaround here:

http://wiki.libvirt.org/page/Guest_can_reach_outside_network%2C_but_can%27t_reach_host_%28macvtap%29

If that's not the problem, when you reply please give more concrete
details about your topology, and be careful to use "host" only when
talking about the virtualization host, and always use "guest" when
talking about a virtual machine.