mailinglists at florian-kaiser.net
2011-Jul-06 10:02 UTC
[libvirt-users] Guest-networking: Two subnets = Guests cannot connect to the other subnet
Hello list,
We have KVM setup using libvirt (0.9.0) as configuration device on a Debian
Squeeze Host. Networking is done in routed bridge mode and we have two
different subnets, as well as one main ip. Subnets get routed to the main ip
(this is done by our provider).
Network on hosts are done like this:
<network>
<name>subnet_1</name>
<uuid>7c60990d-ab31-c4a5-cd2b-5ce4c7c61d38</uuid>
<forward dev='eth0' mode='route'/>
<bridge name='virtbridge1' stp='off' delay='0'
/>
<mac address='52:54:00:4D:83:2A'/>
<ip address='xx.xx.240.121' netmask='255.255.255.248'>
</ip>
</network>
<network>
<name>subnet_2</name>
<uuid>7c60990d-ab31-c4a5-cd2b-5ce4c7c61d38</uuid>
<forward dev='eth0' mode='route'/>
<bridge name='virtbridge1' stp='off' delay='0'
/>
<mac address='52:54:00:4D:83:2A'/>
<ip address='xx.xx.240.200' netmask='255.255.255.248'>
</ip>
</network>
Routing on host
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
xx.xx.122.33 0.0.0.0 255.255.255.255 UH 0 0 0
eth0
xx.xx.240.120 0.0.0.0 255.255.255.248 U 0 0 0
virtbridge1
xx.xx.239.200 0.0.0.0 255.255.255.248 U 0 0 0
virtbridge2
0.0.0.0 xx.xx.122.33 0.0.0.0 UG 0 0 0
eth0
Guests get this:
<interface type='network'>
<mac address='de:ad:b6:83:85:5b'/>
<source network='subnet_1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
Guests put this into interfaces:
iface eth0 inet static
address xx.xx.239.202
netmask 255.255.255.248
gateway xx.xx.239.201
Guest routing
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
xx.xx.239.200 0.0.0.0 255.255.255.248 U 0 0 0 eth0
0.0.0.0 xx.xx.239.201 0.0.0.0 UG 0 0 0 eth0
Everything seemed to work just fine, that is: Guests could connect
- to host,
- to internet
- to other virtual maschines in the same subnet - all vice-versa.
Internet can connect to any guest and our host can connect to any guest,
too.
Then, we needed one guest to connect to a guest from the other subnet. And
this does not work, what I get follows:
PING xx.xx.240.122 (xx.xx.240.122) 56(84) bytes of data.>From xx.xx.239.201 icmp_seq=1 Destination Port Unreachable
>From xx.xx.239.201 icmp_seq=2 Destination Port Unreachable
Note, that I can ping the other subnets gateway just fine! But I cannot
connect to any of the other subnet IPs.
I believe this is just a routing problem, but I cannot figure out what route
to add where. Please ask if you need any more information.
Laine Stump
2011-Jul-07 06:55 UTC
[libvirt-users] Guest-networking: Two subnets = Guests cannot connect to the other subnet
On 07/06/2011 06:02 AM, mailinglists at florian-kaiser.net wrote:> Hello list, > > We have KVM setup using libvirt (0.9.0) as configuration device on a Debian > Squeeze Host. Networking is done in routed bridge mode and we have two > different subnets, as well as one main ip. Subnets get routed to the main ip > (this is done by our provider). > > Network on hosts are done like this: > <network> > <name>subnet_1</name> > <uuid>7c60990d-ab31-c4a5-cd2b-5ce4c7c61d38</uuid> > <forward dev='eth0' mode='route'/> > <bridge name='virtbridge1' stp='off' delay='0' /> > <mac address='52:54:00:4D:83:2A'/> > <ip address='xx.xx.240.121' netmask='255.255.255.248'> > </ip> > </network> > > <network> > <name>subnet_2</name> > <uuid>7c60990d-ab31-c4a5-cd2b-5ce4c7c61d38</uuid> > <forward dev='eth0' mode='route'/> > <bridge name='virtbridge1' stp='off' delay='0' /> > <mac address='52:54:00:4D:83:2A'/> > <ip address='xx.xx.240.200' netmask='255.255.255.248'> > </ip> > </network> > > Routing on host > Kernel IP routing table > Destination Gateway Genmask Flags Metric Ref Use > Iface > xx.xx.122.33 0.0.0.0 255.255.255.255 UH 0 0 0 > eth0 > xx.xx.240.120 0.0.0.0 255.255.255.248 U 0 0 0 > virtbridge1 > xx.xx.239.200 0.0.0.0 255.255.255.248 U 0 0 0 > virtbridge2 > 0.0.0.0 xx.xx.122.33 0.0.0.0 UG 0 0 0 > eth0 > > > Guests get this: > <interface type='network'> > <mac address='de:ad:b6:83:85:5b'/> > <source network='subnet_1'/> > <model type='virtio'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> > </interface> > > Guests put this into interfaces: > iface eth0 inet static > address xx.xx.239.202 > netmask 255.255.255.248 > gateway xx.xx.239.201 > > Guest routing > Kernel IP routing table > Destination Gateway Genmask Flags Metric Ref Use > Iface > xx.xx.239.200 0.0.0.0 255.255.255.248 U 0 0 0 eth0 > 0.0.0.0 xx.xx.239.201 0.0.0.0 UG 0 0 0 eth0 > > > Everything seemed to work just fine, that is: Guests could connect > - to host, > - to internet > - to other virtual maschines in the same subnet - all vice-versa. > > Internet can connect to any guest and our host can connect to any guest, > too. > > Then, we needed one guest to connect to a guest from the other subnet. And > this does not work, what I get follows: > > PING xx.xx.240.122 (xx.xx.240.122) 56(84) bytes of data. > > From xx.xx.239.201 icmp_seq=1 Destination Port Unreachable > > From xx.xx.239.201 icmp_seq=2 Destination Port Unreachable > > Note, that I can ping the other subnets gateway just fine! But I cannot > connect to any of the other subnet IPs. > > I believe this is just a routing problem, but I cannot figure out what route > to add where. Please ask if you need any more information.(I think you are the same person who asked about this on IRC later in the day. Still, for archival purposes, here is an answer): By design, libvirt's virtual networks can only be reached from the host itself. The only other connections allowed are outgoing connections frmo the virtual network. This applies to communications between two virtual networks on the same host - the network with the initiating guest will allow an outbound connection, but the the network with the receiving host will not allow this (from its point of view) inbound connection. If you want the guests on two subnets to be able to talk to each other, there are two possible solutions: 1) create host bridges yourself, outside of libvirt. You'll want to assign an IP address to each bridge, but not bind any physical interface to it - let the host's IP stack handle the forwarding of the traffic. You'll also need to add some iptables rules. 2) define multiple IP addresses (each address on a different subnet) on a single libvirt virtual network. The examples in the domain XML document section of the website give examples of this. The only caveat is that only the 1st IP address can have a dhcp server associated with it, so guests that will be on the other subnets connected to this same bridge will need to have their IP addresses statically configured.