thr3ads.net - libvirt users - Re: [libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa [Jun 2017]

If this information is useful, please help other people find it:
Share via:

Travis S. Johnson

2017-Jun-20 06:26 UTC

[libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa

Hello,

I came across an interesting problem in my home lab a few weeks ago as I'm
prepping for my RHCE exam using Michael Jang study guide. I've been at this
for days now, and I still can't wrap my head around how two or more virtual
networks in default NAT configuration are even allowed to communicate with
each other despite what the libvirt documentation said.


Here's the excerpt I'm referring to in the wiki link here:
http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
> By default, guests that are connected via a virtual network with
<forward
> mode='nat'/> can make any outgoing network connection they like.
Incoming
> connections are allowed from the host, and from other guests connected to
> the same libvirt network, but all other incoming connections are blocked by
> iptables rules.

Also here's another assertion from 'The virtual network driver'
section in
http://libvirt.org/firewall.html:
> type=nat
>
> Allow inbound related to an established connection. Allow outbound, but
> only from our expected subnet. Allow traffic between guests. Deny all other
> inbound. Deny all other outbound.


I have three virtual networks with the following configs:
-----------------------------------------------------------------------------
<network connections='1'>
  <name>default</name>
  <uuid>9c6796be-d54e-42bc-bcbe-2e4feee7154a</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:5a:5d:0e'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

<network connections='1'>
  <name>outsider</name>
  <uuid>247e380a-8795-466a-b94a-5be2d05267bb</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:7f:a1:fb'/>
  <domain name='outsider'/>
  <ip address='192.168.100.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.100.2' end='192.168.100.254'/>
    </dhcp>
  </ip>
</network>

<network connections='1'>
  <name>besider</name>
  <uuid>cc714cce-dbba-452d-b2bf-d36084dcb723</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr2' stp='on' delay='0'/>
  <mac address='52:54:00:59:67:7f'/>
  <domain name='besider'/>
  <ip address='192.168.110.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.110.2' end='192.168.110.254'/>
    </dhcp>
  </ip>
</network>
----------------------------------------------------------------------------


Here is the output of the 'FORWARD' iptables chain rules on my host
(still
using firewall-cmd):
------------------------------------------------------------------------
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination
 8967   14M ACCEPT     all  --  *      virbr2  0.0.0.0/0
192.168.110.0/24     ctstate RELATED,ESTABLISHED
 5262  279K ACCEPT     all  --  virbr2 *       192.168.110.0/24
0.0.0.0/0
    0     0 ACCEPT     all  --  virbr2 virbr2  0.0.0.0/0
0.0.0.0/0
   70  5832 REJECT     all  --  *      virbr2  0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr2 *       0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable
 8510   13M ACCEPT     all  --  *      virbr0  0.0.0.0/0
192.168.122.0/24     ctstate RELATED,ESTABLISHED
 5177  275K ACCEPT     all  --  virbr0 *       192.168.122.0/24
0.0.0.0/0
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0
0.0.0.0/0
   61  5100 REJECT     all  --  *      virbr0  0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable
 8612   13M ACCEPT     all  --  *      virbr1  0.0.0.0/0
192.168.100.0/24     ctstate RELATED,ESTABLISHED
 5172  273K ACCEPT     all  --  virbr1 *       192.168.100.0/24
0.0.0.0/0
    0     0 ACCEPT     all  --  virbr1 virbr1  0.0.0.0/0
0.0.0.0/0
    0     0 REJECT     all  --  *      virbr1  0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr1 *       0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0
0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0
0.0.0.0/0
    0     0 FORWARD_direct  all  --  *      *       0.0.0.0/0
0.0.0.0/0
    0     0 FORWARD_IN_ZONES_SOURCE  all  --  *      *       0.0.0.0/0
       0.0.0.0/0
    0     0 FORWARD_IN_ZONES  all  --  *      *       0.0.0.0/0
0.0.0.0/0
    0     0 FORWARD_OUT_ZONES_SOURCE  all  --  *      *       0.0.0.0/0
       0.0.0.0/0
    0     0 FORWARD_OUT_ZONES  all  --  *      *       0.0.0.0/0
0.0.0.0/0
    0     0 DROP       all  --  *      *       0.0.0.0/0
0.0.0.0/0            ctstate INVALID
    0     0 REJECT     all  --  *      *       0.0.0.0/0
0.0.0.0/0            reject-with icmp-host-prohibited
--------------------------------------------------------------------------

I have a VM in each network:
nest1.example.com (virbr0) - 192.168.122.50
nest2.example.org (virbr1) - 192.168.100.100
nest3.example.net (virbr2) - 192.168.110.25

I'm quite aware the above iptables rules were added by libvirt, but I'm
still managing the firewall primarily through the *firewall-cmd* command.
>From what I gathered...----------------------------------------------------------
nest3 can ping nest1 and nest2.

Nest3 ping nest1 and made a hit here:
 8967   14M ACCEPT     all  --  *      virbr2  0.0.0.0/0
192.168.110.0/24     ctstate RELATED,ESTABLISHED
 5262  279K ACCEPT     all  --  virbr2 *       192.168.110.0/24
0.0.0.0/0

Nest3 ping nest2 and made a hit here:
 8967   14M ACCEPT     all  --  *      virbr2  0.0.0.0/0
192.168.110.0/24     ctstate RELATED,ESTABLISHED
 5262  279K ACCEPT     all  --  virbr2 *       192.168.110.0/24
0.0.0.0/0
----------------------------------------------------------


----------------------------------------------------------
Nest1 can ping nest2, but cannot ping nest3.

Nest1 ping nest2 and made a hit here:
8510   13M ACCEPT     all  --  *      virbr0  0.0.0.0/0
192.168.122.0/24     ctstate RELATED,ESTABLISHED
 5177  275K ACCEPT     all  --  virbr0 *       192.168.122.0/24
0.0.0.0/0

Nest1 ping nest3 and made a hit here:
   70  5832 REJECT     all  --  *      virbr2  0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable
----------------------------------------------------------


----------------------------------------------------------
Nest2 cannot ping nest1 and nest3.

Nest2 ping nest1 and made a hit here:
   61  5100 REJECT     all  --  *      virbr0  0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable

Nest2 ping test3 and made a hit here:
   70  5832 REJECT     all  --  *      virbr2  0.0.0.0/0
0.0.0.0/0            reject-with icmp-port-unreachable
----------------------------------------------------------

>From my observation, I see that the order of the virtual networks in theiptables FORWARD chain makes a difference. Each chunk associated with the
virtual network in the chain, consisting of five lines, is exactly as
described in the aforementioned link (http://libvirt.org/firewall.html).
The virtual network in the top chunk of the chain can communicate with
virtually all networks as opposed to the network in the last chunk that is
consistent with the intention of the original design.

I'm using CentOS 7.3 with libvirt 2.0. I even tried reproducing this with
CentOS 6.9 as I thought it was possible the firewalld may have influenced
the change, but I was still getting similar result.

Right now, I'm not certain if this is already a reported known bug, but
I'm
highly convinced this configuration is unofficially unsupported for quite a
while. Can someone confirm this? This is my very first mailing list
submission ever, and I apologize in advance if I couldn't figure out how to
conveniently search up a similar discussion as this one in the archive. If
this is in fact near impossible to implement in accordance to the intended
design, then I'd like to have this confirmation publicly included in the
docs.

Thanks,

Travis Johnson

Martin Kletzander

2017-Jun-20 08:05 UTC

head link

Re: [libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa

On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson
wrote:>Hello,
>
>I came across an interesting problem in my home lab a few weeks ago as
I'm
>prepping for my RHCE exam using Michael Jang study guide. I've been at
this
>for days now, and I still can't wrap my head around how two or more
virtual
>networks in default NAT configuration are even allowed to communicate with
>each other despite what the libvirt documentation said.
>
>
>Here's the excerpt I'm referring to in the wiki link here:
>http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
>
>> By default, guests that are connected via a virtual network with
<forward
>> mode='nat'/> can make any outgoing network connection they
like. Incoming
>> connections are allowed from the host, and from other guests connected
to
>> the same libvirt network, but all other incoming connections are
blocked by
>> iptables rules.
>
>
>Also here's another assertion from 'The virtual network driver'
section in
>http://libvirt.org/firewall.html:
>
>> type=nat
>>
>> Allow inbound related to an established connection. Allow outbound, but
>> only from our expected subnet. Allow traffic between guests. Deny all
other
>> inbound. Deny all other outbound.
>
>
>
>I have three virtual networks with the following configs:
>-----------------------------------------------------------------------------
><network connections='1'>
>  <name>default</name>
>  <uuid>9c6796be-d54e-42bc-bcbe-2e4feee7154a</uuid>
>  <forward mode='nat'>
>    <nat>
>      <port start='1024' end='65535'/>
>    </nat>
>  </forward>
>  <bridge name='virbr0' stp='on' delay='0'/>
>  <mac address='52:54:00:5a:5d:0e'/>
>  <ip address='192.168.122.1' netmask='255.255.255.0'>
>    <dhcp>
>      <range start='192.168.122.2'
end='192.168.122.254'/>
>    </dhcp>
>  </ip>
></network>
>
><network connections='1'>
>  <name>outsider</name>
>  <uuid>247e380a-8795-466a-b94a-5be2d05267bb</uuid>
>  <forward mode='nat'>
>    <nat>
>      <port start='1024' end='65535'/>
>    </nat>
>  </forward>
>  <bridge name='virbr1' stp='on' delay='0'/>
>  <mac address='52:54:00:7f:a1:fb'/>
>  <domain name='outsider'/>
>  <ip address='192.168.100.1' netmask='255.255.255.0'>
>    <dhcp>
>      <range start='192.168.100.2'
end='192.168.100.254'/>
>    </dhcp>
>  </ip>
></network>
>
><network connections='1'>
>  <name>besider</name>
>  <uuid>cc714cce-dbba-452d-b2bf-d36084dcb723</uuid>
>  <forward mode='nat'>
>    <nat>
>      <port start='1024' end='65535'/>
>    </nat>
>  </forward>
>  <bridge name='virbr2' stp='on' delay='0'/>
>  <mac address='52:54:00:59:67:7f'/>
>  <domain name='besider'/>
>  <ip address='192.168.110.1' netmask='255.255.255.0'>
>    <dhcp>
>      <range start='192.168.110.2'
end='192.168.110.254'/>
>    </dhcp>
>  </ip>
></network>
>----------------------------------------------------------------------------
>
>
>Here is the output of the 'FORWARD' iptables chain rules on my host
(still
>using firewall-cmd):
>------------------------------------------------------------------------
>Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> pkts bytes target     prot opt in     out     source
>destination
> 8967   14M ACCEPT     all  --  *      virbr2  0.0.0.0/0
>192.168.110.0/24     ctstate RELATED,ESTABLISHED
> 5262  279K ACCEPT     all  --  virbr2 *       192.168.110.0/24
>0.0.0.0/0
>    0     0 ACCEPT     all  --  virbr2 virbr2  0.0.0.0/0
>0.0.0.0/0
>   70  5832 REJECT     all  --  *      virbr2  0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
>    0     0 REJECT     all  --  virbr2 *       0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
> 8510   13M ACCEPT     all  --  *      virbr0  0.0.0.0/0
>192.168.122.0/24     ctstate RELATED,ESTABLISHED
> 5177  275K ACCEPT     all  --  virbr0 *       192.168.122.0/24
>0.0.0.0/0
>    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0
>0.0.0.0/0
>   61  5100 REJECT     all  --  *      virbr0  0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
>    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
> 8612   13M ACCEPT     all  --  *      virbr1  0.0.0.0/0
>192.168.100.0/24     ctstate RELATED,ESTABLISHED
> 5172  273K ACCEPT     all  --  virbr1 *       192.168.100.0/24
>0.0.0.0/0
>    0     0 ACCEPT     all  --  virbr1 virbr1  0.0.0.0/0
>0.0.0.0/0
>    0     0 REJECT     all  --  *      virbr1  0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
>    0     0 REJECT     all  --  virbr1 *       0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
>    0     0 ACCEPT     all  --  *      *       0.0.0.0/0
>0.0.0.0/0            ctstate RELATED,ESTABLISHED
>    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0
>0.0.0.0/0
>    0     0 FORWARD_direct  all  --  *      *       0.0.0.0/0
>0.0.0.0/0
>    0     0 FORWARD_IN_ZONES_SOURCE  all  --  *      *       0.0.0.0/0
>       0.0.0.0/0
>    0     0 FORWARD_IN_ZONES  all  --  *      *       0.0.0.0/0
>0.0.0.0/0
>    0     0 FORWARD_OUT_ZONES_SOURCE  all  --  *      *       0.0.0.0/0
>       0.0.0.0/0
>    0     0 FORWARD_OUT_ZONES  all  --  *      *       0.0.0.0/0
>0.0.0.0/0
>    0     0 DROP       all  --  *      *       0.0.0.0/0
>0.0.0.0/0            ctstate INVALID
>    0     0 REJECT     all  --  *      *       0.0.0.0/0
>0.0.0.0/0            reject-with icmp-host-prohibited
>--------------------------------------------------------------------------
>
>I have a VM in each network:
>nest1.example.com (virbr0) - 192.168.122.50
>nest2.example.org (virbr1) - 192.168.100.100
>nest3.example.net (virbr2) - 192.168.110.25
>
>I'm quite aware the above iptables rules were added by libvirt, but
I'm
>still managing the firewall primarily through the *firewall-cmd* command.
>
>>From what I gathered...
>----------------------------------------------------------
>nest3 can ping nest1 and nest2.
>
>Nest3 ping nest1 and made a hit here:
> 8967   14M ACCEPT     all  --  *      virbr2  0.0.0.0/0
>192.168.110.0/24     ctstate RELATED,ESTABLISHED
> 5262  279K ACCEPT     all  --  virbr2 *       192.168.110.0/24
>0.0.0.0/0
>
>Nest3 ping nest2 and made a hit here:
> 8967   14M ACCEPT     all  --  *      virbr2  0.0.0.0/0
>192.168.110.0/24     ctstate RELATED,ESTABLISHED
> 5262  279K ACCEPT     all  --  virbr2 *       192.168.110.0/24
>0.0.0.0/0
>----------------------------------------------------------
>
>
>----------------------------------------------------------
>Nest1 can ping nest2, but cannot ping nest3.
>
>Nest1 ping nest2 and made a hit here:
>8510   13M ACCEPT     all  --  *      virbr0  0.0.0.0/0
>192.168.122.0/24     ctstate RELATED,ESTABLISHED
> 5177  275K ACCEPT     all  --  virbr0 *       192.168.122.0/24
>0.0.0.0/0
>
>Nest1 ping nest3 and made a hit here:
>   70  5832 REJECT     all  --  *      virbr2  0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
>----------------------------------------------------------
>
>
>----------------------------------------------------------
>Nest2 cannot ping nest1 and nest3.
>
>Nest2 ping nest1 and made a hit here:
>   61  5100 REJECT     all  --  *      virbr0  0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
>
>Nest2 ping test3 and made a hit here:
>   70  5832 REJECT     all  --  *      virbr2  0.0.0.0/0
>0.0.0.0/0            reject-with icmp-port-unreachable
>----------------------------------------------------------
>
>
>>From my observation, I see that the order of the virtual networks in the
>iptables FORWARD chain makes a difference. Each chunk associated with the
>virtual network in the chain, consisting of five lines, is exactly as
>described in the aforementioned link (http://libvirt.org/firewall.html).
>The virtual network in the top chunk of the chain can communicate with
>virtually all networks as opposed to the network in the last chunk that is
>consistent with the intention of the original design.
>
>I'm using CentOS 7.3 with libvirt 2.0. I even tried reproducing this
with
>CentOS 6.9 as I thought it was possible the firewalld may have influenced
>the change, but I was still getting similar result.
>
>Right now, I'm not certain if this is already a reported known bug, but
I'm
>highly convinced this configuration is unofficially unsupported for quite a
>while. Can someone confirm this? This is my very first mailing list
>submission ever, and I apologize in advance if I couldn't figure out how
to
>conveniently search up a similar discussion as this one in the archive. If
>this is in fact near impossible to implement in accordance to the intended
>design, then I'd like to have this confirmation publicly included in the
>docs.
>
Thanks for reporting this.  It's clearly a bug in libvirt.  The rules
are in this order:

  all rules for virbr0
  all rules for virbr1
  all rules for virbr2

But what we should do instead is:

  input rules for all networks
  local rules for all networks
  output rules for all networks
  reject rules for all networks

The problem is that we do not know how other rules look like.  So what
we might need to do is create chains where rules for the first network
are, then only append network rules into those chains.

Would you mind filing a bug for this issue, so we can properly track it
and don't forget about it?  I'll have a look at it in the meantime, but
don't promise anything since I'm not that familiar with that part of the
codebase.
>Thanks,
>
>Travis Johnson
>_______________________________________________
>libvirt-users mailing list
>libvirt-users@redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users

Martin Kletzander

2017-Jun-20 09:27 UTC

head link

Re: [libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa

On Tue, Jun 20, 2017 at 10:05:19AM +0200, Martin Kletzander
wrote:>On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
>>Hello,
>>
>>I came across an interesting problem in my home lab a few weeks ago as
I'm
>>prepping for my RHCE exam using Michael Jang study guide. I've been
at this
>>for days now, and I still can't wrap my head around how two or more
virtual
>>networks in default NAT configuration are even allowed to communicate
with
>>each other despite what the libvirt documentation said.
>>
>>
>>Here's the excerpt I'm referring to in the wiki link here:
>>http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
>>
>>> By default, guests that are connected via a virtual network with
<forward
>>> mode='nat'/> can make any outgoing network connection
they like. Incoming
>>> connections are allowed from the host, and from other guests
connected to
>>> the same libvirt network, but all other incoming connections are
blocked by
>>> iptables rules.
>>
I did not read this properly...
>>
>>Also here's another assertion from 'The virtual network
driver' section in
>>http://libvirt.org/firewall.html:
>>
>>> type=nat
>>>
>>> Allow inbound related to an established connection. Allow outbound,
but
>>> only from our expected subnet. Allow traffic between guests. Deny
all other
I would expect the 'traffic between guests' to mean only guests on the
same network.  Also that lines up with what's written above.
>>> inbound. Deny all other outbound.
>>
[...]
>
>Thanks for reporting this.  It's clearly a bug in libvirt.  The rules
>are in this order:
>
>  all rules for virbr0
>  all rules for virbr1
>  all rules for virbr2
>
>But what we should do instead is:
>
>  input rules for all networks
>  local rules for all networks
>  output rules for all networks
>  reject rules for all networks
>
>The problem is that we do not know how other rules look like.  So what
>we might need to do is create chains where rules for the first network
>are, then only append network rules into those chains.
>
>Would you mind filing a bug for this issue, so we can properly track it
>and don't forget about it?  I'll have a look at it in the meantime,
but
>don't promise anything since I'm not that familiar with that part of
the
>codebase.
>
Actually, now that I'm thinking about it, the problem is not that we
disallow the communication from virbr1 to virbr0, but the problem is
that we allow the connection from virbr0 to everywhere.

Maybe the solution would be: for each network, insert the rules on top
of the forward chain and for each started network, explicitly reject
that one.  So that after one network starts it would look like this (I'm
writing this from memory just to illustrate the idea, not actually
looking up how stuff looks):

  ACCEPT any virbr0 RELATED,ESTABLISHED
  ACCEPT virbr0 virbr0
  ACCEPT virbr0 any src:192.168.122.0/24
  REJECT any virbr0
  REJECT virbr0 any

And after second network is started, we'd have:

  # This one is new:
  REJECT virbr1 virbr0
  # These ones would be normally at the end, IIRC:
  ACCEPT any virbr1 RELATED,ESTABLISHED
  ACCEPT virbr1 virbr1
  ACCEPT virbr1 any src:192.168.122.0/24
  REJECT any virbr1
  REJECT virbr1 any
  # These are left as they were:
  ACCEPT any virbr0 RELATED,ESTABLISHED
  ACCEPT virbr0 virbr0
  ACCEPT virbr0 any src:192.168.122.0/24
  REJECT any virbr0
  REJECT virbr0 any

Cc'ing Laine so that he can weigh in as he has way more knowledge of
this part of the code =)
>>Thanks,
>>
>>Travis Johnson
>
>>_______________________________________________
>>libvirt-users mailing list
>>libvirt-users@redhat.com
>>https://www.redhat.com/mailman/listinfo/libvirt-users

>_______________________________________________
>libvirt-users mailing list
>libvirt-users@redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users

Maybe Matching Threads

Search for more reasonably related threads

libvirt users - Jun 2017 - Re: guest A from virbr0 can talk to guest B in virbr1 but not vice versa

[libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa

Re: [libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa

Re: [libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa

Maybe Matching Threads