libvirt users - Jun 2017 - Re: guest A from virbr0 can talk to guest B in virbr1 but not vice versa

On Tue, Jun 20, 2017 at 10:05:19AM +0200, Martin Kletzander
wrote:
>On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
>>Hello,
>>
>>I came across an interesting problem in my home lab a few weeks ago as
I'm
>>prepping for my RHCE exam using Michael Jang study guide. I've been
at this
>>for days now, and I still can't wrap my head around how two or more
virtual
>>networks in default NAT configuration are even allowed to communicate
with
>>each other despite what the libvirt documentation said.
>>
>>
>>Here's the excerpt I'm referring to in the wiki link here:
>>http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
>>
>>> By default, guests that are connected via a virtual network with
<forward
>>> mode='nat'/> can make any outgoing network connection
they like. Incoming
>>> connections are allowed from the host, and from other guests
connected to
>>> the same libvirt network, but all other incoming connections are
blocked by
>>> iptables rules.
>>
I did not read this properly...
>>
>>Also here's another assertion from 'The virtual network
driver' section in
>>http://libvirt.org/firewall.html:
>>
>>> type=nat
>>>
>>> Allow inbound related to an established connection. Allow outbound,
but
>>> only from our expected subnet. Allow traffic between guests. Deny
all other
I would expect the 'traffic between guests' to mean only guests on the
same network.  Also that lines up with what's written above.
>>> inbound. Deny all other outbound.
>>
[...]
>
>Thanks for reporting this.  It's clearly a bug in libvirt.  The rules
>are in this order:
>
>  all rules for virbr0
>  all rules for virbr1
>  all rules for virbr2
>
>But what we should do instead is:
>
>  input rules for all networks
>  local rules for all networks
>  output rules for all networks
>  reject rules for all networks
>
>The problem is that we do not know how other rules look like.  So what
>we might need to do is create chains where rules for the first network
>are, then only append network rules into those chains.
>
>Would you mind filing a bug for this issue, so we can properly track it
>and don't forget about it?  I'll have a look at it in the meantime,
but
>don't promise anything since I'm not that familiar with that part of
the
>codebase.
>
Actually, now that I'm thinking about it, the problem is not that we
disallow the communication from virbr1 to virbr0, but the problem is
that we allow the connection from virbr0 to everywhere.

Maybe the solution would be: for each network, insert the rules on top
of the forward chain and for each started network, explicitly reject
that one.  So that after one network starts it would look like this (I'm
writing this from memory just to illustrate the idea, not actually
looking up how stuff looks):

  ACCEPT any virbr0 RELATED,ESTABLISHED
  ACCEPT virbr0 virbr0
  ACCEPT virbr0 any src:192.168.122.0/24
  REJECT any virbr0
  REJECT virbr0 any

And after second network is started, we'd have:

  # This one is new:
  REJECT virbr1 virbr0
  # These ones would be normally at the end, IIRC:
  ACCEPT any virbr1 RELATED,ESTABLISHED
  ACCEPT virbr1 virbr1
  ACCEPT virbr1 any src:192.168.122.0/24
  REJECT any virbr1
  REJECT virbr1 any
  # These are left as they were:
  ACCEPT any virbr0 RELATED,ESTABLISHED
  ACCEPT virbr0 virbr0
  ACCEPT virbr0 any src:192.168.122.0/24
  REJECT any virbr0
  REJECT virbr0 any

Cc'ing Laine so that he can weigh in as he has way more knowledge of
this part of the code =)
>>Thanks,
>>
>>Travis Johnson
>
>>_______________________________________________
>>libvirt-users mailing list
>>libvirt-users@redhat.com
>>https://www.redhat.com/mailman/listinfo/libvirt-users

>_______________________________________________
>libvirt-users mailing list
>libvirt-users@redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users

On 06/20/2017 05:27 AM, Martin Kletzander wrote:
> On Tue, Jun 20, 2017 at 10:05:19AM +0200, Martin Kletzander wrote:
>> On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
>>> Hello,
>>>
>>> I came across an interesting problem in my home lab a few weeks ago
>>> as I'm
>>> prepping for my RHCE exam using Michael Jang study guide. I've
been
>>> at this
>>> for days now, and I still can't wrap my head around how two or
more
>>> virtual
>>> networks in default NAT configuration are even allowed to
communicate
>>> with
>>> each other despite what the libvirt documentation said.
>>>
>>>
>>> Here's the excerpt I'm referring to in the wiki link here:
>>>
http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
>>>
>>>> By default, guests that are connected via a virtual network
with
>>>> <forward
>>>> mode='nat'/> can make any outgoing network
connection they like.
>>>> Incoming
>>>> connections are allowed from the host, and from other guests
>>>> connected to
>>>> the same libvirt network, but all other incoming connections
are
>>>> blocked by
>>>> iptables rules.
>>>
> 
> I did not read this properly...
> 
>>>
>>> Also here's another assertion from 'The virtual network
driver'
>>> section in
>>> http://libvirt.org/firewall.html:
>>>
>>>> type=nat
>>>>
>>>> Allow inbound related to an established connection. Allow
outbound, but
>>>> only from our expected subnet. Allow traffic between guests.
Deny
>>>> all other
> 
> I would expect the 'traffic between guests' to mean only guests on
the
> same network.  Also that lines up with what's written above.
That is correct.
> 
>>>> inbound. Deny all other outbound.
>>>
> 
> [...]
> 
>>
>> Thanks for reporting this.  It's clearly a bug in libvirt.  The
rules
>> are in this order:
>>
>>  all rules for virbr0
>>  all rules for virbr1
>>  all rules for virbr2
>>
>> But what we should do instead is:
>>
>>  input rules for all networks
>>  local rules for all networks
>>  output rules for all networks
>>  reject rules for all networks
That doesn't achieve the desired results (as you figured out later,
according to your followup comments.

>>
>> The problem is that we do not know how other rules look like.  So what
>> we might need to do is create chains where rules for the first network
>> are, then only append network rules into those chains.
>>
>> Would you mind filing a bug for this issue, so we can properly track it
>> and don't forget about it?  I'll have a look at it in the
meantime, but
>> don't promise anything since I'm not that familiar with that
part of the
>> codebase.
>>
> 
> Actually, now that I'm thinking about it, the problem is not that we
> disallow the communication from virbr1 to virbr0, but the problem is
> that we allow the connection from virbr0 to everywhere.
Yep!

That's actually a known bug that has been there "forever". It has
always
been brought up in the context of "I want communication between my NATed
networks, and it only works in one direction!", rather than "I want to
block communication between my NATed networks, but it gets past in one
direction!". When people learn that it's (kind of) blocked, they work
around it by adding a new network that is shared by all the guests.
Because all of the reports have been in this direction, fixing it
properly has been very low on (my) priority list.

> 
> Maybe the solution would be: for each network, insert the rules on top
> of the forward chain and for each started network, explicitly reject
> that one.  So that after one network starts it would look like this
(I'm
> writing this from memory just to illustrate the idea, not actually
> looking up how stuff looks):
> 
>  ACCEPT any virbr0 RELATED,ESTABLISHED
>  ACCEPT virbr0 virbr0
>  ACCEPT virbr0 any src:192.168.122.0/24
>  REJECT any virbr0
>  REJECT virbr0 any
> 
> And after second network is started, we'd have:
> 
>  # This one is new:
>  REJECT virbr1 virbr0
Yeah, it's (mostly) this rule that makes the solution complicated.
Essentially you need one of these rules for each pair of virtual
networks, so if you have 10 networks, you'll need 45 rules, for 11
you'll need 55 (in general, for "n" networks, you'll need (n *
(n-1) /
2) extra rules).

Aside from the extra packet processing overhead caused by the high
number of rules needed (which may be completely insignificant), and the
necessity to add/remove rules at specific labels rather than just
inserting them all at the beginning, there is the issue that currently
each iptables rule added for a libvirt network is dependent only on that
network, so it's simple to determine which rules to remove when a
network is destroyed. If we start adding rules that are dependent on two
networks, then proper cleanup will be more complicated.

None of that should prevent a properly motivated person from writing the
code to do it, of course!
>  # These ones would be normally at the end, IIRC:
>  ACCEPT any virbr1 RELATED,ESTABLISHED
>  ACCEPT virbr1 virbr1
>  ACCEPT virbr1 any src:192.168.122.0/24
>  REJECT any virbr1
>  REJECT virbr1 any
>  # These are left as they were:
>  ACCEPT any virbr0 RELATED,ESTABLISHED
>  ACCEPT virbr0 virbr0
>  ACCEPT virbr0 any src:192.168.122.0/24
>  REJECT any virbr0
>  REJECT virbr0 any
> 
> Cc'ing Laine so that he can weigh in as he has way more knowledge of
> this part of the code =)

Yep!
> That's actually a known bug that has been there "forever". It
has always
> been brought up in the context of "I want communication between my
NATed
> networks, and it only works in one direction!", rather than "I
want to
> block communication between my NATed networks, but it gets past in one
> direction!". When people learn that it's (kind of) blocked, they
work
> around it by adding a new network that is shared by all the guests.
> Because all of the reports have been in this direction, fixing it
> properly has been very low on (my) priority list.


Thanks for confirming the bug as I suspected, Laine! I knew it was too good
to be true that I'd be the first to discover this. Thanks for your huge
help, Martin.

I tried searching for this age-old you may be speaking of and I was only
able to find this:
https://bugzilla.redhat.com/show_bug.cgi?id=533193

Also found another somewhat-related bug affecting iptables (port-forwarding
to VM):
https://bugzilla.redhat.com/show_bug.cgi?id=1079088


If it's not too much trouble, can you provide me the ID number so I can
track it?


Aside from the extra packet processing overhead caused by the
high
> number of rules needed (which may be completely insignificant), and the
> necessity to add/remove rules at specific labels rather than just
> inserting them all at the beginning, there is the issue that currently
> each iptables rule added for a libvirt network is dependent only on that
> network, so it's simple to determine which rules to remove when a
> network is destroyed. If we start adding rules that are dependent on two
> networks, then proper cleanup will be more complicated.
> None of that should prevent a properly motivated person from writing the
> code to do it, of course!


At least now I know this is quite feasible to fix. Thank you both for the
well-elaborated responses.


Cheers,

-Travis



On Tue, Jun 20, 2017 at 12:46 PM, Laine Stump <laine@laine.org> wrote:
> On 06/20/2017 05:27 AM, Martin Kletzander wrote:
> > On Tue, Jun 20, 2017 at 10:05:19AM +0200, Martin Kletzander wrote:
> >> On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
> >>> Hello,
> >>>
> >>> I came across an interesting problem in my home lab a few
weeks ago
> >>> as I'm
> >>> prepping for my RHCE exam using Michael Jang study guide.
I've been
> >>> at this
> >>> for days now, and I still can't wrap my head around how
two or more
> >>> virtual
> >>> networks in default NAT configuration are even allowed to
communicate
> >>> with
> >>> each other despite what the libvirt documentation said.
> >>>
> >>>
> >>> Here's the excerpt I'm referring to in the wiki link
here:
> >>> http://wiki.libvirt.org/page/Networking#Forwarding_
> Incoming_Connections:
> >>>
> >>>> By default, guests that are connected via a virtual
network with
> >>>> <forward
> >>>> mode='nat'/> can make any outgoing network
connection they like.
> >>>> Incoming
> >>>> connections are allowed from the host, and from other
guests
> >>>> connected to
> >>>> the same libvirt network, but all other incoming
connections are
> >>>> blocked by
> >>>> iptables rules.
> >>>
> >
> > I did not read this properly...
> >
> >>>
> >>> Also here's another assertion from 'The virtual
network driver'
> >>> section in
> >>> http://libvirt.org/firewall.html:
> >>>
> >>>> type=nat
> >>>>
> >>>> Allow inbound related to an established connection. Allow
outbound,
> but
> >>>> only from our expected subnet. Allow traffic between
guests. Deny
> >>>> all other
> >
> > I would expect the 'traffic between guests' to mean only
guests on the
> > same network.  Also that lines up with what's written above.
>
> That is correct.
>
> >
> >>>> inbound. Deny all other outbound.
> >>>
> >
> > [...]
> >
> >>
> >> Thanks for reporting this.  It's clearly a bug in libvirt. 
The rules
> >> are in this order:
> >>
> >>  all rules for virbr0
> >>  all rules for virbr1
> >>  all rules for virbr2
> >>
> >> But what we should do instead is:
> >>
> >>  input rules for all networks
> >>  local rules for all networks
> >>  output rules for all networks
> >>  reject rules for all networks
>
> That doesn't achieve the desired results (as you figured out later,
> according to your followup comments.
>
>
> >>
> >> The problem is that we do not know how other rules look like.  So
what
> >> we might need to do is create chains where rules for the first
network
> >> are, then only append network rules into those chains.
> >>
> >> Would you mind filing a bug for this issue, so we can properly
track it
> >> and don't forget about it?  I'll have a look at it in the
meantime, but
> >> don't promise anything since I'm not that familiar with
that part of the
> >> codebase.
> >>
> >
> > Actually, now that I'm thinking about it, the problem is not that
we
> > disallow the communication from virbr1 to virbr0, but the problem is
> > that we allow the connection from virbr0 to everywhere.
>
> Yep!
>
> That's actually a known bug that has been there "forever". It
has always
> been brought up in the context of "I want communication between my
NATed
> networks, and it only works in one direction!", rather than "I
want to
> block communication between my NATed networks, but it gets past in one
> direction!". When people learn that it's (kind of) blocked, they
work
> around it by adding a new network that is shared by all the guests.
> Because all of the reports have been in this direction, fixing it
> properly has been very low on (my) priority list.
>
>
> >
> > Maybe the solution would be: for each network, insert the rules on top
> > of the forward chain and for each started network, explicitly reject
> > that one.  So that after one network starts it would look like this
(I'm
> > writing this from memory just to illustrate the idea, not actually
> > looking up how stuff looks):
> >
> >  ACCEPT any virbr0 RELATED,ESTABLISHED
> >  ACCEPT virbr0 virbr0
> >  ACCEPT virbr0 any src:192.168.122.0/24
> >  REJECT any virbr0
> >  REJECT virbr0 any
> >
> > And after second network is started, we'd have:
> >
> >  # This one is new:
> >  REJECT virbr1 virbr0
>
> Yeah, it's (mostly) this rule that makes the solution complicated.
> Essentially you need one of these rules for each pair of virtual
> networks, so if you have 10 networks, you'll need 45 rules, for 11
> you'll need 55 (in general, for "n" networks, you'll need
(n * (n-1) /
> 2) extra rules).
>
> Aside from the extra packet processing overhead caused by the high
> number of rules needed (which may be completely insignificant), and the
> necessity to add/remove rules at specific labels rather than just
> inserting them all at the beginning, there is the issue that currently
> each iptables rule added for a libvirt network is dependent only on that
> network, so it's simple to determine which rules to remove when a
> network is destroyed. If we start adding rules that are dependent on two
> networks, then proper cleanup will be more complicated.
>
> None of that should prevent a properly motivated person from writing the
> code to do it, of course!
>
> >  # These ones would be normally at the end, IIRC:
> >  ACCEPT any virbr1 RELATED,ESTABLISHED
> >  ACCEPT virbr1 virbr1
> >  ACCEPT virbr1 any src:192.168.122.0/24
> >  REJECT any virbr1
> >  REJECT virbr1 any
> >  # These are left as they were:
> >  ACCEPT any virbr0 RELATED,ESTABLISHED
> >  ACCEPT virbr0 virbr0
> >  ACCEPT virbr0 any src:192.168.122.0/24
> >  REJECT any virbr0
> >  REJECT virbr0 any
> >
> > Cc'ing Laine so that he can weigh in as he has way more knowledge
of
> > this part of the code =)
>