libvirt users - Nov 2012 - unable to ping from guests in virbr0 to guests in virbr1 network

Hi all,

I have 3 guests (2-RHEL4 and 1 RHEL6) and have some issues regarding
networking between them. The 2 RHEL4 system's use default bridge virbr0 and
get ip's of range 192.168.122.0/24  (192.168.122.207, 192.168.122.167)

I created another bridge (virbr1) with NAT forwarding (no dhcp). The
network i choose was 192.168.100.0/24. And the third system (RHEL6) was
assigned static ip-addres 192.168.100.101,
>From the RHEL6 system which uses virbr1 is able to ping systems in
192.168.122.0/24 series , but  guest systems in 192.168.122.0/24 are not
able to ping RHEL6 system (in virbr1) network.
>From the RHEL4 guests i am able to ping the gateway ip's (192.168.122.1,
192.168.100.1) , but not the RHEL6 system


Versions:
Fedora release 16 (Verne)
libvirt-0.9.6.3-1.fc16.x86_64
qemu-kvm-0.15.1-8.fc16.x86_64

Any hints on what could be the problem


Regards
Niranjan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20121119/3ec61111/attachment.htm>

On Mon, Nov 19, 2012 at 12:09:53PM +0530, mallapadi niranjan
wrote:
> Hi all,
> 
> I have 3 guests (2-RHEL4 and 1 RHEL6) and have some issues regarding
> networking between them. The 2 RHEL4 system's use default bridge virbr0
and
> get ip's of range 192.168.122.0/24  (192.168.122.207, 192.168.122.167)
> 
> I created another bridge (virbr1) with NAT forwarding (no dhcp). The
> network i choose was 192.168.100.0/24. And the third system (RHEL6) was
> assigned static ip-addres 192.168.100.101,
> 
> >From the RHEL6 system which uses virbr1 is able to ping systems in
> 192.168.122.0/24 series , but  guest systems in 192.168.122.0/24 are not
> able to ping RHEL6 system (in virbr1) network.
> 
> >From the RHEL4 guests i am able to ping the gateway ip's
(192.168.122.1,
> 192.168.100.1) , but not the RHEL6 system
> 
That's correct behavior, because the networks are in different ranges
they need to have a way to talk to each other, either via a gateway or
fixed routing.

btw, Why do you need your hosts in separate networks?

You may want to check this page on libvirt networking.
http://wiki.libvirt.org/page/VirtualNetworking

regards.

-- 
Rudy Godoy
http://stone-head.org

On Mon, Nov 19, 2012 at 12:09:53PM +0530, mallapadi niranjan
wrote:
> Hi all,
> 
> I have 3 guests (2-RHEL4 and 1 RHEL6) and have some issues regarding
> networking between them. The 2 RHEL4 system's use default bridge virbr0
and
> get ip's of range 192.168.122.0/24  (192.168.122.207, 192.168.122.167)
> 
> I created another bridge (virbr1) with NAT forwarding (no dhcp). The
> network i choose was 192.168.100.0/24. And the third system (RHEL6) was
> assigned static ip-addres 192.168.100.101,
> 
> >>From the RHEL6 system which uses virbr1 is able to ping systems in
> 192.168.122.0/24 series , but  guest systems in 192.168.122.0/24 are not
> able to ping RHEL6 system (in virbr1) network.
> 
> >>From the RHEL4 guests i am able to ping the gateway ip's
(192.168.122.1,
> 192.168.100.1) , but not the RHEL6 system
> 
> 
> Versions:
> Fedora release 16 (Verne)
> libvirt-0.9.6.3-1.fc16.x86_64
> qemu-kvm-0.15.1-8.fc16.x86_64
> 
> Any hints on what could be the problem
The problem is caused by the relative order of the iptables rules for those two 
networks.

When libvirt created virbr1 for the network 192.168.100.0/24 it inserted a 
couple of iptables rules on the FORWARDING chain for this interface, but it 
added them before the rules of virbr0 on the same chain. Those rules basically 
are ordered as follows (at least on my system which I suspect is different from 
yours since I also had the same problem):

1. Forward packets destined for the interface that are part of an established 
connection.
2. Forward packets coming from the interface.
3. Forward packets coming from and destined to the same interface (loopback).
4. Reject forwarding anything else to the interface.
5. Reject forwarding anything else from the interface.

Since those rules are inserted for every interface libvirt adds in this order, 
you can only ping from one interface to another if the set of rules for the 
interface you are pinging from comes before that of the one you're pinging
to.
Otherwise, rule number 4 of the destination interface will get in the way since 
iptables matches the packets against the rules in a chain according to their 
order on it.

A possible workaround:

iptables-save >/path/to/iptables/dump/file

then, edit the file to move all relevant rules that REJECT things on the 
FORWARDING chain to the end of the chain.

then put this command in your rc.local script:

cat /path/to/iptables/dump/file | iptables-restore

This solution is not reliable though because you will need to update the file 
containing the rules every time you add a new virtual network or update the 
iptables rules for some other reason.

Also, I hinted before at this problem on the list and someone pointed me that 
it's a bug, but nobody confirmed. Read my message on this topic with the 
subject: Inconsistent iptables forwarding rules for virtual networks, to get a 
clearer picture.

		
	Marwan

---------- Forwarded message ----------
From: mallapadi niranjan <niranjan.ashok at gmail.com>
Date: Mon, Nov 19, 2012 at 7:30 PM
Subject: Re: [libvirt-users] unable to ping from guests in virbr0 to guests
in virbr1 network
To: Marwan Tanager <marwan.tngr at gmail.com>




On Mon, Nov 19, 2012 at 7:51 PM, Marwan Tanager <marwan.tngr at
gmail.com>wrote:
> On Mon, Nov 19, 2012 at 12:09:53PM +0530, mallapadi niranjan wrote:
> > Hi all,
> >
> > I have 3 guests (2-RHEL4 and 1 RHEL6) and have some issues regarding
> > networking between them. The 2 RHEL4 system's use default bridge
virbr0
> and
> > get ip's of range 192.168.122.0/24  (192.168.122.207,
192.168.122.167)
> >
> > I created another bridge (virbr1) with NAT forwarding (no dhcp). The
> > network i choose was 192.168.100.0/24. And the third system (RHEL6)
was
> > assigned static ip-addres 192.168.100.101,
> >
> > >>From the RHEL6 system which uses virbr1 is able to ping
systems in
> > 192.168.122.0/24 series , but  guest systems in 192.168.122.0/24 are
not
> > able to ping RHEL6 system (in virbr1) network.
> >
> > >>From the RHEL4 guests i am able to ping the gateway ip's
> (192.168.122.1,
> > 192.168.100.1) , but not the RHEL6 system
> >
> >
> > Versions:
> > Fedora release 16 (Verne)
> > libvirt-0.9.6.3-1.fc16.x86_64
> > qemu-kvm-0.15.1-8.fc16.x86_64
> >
> > Any hints on what could be the problem
>
> The problem is caused by the relative order of the iptables rules for
> those two
> networks.
>
> When libvirt created virbr1 for the network 192.168.100.0/24 it inserted a
> couple of iptables rules on the FORWARDING chain for this interface, but it
> added them before the rules of virbr0 on the same chain. Those rules
> basically
> are ordered as follows (at least on my system which I suspect is different
> from
> yours since I also had the same problem):
>
> 1. Forward packets destined for the interface that are part of an
> established
> connection.
> 2. Forward packets coming from the interface.
> 3. Forward packets coming from and destined to the same interface
> (loopback).
> 4. Reject forwarding anything else to the interface.
> 5. Reject forwarding anything else from the interface.
>
okay that seems to be the behaviour:
hain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            192.168.100.0/24     state
RELATED,ESTABLISHED
ACCEPT     all  --  192.168.100.0/24     0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with
icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with
icmp-port-unreachable
ACCEPT     all  --  0.0.0.0/0            192.168.200.0/24     state
RELATED,ESTABLISHED
ACCEPT     all  --  192.168.200.0/24     0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with
icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with
icmp-port-unreachable
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     state
RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with
icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with
icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with
icmp-host-prohibited

I can ping from 192.168.100.0/24 -> 192.168.122.0/24 (but not the viceversa)


>
> Since those rules are inserted for every interface libvirt adds in this
> order,
> you can only ping from one interface to another if the set of rules for the
> interface you are pinging from comes before that of the one you're
pinging
> to.
> Otherwise, rule number 4 of the destination interface will get in the way
> since
> iptables matches the packets against the rules in a chain according to
> their
> order on it.
>
> A possible workaround:
>
> iptables-save >/path/to/iptables/dump/file
>
> then, edit the file to move all relevant rules that REJECT things on the
> FORWARDING chain to the end of the chain.
>
> then put this command in your rc.local script:
>
> cat /path/to/iptables/dump/file | iptables-restore
>
> This solution is not reliable though because you will need to update the
> file
> containing the rules every time you add a new virtual network or update the
> iptables rules for some other reason.
>
Right ,

>
> Also, I hinted before at this problem on the list and someone pointed me
> that
> it's a bug, but nobody confirmed. Read my message on this topic with
the
> subject: Inconsistent iptables forwarding rules for virtual networks, to
> get a
> clearer picture.
>
> Okay, will look in to that mail
>
>         Marwan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20121119/3b5ec0b3/attachment.htm>