libvirt users - Jul 2014 - Re: multicast traffic no longer working after reboot

Hi,
today a couple of Hypervisors got restarted due to a power outage. While
the systems and the VMs on the are running fine multicast traffic
between the guests is no longer working.

I generate multicast udp packet using iperf in the guest and check on
the host using tcpdump:

tcpdump vnet0 => multicast packets are visible
tcpdump vlanbr8 => multicast packets are visible
tcpdump em1.8 => no multicast packets visible

vlanbr8 is the bridge, vnet0 the interface of the guest and em1.8 the
physical interface for vlan 8.

I compared the settings in /sys/class/net/vlanbr8/bridge/multicast_*
with a system where multicast works fine and see no differences. brctl
showstp also shows "mc router" to be 1 for both interfaces and the
bridge. All settings in /proc/sys/net/bridge/* are 0.

Any ideas why the bridge doesn't seem to forward multicast packets?
iperf with unicast udp also works fine so it's just multicast that is
the problem.

Regards,
  Dennis

On 31.07.2014 15:14, Dennis Jacobfeuerborn wrote:
> Hi,
> today a couple of Hypervisors got restarted due to a power outage. While
> the systems and the VMs on the are running fine multicast traffic
> between the guests is no longer working.
> 
> I generate multicast udp packet using iperf in the guest and check on
> the host using tcpdump:
> 
> tcpdump vnet0 => multicast packets are visible
> tcpdump vlanbr8 => multicast packets are visible
> tcpdump em1.8 => no multicast packets visible
> 
> vlanbr8 is the bridge, vnet0 the interface of the guest and em1.8 the
> physical interface for vlan 8.
> 
> I compared the settings in /sys/class/net/vlanbr8/bridge/multicast_*
> with a system where multicast works fine and see no differences. brctl
> showstp also shows "mc router" to be 1 for both interfaces and
the
> bridge. All settings in /proc/sys/net/bridge/* are 0.
> 
> Any ideas why the bridge doesn't seem to forward multicast packets?
> iperf with unicast udp also works fine so it's just multicast that is
> the problem.
I did some further testing and strangely enough after booting the HV and
the guest things work as expected but after 2-3 minutes I suddenly don't
see the the packets anymore on em1.8. This is really bizarre behavior.

Regards,
  Dennis

On 31.07.2014 17:23, Dennis Jacobfeuerborn wrote:
> On 31.07.2014 15:14, Dennis Jacobfeuerborn wrote:
>> Hi,
>> today a couple of Hypervisors got restarted due to a power outage.
While
>> the systems and the VMs on the are running fine multicast traffic
>> between the guests is no longer working.
>>
>> I generate multicast udp packet using iperf in the guest and check on
>> the host using tcpdump:
>>
>> tcpdump vnet0 => multicast packets are visible
>> tcpdump vlanbr8 => multicast packets are visible
>> tcpdump em1.8 => no multicast packets visible
>>
>> vlanbr8 is the bridge, vnet0 the interface of the guest and em1.8 the
>> physical interface for vlan 8.
>>
>> I compared the settings in /sys/class/net/vlanbr8/bridge/multicast_*
>> with a system where multicast works fine and see no differences. brctl
>> showstp also shows "mc router" to be 1 for both interfaces
and the
>> bridge. All settings in /proc/sys/net/bridge/* are 0.
>>
>> Any ideas why the bridge doesn't seem to forward multicast packets?
>> iperf with unicast udp also works fine so it's just multicast that
is
>> the problem.
> 
> I did some further testing and strangely enough after booting the HV and
> the guest things work as expected but after 2-3 minutes I suddenly
don't
> see the the packets anymore on em1.8. This is really bizarre behavior.
So after a lot of digging I seem to have found the cause of the issue
but I'm not sure what the proper fix is.

The cause seems to be a kernel "regression":
https://bugzilla.redhat.com/show_bug.cgi?id=880035#c56

It seems that multicast now requires explicitly setting
/sys/class/<bridge>/bridge/multicast_querier to 1.

What is puzzling about this is that im wonder why this behavior
changeing patch was included in the RHEL/CentOS kernel. This is the
first time I ever ran into this kind of problem and previously multicast
seemed to work just out of the box.

The other issue is that I'm not sure how to make this change permanent.
When changing somthing under /proc/sys I can put the value in
/etc/sysctl.conf but no such config file exists for /sys/.

There is a snippet in the bug report but that only sets the flag for
specific bridges. What I'm really looking for is a way to get back to
the old behaviour where I can start a bridge and have multicast work by
default.

Regards,
  Dennis