libvirt users - Apr 2016 - Re: Networking issues with lxc containers in AWS EC2

On 04/11/2016 11:33 AM, Laine Stump wrote:
> Interesting. That functionality was moved out of the kernel's bridge 
> module into br_netfilter some time back, but that was done later than 
> the kernel 3.10 that is used by CentOS 7. Are you running some later 
> kernel version?
>
> If your kernel doesn't have a message in dmesg that looks like this:
>
>    bridge: automatic filtering via arp/ip/ip6tables has been deprecated.
>    Update your scripts to load br_netfilter if you need this.
>
> and the bridge driver is loaded, then that key should be available. Of 
> course if you don't have it, that's equivalent to having it set to
0,
> so you should be okay regardless of why it's missing.
>
Ah, you were right. I'd forgot that the AMI I've using was one running 
the 4.0.5 ml kernel. We discovered that bonded interfaces running with 
mode 5 or 6 do not work with lxc containers (the host's ARP table does 
not get updated). The issue was fixed in the 4.0.5 kernel so we ran for 
a short time with this kernel, only to later abandon this kernel due to 
a bug with software RAID.

I've reverted the kernel back to 3.10 on the AWS instances I'm using the
net.bridge.bridge-nf-call-iptables key is now present. It's already set 
to 0 though so there is nothing that needs to be done
here.
>
> I wouldn't be too quick to judgement. First take a look at tcpdump on 
> the bridge interface that the containers are attached to, and on the 
> ethernet device that connects the bridge to the rest of Amazon's 
> infrastructure. If you see packets from the container's IP going out 
> but not coming back in, check the iptables rules (again - firewalld 
> uses iptables to setup its filtering) for a REJECT or DISCARD rule 
> that has an incrementing count. I use something like this to narrow 
> down the list I need to check:
>
> while true; do iptables -v -S -Z | grep -v '^Zeroing' | grep -v
"c 0
> 0" | grep -e '-c'; echo '**************'; sleep 1;
>
> If you don't see any REJECT or DISCARD rules being triggered, then 
> maybe the problem is that AWS is providing an IP address to your 
> container's MAC, but isn't actually allowing traffic from that MAC
out
> onto the network.
>
I'll get this test setup. Unfortunately I'm not particularly 
knowledgeable with iptables; we don't use it in our product so I've 
never had to deal with it. I think you are right though about what's 
happening--AWS doesn't recognize the MAC addresses for containers 
running under another instance.

On 04/12/2016 01:37 PM, Peter Steele wrote:
> On 04/11/2016 11:33 AM, Laine Stump wrote: I wouldn't be too quick to 
> judgement. First take a look at tcpdump on the bridge interface that 
> the containers are attached to, and on the ethernet device that 
> connects the bridge to the rest of Amazon's infrastructure. If you see 
> packets from the container's IP going out but not coming back in, 
> check the iptables rules (again - firewalld uses iptables to setup its 
> filtering) for a REJECT or DISCARD rule that has an incrementing 
> count. I use something like this to narrow down the list I need to check:
>>
>> while true; do iptables -v -S -Z | grep -v '^Zeroing' | grep -v
"c 0
>> 0" | grep -e '-c'; echo '**************'; sleep 1;
>>
>> If you don't see any REJECT or DISCARD rules being triggered, then 
>> maybe the problem is that AWS is providing an IP address to your 
>> container's MAC, but isn't actually allowing traffic from that
MAC
>> out onto the network.
>>
> I'll get this test setup. Unfortunately I'm not particularly 
> knowledgeable with iptables; we don't use it in our product so I've
> never had to deal with it. I think you are right though about what's 
> happening--AWS doesn't recognize the MAC addresses for containers 
> running under another instance.
>
I did this test and there were no REJECT or DISCARD rules being 
triggered. I did discover something interesting though. I had two AWS 
instances running with some libvirt containers on each. I did a ping 
from one AWS instance to an IP assigned to a container on another AWS 
instance.  The ping failed, and when I checked the source host's arp 
table the mac address that was recorded for the container being pinged 
was that of the container's host instance's br0 interface, not the mac 
address of the container's eth0 interface.

Doing the same test on premise using KVM based instances, when a ping 
was run from one VM to a container hosted on another VM, the arp table 
of the source VM contained the mac address of the eth0 interface bound 
to the container, not the mac address of its host VM.

This indicates to me that AWS thinks all of the IP addresses that have 
been allocated to an instance will be bound to that instance and it 
doesn't try to go any further than that. I'm not exactly sure how to get
AWS to route these addresses properly, but it doesn't seem to be an 
issue with libvirt per se.

Peter

On 04/14/2016 03:35 PM, Peter Steele wrote:
> On 04/12/2016 01:37 PM, Peter Steele wrote:
>> On 04/11/2016 11:33 AM, Laine Stump wrote: I wouldn't be too quick
to
>> judgement. First take a look at tcpdump on the bridge interface that 
>> the containers are attached to, and on the ethernet device that 
>> connects the bridge to the rest of Amazon's infrastructure. If you 
>> see packets from the container's IP going out but not coming back
in,
>> check the iptables rules (again - firewalld uses iptables to setup 
>> its filtering) for a REJECT or DISCARD rule that has an incrementing 
>> count. I use something like this to narrow down the list I need to 
>> check:
>>>
>>> while true; do iptables -v -S -Z | grep -v '^Zeroing' |
grep -v "c 0
>>> 0" | grep -e '-c'; echo '**************';
sleep 1;
>>>
>>> If you don't see any REJECT or DISCARD rules being triggered,
then
>>> maybe the problem is that AWS is providing an IP address to your 
>>> container's MAC, but isn't actually allowing traffic from
that MAC
>>> out onto the network.
>>>
>> I'll get this test setup. Unfortunately I'm not particularly 
>> knowledgeable with iptables; we don't use it in our product so
I've
>> never had to deal with it. I think you are right though about
what's
>> happening--AWS doesn't recognize the MAC addresses for containers 
>> running under another instance.
>>
>
> I did this test and there were no REJECT or DISCARD rules being 
> triggered. I did discover something interesting though. I had two AWS 
> instances running with some libvirt containers on each. I did a ping 
> from one AWS instance to an IP assigned to a container on another AWS 
> instance.  The ping failed, and when I checked the source host's arp 
> table the mac address that was recorded for the container being pinged 
> was that of the container's host instance's br0 interface, not the
mac
> address of the container's eth0 interface.
>
> Doing the same test on premise using KVM based instances, when a ping 
> was run from one VM to a container hosted on another VM, the arp table 
> of the source VM contained the mac address of the eth0 interface bound 
> to the container, not the mac address of its host VM.
>
> This indicates to me that AWS thinks all of the IP addresses that have 
> been allocated to an instance will be bound to that instance and it 
> doesn't try to go any further than that. I'm not exactly sure how
to
> get AWS to route these addresses properly, but it doesn't seem to be 
> an issue with libvirt per se.
>
> Peter
>
I finally got this to work, using proxy arp. I just need to apply the 
following settings on each EC2 instance:

echo 1 > /proc/sys/net/ipv4/conf/br0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp_pvlan
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/br0/send_redirects

With these settings my containers and hosts have full connectivity and 
behave just like they are on the same subnet on-premise. This works for 
CentOS 7 at least, but I assume the same solution would work for Ubuntu.

Peter