libvirt users - Apr 2016 - Re: Networking issues with lxc containers in AWS EC2

On 04/01/2016 02:07 PM, Laine Stump wrote:
> On 03/31/2016 06:43 PM, Peter Steele wrote:
>> I've created an EC2 AMI for AWS that essentially represents a
CentOS
>> 7 "hypervisor" image. I deploy instances of these in AWS and
create
>> an number of libvirt based lxc containers on each of these instances. 
>> The containers run fine within a single host and have no problem 
>> communicating with themselves as well as with their host, and vice 
>> versa. However, containers hosted in one EC2 instance cannot 
>> communicate with containers hosted in another EC2 instance.
>>
>> We've tried various tweaks with our Amazon VPC but have been unable
>> to find a way to solve this networking issue. If I use something like 
>> VMware or KVM and create VMs using this same hypervisor image, the 
>> containers running under these VMs can communicate with with each 
>> other, even across different hosts.
>
> What is the <interface> config of your nested containers? Do they
each
> get a public IP address?
Yes, they all have public IPs on the same subnet. When deployed in a VM 
environment on premises, the containers have no problems. Amazon clearly 
does something with the packets though and the containers can't talk to 
each other.
>
>>
>> My real question is has anyone tried deploying EC2 images that host 
>> containers and have figured out how to successfully communicate 
>> between containers on different hosts?
>>
>
> No experience with EC2, sorry.
>
I think we'll need to go to Amazon themselves to resolve this issue. 
There is very little information out there about how to get lxc 
containers to work properly in EC2.

On 04/01/2016 07:04 PM, Peter Steele wrote:
> On 04/01/2016 02:07 PM, Laine Stump wrote:
>> On 03/31/2016 06:43 PM, Peter Steele wrote:
>>> I've created an EC2 AMI for AWS that essentially represents a
CentOS
>>> 7 "hypervisor" image. I deploy instances of these in AWS
and create
>>> an number of libvirt based lxc containers on each of these 
>>> instances. The containers run fine within a single host and have no
>>> problem communicating with themselves as well as with their host, 
>>> and vice versa. However, containers hosted in one EC2 instance 
>>> cannot communicate with containers hosted in another EC2 instance.
>>>
>>> We've tried various tweaks with our Amazon VPC but have been
unable
>>> to find a way to solve this networking issue. If I use something 
>>> like VMware or KVM and create VMs using this same hypervisor image,
>>> the containers running under these VMs can communicate with with 
>>> each other, even across different hosts.
>>
>> What is the <interface> config of your nested containers? Do they
>> each get a public IP address?
> Yes, they all have public IPs on the same subnet. When deployed in a 
> VM environment on premises, the containers have no problems. Amazon 
> clearly does something with the packets though and the containers 
> can't talk to each other.

You say they can talk among containers on the same host, and with their 
own host (I guess you mean the virtual machine that is hosting the 
containers), but not to containers on another host. Can the containers 
communicate outside of the host at all? If not, perhaps the problem is 
iptables rules for the bridge device the containers are using - try 
running this command:

   sysctl net.bridge.bridge-nf-call-iptables

If that returns:

   net.bridge.bridge-nf-call-iptables = 1

then run this command and see if the containers can now communicate with 
the outside:

   sysctl -w net.bridge.bridge-nf-call-iptables=0

>>>
>>> My real question is has anyone tried deploying EC2 images that host
>>> containers and have figured out how to successfully communicate 
>>> between containers on different hosts?
>>>
>>
>> No experience with EC2, sorry.
>>
> I think we'll need to go to Amazon themselves to resolve this issue. 
> There is very little information out there about how to get lxc 
> containers to work properly in EC2.
Well, if they've allowed your virtual machine to acquire multiple IP 
addresses, then it would make sense that they would allow them to 
actually use those IP addresses. I'm actually more inclined to think 
that the packets simply aren't getting out of the virtual machine (or 
the responses aren't getting back in).

On 04/02/2016 05:20 PM, Laine Stump wrote:
> You say they can talk among containers on the same host, and with their
> own host (I guess you mean the virtual machine that is hosting the
> containers), but not to containers on another host. Can the containers
> communicate outside of the host at all? If not, perhaps the problem is
> iptables rules for the bridge device the containers are using - try
> running this command:
>
>   sysctl net.bridge.bridge-nf-call-iptables
>
> If that returns:
>
>   net.bridge.bridge-nf-call-iptables = 1
>
> then run this command and see if the containers can now communicate with
> the outside:
>
>   sysctl -w net.bridge.bridge-nf-call-iptables=0
This key doesn't exist in the CentOS 7 image I'm running. I do have a 
bridge interface defined of course, although we do not run iptables. We 
don't need this service when running our software on premise. Actually, 
in CentOS 7 the iptables service doesn't exist; there's a new service 
called firewalld that serves the same purpose. We don't run this either 
at present.
> Well, if they've allowed your virtual machine to acquire multiple IP
> addresses, then it would make sense that they would allow them to
> actually use those IP addresses. I'm actually more inclined to think
> that the packets simply aren't getting out of the virtual machine (or
> the responses aren't getting back in).
>
The difference is that the virtual machine itself isn't assigned the IPs 
but rather containers under the AWS instance and something with how 
Amazon manages their stack prevents the packets from one container to 
the other. The very fact that the exact same software runs fine in VMs 
under say VMware or KVM but not VMs under AWS clearly points to AWS as 
the ultimate source of the problem.