libvirt users - Mar 2014 - Re: method for communication between containers

Dear Daniel,
Thank you for your email.
According to your suggestion, I tried to use the shared memory filesystem for
data exchange between the host machine and the container. The steps are as
follows:

1) create an empty file in the host machine (/tmp1/file1).
2) run #chmod 777 /tmp1 and #chmod 777 /tmp1/file1.
3) expose /tmp1 to the container by configuring the container with the following
XML scripts.
     <filesystem type="mount"
accessmode="passthrough">
      <source dir="/tmp1"/>
      <target dir="/tmp1"/>
    </filesystem>
4) write two programs: program1 writes a string to the shared memory
                   program2 reads the string from the shared memory
   and compile both programs in the host machine.
   both of the two programs use ftok() and shmget() to obtain the shmfs id.
5) a. if we run both program1 and program2 in the host machine, the program2 can
read the string that program1 has written.
  b. if we run both program1 and program2 in the container, both programs also
work well.
  c. if we run program1 in the host and run program2 in the container, program2
fails. the ftok() function can return the expected value, but shmget(...,0,0)
fails and prompts "No such files". It seems that the exposed file
cannot be recognized by shmget() function, although ftok() can work and return
the value.

Do you have an idea?

Best wishes.
Cheng Wang

-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com] 
Sent: 2014年2月27日 18:30
To: WANG Cheng D
Cc: libvirt-users@redhat.com
Subject: Re: [libvirt-users] method for communication between containers

On Thu, Feb 27, 2014 at 06:28:46AM +0000, WANG Cheng D
wrote:
> Dear all,
> In my system, two containers need to exchange data as quick as 
> possible and the two containers are hosted by the same physical 
> machine, I wonder if socket is the only method for communication between
containers?
If you have an area of filesystem that is shared + writable to both containers,
they could create UNIX sockets in that filesystem, rather than TCP sockets. Also
they could create named FIFO pipes.

Another option would be to create a shared memory backed file in shmfs or
hugepagefs and expose it to both containers.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

On Mon, Mar 03, 2014 at 01:39:03PM +0000, WANG Cheng D
wrote:
> Dear Daniel,
> Thank you for your email.
> According to your suggestion, I tried to use the shared memory filesystem
for data exchange between the host machine and the container. The steps are as
follows:
> 
> 1) create an empty file in the host machine (/tmp1/file1).
> 2) run #chmod 777 /tmp1 and #chmod 777 /tmp1/file1.
> 3) expose /tmp1 to the container by configuring the container with the
following XML scripts.
>      <filesystem type="mount"
accessmode="passthrough">
>       <source dir="/tmp1"/>
>       <target dir="/tmp1"/>
>     </filesystem>
> 4) write two programs: program1 writes a string to the shared memory
>                    program2 reads the string from the shared memory
>    and compile both programs in the host machine.
>    both of the two programs use ftok() and shmget() to obtain the shmfs id.
> 5) a. if we run both program1 and program2 in the host machine, the
program2 can read the string that program1 has written.
>   b. if we run both program1 and program2 in the container, both programs
also work well.
>   c. if we run program1 in the host and run program2 in the container,
>    program2 fails. the ftok() function can return the expected value,
>    but shmget(...,0,0) fails and prompts "No such files". It
seems that
>    the exposed file cannot be recognized by shmget() function, although
>    ftok() can work and return the value.
> 
> Do you have an idea?
Yes, the IPCS namespace causes the container's 'shmget' call to be
isolated
from the host, so using shmget & other similar SysV-IPC APIs will fail.

What I was thinking though was that you'd be able to open() the file in
the shared memory fs, and then use mmap() to map its data into RAM in
the container. 

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Dear Daniel,
Your suggestion is right. I followed your method and used mmap(), it works well.
Now I can realize data exchange between containers.
Thank you for your help.
Cheng

-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com] 
Sent: 2014年3月3日 21:42
To: WANG Cheng D
Cc: libvirt-users@redhat.com
Subject: Re: [libvirt-users] method for communication between containers

On Mon, Mar 03, 2014 at 01:39:03PM +0000, WANG Cheng D
wrote:
> Dear Daniel,
> Thank you for your email.
> According to your suggestion, I tried to use the shared memory filesystem
for data exchange between the host machine and the container. The steps are as
follows:
> 
> 1) create an empty file in the host machine (/tmp1/file1).
> 2) run #chmod 777 /tmp1 and #chmod 777 /tmp1/file1.
> 3) expose /tmp1 to the container by configuring the container with the
following XML scripts.
>      <filesystem type="mount"
accessmode="passthrough">
>       <source dir="/tmp1"/>
>       <target dir="/tmp1"/>
>     </filesystem>
> 4) write two programs: program1 writes a string to the shared memory
>                    program2 reads the string from the shared memory
>    and compile both programs in the host machine.
>    both of the two programs use ftok() and shmget() to obtain the shmfs id.
> 5) a. if we run both program1 and program2 in the host machine, the
program2 can read the string that program1 has written.
>   b. if we run both program1 and program2 in the container, both programs
also work well.
>   c. if we run program1 in the host and run program2 in the container,
>    program2 fails. the ftok() function can return the expected value,
>    but shmget(...,0,0) fails and prompts "No such files". It
seems that
>    the exposed file cannot be recognized by shmget() function, although
>    ftok() can work and return the value.
> 
> Do you have an idea?
Yes, the IPCS namespace causes the container's 'shmget' call to be
isolated from the host, so using shmget & other similar SysV-IPC APIs will
fail.

What I was thinking though was that you'd be able to open() the file in the
shared memory fs, and then use mmap() to map its data into RAM in the container.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Dear Daniel,
As you know, following your suggestion, I can expose shared memory to two
containers hosted by the same machine, thus the two containers can exchange
data. Now I have a new problem: how does the containers avoid concurrent
accesses to the shared memory? In a native system, this can be achieved by using
semaphore. But I don't know how the two containers can see the same
semaphore. Do I need to design a mechanism to control the access to the shared
memory?
With my warmest regards,
Cheng

-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com] 
Sent: 2014年3月3日 21:42
To: WANG Cheng D
Cc: libvirt-users@redhat.com
Subject: Re: [libvirt-users] method for communication between containers

On Mon, Mar 03, 2014 at 01:39:03PM +0000, WANG Cheng D
wrote:
> Dear Daniel,
> Thank you for your email.
> According to your suggestion, I tried to use the shared memory filesystem
for data exchange between the host machine and the container. The steps are as
follows:
> 
> 1) create an empty file in the host machine (/tmp1/file1).
> 2) run #chmod 777 /tmp1 and #chmod 777 /tmp1/file1.
> 3) expose /tmp1 to the container by configuring the container with the
following XML scripts.
>      <filesystem type="mount"
accessmode="passthrough">
>       <source dir="/tmp1"/>
>       <target dir="/tmp1"/>
>     </filesystem>
> 4) write two programs: program1 writes a string to the shared memory
>                    program2 reads the string from the shared memory
>    and compile both programs in the host machine.
>    both of the two programs use ftok() and shmget() to obtain the shmfs id.
> 5) a. if we run both program1 and program2 in the host machine, the
program2 can read the string that program1 has written.
>   b. if we run both program1 and program2 in the container, both programs
also work well.
>   c. if we run program1 in the host and run program2 in the container,
>    program2 fails. the ftok() function can return the expected value,
>    but shmget(...,0,0) fails and prompts "No such files". It
seems that
>    the exposed file cannot be recognized by shmget() function, although
>    ftok() can work and return the value.
> 
> Do you have an idea?
Yes, the IPCS namespace causes the container's 'shmget' call to be
isolated from the host, so using shmget & other similar SysV-IPC APIs will
fail.

What I was thinking though was that you'd be able to open() the file in the
shared memory fs, and then use mmap() to map its data into RAM in the container.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|