libvirt users - Jul 2013 - Re: Permission problem with /dev/net/tun

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi lxc folks,

the symptom my libvirt LXC container suffers from is:

root@depot:/dev/net# ls -la
total 0
drwxr-xr-x 2 root root  40 Jun 29 16:26 .
drwxr-xr-x 5 root root 480 Jun 29 16:26 ..

root@depot:/dev/net# mknod tun c 10 200
mknod: `tun': Operation not permitted

The host is an up-to-date AMD64 Ubuntu raring on 3.8.0-25-generic that
was formerly installed from precise and then upgraded. The guest is
Ubuntu precise; however, I see the same symptom in another raring
container on the same host.

What I tried to resolve this:

1) On the host, I echoed various stuff to the cgroup device files:

cd /sys/fs/cgroup/devices/libvirt/lxc
echo "c 10:200 rwm" > devices.allow
echo "c 10:200 rwm" > depot/devices.allow
echo a > depot/devices.allow

... and I see the successful results in depot/devices.list, but no
success.

2) I inserted a line "/dev/net/tun rwk," into
/etc/apparmor.d/abstractions/lxc/container-base - no change. (I know,
it seems kind of pointless - because it's about permissions to a
device, not a path. You may deduce my desperation from this ...

SELinux is not active. Mounts on /dev look normal to me:

devfs on /dev type tmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
devpts on /dev/ptmx type devpts
(rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)

Now comes the weird part: Once I umount /dev inside the container, the
"hidden" /dev appears which contains a usable /dev/net/tun. So the
mknod problem is probably due to the dropped capabilities - but
why/how mounts the container a more restricted /dev on top of the
prepared one ...?


Thanks & regards
Thomas


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlHYdysACgkQiMyIQtYO79yUMQCfUrbaAWcZsqYkopRL6F2evWdq
XnMAoOq6/wopgBGZniWDGnIiYBEsFX61
=BIr9
-----END PGP SIGNATURE-----

On Sat, Jul 06, 2013 at 09:59:39PM +0200, Thomas Karcher
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi lxc folks,
> 
> the symptom my libvirt LXC container suffers from is:
> 
> root@depot:/dev/net# ls -la
> total 0
> drwxr-xr-x 2 root root  40 Jun 29 16:26 .
> drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
> 
> root@depot:/dev/net# mknod tun c 10 200
> mknod: `tun': Operation not permitted
> 
> The host is an up-to-date AMD64 Ubuntu raring on 3.8.0-25-generic that
> was formerly installed from precise and then upgraded. The guest is
> Ubuntu precise; however, I see the same symptom in another raring
> container on the same host.
> 
> What I tried to resolve this:
> 
> 1) On the host, I echoed various stuff to the cgroup device files:
> 
> cd /sys/fs/cgroup/devices/libvirt/lxc
> echo "c 10:200 rwm" > devices.allow
> echo "c 10:200 rwm" > depot/devices.allow
> echo a > depot/devices.allow
> 
> ... and I see the successful results in depot/devices.list, but no
> success.
> 
> 2) I inserted a line "/dev/net/tun rwk," into
> /etc/apparmor.d/abstractions/lxc/container-base - no change. (I know,
> it seems kind of pointless - because it's about permissions to a
> device, not a path. You may deduce my desperation from this ...
> 
> SELinux is not active. Mounts on /dev look normal to me:
> 
> devfs on /dev type tmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> devpts on /dev/ptmx type devpts
> (rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
> cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
> 
> Now comes the weird part: Once I umount /dev inside the container, the
> "hidden" /dev appears which contains a usable /dev/net/tun. So
the
> mknod problem is probably due to the dropped capabilities - but
> why/how mounts the container a more restricted /dev on top of the
> prepared one ...?
Allowing the container direct access to the hosts' /dev would be
a security flaw, so libvirt sets up a private /dev for the container.
Allowing the container to use mknod would also be insecure, so we
blocking mknod using both cgroups device ACL, and also droping the
CAP_MKNOD capability.

  http://libvirt.org/drvlxc.html#devnodes

Any device that the container is authorized to access per the XML
configuration, will be pre-created in the container's /dev. To
explicitly allow /dev/net/tun you need to tell libvirt about it.

  http://libvirt.org/formatdomain.html#elementsHostDevCaps


Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Daniel,

On 07/08/2013 11:41 AM, Daniel P. Berrange wrote:
>> the symptom my libvirt LXC container suffers from is: 
>> root@depot:/dev/net# ls -la total 0 drwxr-xr-x 2 root root  40
>> Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 .. 
>> root@depot:/dev/net# mknod tun c 10 200 mknod: `tun': Operation
>> not permitted
> Allowing the container direct access to the hosts' /dev would be a
> security flaw, so libvirt sets up a private /dev for the
> container. Allowing the container to use mknod would also be
> insecure, so we blocking mknod using both cgroups device ACL, and
> also droping the CAP_MKNOD capability. 
> http://libvirt.org/drvlxc.html#devnodes
Good to know.
> Any device that the container is authorized to access per the XML 
> configuration, will be pre-created in the container's /dev. To 
> explicitly allow /dev/net/tun you need to tell libvirt about it. 
> http://libvirt.org/formatdomain.html#elementsHostDevCaps
Thanks!

I extended the 'devices' section as follows:

    <hostdev mode='capabilities' type='misc'>
      <source>
        <char>/dev/net/tun</char>
      </source>
    </hostdev>

... because even though /dev/net/tun is used for networking, it
appears as a character device. (Btw: The documentation says in the
hostdev section: ''For block/character device passthrough mode is
always "capabilities" and type is "block" for a block
device, "char"
for a character device and "net" for a host network
interface.'' When
I specify type='char', I get an error from virsh.)

With this XML, I can define the container. But upon start, I get the
following error message:

Fehler: internal error guest failed to start: PATH=/bin:/sbin
TERM=linux container=lxc-libvirt
container_uuid=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_UUID=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_NAME=depot /sbin/init
2013-07-08 21:36:50.735+0000: 1: info : libvirt version: 1.0.2
2013-07-08 21:36:50.735+0000: 1: error :
lxcContainerSetupHostdevCapsMisc:1490 : Unable to create device
/dev/net/tun: No such file or directory
2013-07-08 21:36:50.744+0000: 19537: info : libvirt version: 1.0.2
2013-07-08 21:36:50.744+0000: 19537: error : virCommandWait:2287 :
internal error Child process (ip link set veth6 netns 19538)
unexpected exit status 2: RTNETLINK answers: No such process

2013-07-08 21:36:50.786+0000: 19537: error : virCommandWait:2287 :
internal error Child process (ip link del veth4) unexpected exit
status 1: Cannot find device "veth4"

On the host, /dev/net/tun exists as character device:

root@main:~# ls -la /dev/net/tun
crw-rw-rwT 1 root root 10, 200 Jul  8 23:45 /dev/net/tun

What am I doing wrong ...?


Thanks
Thomas

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlHbNGwACgkQiMyIQtYO79zOwgCdFVzn0JopHK+ZY2ZshgZnuz6L
Yx8An3BL/2sfLTFSs39yNkB0FXzq9K/4
=xRtO
-----END PGP SIGNATURE-----