libvirt users - Jan 2018 - Re: How to use /dev/net/tun from libvirt-lxc with user namespacing enabled

I have a container rootfs that I use to keep all work-related stuff. This
container was originally created by lxd (which creates all containers for
use with user namespacing), but now I decided to start using libvirt for
container management since I already use it for virtual machines, which
will spare me from dealing with multiple hypervisor technologies.

I managed to create a working domain xml for the container, and everything
seems to be working very well except one thing: I cannot start openconnect
(VPN software) inside the container.  I noticed that by default libvirt
won't create /dev/net/tun for the container, so I added this to the domain
xml:

    <filesystem type='mount'>
      <source dir='/dev/net'/>
      <target dir='/dev/net'/>
    </filesystem>

This successfully created /dev/net/tun in the container, but openconnect
still can't open it even though it has 666 permissions. It seems this is
exactly what lxd does to allow VPNs for their unprivileged containers, as
shown by the output of ls -l /dev/net

total 0
crw-rw-rw- 1 nobody nogroup 10, 200 Jan 29 13:23 tun

The same container can also be successfully booted with systemd-nspawn,
also allowing openconnect to create its VPN (though systemd-nspawn appears
to create a new device node, owned by root relative to the user namespace).

I already tried setting security driver to "none" in
/etc/libvirt/lxc.conf,
but it had no effect. I get "Operation not permitted" when trying to
open
/dev/net/tun, which is also the message openconnect displays in its logs.

Can someone guide me on how I might debug what is causing this error? BTW,
here's the full xml:

<domain type='lxc'>
  <name>work-stuff</name>
  <uuid>ffee008c-ec6b-48ab-af6d-4aba830847a1</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>16</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <idmap>
    <uid start='0' target='165536' count='65536'/>
    <gid start='0' target='165536' count='65536'/>
  </idmap>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source
dir='/var/lib/libvirt/containers/work-stuff/rootfs'/>
      <target dir='/'/>
    </filesystem>
    <filesystem type='mount'>
      <source dir='/dev/net'/>
      <target dir='/dev/net'/>
    </filesystem>
    <interface type='network'>
      <mac address='52:54:00:3e:59:e9'/>
      <source network='default'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>

On Mon, Jan 29, 2018 at 07:01:15PM -0300, Thiago Padilha
wrote:
> I have a container rootfs that I use to keep all work-related stuff. This
> container was originally created by lxd (which creates all containers for
> use with user namespacing), but now I decided to start using libvirt for
> container management since I already use it for virtual machines, which
> will spare me from dealing with multiple hypervisor technologies.
> 
> I managed to create a working domain xml for the container, and everything
> seems to be working very well except one thing: I cannot start openconnect
> (VPN software) inside the container.  I noticed that by default libvirt
> won't create /dev/net/tun for the container, so I added this to the
domain
> xml:
> 
>     <filesystem type='mount'>
>       <source dir='/dev/net'/>
>       <target dir='/dev/net'/>
>     </filesystem>
> 
> This successfully created /dev/net/tun in the container, but openconnect
> still can't open it even though it has 666 permissions. It seems this
is
> exactly what lxd does to allow VPNs for their unprivileged containers, as
> shown by the output of ls -l /dev/net
That config makes the filesystem containing the device node visible, but
does not grant access to device nodes themselves.

You instead need device passthrough

<hostdev mode='capabilities' type='misc'>
  <source>
    <char>/dev/net/tun</char>
  </source>
</hostdev>



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Wed, Jan 31, 2018 at 6:18 AM, Daniel P. Berrangé <berrange@redhat.com>
wrote:
> That config makes the filesystem containing the device node visible, but
> does not grant access to device nodes themselves.
>
> You instead need device passthrough
>
> <hostdev mode='capabilities' type='misc'>
>   <source>
>     <char>/dev/net/tun</char>
>   </source>
> </hostdev>
Just tried adding the suggested <hostdev> snippet but /dev/net/tun is
still not accessible:

    $ cat /dev/net/tun
    cat: /dev/net/tun: Operation not permitted

Where outside the container or when in LXD or systemd-nspawn I see:

    $ cat /dev/net/tun
    cat: /dev/net/tun: File descriptor in bad state

(Which is the expected output)