libvirt users - Nov 2020 - consume existing tap device when libvirt / qemu run as different users

Hello,

I'm having some doubts about consuming an existing - already
configured - tap device from libvirt (with `managed='no' ` attribute
set).

In KubeVirt, we want to have the consumer side of the tap device run
without the NET_ADMIN capability, which requires the UID / GID of the
tap creator / opener to match, as per the kernel code in [0]. As such,
we create the tap device (with the qemu user / group on behalf of
qemu), which will ultimately be the tap consumer.

This leads me to question: why is libvirt opening / calling
`ioctl(..., TUNSETIFF, ...) ` on the tap device when it already exists
- [1] & [2] ? Why can't the tap device (already configured) be left
alone, and let qemu consume it ?

The above is problematic for KubeVirt, since our setup currently has
libvirt running as root (while qemu runs as a different user), which
is preventing us from removing NET_ADMIN (libvirt & qemu run as
different users).

Thanks in advance for your time,
Miguel

[0] -
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/tun.c?id=4ef8451b332662d004df269d4cdeb7d9f31419b5#n574

[1] -
https://github.com/libvirt/libvirt/blob/99a1cfc43889c6d425a64013a12b234dde8cff1e/src/qemu/qemu_interface.c#L453

[2] -
https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L274

On 11/4/20 6:34 AM, Miguel Duarte de Mora Barroso wrote:
> Hello,
>
> I'm having some doubts about consuming an existing - already
> configured - tap device from libvirt (with `managed='no' `
attribute
> set).
>
> In KubeVirt, we want to have the consumer side of the tap device run
> without the NET_ADMIN capability, which requires the UID / GID of the
> tap creator / opener to match, as per the kernel code in [0]. As such,
> we create the tap device (with the qemu user / group on behalf of
> qemu), which will ultimately be the tap consumer.
>
> This leads me to question: why is libvirt opening / calling
> `ioctl(..., TUNSETIFF, ...) ` on the tap device when it already exists
> - [1] & [2] ? Why can't the tap device (already configured) be left
> alone, and let qemu consume it ?
>
> The above is problematic for KubeVirt, since our setup currently has
> libvirt running as root (while qemu runs as a different user), which
> is preventing us from removing NET_ADMIN (libvirt & qemu run as
> different users).

Miguel also brought this question up in the #virt channel on 
irc.oftc.net, and we discussed it a bit there.


I wondered if possibly the uid of the qemu process was irrelevant, since 
it is libvirtd that's opening and configuring the device (and in his 
case libvirtd is running as root, just without the NET_ADMIN capability) 
- as Miguel points out in his references, the kernel allows a process 
running under the same uid as the owner of net device to perform ioctls 
on that device, even if that process doesn't have NET_ADMIN.


Miguel tried setting (actually "leaving" :-)) ownership of the tap 
device to root when it was created; libvirtd was then able to open and 
configure the device; it passed the open file descriptor to qemu 
(running as user qemu), which consumed and used it without problem.


So, the answer is that the pre-created / unmanaged tap/macvtap devices 
must be owned by the same uid as the libvirtd process, *not* the same 
uid as the qemu process, because libvirtd is the process that operates 
on the device itself (qemu just sends and receives on an already-opened 
file descriptor).

> Thanks in advance for your time,
> Miguel
>
> [0] -
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/tun.c?id=4ef8451b332662d004df269d4cdeb7d9f31419b5#n574
>
> [1] -
https://github.com/libvirt/libvirt/blob/99a1cfc43889c6d425a64013a12b234dde8cff1e/src/qemu/qemu_interface.c#L453
>
> [2] -
https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L274
>