libvirt users - Sep 2020 - consuming pre-created tap - with multiqueue

Hello,

On KubeVirt, we are trying to pre-create a tap device, then instruct
libvirt to consume it (via the type=ethernet , managed='no'
attributes).

It works as expected, **unless** when we create a multi-queue tap device.

The difference when creating the tap device is that we set the
multi-queue flag; libvirt throws the following error when consuming
it:

```
LibvirtError(Code=38, Domain=0, Message='Unable to create tap device
tap0: Invalid argument')
```

After digging a bit on the libvirt code (we're using libvirt 6.0.0), I
see this on the logs (immediately before the error):
{"component":"virt-launcher","level":"info","msg":"Enabling
IFF_VNET_HDR","pos":"virNetDevProbeVnetHdr:190","subcomponent":"libvirt","thread":"33"
,"timestamp":"2020-09-22T10:34:29.335000Z"}

I do not understand how it can try to set the VNET_HDR flag, since I
have not set it when I created it, which, as per [0] should only
happen when requested. Here's the tap device I'm creating: (output of
`ip tuntap show`)
  - tap0: tap persist0x100 user 107 group 107

I'm confused by this, since on [1] it says that only one flag can be
used (persist *or* vnet_hdr). Also, from my limited understanding, in
order to open a pre-created tap device, it must be set to persistent.

Am I right (in the sense that these flags cannot co-exist) ? If so,
why would libvirt be setting the VNET_HDR flag on the tap device ?

[0] -
https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L261
[1] -
https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L209

On Tue, Sep 22, 2020 at 01:48:08PM +0200, Miguel Duarte de Mora Barroso
wrote:
> Hello,
> 
> On KubeVirt, we are trying to pre-create a tap device, then instruct
> libvirt to consume it (via the type=ethernet , managed='no'
> attributes).
> 
> It works as expected, **unless** when we create a multi-queue tap device.
> 
> The difference when creating the tap device is that we set the
> multi-queue flag; libvirt throws the following error when consuming
> it:
> 
> ```
> LibvirtError(Code=38, Domain=0, Message='Unable to create tap device
> tap0: Invalid argument')
> ```
> 
> After digging a bit on the libvirt code (we're using libvirt 6.0.0), I
> see this on the logs (immediately before the error):
>
{"component":"virt-launcher","level":"info","msg":"Enabling
>
IFF_VNET_HDR","pos":"virNetDevProbeVnetHdr:190","subcomponent":"libvirt","thread":"33"
> ,"timestamp":"2020-09-22T10:34:29.335000Z"}
> 
> I do not understand how it can try to set the VNET_HDR flag, since I
> have not set it when I created it, which, as per [0] should only
> happen when requested. Here's the tap device I'm creating: (output
of
> `ip tuntap show`)
>   - tap0: tap persist0x100 user 107 group 107
IIUC the kernel code correctly, the VNET_HDR flag is not required
to be set when you first create the tap device - it appears to be
permitted any time you open a file descriptor for it.

AFAIK, there is no problem with VNET_HDR, as it is a standard flag
we've set on all tap devices on Linux for 10 years.
> I'm confused by this, since on [1] it says that only one flag can be
> used (persist *or* vnet_hdr). Also, from my limited understanding, in
> order to open a pre-created tap device, it must be set to persistent.
That comment you're pointing to is saying that the method only
has one named flag defined in the enum. It should have been updated
when we added the second PERSIST flag. They are not mutually exclusive
from either libvirt or kernel POV AFAICT.
> Am I right (in the sense that these flags cannot co-exist) ? If so,
> why would libvirt be setting the VNET_HDR flag on the tap device ?
> 
> [0] -
https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L261
> [1] -
https://github.com/libvirt/libvirt/blob/v6.0.0/src/util/virnetdevtap.c#L209

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Wed, Sep 23, 2020 at 05:44:28PM +0100, Daniel P. Berrangé
wrote:
> On Tue, Sep 22, 2020 at 01:48:08PM +0200, Miguel Duarte de Mora Barroso
wrote:
> > Hello,
> > 
> > On KubeVirt, we are trying to pre-create a tap device, then instruct
> > libvirt to consume it (via the type=ethernet , managed='no'
> > attributes).
> > 
> > It works as expected, **unless** when we create a multi-queue tap
device.
> > 
> > The difference when creating the tap device is that we set the
> > multi-queue flag; libvirt throws the following error when consuming
> > it:
> > 
> > ```
> > LibvirtError(Code=38, Domain=0, Message='Unable to create tap
device
> > tap0: Invalid argument')
> > ```
> > 
> > After digging a bit on the libvirt code (we're using libvirt
6.0.0), I
> > see this on the logs (immediately before the error):
> >
{"component":"virt-launcher","level":"info","msg":"Enabling
> >
IFF_VNET_HDR","pos":"virNetDevProbeVnetHdr:190","subcomponent":"libvirt","thread":"33"
> > ,"timestamp":"2020-09-22T10:34:29.335000Z"}
> > 
> > I do not understand how it can try to set the VNET_HDR flag, since I
> > have not set it when I created it, which, as per [0] should only
> > happen when requested. Here's the tap device I'm creating:
(output of
> > `ip tuntap show`)
> >   - tap0: tap persist0x100 user 107 group 107
> 
> IIUC the kernel code correctly, the VNET_HDR flag is not required
> to be set when you first create the tap device - it appears to be
> permitted any time you open a file descriptor for it.
> 
> AFAIK, there is no problem with VNET_HDR, as it is a standard flag
> we've set on all tap devices on Linux for 10 years.
Looking at the kernel code, you need to set the MULTI_QUEUE flag
at time you create the device and also set it when opening the
device. In tun_set_iff():


                if (!!(ifr->ifr_flags & IFF_MULTI_QUEUE) !               
!!(tun->flags & IFF_MULTI_QUEUE))
                        return -EINVAL;


so if you've configured QEMU to use multiqueue, the you need
to use:

  $ ip tuntap add dev mytap mode tap vnet_hdr multi_queue

actually vnet_hdr doesn't matter as that can be set on the fly
but multi_queue is mandatory.  Without it, I get the same EINVAL
error as you mention.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|