libvirt users - Jun 2020 - Re: plug pre-created tap devices to libvirt guests

On 4/6/20 9:54 AM, Daniel P. Berrangé wrote:
> On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora Barroso
wrote:
>> Hi all,
>>
>> I'm aware that it is possible to plug pre-created macvtap devices
to
>> libvirt guests - tracked in RFE [0].
>>
>> My interpretation of the wording in [1] and [2] is that it is also
>> possible to plug pre-created tap devices into libvirt guests - that
>> would be a requirement to allow kubevirt to run with less capabilities
>> in the pods that encapsulate the VMs.
>>
>> I took a look at the libvirt code ([3] & [4]), and, from my limited
>> understanding, I got the impression that plugging existing interfaces
>> via `managed='no' ` is only possible for macvtap interfaces.

No, it works for standard tap devices as well.


The reason the BZs and commit logs talk mostly about macvtap rather than 
tap is because 1) that's what kubevirt people had asked for and 2) it 
already *mostly* worked for tap devices, so most of the work was related 
to macvtap (my memory is already fuzzy, but I think there were a couple 
privileged operations we still tried to do for standard tap devices even 
if they were precreated (standard disclaimer: I often misremember, so 
this memory could be wrong! But definitely precreated tap devices do work).


I think though that when someone from kubevirt actually tried using a 
precreated macvtap device, they found that their precreated device 
wasn't visible at all to the unprivileged libvirtd in the pod, because 
it was in a different network namespace, or something like that. So 
there may still be more work to do (or, again, my info might be out of 
date and they figured out a proper solution).

>>
>> Would you be able to shed some light into this ? Is it possible on
>> libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ?
>>
>> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367
> This links to the following message, which illustrates how to use
pre-create
> tap and macvtap devices:
>
>    https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html
>
> Laine: it would be useful to add something like this short guide to the
> knowledge base docs

You mean the wiki? Sure, I can do that.


(BTW - that was admirable reading / searching / responding - 7 minutes 
and it wasn't even your patch! How do you do that? :-))

On Mon, Apr 06, 2020 at 10:03:41AM -0400, Laine Stump
wrote:
> On 4/6/20 9:54 AM, Daniel P. Berrangé wrote:
> > > Would you be able to shed some light into this ? Is it possible
on
> > > libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ?
> > > 
> > > [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367
> > This links to the following message, which illustrates how to use
pre-create
> > tap and macvtap devices:
> > 
> >   
https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html
> > 
> > Laine: it would be useful to add something like this short guide to
the
> > knowledge base docs
> 
> 
> You mean the wiki? Sure, I can do that.
No, i mean docs/kbase/
> (BTW - that was admirable reading / searching / responding - 7 minutes and
> it wasn't even your patch! How do you do that? :-))
I just read the BZ & followed the link :-)

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com>
wrote:
>
> On 4/6/20 9:54 AM, Daniel P. Berrangé wrote:
> > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora
Barroso wrote:
> >> Hi all,
> >>
> >> I'm aware that it is possible to plug pre-created macvtap
devices to
> >> libvirt guests - tracked in RFE [0].
> >>
> >> My interpretation of the wording in [1] and [2] is that it is also
> >> possible to plug pre-created tap devices into libvirt guests -
that
> >> would be a requirement to allow kubevirt to run with less
capabilities
> >> in the pods that encapsulate the VMs.
> >>
> >> I took a look at the libvirt code ([3] & [4]), and, from my
limited
> >> understanding, I got the impression that plugging existing
interfaces
> >> via `managed='no' ` is only possible for macvtap
interfaces.
>
>
> No, it works for standard tap devices as well.
>
>
> The reason the BZs and commit logs talk mostly about macvtap rather than
> tap is because 1) that's what kubevirt people had asked for and 2) it
> already *mostly* worked for tap devices, so most of the work was related
> to macvtap (my memory is already fuzzy, but I think there were a couple
> privileged operations we still tried to do for standard tap devices even
> if they were precreated (standard disclaimer: I often misremember, so
> this memory could be wrong! But definitely precreated tap devices do work).
>
It's been a while since I've started this thread, but lately I've
understood better how tap devices work, and that new insight makes me
wonder about a couple of things.

Our ultimate goal In kubevirt is to consume a pre-created tap device
by a kubernetes pod that doesn't have the NET_ADMIN capability.

After looking at the current libvirt code, I don't think that is
currently supported, since we'll *always* enter the
`virNetDevTapCreate` function in [1] (I'm interested in the *tap*
scenario).

The tap device is effectively created in that function - [2] - by
opening the clone device (/dev/net/tun), and calling `ioctl(fd,
TUNSETIFF,...)` in it. AFAIK, both of those operations *require* the
NET_ADMIN capability. If I'm correct, this means that the current
libvirt implementation makes our goals impossible to achieve.

I'd first like to know if I read the code correctly, and if I'm
ultimately right - i.e. since libvirt's implementation for consuming a
pre-existent tap device opens the clone device, where ever libvirt is
run on will  *always* require the NET_ADMIN capability.

>
> I think though that when someone from kubevirt actually tried using a
> precreated macvtap device, they found that their precreated device
> wasn't visible at all to the unprivileged libvirtd in the pod, because
> it was in a different network namespace, or something like that. So
> there may still be more work to do (or, again, my info might be out of
> date and they figured out a proper solution).
>
>
> >>
> >> Would you be able to shed some light into this ? Is it possible on
> >> libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ?
> >>
> >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367
> > This links to the following message, which illustrates how to use
pre-create
> > tap and macvtap devices:
> >
> >   
https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html
> >
> > Laine: it would be useful to add something like this short guide to
the
> > knowledge base docs
>
>
> You mean the wiki? Sure, I can do that.
>
>
> (BTW - that was admirable reading / searching / responding - 7 minutes
> and it wasn't even your patch! How do you do that? :-))
>
>
On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com>
wrote:
>
> On 4/6/20 9:54 AM, Daniel P. Berrangé wrote:
> > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora
Barroso wrote:
> >> Hi all,
> >>
> >> I'm aware that it is possible to plug pre-created macvtap
devices to
> >> libvirt guests - tracked in RFE [0].
> >>
> >> My interpretation of the wording in [1] and [2] is that it is also
> >> possible to plug pre-created tap devices into libvirt guests -
that
> >> would be a requirement to allow kubevirt to run with less
capabilities
> >> in the pods that encapsulate the VMs.
> >>
> >> I took a look at the libvirt code ([3] & [4]), and, from my
limited
> >> understanding, I got the impression that plugging existing
interfaces
> >> via `managed='no' ` is only possible for macvtap
interfaces.
>
>
> No, it works for standard tap devices as well.
>
>
> The reason the BZs and commit logs talk mostly about macvtap rather than
> tap is because 1) that's what kubevirt people had asked for and 2) it
> already *mostly* worked for tap devices, so most of the work was related
> to macvtap (my memory is already fuzzy, but I think there were a couple
> privileged operations we still tried to do for standard tap devices even
> if they were precreated (standard disclaimer: I often misremember, so
> this memory could be wrong! But definitely precreated tap devices do work).
>
>
> I think though that when someone from kubevirt actually tried using a
> precreated macvtap device, they found that their precreated device
> wasn't visible at all to the unprivileged libvirtd in the pod, because
> it was in a different network namespace, or something like that. So
> there may still be more work to do (or, again, my info might be out of
> date and they figured out a proper solution).
>
>
> >>
> >> Would you be able to shed some light into this ? Is it possible on
> >> libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ?
> >>
> >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367
> > This links to the following message, which illustrates how to use
pre-create
> > tap and macvtap devices:
> >
> >   
https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html
> >
> > Laine: it would be useful to add something like this short guide to
the
> > knowledge base docs
>
>
> You mean the wiki? Sure, I can do that.
>
>
> (BTW - that was admirable reading / searching / responding - 7 minutes
> and it wasn't even your patch! How do you do that? :-))
>
>

On Tue, Jun 30, 2020 at 12:59 PM Miguel Duarte de Mora Barroso
<mdbarroso@redhat.com> wrote:
>
> On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com> wrote:
> >
> > On 4/6/20 9:54 AM, Daniel P. Berrangé wrote:
> > > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora
Barroso wrote:
> > >> Hi all,
> > >>
> > >> I'm aware that it is possible to plug pre-created macvtap
devices to
> > >> libvirt guests - tracked in RFE [0].
> > >>
> > >> My interpretation of the wording in [1] and [2] is that it is
also
> > >> possible to plug pre-created tap devices into libvirt guests
- that
> > >> would be a requirement to allow kubevirt to run with less
capabilities
> > >> in the pods that encapsulate the VMs.
> > >>
> > >> I took a look at the libvirt code ([3] & [4]), and, from
my limited
> > >> understanding, I got the impression that plugging existing
interfaces
> > >> via `managed='no' ` is only possible for macvtap
interfaces.
> >
> >
> > No, it works for standard tap devices as well.
> >
> >
> > The reason the BZs and commit logs talk mostly about macvtap rather
than
> > tap is because 1) that's what kubevirt people had asked for and 2)
it
> > already *mostly* worked for tap devices, so most of the work was
related
> > to macvtap (my memory is already fuzzy, but I think there were a
couple
> > privileged operations we still tried to do for standard tap devices
even
> > if they were precreated (standard disclaimer: I often misremember, so
> > this memory could be wrong! But definitely precreated tap devices do
work).
> >
>
> It's been a while since I've started this thread, but lately
I've
> understood better how tap devices work, and that new insight makes me
> wonder about a couple of things.
>
> Our ultimate goal In kubevirt is to consume a pre-created tap device
> by a kubernetes pod that doesn't have the NET_ADMIN capability.
>
> After looking at the current libvirt code, I don't think that is
> currently supported, since we'll *always* enter the
> `virNetDevTapCreate` function in [1] (I'm interested in the *tap*
> scenario).
>
> The tap device is effectively created in that function - [2] - by
> opening the clone device (/dev/net/tun), and calling `ioctl(fd,
> TUNSETIFF,...)` in it. AFAIK, both of those operations *require* the
> NET_ADMIN capability. If I'm correct, this means that the current
> libvirt implementation makes our goals impossible to achieve.
>
> I'd first like to know if I read the code correctly, and if I'm
> ultimately right - i.e. since libvirt's implementation for consuming a
> pre-existent tap device opens the clone device, where ever libvirt is
> run on will  *always* require the NET_ADMIN capability.
Adding the links that I've forgotten to add ...

[0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367
[1] -
https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_interface.c#L443
[2] -
https://github.com/libvirt/libvirt/blob/master/src/util/virnetdevtap.c#L243
>
>
> >
> > I think though that when someone from kubevirt actually tried using a
> > precreated macvtap device, they found that their precreated device
> > wasn't visible at all to the unprivileged libvirtd in the pod,
because
> > it was in a different network namespace, or something like that. So
> > there may still be more work to do (or, again, my info might be out of
> > date and they figured out a proper solution).
> >
> >
> > >>
> > >> Would you be able to shed some light into this ? Is it
possible on
> > >> libvirt-5.6.0 to plug pre-created tap devices to libvirt
guests ?
> > >>
> > >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367
> > > This links to the following message, which illustrates how to use
pre-create
> > > tap and macvtap devices:
> > >
> > >   
https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html
> > >
> > > Laine: it would be useful to add something like this short guide
to the
> > > knowledge base docs
> >
> >
> > You mean the wiki? Sure, I can do that.
> >
> >
> > (BTW - that was admirable reading / searching / responding - 7 minutes
> > and it wasn't even your patch! How do you do that? :-))
> >
> >
>
> On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com> wrote:
> >
> > On 4/6/20 9:54 AM, Daniel P. Berrangé wrote:
> > > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora
Barroso wrote:
> > >> Hi all,
> > >>
> > >> I'm aware that it is possible to plug pre-created macvtap
devices to
> > >> libvirt guests - tracked in RFE [0].
> > >>
> > >> My interpretation of the wording in [1] and [2] is that it is
also
> > >> possible to plug pre-created tap devices into libvirt guests
- that
> > >> would be a requirement to allow kubevirt to run with less
capabilities
> > >> in the pods that encapsulate the VMs.
> > >>
> > >> I took a look at the libvirt code ([3] & [4]), and, from
my limited
> > >> understanding, I got the impression that plugging existing
interfaces
> > >> via `managed='no' ` is only possible for macvtap
interfaces.
> >
> >
> > No, it works for standard tap devices as well.
> >
> >
> > The reason the BZs and commit logs talk mostly about macvtap rather
than
> > tap is because 1) that's what kubevirt people had asked for and 2)
it
> > already *mostly* worked for tap devices, so most of the work was
related
> > to macvtap (my memory is already fuzzy, but I think there were a
couple
> > privileged operations we still tried to do for standard tap devices
even
> > if they were precreated (standard disclaimer: I often misremember, so
> > this memory could be wrong! But definitely precreated tap devices do
work).
> >
> >
> > I think though that when someone from kubevirt actually tried using a
> > precreated macvtap device, they found that their precreated device
> > wasn't visible at all to the unprivileged libvirtd in the pod,
because
> > it was in a different network namespace, or something like that. So
> > there may still be more work to do (or, again, my info might be out of
> > date and they figured out a proper solution).
> >
> >
> > >>
> > >> Would you be able to shed some light into this ? Is it
possible on
> > >> libvirt-5.6.0 to plug pre-created tap devices to libvirt
guests ?
> > >>
> > >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367
> > > This links to the following message, which illustrates how to use
pre-create
> > > tap and macvtap devices:
> > >
> > >   
https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html
> > >
> > > Laine: it would be useful to add something like this short guide
to the
> > > knowledge base docs
> >
> >
> > You mean the wiki? Sure, I can do that.
> >
> >
> > (BTW - that was admirable reading / searching / responding - 7 minutes
> > and it wasn't even your patch! How do you do that? :-))
> >
> >

On Tue, Jun 30, 2020 at 12:59:03PM +0200, Miguel Duarte de Mora Barroso
wrote:
> On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com> wrote:
> >
> > On 4/6/20 9:54 AM, Daniel P. Berrangé wrote:
> > > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora
Barroso wrote:
> > >> Hi all,
> > >>
> > >> I'm aware that it is possible to plug pre-created macvtap
devices to
> > >> libvirt guests - tracked in RFE [0].
> > >>
> > >> My interpretation of the wording in [1] and [2] is that it is
also
> > >> possible to plug pre-created tap devices into libvirt guests
- that
> > >> would be a requirement to allow kubevirt to run with less
capabilities
> > >> in the pods that encapsulate the VMs.
> > >>
> > >> I took a look at the libvirt code ([3] & [4]), and, from
my limited
> > >> understanding, I got the impression that plugging existing
interfaces
> > >> via `managed='no' ` is only possible for macvtap
interfaces.
> >
> >
> > No, it works for standard tap devices as well.
> >
> >
> > The reason the BZs and commit logs talk mostly about macvtap rather
than
> > tap is because 1) that's what kubevirt people had asked for and 2)
it
> > already *mostly* worked for tap devices, so most of the work was
related
> > to macvtap (my memory is already fuzzy, but I think there were a
couple
> > privileged operations we still tried to do for standard tap devices
even
> > if they were precreated (standard disclaimer: I often misremember, so
> > this memory could be wrong! But definitely precreated tap devices do
work).
> >
> 
> It's been a while since I've started this thread, but lately
I've
> understood better how tap devices work, and that new insight makes me
> wonder about a couple of things.
> 
> Our ultimate goal In kubevirt is to consume a pre-created tap device
> by a kubernetes pod that doesn't have the NET_ADMIN capability.
> 
> After looking at the current libvirt code, I don't think that is
> currently supported, since we'll *always* enter the
> `virNetDevTapCreate` function in [1] (I'm interested in the *tap*
> scenario).
> 
> The tap device is effectively created in that function - [2] - by
> opening the clone device (/dev/net/tun), and calling `ioctl(fd,
> TUNSETIFF,...)` in it. AFAIK, both of those operations *require* the
> NET_ADMIN capability. If I'm correct, this means that the current
> libvirt implementation makes our goals impossible to achieve.
AFAIK, that is not correct - CAP_NET_ADMIN isn't required to open
or create a tap device - only to add the tap device to a bridge.

So if you create the tap device & attach it to a bridge ahead of
time, libvirt should then be able to open it and give it to QEMU


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|