On 4/6/20 9:54 AM, Daniel P. Berrangé wrote:> On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora Barroso wrote: >> Hi all, >> >> I'm aware that it is possible to plug pre-created macvtap devices to >> libvirt guests - tracked in RFE [0]. >> >> My interpretation of the wording in [1] and [2] is that it is also >> possible to plug pre-created tap devices into libvirt guests - that >> would be a requirement to allow kubevirt to run with less capabilities >> in the pods that encapsulate the VMs. >> >> I took a look at the libvirt code ([3] & [4]), and, from my limited >> understanding, I got the impression that plugging existing interfaces >> via `managed='no' ` is only possible for macvtap interfaces.No, it works for standard tap devices as well. The reason the BZs and commit logs talk mostly about macvtap rather than tap is because 1) that's what kubevirt people had asked for and 2) it already *mostly* worked for tap devices, so most of the work was related to macvtap (my memory is already fuzzy, but I think there were a couple privileged operations we still tried to do for standard tap devices even if they were precreated (standard disclaimer: I often misremember, so this memory could be wrong! But definitely precreated tap devices do work). I think though that when someone from kubevirt actually tried using a precreated macvtap device, they found that their precreated device wasn't visible at all to the unprivileged libvirtd in the pod, because it was in a different network namespace, or something like that. So there may still be more work to do (or, again, my info might be out of date and they figured out a proper solution).>> >> Would you be able to shed some light into this ? Is it possible on >> libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ? >> >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367 > This links to the following message, which illustrates how to use pre-create > tap and macvtap devices: > > https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html > > Laine: it would be useful to add something like this short guide to the > knowledge base docsYou mean the wiki? Sure, I can do that. (BTW - that was admirable reading / searching / responding - 7 minutes and it wasn't even your patch! How do you do that? :-))
Daniel P. Berrangé
2020-Apr-06 14:05 UTC
Re: plug pre-created tap devices to libvirt guests
On Mon, Apr 06, 2020 at 10:03:41AM -0400, Laine Stump wrote:> On 4/6/20 9:54 AM, Daniel P. Berrangé wrote: > > > Would you be able to shed some light into this ? Is it possible on > > > libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ? > > > > > > [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367 > > This links to the following message, which illustrates how to use pre-create > > tap and macvtap devices: > > > > https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html > > > > Laine: it would be useful to add something like this short guide to the > > knowledge base docs > > > You mean the wiki? Sure, I can do that.No, i mean docs/kbase/> (BTW - that was admirable reading / searching / responding - 7 minutes and > it wasn't even your patch! How do you do that? :-))I just read the BZ & followed the link :-) Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Miguel Duarte de Mora Barroso
2020-Jun-30 10:59 UTC
Re: plug pre-created tap devices to libvirt guests
On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com> wrote:> > On 4/6/20 9:54 AM, Daniel P. Berrangé wrote: > > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora Barroso wrote: > >> Hi all, > >> > >> I'm aware that it is possible to plug pre-created macvtap devices to > >> libvirt guests - tracked in RFE [0]. > >> > >> My interpretation of the wording in [1] and [2] is that it is also > >> possible to plug pre-created tap devices into libvirt guests - that > >> would be a requirement to allow kubevirt to run with less capabilities > >> in the pods that encapsulate the VMs. > >> > >> I took a look at the libvirt code ([3] & [4]), and, from my limited > >> understanding, I got the impression that plugging existing interfaces > >> via `managed='no' ` is only possible for macvtap interfaces. > > > No, it works for standard tap devices as well. > > > The reason the BZs and commit logs talk mostly about macvtap rather than > tap is because 1) that's what kubevirt people had asked for and 2) it > already *mostly* worked for tap devices, so most of the work was related > to macvtap (my memory is already fuzzy, but I think there were a couple > privileged operations we still tried to do for standard tap devices even > if they were precreated (standard disclaimer: I often misremember, so > this memory could be wrong! But definitely precreated tap devices do work). >It's been a while since I've started this thread, but lately I've understood better how tap devices work, and that new insight makes me wonder about a couple of things. Our ultimate goal In kubevirt is to consume a pre-created tap device by a kubernetes pod that doesn't have the NET_ADMIN capability. After looking at the current libvirt code, I don't think that is currently supported, since we'll *always* enter the `virNetDevTapCreate` function in [1] (I'm interested in the *tap* scenario). The tap device is effectively created in that function - [2] - by opening the clone device (/dev/net/tun), and calling `ioctl(fd, TUNSETIFF,...)` in it. AFAIK, both of those operations *require* the NET_ADMIN capability. If I'm correct, this means that the current libvirt implementation makes our goals impossible to achieve. I'd first like to know if I read the code correctly, and if I'm ultimately right - i.e. since libvirt's implementation for consuming a pre-existent tap device opens the clone device, where ever libvirt is run on will *always* require the NET_ADMIN capability.> > I think though that when someone from kubevirt actually tried using a > precreated macvtap device, they found that their precreated device > wasn't visible at all to the unprivileged libvirtd in the pod, because > it was in a different network namespace, or something like that. So > there may still be more work to do (or, again, my info might be out of > date and they figured out a proper solution). > > > >> > >> Would you be able to shed some light into this ? Is it possible on > >> libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ? > >> > >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367 > > This links to the following message, which illustrates how to use pre-create > > tap and macvtap devices: > > > > https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html > > > > Laine: it would be useful to add something like this short guide to the > > knowledge base docs > > > You mean the wiki? Sure, I can do that. > > > (BTW - that was admirable reading / searching / responding - 7 minutes > and it wasn't even your patch! How do you do that? :-)) > >On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com> wrote:> > On 4/6/20 9:54 AM, Daniel P. Berrangé wrote: > > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora Barroso wrote: > >> Hi all, > >> > >> I'm aware that it is possible to plug pre-created macvtap devices to > >> libvirt guests - tracked in RFE [0]. > >> > >> My interpretation of the wording in [1] and [2] is that it is also > >> possible to plug pre-created tap devices into libvirt guests - that > >> would be a requirement to allow kubevirt to run with less capabilities > >> in the pods that encapsulate the VMs. > >> > >> I took a look at the libvirt code ([3] & [4]), and, from my limited > >> understanding, I got the impression that plugging existing interfaces > >> via `managed='no' ` is only possible for macvtap interfaces. > > > No, it works for standard tap devices as well. > > > The reason the BZs and commit logs talk mostly about macvtap rather than > tap is because 1) that's what kubevirt people had asked for and 2) it > already *mostly* worked for tap devices, so most of the work was related > to macvtap (my memory is already fuzzy, but I think there were a couple > privileged operations we still tried to do for standard tap devices even > if they were precreated (standard disclaimer: I often misremember, so > this memory could be wrong! But definitely precreated tap devices do work). > > > I think though that when someone from kubevirt actually tried using a > precreated macvtap device, they found that their precreated device > wasn't visible at all to the unprivileged libvirtd in the pod, because > it was in a different network namespace, or something like that. So > there may still be more work to do (or, again, my info might be out of > date and they figured out a proper solution). > > > >> > >> Would you be able to shed some light into this ? Is it possible on > >> libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ? > >> > >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367 > > This links to the following message, which illustrates how to use pre-create > > tap and macvtap devices: > > > > https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html > > > > Laine: it would be useful to add something like this short guide to the > > knowledge base docs > > > You mean the wiki? Sure, I can do that. > > > (BTW - that was admirable reading / searching / responding - 7 minutes > and it wasn't even your patch! How do you do that? :-)) > >
Miguel Duarte de Mora Barroso
2020-Jun-30 11:02 UTC
Re: plug pre-created tap devices to libvirt guests
On Tue, Jun 30, 2020 at 12:59 PM Miguel Duarte de Mora Barroso <mdbarroso@redhat.com> wrote:> > On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com> wrote: > > > > On 4/6/20 9:54 AM, Daniel P. Berrangé wrote: > > > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora Barroso wrote: > > >> Hi all, > > >> > > >> I'm aware that it is possible to plug pre-created macvtap devices to > > >> libvirt guests - tracked in RFE [0]. > > >> > > >> My interpretation of the wording in [1] and [2] is that it is also > > >> possible to plug pre-created tap devices into libvirt guests - that > > >> would be a requirement to allow kubevirt to run with less capabilities > > >> in the pods that encapsulate the VMs. > > >> > > >> I took a look at the libvirt code ([3] & [4]), and, from my limited > > >> understanding, I got the impression that plugging existing interfaces > > >> via `managed='no' ` is only possible for macvtap interfaces. > > > > > > No, it works for standard tap devices as well. > > > > > > The reason the BZs and commit logs talk mostly about macvtap rather than > > tap is because 1) that's what kubevirt people had asked for and 2) it > > already *mostly* worked for tap devices, so most of the work was related > > to macvtap (my memory is already fuzzy, but I think there were a couple > > privileged operations we still tried to do for standard tap devices even > > if they were precreated (standard disclaimer: I often misremember, so > > this memory could be wrong! But definitely precreated tap devices do work). > > > > It's been a while since I've started this thread, but lately I've > understood better how tap devices work, and that new insight makes me > wonder about a couple of things. > > Our ultimate goal In kubevirt is to consume a pre-created tap device > by a kubernetes pod that doesn't have the NET_ADMIN capability. > > After looking at the current libvirt code, I don't think that is > currently supported, since we'll *always* enter the > `virNetDevTapCreate` function in [1] (I'm interested in the *tap* > scenario). > > The tap device is effectively created in that function - [2] - by > opening the clone device (/dev/net/tun), and calling `ioctl(fd, > TUNSETIFF,...)` in it. AFAIK, both of those operations *require* the > NET_ADMIN capability. If I'm correct, this means that the current > libvirt implementation makes our goals impossible to achieve. > > I'd first like to know if I read the code correctly, and if I'm > ultimately right - i.e. since libvirt's implementation for consuming a > pre-existent tap device opens the clone device, where ever libvirt is > run on will *always* require the NET_ADMIN capability.Adding the links that I've forgotten to add ... [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367 [1] - https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_interface.c#L443 [2] - https://github.com/libvirt/libvirt/blob/master/src/util/virnetdevtap.c#L243> > > > > > I think though that when someone from kubevirt actually tried using a > > precreated macvtap device, they found that their precreated device > > wasn't visible at all to the unprivileged libvirtd in the pod, because > > it was in a different network namespace, or something like that. So > > there may still be more work to do (or, again, my info might be out of > > date and they figured out a proper solution). > > > > > > >> > > >> Would you be able to shed some light into this ? Is it possible on > > >> libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ? > > >> > > >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367 > > > This links to the following message, which illustrates how to use pre-create > > > tap and macvtap devices: > > > > > > https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html > > > > > > Laine: it would be useful to add something like this short guide to the > > > knowledge base docs > > > > > > You mean the wiki? Sure, I can do that. > > > > > > (BTW - that was admirable reading / searching / responding - 7 minutes > > and it wasn't even your patch! How do you do that? :-)) > > > > > > On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com> wrote: > > > > On 4/6/20 9:54 AM, Daniel P. Berrangé wrote: > > > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora Barroso wrote: > > >> Hi all, > > >> > > >> I'm aware that it is possible to plug pre-created macvtap devices to > > >> libvirt guests - tracked in RFE [0]. > > >> > > >> My interpretation of the wording in [1] and [2] is that it is also > > >> possible to plug pre-created tap devices into libvirt guests - that > > >> would be a requirement to allow kubevirt to run with less capabilities > > >> in the pods that encapsulate the VMs. > > >> > > >> I took a look at the libvirt code ([3] & [4]), and, from my limited > > >> understanding, I got the impression that plugging existing interfaces > > >> via `managed='no' ` is only possible for macvtap interfaces. > > > > > > No, it works for standard tap devices as well. > > > > > > The reason the BZs and commit logs talk mostly about macvtap rather than > > tap is because 1) that's what kubevirt people had asked for and 2) it > > already *mostly* worked for tap devices, so most of the work was related > > to macvtap (my memory is already fuzzy, but I think there were a couple > > privileged operations we still tried to do for standard tap devices even > > if they were precreated (standard disclaimer: I often misremember, so > > this memory could be wrong! But definitely precreated tap devices do work). > > > > > > I think though that when someone from kubevirt actually tried using a > > precreated macvtap device, they found that their precreated device > > wasn't visible at all to the unprivileged libvirtd in the pod, because > > it was in a different network namespace, or something like that. So > > there may still be more work to do (or, again, my info might be out of > > date and they figured out a proper solution). > > > > > > >> > > >> Would you be able to shed some light into this ? Is it possible on > > >> libvirt-5.6.0 to plug pre-created tap devices to libvirt guests ? > > >> > > >> [0] - https://bugzilla.redhat.com/show_bug.cgi?id=1723367 > > > This links to the following message, which illustrates how to use pre-create > > > tap and macvtap devices: > > > > > > https://www.redhat.com/archives/libvir-list/2019-August/msg01256.html > > > > > > Laine: it would be useful to add something like this short guide to the > > > knowledge base docs > > > > > > You mean the wiki? Sure, I can do that. > > > > > > (BTW - that was admirable reading / searching / responding - 7 minutes > > and it wasn't even your patch! How do you do that? :-)) > > > >
Daniel P. Berrangé
2020-Jun-30 15:02 UTC
Re: plug pre-created tap devices to libvirt guests
On Tue, Jun 30, 2020 at 12:59:03PM +0200, Miguel Duarte de Mora Barroso wrote:> On Mon, Apr 6, 2020 at 4:03 PM Laine Stump <lstump@redhat.com> wrote: > > > > On 4/6/20 9:54 AM, Daniel P. Berrangé wrote: > > > On Mon, Apr 06, 2020 at 03:47:01PM +0200, Miguel Duarte de Mora Barroso wrote: > > >> Hi all, > > >> > > >> I'm aware that it is possible to plug pre-created macvtap devices to > > >> libvirt guests - tracked in RFE [0]. > > >> > > >> My interpretation of the wording in [1] and [2] is that it is also > > >> possible to plug pre-created tap devices into libvirt guests - that > > >> would be a requirement to allow kubevirt to run with less capabilities > > >> in the pods that encapsulate the VMs. > > >> > > >> I took a look at the libvirt code ([3] & [4]), and, from my limited > > >> understanding, I got the impression that plugging existing interfaces > > >> via `managed='no' ` is only possible for macvtap interfaces. > > > > > > No, it works for standard tap devices as well. > > > > > > The reason the BZs and commit logs talk mostly about macvtap rather than > > tap is because 1) that's what kubevirt people had asked for and 2) it > > already *mostly* worked for tap devices, so most of the work was related > > to macvtap (my memory is already fuzzy, but I think there were a couple > > privileged operations we still tried to do for standard tap devices even > > if they were precreated (standard disclaimer: I often misremember, so > > this memory could be wrong! But definitely precreated tap devices do work). > > > > It's been a while since I've started this thread, but lately I've > understood better how tap devices work, and that new insight makes me > wonder about a couple of things. > > Our ultimate goal In kubevirt is to consume a pre-created tap device > by a kubernetes pod that doesn't have the NET_ADMIN capability. > > After looking at the current libvirt code, I don't think that is > currently supported, since we'll *always* enter the > `virNetDevTapCreate` function in [1] (I'm interested in the *tap* > scenario). > > The tap device is effectively created in that function - [2] - by > opening the clone device (/dev/net/tun), and calling `ioctl(fd, > TUNSETIFF,...)` in it. AFAIK, both of those operations *require* the > NET_ADMIN capability. If I'm correct, this means that the current > libvirt implementation makes our goals impossible to achieve.AFAIK, that is not correct - CAP_NET_ADMIN isn't required to open or create a tap device - only to add the tap device to a bridge. So if you create the tap device & attach it to a bridge ahead of time, libvirt should then be able to open it and give it to QEMU Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|