thr3ads.net - libvirt users - multiple vms with same PCI passthrough [Aug 2020]

If this information is useful, please help other people find it:
Share via:

Daniel Black

2020-Aug-09 03:53 UTC

multiple vms with same PCI passthrough

In attempting to isolate vfio-pci problems between two different guest
instances, the creation of a second guest (with existing guest shutdown)
resulted in:.

Aug 09 12:43:23 grit libvirtd[6716]: internal error: Device 0000:01:00.3 is
already in use
Aug 09 12:43:23 grit libvirtd[6716]: internal error: Device 0000:01:00.3 is
already in use
Aug 09 12:43:23 grit libvirtd[6716]: Failed to allocate PCI device list:
internal error: Device 0000:01:00.3 is already in use

Compiled against library: libvirt 6.1.0
Using library: libvirt 6.1.0
Using API: QEMU 6.1.0
Running hypervisor: QEMU 4.2.1
(fc32 default install)

The upstream code seems  also to test definitions rather than active uses
of the PCI device.

My potentially naive patch to correct this (but not the failing test cases)
would be:

diff --git a/src/util/virpci.c b/src/util/virpci.c
index 47c671daa0..a00c5e6f44 100644
--- a/src/util/virpci.c
+++ b/src/util/virpci.c
@@ -1597,7 +1597,7 @@ int
 virPCIDeviceListAdd(virPCIDeviceListPtr list,
                     virPCIDevicePtr dev)
 {
-    if (virPCIDeviceListFind(list, dev)) {
+    if (virPCIDeviceBusContainsActiveDevices(dev, list)) {
         virReportError(VIR_ERR_INTERNAL_ERROR,
                        _("Device %s is already in use"),
dev->name);
         return -1;

Is this too simplistic or undesirable a feature request/implementation?

I'd be more than grateful if someone carries this through as I'm unsure
when I may get time for this.

Daniel Black

2020-Aug-16 01:46 UTC

head link

multiple vms with same PCI passthrough

In attempting to isolate vfio-pci problems between two different guest
instances, the creation of a second guest (with existing guest shutdown)
resulted in:.

Aug 09 12:43:23 grit libvirtd[6716]: internal error: Device 0000:01:00.3 is
already in use
Aug 09 12:43:23 grit libvirtd[6716]: internal error: Device 0000:01:00.3 is
already in use
Aug 09 12:43:23 grit libvirtd[6716]: Failed to allocate PCI device list:
internal error: Device 0000:01:00.3 is already in use

Compiled against library: libvirt 6.1.0
Using library: libvirt 6.1.0
Using API: QEMU 6.1.0
Running hypervisor: QEMU 4.2.1
(fc32 default install)

The upstream code seems  also to test definitions rather than active uses
of the PCI device.

My potentially naive patch to correct this (but not the failing test cases)
would be:

diff --git a/src/util/virpci.c b/src/util/virpci.c
index 47c671daa0..a00c5e6f44 100644
--- a/src/util/virpci.c
+++ b/src/util/virpci.c
@@ -1597,7 +1597,7 @@ int
 virPCIDeviceListAdd(virPCIDeviceListPtr list,
                     virPCIDevicePtr dev)
 {
-    if (virPCIDeviceListFind(list, dev)) {
+    if (virPCIDeviceBusContainsActiveDevices(dev, list)) {
         virReportError(VIR_ERR_INTERNAL_ERROR,
                        _("Device %s is already in use"),
dev->name);
         return -1;

Is this too simplistic or undesirable a feature request/implementation?

I'd be grateful if someone carries this through as I'm unsure when I may
get time for this (busy handling other open source ecosystems). With an
install and test mechanisms that doesn't create a mess of overwritten
packages I"m happy to test a git branch or patches.

Laine Stump

2020-Aug-17 15:00 UTC

head link

Re: multiple vms with same PCI passthrough

On 8/8/20 11:53 PM, Daniel Black wrote:> 
> In attempting to isolate vfio-pci problems between two different guest 
> instances, the creation of a second guest (with existing guest shutdown) 
> resulted in:.
> 
> Aug 09 12:43:23 grit libvirtd[6716]: internal error: Device 0000:01:00.3 
> is already in use
> Aug 09 12:43:23 grit libvirtd[6716]: internal error: Device 0000:01:00.3 
> is already in use
> Aug 09 12:43:23 grit libvirtd[6716]: Failed to allocate PCI device list: 
> internal error: Device 0000:01:00.3 is already in use
Hmm. Normally the error that would be logged if a device is already in 
use would say something like this:

error: Failed to start domain Win10-GPU
error: Requested operation is not valid: PCI device 0000:05:00.0 is in
        use by driver QEMU, domain F30

So you're encountering this in an unexpected place.
> 
> Compiled against library: libvirt 6.1.0
> Using library: libvirt 6.1.0
> Using API: QEMU 6.1.0
> Running hypervisor: QEMU 4.2.1
> (fc32 default install)
> 
> The upstream code seems  also to test definitions rather than active 
> uses of the PCI device.

That isn't the case. You're misunderstanding what devices are on the 
list. (see below for details)
> 
> My potentially naive patch to correct this (but not the failing test 
> cases) would be:
> 
> diff --git a/src/util/virpci.c b/src/util/virpci.c
> index 47c671daa0..a00c5e6f44 100644
> --- a/src/util/virpci.c
> +++ b/src/util/virpci.c
> @@ -1597,7 +1597,7 @@ int
>   virPCIDeviceListAdd(virPCIDeviceListPtr list,
>                       virPCIDevicePtr dev)
>   {
> -    if (virPCIDeviceListFind(list, dev)) {
> +    if (virPCIDeviceBusContainsActiveDevices(dev, list)) {
>           virReportError(VIR_ERR_INTERNAL_ERROR,
>                          _("Device %s is already in use"),
dev->name);
>           return -1;
> 
> Is this too simplistic or undesirable a feature request/implementation?
Only devices that are currently in use by a guest (activePCIHostdevs), 
or that libvirt is in the process of detaching from the guest + vfio and 
rebinding to the device's host driver (inactivePCIHostdevs) are on 
either list of PCI devices maintained by libvirt. Once a device is 
completely detached from the guest and (if "managed='yes'" was
set in
the XML config) re-binded to the natural host driver for the device, it 
is removed from the list and can be used elsewhere.

I just tested this with an assigned GPU + soundcard on two guests to 
verify that it works properly. (I'm running the latest upstream master 
though, so it's not an exact replication of your test)

> 
> I'd be more than grateful if someone carries this through as I'm
unsure
> when I may get time for this.

Can you provide the XML for your <hostdev> in the two guests, and the 
exact sequence of commands that lead to this error? There is definitely 
either a bug in the code, or a bug in what you're doing. By seeing the 
sequence of events, we can either attempt to replicate it, or let you 
know what change you need to make to your workflow to eliminate the error.

Daniel Black

2020-Aug-18 00:40 UTC

head link

Re: multiple vms with same PCI passthrough

Thanks Laine,

Mea Culpa,

Couldn't reproduce, or found the multiple duplicate entries in the guest
pci domain/bus/slot/function space which got resolved easily.




[root@grit tmp]# virsh list
 Id   Name   State
--------------------

[root@grit tmp]# virsh list --all
 Id   Name          State
------------------------------
 -    openbsd6.7    shut off
 -    openindiana   shut off
 -    proxmox6.2    shut off
 -    ubuntu20.04   shut off
 -    win2k19       shut off

[root@grit tmp]# virsh version
Compiled against library: libvirt 6.1.0
Using library: libvirt 6.1.0
Using API: QEMU 6.1.0
Running hypervisor: QEMU 4.2.1

[root@grit tmp]# virsh dumpxml ubuntu20.04 | gzip -c >  ubunt2004.xml.gz
[root@grit tmp]# chown dan: ubunt2004.xml.gz
[root@grit tmp]# virsh edit win2k19

(paste the 4 <hostdev mode='subsystem' type='pci'
managed='yes'> devices
from ubuntu2004 to the devices section)
save and exit.

error: XML error: Attempted double use of PCI Address 0000:08:00.0
Failed. Try again? [y,n,i,f,?]:

[root@grit tmp]# virsh dumpxml win2k19  | gzip -c >  win2k19.xml.gz


Looking closer it seemed the 0000:08:00.0 was the guest address that
already had a device defined on it.


Removing those in virt-manager and readding them generated a warning that
they were in another definition, however
it did accept them.

This, for 4 pci devices, confused libvrtd in the meantime however it was
still functional.

Aug 18 10:31:27 grit libvirtd[106082]: internal error: failed to get number
of host interfaces: unspecified error - errors in loading some config files
Aug 18 10:31:55 grit libvirtd[106082]: internal error: failed to get number
of host interfaces: unspecified error - errors in loading some config files
Aug 18 10:32:17 grit libvirtd[106082]: internal error: failed to get number
of host interfaces: unspecified error - errors in loading some config files
Aug 18 10:32:32 grit libvirtd[106082]: internal error: failed to get number
of host interfaces: unspecified error - errors in loading some config files




On Tue, Aug 18, 2020 at 1:00 AM Laine Stump <laine@redhat.com> wrote:
> On 8/8/20 11:53 PM, Daniel Black wrote:
> >
> > In attempting to isolate vfio-pci problems between two different guest
> > instances, the creation of a second guest (with existing guest
shutdown)
> > resulted in:.
> >
> > Aug 09 12:43:23 grit libvirtd[6716]: internal error: Device
0000:01:00.3
> > is already in use
> > Aug 09 12:43:23 grit libvirtd[6716]: internal error: Device
0000:01:00.3
> > is already in use
> > Aug 09 12:43:23 grit libvirtd[6716]: Failed to allocate PCI device
list:
> > internal error: Device 0000:01:00.3 is already in use
>
> Hmm. Normally the error that would be logged if a device is already in
> use would say something like this:
>
> error: Failed to start domain Win10-GPU
> error: Requested operation is not valid: PCI device 0000:05:00.0 is in
>         use by driver QEMU, domain F30
>
> So you're encountering this in an unexpected place.
>
> >
> > Compiled against library: libvirt 6.1.0
> > Using library: libvirt 6.1.0
> > Using API: QEMU 6.1.0
> > Running hypervisor: QEMU 4.2.1
> > (fc32 default install)
> >
> > The upstream code seems  also to test definitions rather than active
> > uses of the PCI device.
>
>
> That isn't the case. You're misunderstanding what devices are on
the
> list. (see below for details)
>
> >
> > My potentially naive patch to correct this (but not the failing test
> > cases) would be:
> >
> > diff --git a/src/util/virpci.c b/src/util/virpci.c
> > index 47c671daa0..a00c5e6f44 100644
> > --- a/src/util/virpci.c
> > +++ b/src/util/virpci.c
> > @@ -1597,7 +1597,7 @@ int
> >   virPCIDeviceListAdd(virPCIDeviceListPtr list,
> >                       virPCIDevicePtr dev)
> >   {
> > -    if (virPCIDeviceListFind(list, dev)) {
> > +    if (virPCIDeviceBusContainsActiveDevices(dev, list)) {
> >           virReportError(VIR_ERR_INTERNAL_ERROR,
> >                          _("Device %s is already in use"),
dev->name);
> >           return -1;
> >
> > Is this too simplistic or undesirable a feature
request/implementation?
>
> Only devices that are currently in use by a guest (activePCIHostdevs),
> or that libvirt is in the process of detaching from the guest + vfio and
> rebinding to the device's host driver (inactivePCIHostdevs) are on
> either list of PCI devices maintained by libvirt. Once a device is
> completely detached from the guest and (if
"managed='yes'" was set in
> the XML config) re-binded to the natural host driver for the device, it
> is removed from the list and can be used elsewhere.
>
> I just tested this with an assigned GPU + soundcard on two guests to
> verify that it works properly. (I'm running the latest upstream master
> though, so it's not an exact replication of your test)
>
>
> >
> > I'd be more than grateful if someone carries this through as
I'm unsure
> > when I may get time for this.
>
>
> Can you provide the XML for your <hostdev> in the two guests, and the
> exact sequence of commands that lead to this error? There is definitely
> either a bug in the code, or a bug in what you're doing. By seeing the
> sequence of events, we can either attempt to replicate it, or let you
> know what change you need to make to your workflow to eliminate the error.
>
>

Seemingly Similar Threads

Search for more seemingly similar threads

libvirt users - Aug 2020 - multiple vms with same PCI passthrough

multiple vms with same PCI passthrough

multiple vms with same PCI passthrough

Re: multiple vms with same PCI passthrough

Re: multiple vms with same PCI passthrough

Seemingly Similar Threads