libvirt users - Oct 2018 - Re: pcie-expander-bus doesn't support pcie-pci-bridge and pcie-switch-upstream-port

In libvirt, I found pcie-expander-bus controller doesn't support
pcie-to-pci-bridge and pcie-switch-upstream-port.

Version: libvirt-4.9

# cat /tmp/c.xml
...
    <controller type='pci' index='0'
model='pcie-root'/>
    <controller type='pci' index='1'
model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='3'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
    </controller>
    <controller type='pci' index='2'
model='pcie-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01'
slot='0x00'
function='0x0'/>
    </controller>
...
# virsh -k0 -K0 define
/tmp/c.xml

error: Failed to define domain from /tmp/c.xml
error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged
into the PCI controller with index='1'. It requires a controller that
accepts a pcie-to-pci-bridge.

# cat /tmp/b.xml
...
    <controller type='pci' index='0'
model='pcie-root'/>
    <controller type='pci' index='1'
model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='3'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
    </controller>
    <controller type='pci' index='2'
model='pcie-switch-upstream-port'>
      <model name='x3130-upstream'/>
      <address type='pci' domain='0x0000' bus='0x01'
slot='0x00'
function='0x0'/>
    </controller>
...
# virsh -k0 -K0 define /tmp/b.xml
error: Failed to define domain from /tmp/b.xml
error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged
into the PCI controller with index='1'. It requires a controller that
accepts a pci-switch-upstream-port.

In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus only
supports pcie-root-port and dmi-to-pci-bridge
   353     case
VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS:
   354     ┆   /* 32 slots, no hotplug, only accepts pcie-root-port
or
   355     ┆   ┆*
dmi-to-pci-bridge
   356     ┆
┆*/
   357     ┆   bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT
|
   358     ┆   ┆   ┆   ┆   ┆
VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE);
   359     ┆   bus->minSlot 0;
   360     ┆   bus->maxSlot VIR_PCI_ADDRESS_SLOT_LAST;
   361     ┆   break;

But it works in qemu:
# /usr/libexec/qemu-kvm -machine q35 -m 1024 -device
pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device
pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device
pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice
port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2

So why pcie-expander-bus doesn't these two controllers? For any concerns in
libvirt?
Thank you~


-- 
Best regards,
-----------------------------------
Han Han
Quality Engineer
Redhat.

Email: hhan@redhat.com
Phone: +861065339333

On Wed, 2018-10-17 at 10:50 +0800, Han Han wrote:
> In libvirt, I found pcie-expander-bus controller doesn't support
pcie-to-pci-bridge and pcie-switch-upstream-port.
[...]
> # virsh -k0 -K0 define /tmp/c.xml
Aside: the -k and -K virsh options are documented as

  -k | --keepalive-interval=NUM
                      keepalive interval in seconds, 0 for disable
  -K | --keepalive-count=NUM
                      number of possible missed keepalive messages

So -k0 disables keepalive entirely, making -K0 unnecessary :)
> error: Failed to define domain from /tmp/c.xml
> error: XML error: The device at PCI address 0000:01:00.0 cannot be plugged
into the PCI controller with index='1'. It requires a controller that
accepts a pcie-to-pci-bridge.
[...]
> In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus only
supports pcie-root-port and dmi-to-pci-bridge
>    353     case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS:
>    354     ┆   /* 32 slots, no hotplug, only accepts pcie-root-port or
>    355     ┆   ┆* dmi-to-pci-bridge
>    356     ┆   ┆*/
>    357     ┆   bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT |
>    358     ┆   ┆   ┆   ┆   ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE);
>    359     ┆   bus->minSlot = 0;
>    360     ┆   bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST;
>    361     ┆   break;
The above is consistent with pcie-root itself, which also doesn't
support plugging a pcie-to-pci-bridge directly into it but requires
a pcie-root <- pcie-root-port <- pcie-to-pci-bridge topology.

Now, I don't quite recall *why* that is the case - perhaps Laine
does? - but I'm sure we had very compelling reasons O:-)
> But it works in qemu:
> # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device
pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device
pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device
pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice
port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2
It should be noted that QEMU in general allows users to make pretty
unwise device placement choices without so much of a warning, so I
would not take the above as proof the pcie-to-pci-bridge should be
allowed to plug into pcie-expander-bus (or pcie-root) directly :)

-- 
Andrea Bolognani / Red Hat / Virtualization

On 10/17/2018 08:56 AM, Andrea Bolognani wrote:
> On Wed, 2018-10-17 at 10:50 +0800, Han Han wrote:
>> In libvirt, I found pcie-expander-bus controller doesn't support
pcie-to-pci-bridge and pcie-switch-upstream-port.
> [...]
>> # virsh -k0 -K0 define /tmp/c.xml
> Aside: the -k and -K virsh options are documented as
>
>   -k | --keepalive-interval=NUM
>                       keepalive interval in seconds, 0 for disable
>   -K | --keepalive-count=NUM
>                       number of possible missed keepalive messages
>
> So -k0 disables keepalive entirely, making -K0 unnecessary :)
>
>> error: Failed to define domain from /tmp/c.xml
>> error: XML error: The device at PCI address 0000:01:00.0 cannot be
plugged into the PCI controller with index='1'. It requires a controller
that accepts a pcie-to-pci-bridge.
> [...]
>> In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus
only supports pcie-root-port and dmi-to-pci-bridge
>>    353     case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS:
>>    354     ┆   /* 32 slots, no hotplug, only accepts pcie-root-port or
>>    355     ┆   ┆* dmi-to-pci-bridge
>>    356     ┆   ┆*/
>>    357     ┆   bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT |
>>    358     ┆   ┆   ┆   ┆   ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE);
>>    359     ┆   bus->minSlot = 0;
>>    360     ┆   bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST;
>>    361     ┆   break;
> The above is consistent with pcie-root itself, which also doesn't
> support plugging a pcie-to-pci-bridge directly into it but requires
> a pcie-root <- pcie-root-port <- pcie-to-pci-bridge topology.
>
> Now, I don't quite recall *why* that is the case - perhaps Laine
> does? - but I'm sure we had very compelling reasons O:-)
Umm, because Marcel told us that was the proper behavior ? Right, Marcel?

>
>> But it works in qemu:
>> # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device
pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device
pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device
pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice
port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2
> It should be noted that QEMU in general allows users to make pretty
> unwise device placement choices without so much of a warning, so I
> would not take the above as proof the pcie-to-pci-bridge should be
> allowed to plug into pcie-expander-bus (or pcie-root) directly :)
>

On Wed, Oct 17, 2018 at 8:56 PM Andrea Bolognani <abologna@redhat.com>
wrote:
> On Wed, 2018-10-17 at 10:50 +0800, Han Han wrote:
> > In libvirt, I found pcie-expander-bus controller doesn't support
> pcie-to-pci-bridge and pcie-switch-upstream-port.
> [...]
> > # virsh -k0 -K0 define /tmp/c.xml
>
> Aside: the -k and -K virsh options are documented as
>
>   -k | --keepalive-interval=NUM
>                       keepalive interval in seconds, 0 for disable
>   -K | --keepalive-count=NUM
>                       number of possible missed keepalive messages
>
> So -k0 disables keepalive entirely, making -K0 unnecessary :)
>
> Thanks for your advice.
> > error: Failed to define domain from /tmp/c.xml
> > error: XML error: The device at PCI address 0000:01:00.0 cannot be
> plugged into the PCI controller with index='1'. It requires a
controller
> that accepts a pcie-to-pci-bridge.
> [...]
> > In function virDomainPCIAddressBusSetModel, I find pcie-expander-bus
> only supports pcie-root-port and dmi-to-pci-bridge
> >    353     case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS:
> >    354     ┆   /* 32 slots, no hotplug, only accepts pcie-root-port or
> >    355     ┆   ┆* dmi-to-pci-bridge
> >    356     ┆   ┆*/
> >    357     ┆   bus->flags = (VIR_PCI_CONNECT_TYPE_PCIE_ROOT_PORT |
> >    358     ┆   ┆   ┆   ┆   ┆ VIR_PCI_CONNECT_TYPE_DMI_TO_PCI_BRIDGE);
> >    359     ┆   bus->minSlot = 0;
> >    360     ┆   bus->maxSlot = VIR_PCI_ADDRESS_SLOT_LAST;
> >    361     ┆   break;
>
> The above is consistent with pcie-root itself, which also doesn't
> support plugging a pcie-to-pci-bridge directly into it but requires
> a pcie-root <- pcie-root-port <- pcie-to-pci-bridge topology.
>
> Now, I don't quite recall *why* that is the case - perhaps Laine
> does? - but I'm sure we had very compelling reasons O:-)
>
> > But it works in qemu:
> > # /usr/libexec/qemu-kvm -machine q35 -m 1024 -device
> pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x4 -device
> pcie-pci-bridge,id=pci.250,bus=pci.1,addr=0x0 -device
> pcie-pci-bridge,id=pci.251,bus=pci.1,addr=0x1 -spice
> port=5902,addr=0.0.0.0,disable-ticketing /var/lib/libvirt/images/q35.qcow2
>
> It should be noted that QEMU in general allows users to make pretty
> unwise device placement choices without so much of a warning, so I
> would not take the above as proof the pcie-to-pci-bridge should be
> allowed to plug into pcie-expander-bus (or pcie-root) directly :)
>
> If  pcie<-pcie-expander-bus<-(pcie-to-pci-bridge |
pcie-switch-upstream-port) is an unwise device placement, we'd better to
add the reason to code comments or commit msg or documents.
> --
> Andrea Bolognani / Red Hat / Virtualization
>
>
-- 
Best regards,
-----------------------------------
Han Han
Quality Engineer
Redhat.

Email: hhan@redhat.com
Phone: +861065339333