libvirt users - Jan 2019 - Re: assigning PCI addresses with bus > 0x09

On 12/20/18 9:20 AM, Daniel P. Berrangé wrote:
> On Thu, Dec 20, 2018 at 03:15:34PM +0100, Riccardo Ravaioli wrote:
>> Hi,
>>
>> My goal is to assign PCI addresses to a number of devices (network
>> interfaces, disks and PCI devices in PCI-passthrough) without
delegating to
>> libvirt the generation of those values. This should give me more
control
>> and for sure more predictability over the hardware configuration of a
>> virtual machine and consequently the name of the interfaces in it. 
I'm
>> using libvirt 4.3.0 to create qemu/KVM virtual machines running Linux
>> (Debian Stretch).
>>
>> So, for every device of the type mentioned above, I add this line:
>> <address type='pci' domain='0x0000'
bus='0x__' slot='0x__' function='0x0'/>,
>> ... with values from 00 to ff in the bus field, and from 00 to 1f in
the
>> slot field, as described in the documentation.
>>
>> Long story short, I noticed that as soon as I assign values > 0x09
to the
>> bus field, the serial console hangs indefinitely, in both Debian and
>> Ubuntu. The VM seems to be started correctly and its state is
"running"; in
>> the XML file created by libvirt, I see all controllers from 0 the
largest
>> bus value I assigned, so everything from that side seems ok.
> 
> I guess the hang is that you hit some limit in PCI buses.
I think you're right. Each bus requires some amount of IO space, and I 
thought I recalled someone saying that all of the available IO space is 
exhausted after 7 or 8 buses. This was in relation to PCIe, where each 
root port is a bus, and can potentially take up IO space, so possibly in 
that context they were talking about the buses *after* the root bus and 
pcie-pci-bridge, which would bring us back to the same number you're 
getting.

For PCIe our solution was to turn off IO space usage on the 
pcie-root-ports, but you can't do that for conventional PCI buses, since 
the devices might actually need IO space to function properly and the 
standard requires that you provide it.

> 
> The real question though is why you need to create sooooo many PCI buses.
> Each bus can do 31 devices.  Do you really need to have more than 279
> devices in your VM ?
And if you do need more than 279 devices, do they all need to be 
hot-pluggable? If not, then you can put up to 8 devices on each slot 
(functions 0 - 7).
> 
> Regards,
> Daniel
>

On Thu, 20 Dec 2018 at 15:39, Laine Stump <laine@redhat.com> wrote:
> [...]
> I think you're right. Each bus requires some amount of IO space, and I
> thought I recalled someone saying that all of the available IO space is
> exhausted after 7 or 8 buses. This was in relation to PCIe, where each
> root port is a bus, and can potentially take up IO space, so possibly in
> that context they were talking about the buses *after* the root bus and
> pcie-pci-bridge, which would bring us back to the same number you're
> getting.
>
> For PCIe our solution was to turn off IO space usage on the
> pcie-root-ports, but you can't do that for conventional PCI buses,
since
> the devices might actually need IO space to function properly and the
> standard requires that you provide it.
>
Ok, that makes sense.

> > The real question though is why you need to create sooooo many PCI
buses.
> > Each bus can do 31 devices.  Do you really need to have more than 279
> > devices in your VM ?
>
> And if you do need more than 279 devices, do they all need to be
> hot-pluggable? If not, then you can put up to 8 devices on each slot
> (functions 0 - 7).
>
True. I'll use the function field too, then.

Thanks a lot!

Riccardo

On Thu, 20 Dec 2018 at 15:39, Laine Stump <laine@redhat.com> wrote:
> I think you're right. Each bus requires some amount of IO space, and I
> thought I recalled someone saying that all of the available IO space is
> exhausted after 7 or 8 buses. [...]
>
Laine,

Do you have by any chance a link to a page explaining this in more details?
Thanks again! :)

Riccardo

On 1/3/19 5:22 AM, Riccardo Ravaioli wrote:
> On Thu, 20 Dec 2018 at 15:39, Laine Stump <laine@redhat.com 
> <mailto:laine@redhat.com>> wrote:
> 
>     I think you're right. Each bus requires some amount of IO space,
and I
>     thought I recalled someone saying that all of the available IO space is
>     exhausted after 7 or 8 buses. [...]
> 
> 
> Laine,
> 
> Do you have by any chance a link to a page explaining this in more details?
> Thanks again! :)
No, sorry. There are mentions of it in bugzilla records (e.g. 
https://bugzilla.redhat.com/show_bug.cgi?id=1067777 ) but all the info I 
have is just recalled from email and irc conversations over the last 3-4 
years. Basically the amount of IO address space is limited, and SeaBIOS 
allocates some minimum-sized chunk of that for each PCI controller that 
is probed and "seems to need IO address space for its devices". After
8
or so controllers, all the space is used up.

You can avoid the IO address space limit if you're using a PCI Express 
based machinetype, *and* all PCIe devices, but you'll run up against 
different limits that need to be worked around in a different way - PCIe 
devices are required to be usable with no IO address space (and so QEMU 
specifically creates the PCIe controllers beyond pcie-root without any). 
But each PCIe controller only has a single slot (with 8 functions), each 
consumes a "bus number" and the bus number is an 8-bit value, so
you're
limited to 256 total (x8 if you don't need hotplug and don't mind 
manually assigning addresses).