libvirt users - Nov 2022 - Predictable and consistent net interface naming in guests

On Mon, Oct 31, 2022 at 04:32:27PM +0200, Edward Haas
wrote:
> That discussion mentioned that a guest PCI address may change in two cases:
> - The PCI topology changes.
> - The machine type changes.
>
> Usually, the machine type is not expected to change, especially if one
> wants to allow migrations between nodes.
> I would hope to argue this should not be problematic in practice, because
> guest images would be made per a specific machine type.
The machine type might not change from q35 to i440fx and vice versa,
but since the domain XML is constructed every time a KubeVirt VM is
started, the machine type might be q35-6.0 on one boot and q35-7.0
the next one if a KubeVirt upgrade that comes with a new version of
QEMU has happened in between.

This is unlikely to make a difference in terms of PCI addresses seen
in the guest OS, but it's still not accurate to say that the machine
type will not change.

Live migration is a separate matter, as the machine type will
definitely not change while the VM is running.
> Regarding the PCI topology, I am not sure I understand what changes
> need to occur to the domxml for a defined guest PCI address to change.
> The only think that I can think of is a scenario where hotplug/unplug is
> used,
> but even then I would expect existing devices to preserve their PCI address
> and the plug/unplug device to have a reserved address managed by the one
> acting on it (the management system).
>
> Could you please help clarify in which scenarios the PCI topology can cause
> a mess to the naming of interfaces in the guest?
A change in libvirt (again, due to a KubeVirt upgrade in between two
boots of the same VM) might result in different PCI addresses being
assigned to devices despite the same input XML.

We generally try fairly hard to avoid this kind of situation, but we
can only really guarantee stable PCI addresses for the lifetime of a
VM that has been defined and can't promise that the same input XML
will result in the same guest ABI when using different versions of
libvirt.

-- 
Andrea Bolognani / Red Hat / Virtualization

On Mon, Oct 31, 2022 at 6:55 PM Andrea Bolognani <abologna at redhat.com>
wrote:
> On Mon, Oct 31, 2022 at 04:32:27PM +0200, Edward Haas wrote:
> > That discussion mentioned that a guest PCI address may change in two
> cases:
> > - The PCI topology changes.
> > - The machine type changes.
> >
> > Usually, the machine type is not expected to change, especially if one
> > wants to allow migrations between nodes.
> > I would hope to argue this should not be problematic in practice,
because
> > guest images would be made per a specific machine type.
>
> The machine type might not change from q35 to i440fx and vice versa,
> but since the domain XML is constructed every time a KubeVirt VM is
> started, the machine type might be q35-6.0 on one boot and q35-7.0
> the next one if a KubeVirt upgrade that comes with a new version of
> QEMU has happened in between.
>
> This is unlikely to make a difference in terms of PCI addresses seen
> in the guest OS, but it's still not accurate to say that the machine
> type will not change.
>
Thank you for the clarification.
It makes me wonder now what are the actual implications of
the machine type change.

> Live migration is a separate matter, as the machine type will
> definitely not change while the VM is running.
>
> > Regarding the PCI topology, I am not sure I understand what changes
> > need to occur to the domxml for a defined guest PCI address to change.
> > The only think that I can think of is a scenario where hotplug/unplug
is
> > used,
> > but even then I would expect existing devices to preserve their PCI
> address
> > and the plug/unplug device to have a reserved address managed by the
one
> > acting on it (the management system).
> >
> > Could you please help clarify in which scenarios the PCI topology can
> cause
> > a mess to the naming of interfaces in the guest?
>
> A change in libvirt (again, due to a KubeVirt upgrade in between two
> boots of the same VM) might result in different PCI addresses being
> assigned to devices despite the same input XML.
>
> We generally try fairly hard to avoid this kind of situation, but we
> can only really guarantee stable PCI addresses for the lifetime of a
> VM that has been defined and can't promise that the same input XML
> will result in the same guest ABI when using different versions of
> libvirt.
>
I would expect the PCI addresses that have been explicitly set in the
domxml [2] to be honored. We cannot assume that?
I mainly referred to that input option, not to the expectation that the
generated
configuration (of the domxml) to be identical between different versions.

[2] https://libvirt.org/formatdomain.html#device-addresses
>
> --
> Andrea Bolognani / Red Hat / Virtualization
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20221031/f9387061/attachment.htm>

On 10/31/22 2:21 PM, Edward Haas wrote:
> 
> 
> On Mon, Oct 31, 2022 at 6:55 PM Andrea Bolognani <abologna at redhat.com
> <mailto:abologna at redhat.com>> wrote:
> 
>     On Mon, Oct 31, 2022 at 04:32:27PM +0200, Edward Haas wrote:
>      > That discussion mentioned that a guest PCI address may change in
>     two cases:
>      > - The PCI topology changes.
>      > - The machine type changes.
>      >
>      > Usually, the machine type is not expected to change, especially
>     if one
>      > wants to allow migrations between nodes.
>      > I would hope to argue this should not be problematic in practice,
>     because
>      > guest images would be made per a specific machine type.
> 
>     The machine type might not change from q35 to i440fx and vice versa,
>     but since the domain XML is constructed every time a KubeVirt VM is
>     started, the machine type might be q35-6.0 on one boot and q35-7.0
>     the next one if a KubeVirt upgrade that comes with a new version of
>     QEMU has happened in between.
> 
>     This is unlikely to make a difference in terms of PCI addresses seen
>     in the guest OS, but it's still not accurate to say that the
machine
>     type will not change.
> 
> 
> Thank you for the clarification.
> It makes me wonder now what are the actual implications of
> the machine type change.
> 
> 
>     Live migration is a separate matter, as the machine type will
>     definitely not change while the VM is running.
> 
>      > Regarding the PCI topology, I am not sure I understand what
changes
>      > need to occur to the domxml for a defined guest PCI address to
>     change.
>      > The only think that I can think of is a scenario where
>     hotplug/unplug is
>      > used,
>      > but even then I would expect existing devices to preserve their
>     PCI address
>      > and the plug/unplug device to have a reserved address managed by
>     the one
>      > acting on it (the management system).
>      >
>      > Could you please help clarify in which scenarios the PCI topology
>     can cause
>      > a mess to the naming of interfaces in the guest?
> 
>     A change in libvirt (again, due to a KubeVirt upgrade in between two
>     boots of the same VM) might result in different PCI addresses being
>     assigned to devices despite the same input XML.
> 
>     We generally try fairly hard to avoid this kind of situation, but we
>     can only really guarantee stable PCI addresses for the lifetime of a
>     VM that has been defined and can't promise that the same input XML
>     will result in the same guest ABI when using different versions of
>     libvirt.
> 
> 
> I would expect the PCI addresses that have been explicitly set in the
> domxml [2] to be honored. We cannot assume that?
*If* the PCI address has been set in the original XML, that address will 
be honored any and every time a new domain is defined from that XML. 
Alternately, if a domain is defined once (without explicitly specifying 
any PCI addresses) and then run multiple times from the same definition, 
libvirt will auto-generate PCI addresses at initial definition time, and 
then use those same addresses each time the domain is run.

The issue is that no management application, including KubeVirt, is 
explicitly setting the PCI addresses of devices (and we believe that 
hands-off practice should continue), *AND* KubeVirt is re-defining the 
domain each time it is run (without querying libvirt for (and so never 
saving) the PCI addresses that were assigned to the devices. So each 
time the domain is stopped, all the PCI address info from that run is 
thrown away. And each time the domain is re-started (by re-defining it 
from the original XML that has no PCI address info), libvirt starts from 
scratch assigning addresses based on the information it receives from 
KubeVirt. And if the conditions have changed, then addresses are 
assigned differently.

The potential situation Andrea described, where the PCI addresses could 
be changed merely due to an upgrade of KubeVirt/libvirt/qemu from one 
run to the next in spite of being fed the same (adress-less) XML, is 
actually extremely rare (I don't remember such a case) but theoretically 
it could happen. The more common change would be if a device was added 
or removed during one run of the guest, and then remained added/removed 
the next time it was run - that could change the PCI addresses of one or 
more of the remaining devices, depending on their ordering in the XML).

So, libvirt provides two avenues to maintaining stable PCI addresses 
(and thus, network device names) across multiple runs of a domain 
(either define once, run many, or else query the XML of the running 
domain and use that XML (containing PCI addresses) the next time the 
domain is started, but KubeVirt doesn't use either of these (and if 
memory serves me correctly, it really can't due to its design. And 
delegating management of PCI addresses to KubeVirt is pushing too much 
complexity out to KubeVirt.


> I mainly referred to that input option, not to the expectation that the 
> generated
> configuration (of the domxml) to be identical between different versions.
> 
> [2] https://libvirt.org/formatdomain.html#device-addresses 
> <https://libvirt.org/formatdomain.html#device-addresses>
> 
> 
>     -- 
>     Andrea Bolognani / Red Hat / Virtualization
>