libvirt users - Aug 2022 - Memory locking limit and zero-copy migrations

On Wed, Aug 17, 2022 at 10:56:54 +0200, Milan Zamazal
wrote:
> Hi,
> 
> do I read libvirt sources right that when <memtune> is not used in
the
> libvirt domain then libvirt takes proper care about setting memory
> locking limits when zero-copy is requested for a migration?
Well yes, for a definition of "proper". In this instance qemu can lock
up to the guest-visible memory size of memory for the migration, thus we
set the lockable size to the guest memory size. This is a simple upper
bound which is supposed to work in all scenarios. Qemu is also unlikely
to ever use up all the allowed locking.
> I also wonder whether there are any other situations where memory limits
> could be set by libvirt or QEMU automatically rather than having no
> memory limits?  We had oVirt bugs in the past where certain VMs with
> VFIO devices couldn't be started due to extra requirements on the
amount
> of locked memory and adding <hard_limit> to the domain apparently
> helped.
<hard_limit> is not only an amount of memory qemu can lock into ram, but
an upper bound of all memory the qemu process can consume. This includes
any qemu overhead e.g. used for the emulation layer.

Guessing the correct size of overhead still has the same problems it had
and libvirt is not going to be in the business of doing that.

Peter Krempa <pkrempa at redhat.com> writes:
> On Wed, Aug 17, 2022 at 10:56:54 +0200, Milan Zamazal wrote:
>> Hi,
>> 
>
>> do I read libvirt sources right that when <memtune> is not used
in the
>> libvirt domain then libvirt takes proper care about setting memory
>> locking limits when zero-copy is requested for a migration?
>
> Well yes, for a definition of "proper". In this instance qemu can
lock
> up to the guest-visible memory size of memory for the migration, thus we
> set the lockable size to the guest memory size. This is a simple upper
> bound which is supposed to work in all scenarios. Qemu is also unlikely
> to ever use up all the allowed locking.
Great, thank you for confirmation.
>> I also wonder whether there are any other situations where memory
limits
>> could be set by libvirt or QEMU automatically rather than having no
>> memory limits?  We had oVirt bugs in the past where certain VMs with
>> VFIO devices couldn't be started due to extra requirements on the
amount
>> of locked memory and adding <hard_limit> to the domain apparently
>> helped.
>
> <hard_limit> is not only an amount of memory qemu can lock into ram,
but
> an upper bound of all memory the qemu process can consume. This includes
> any qemu overhead e.g. used for the emulation layer.
>
> Guessing the correct size of overhead still has the same problems it had
> and libvirt is not going to be in the business of doing that.
To clarify, my point was not whether libvirt should, but whether libvirt
or any related component possibly does (or did in the past) impose
memory limits.  Because as I was looking around it seems there are no
real memory limits by default, at least in libvirt, but some limit had
been apparently hit in the reported bugs.