libvirt users - Dec 2021 - Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

Hi,

Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
with upstream kernels.

For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
brings quite some troubles for VMs.

In short, without proper cpuset to bind the VM to either all A72 cores
or all A55 cores, the VM will mostly fail to boot.

Currently the working xml is:

   <vcpu placement='static' cpuset='4-5'>2</vcpu>
   <cpu mode='host-passthrough' check='none'/>

But even with vcpupin, pinning each vcpu to each physical core, VM will
mostly fail to start up due to vcpu initialization failed with -EINVAL.


This brings a problem, in theory RK3399 SoC should out-perform BCM2711
in multi-core performance, but if a VM can only be bind to either A72 or
A55 cores, then the performance is no longer competitive against
BCM2711, wasting the PCIE 2.0 x4 capacity.

I guess with projects like Asahi Linux making progress, there will be
more and more such problems.

Any clue on how to properly pass all physical CPU cores to VM for
little.BIG setup?

Thanks,
Qu

On 12/11/21 02:58, Qu Wenruo wrote:
> Hi,
> 
> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
> with upstream kernels.
> 
> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> brings quite some troubles for VMs.
> 
> In short, without proper cpuset to bind the VM to either all A72 cores
> or all A55 cores, the VM will mostly fail to boot.
> 
> Currently the working xml is:
> 
> ? <vcpu placement='static'
cpuset='4-5'>2</vcpu>
> ? <cpu mode='host-passthrough' check='none'/>
> 
> But even with vcpupin, pinning each vcpu to each physical core, VM will
> mostly fail to start up due to vcpu initialization failed with -EINVAL.
> 
> 
> This brings a problem, in theory RK3399 SoC should out-perform BCM2711
> in multi-core performance, but if a VM can only be bind to either A72 or
> A55 cores, then the performance is no longer competitive against
> BCM2711, wasting the PCIE 2.0 x4 capacity.
> 
> I guess with projects like Asahi Linux making progress, there will be
> more and more such problems.
> 
> Any clue on how to properly pass all physical CPU cores to VM for
> little.BIG setup?
> 
I have never met big.LITTLE but my understanding was that those big
cores are compatible with little ones and the only difference is that
the big ones are shut off if there's no demand (to save energy) leaving
only the little ones running.

Anyway, this is likely too high level forum and I'd ask QEMU developers:

https://www.qemu.org/support/

Michal