libvirt users - Jun 2022 - [Libvirt][emulator-cores][dpdk] Emulatorpin cpuset best practices

Hey folks,

I am curious to understand a bit more the core use of the emulatorpin CPUs
with libvirt.

For example :

    <vcpupin vcpu='0' cpuset='34'/>
    <vcpupin vcpu='1' cpuset='14'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='30'/>
    <emulatorpin cpuset='10,14,30,34'/>

In this case :

   - The VM has 4 vCPUs.
   - Each of the core is pinned to a physical/thread core - 34,14,10,30
   - The emulatorpin is also attached to the same physical CPUs.

In our case :

   - Same setup for cores.
   - Running DPDK inside the virtual machine.
   - The compute/libvirt is using OVS but without DPDK.

What we saw is the following :

   - DPDK is polling two cores at 100% to process packets.
   - From the compute metrics, we see USER %usage at 100% when the
   application starts.
   - From the compute metrics, as traffic increases, we see %SYSTEM going
   up.
   - As %SYSTEM goes up, %USER goes down.

We are not quite sure, but we are wondering if this could cause a direct
impact to the ability of DPDK to process traffic at the same X speed?

   - We can "live" move the emulator CPU to other cores.
   - This seems to be the recommended default setup for VNF/NFV deployments.


Feels like I am missing a piece of the puzzle but gotta start somewhere!

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20220601/b0c304d2/attachment.htm>

On Wed, Jun 01, 2022 at 05:57:14PM -0400, Laurent Dumont
wrote:
>Hey folks,
>
>I am curious to understand a bit more the core use of the emulatorpin CPUs
>with libvirt.
>
>For example :
>
>    <vcpupin vcpu='0' cpuset='34'/>
>    <vcpupin vcpu='1' cpuset='14'/>
>    <vcpupin vcpu='2' cpuset='10'/>
>    <vcpupin vcpu='3' cpuset='30'/>
>    <emulatorpin cpuset='10,14,30,34'/>
>
qemu launched like this is one process with (at least, but let's omit
that) 5 threads.  There is one thread for each vCPU and an extra thread
for all the extra work not running in the guest.  That can be I/O,
serving the monitor etc.  If you pin it onto the same host CPUs as the
vCPUs then you cannot expect to have enough resources allocated for
serving 100% utilisation in the guest.  But it is of course configurable
because overcommitting is sometimes wanted.
>In this case :
>
>   - The VM has 4 vCPUs.
>   - Each of the core is pinned to a physical/thread core - 34,14,10,30
>   - The emulatorpin is also attached to the same physical CPUs.
>
>In our case :
>
>   - Same setup for cores.
>   - Running DPDK inside the virtual machine.
>   - The compute/libvirt is using OVS but without DPDK.
>
>What we saw is the following :
>
>   - DPDK is polling two cores at 100% to process packets.
>   - From the compute metrics, we see USER %usage at 100% when the
>   application starts.
>   - From the compute metrics, as traffic increases, we see %SYSTEM going
>   up.
>   - As %SYSTEM goes up, %USER goes down.
>
>We are not quite sure, but we are wondering if this could cause a direct
>impact to the ability of DPDK to process traffic at the same X speed?
>
>   - We can "live" move the emulator CPU to other cores.
>   - This seems to be the recommended default setup for VNF/NFV deployments.
>
>
>Feels like I am missing a piece of the puzzle but gotta start somewhere!
>
>Thanks!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20220607/bcf24145/attachment.sig>