libvirt users - Sep 2014 - Inconsistent behavior between x86_64 and ppc64 when creating guests with NUMA node placement

Hello all,

I was recently trying out NUMA placement for my guests on both x86_64 
and ppc64 machines. When booting a guest on the x86_64 machine, the 
following specs were valid (obviously, just notable excepts from the xml):

<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
...
   <cpu>
     <topology sockets='4' cores='1' threads='1'/>
     <numa>
       <cell cpus='0-2' memory='6144'/>
       <cell cpus='3' memory='2048'/>
     </numa>
   </cpu>

However, on ppc64 this causes the following error:

error: Failed to create domain from sample_guest.xml
error: internal error: early end of file from monitor: possible problem:
2014-09-11T18:44:25.502140Z qemu-system-ppc64: total memory for NUMA 
nodes (8388608) should equal RAM size (200000000)

The 200000000 is actually 8192 MB in bytes and hexidecimal. This is 
apparently just an issue with the error message.

The following specs work on ppc64:
   <cpu>
     <topology sockets='4' cores='1' threads='1'/>
     <numa>
       <cell cpus='0-2' memory='6291456'/>
       <cell cpus='3' memory='2097152'/>
     </numa>
   </cpu>

Note that the memory for each cell is 6144*1024 and 2048*1024 
respectively. The issue is that the memory size for each NUMA cell 
should be specified in KiB, not MB 
(http://libvirt.org/formatdomain.html#resPartition "|memory| specifies 
the node memory in kibibyte").

In short, it seems that specifying NUMA cell memory in MB works on 
x86_64 but not on ppc64. Does anyone have any insight to what's causing 
this, or if I'm misunderstanding something? Any help is appreciated, 
thank you!

Regards,
Michael Turek

On 12.09.2014 21:10, Michael Turek wrote:
> Hello all,
>
> I was recently trying out NUMA placement for my guests on both x86_64
> and ppc64 machines. When booting a guest on the x86_64 machine, the
> following specs were valid (obviously, just notable excepts from the xml):
>
> <memory unit='KiB'>8388608</memory>
> <currentMemory unit='KiB'>8388608</currentMemory>
>   <vcpu placement='static'>4</vcpu>
> ...
>    <cpu>
>      <topology sockets='4' cores='1'
threads='1'/>
>      <numa>
>        <cell cpus='0-2' memory='6144'/>
>        <cell cpus='3' memory='2048'/>
>      </numa>
>    </cpu>

6144 + 2048 != 8388608.
Either you need to multiply those values by factor of 1024 (6144*1024 + 
2048*1024 == 8388608) or add @unit='MiB' attribute, so libvirt knows 
what units you mean.
>
> However, on ppc64 this causes the following error:
>
> error: Failed to create domain from sample_guest.xml
> error: internal error: early end of file from monitor: possible problem:
> 2014-09-11T18:44:25.502140Z qemu-system-ppc64: total memory for NUMA
> nodes (8388608) should equal RAM size (200000000)
>
> The 200000000 is actually 8192 MB in bytes and hexidecimal. This is
> apparently just an issue with the error message.
>
> The following specs work on ppc64:
>    <cpu>
>      <topology sockets='4' cores='1'
threads='1'/>
>      <numa>
>        <cell cpus='0-2' memory='6291456'/>
>        <cell cpus='3' memory='2097152'/>
>      </numa>
>    </cpu>
>
> Note that the memory for each cell is 6144*1024 and 2048*1024
> respectively. The issue is that the memory size for each NUMA cell
> should be specified in KiB, not MB
> (http://libvirt.org/formatdomain.html#resPartition "|memory| specifies
> the node memory in kibibyte").
>
> In short, it seems that specifying NUMA cell memory in MB works on
> x86_64 but not on ppc64. Does anyone have any insight to what's causing
> this, or if I'm misunderstanding something? Any help is appreciated,
> thank you!
What does the corresponding part of qemu command line look like? In both 
cases. I don't see anything architecture dependent there.

What may happen here is that qemu on x86_64 does the checking while on 
ppc64 it does not.

Michal