thr3ads.net - libvirt users - Re: [libvirt-users] HugePages - can't start guest that requires them [Feb 2015]

If this information is useful, please help other people find it:
Share via:

G. Richard Bellamy

2015-Feb-04 00:59 UTC

Re: [libvirt-users] HugePages - can't start guest that requires them

As I mentioned, I got the instances to launch... but they're only
taking HugePages from "Node 0", when I believe my setup should pull
from both nodes.

[atlas] http://sprunge.us/FSEf
[prometheus] http://sprunge.us/PJcR

2015-02-03 16:51:48
root@eanna i ~ # virsh start atlas
Domain atlas started

2015-02-03 16:51:58
root@eanna i ~ # virsh start prometheus
Domain prometheus started

2015-02-03 16:52:53
root@eanna i ~ # numastat -m

Per-node system memory usage (in MBs):
                          Node 0          Node 2           Total
                 --------------- --------------- ---------------
MemTotal                32113.93        32238.27        64352.20
MemFree                  7030.30         7175.87        14206.17
MemUsed                 25083.63        25062.40        50146.04
Active                   3737.86         4089.77         7827.63
Inactive                 3423.46         2832.16         6255.61
Active(anon)             1658.46         2830.59         4489.05
Inactive(anon)             54.35           64.71          119.05
Active(file)             2079.39         1259.18         3338.57
Inactive(file)           3369.11         2767.45         6136.56
Unevictable                15.68           64.39           80.07
Mlocked                    15.68           64.39           80.07
Dirty                      11.45            6.98           18.43
Writeback                   0.00            0.00            0.00
FilePages                5515.35         4078.96         9594.32
Mapped                    396.62          336.48          733.10
AnonPages                1661.74         2906.95         4568.69
Shmem                      62.90           50.22          113.12
KernelStack                12.89            9.81           22.70
PageTables                 46.08           36.50           82.58
NFS_Unstable                0.00            0.00            0.00
Bounce                      0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00
Slab                      192.31          160.72          353.03
SReclaimable              137.67          118.47          256.14
SUnreclaim                 54.64           42.25           96.89
AnonHugePages               0.00            0.00            0.00
HugePages_Total         17408.00        17408.00        34816.00
HugePages_Free           2048.00            0.00         2048.00
HugePages_Surp              0.00            0.00            0.00
2015-02-03 16:53:47
root@eanna i ~ # numastat -p qemu

Per-node process memory usage (in MBs)
PID                               Node 0          Node 2           Total
-----------------------  --------------- --------------- ---------------
10315 (qemu-system-x86)           589.76            0.00        32391.84
10346 (qemu-system-x86)         14839.83            0.00        18128.85
-----------------------  --------------- --------------- ---------------
Total                           15429.59            0.00        50520.68

On Sun, Feb 1, 2015 at 10:43 PM, Dominique Ramaekers
<dominique.ramaekers@cometal.be> wrote:> Regarding fine tuning my explanation about what system does de actual
> mounting of Hugepages, you’re probably right…. Thanks for the correction.
>
> On upstart systems (like Ubuntu) the mounting of Hugepages is done by the
> init script qemu-kvm.conf
>
>
>
>
>
>
>
>
>
> Van: G. Richard Bellamy [mailto:rbellamy@pteradigm.com]
> Verzonden: zondag 1 februari 2015 0:02
> Aan: Dominique Ramaekers
> CC: libvirt-users@redhat.com
> Onderwerp: Re: [libvirt-users] HugePages - can't start guest that
requires
> them
>
>
>
> Yeah, Dominique, your wiki was one of the many docs I read through
> before/during/after starting down this primrose path... thanks for writing
> it. I'm an Arch user, and I couldn't find anything to indicate
qemu, as its
> compiled for Arch, will look in /etc/default/qemu-kvm. And now that
I've got
> the right page size, the instances are starting...
>
>
>
> The reason I want to use the page element to the hugepages directive is
that
> I want to target a numa node directly - in other words, I like the idea of
> one VM running on Node 0, and the other running on Node 2.
>
>
>
> Your comment about libvirt taking care of the hugepages mount isn't
> consistent with my reading or experience - on a systemd-based system,
> systemd takes care of the hugetlbfs mount to /dev/hugepages, and the
libvirt
> builds the /dev/hugepages/qemu... directory structure. At least that's
what
> I've seen.
>
>
>
> -rb
>
>
>
> On Sat, Jan 31, 2015 at 11:43 AM, Dominique Ramaekers
> <dominique.ramaekers@cometal.be> wrote:
>
> Did you create a mount for the hugepages? If you did, that's maybe the
> problem. I did that also at first but with libvirt it isn't necessary
and in
> my case, it broke hugepages...
>
> If I'm not mistaking, libvirt takes care of the hugepages mount.
>
> A while ago, I've written a wiki to use hugepages in libvirt and
Ubuntu.
> https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepages
>
> Maybe this helps?
>
> ________________________________________
> Van: G. Richard Bellamy [rbellamy@pteradigm.com]
> Verzonden: zaterdag 31 januari 2015 0:33
> Aan: libvirt-users@redhat.com
> Onderwerp: [libvirt-users] HugePages - can't start guest that requires
them
>
>
> Hello All,
>
> I'm trying to enable hugepages, I've turned off THP (Transparent
Huge
> Pages), and enabled hugepages in memoryBacking, and set my 2MB
> hugepages count via sysctl.
>
> I'm getting "libvirtd[5788]: Failed to autostart VM
'atlas': internal
> error: Unable to find any usable hugetlbfs mount for 16777216 KiB"
> where atlas is one of my guests and 16777216 KiB is the amount of
> memory I'm trying to give to the guest.
>
> Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted
> via /dev/hugepages and there is a dir structure
> /dev/hugepages/libvirt/qemu (it's empty).
>
> HugePages is big enough to accommodate the 16G i'm allocating... and
> changing the perms on that directory structure to 777 doesn't work
> either.
>
> Any help is much appreciated.
>
> HOST: http://sprunge.us/SEdc
> GUEST: http://sprunge.us/VCYB
>
> Regards,
> Richard
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>
>

Michal Privoznik

2015-Feb-04 08:17 UTC

head link

Re: [libvirt-users] HugePages - can't start guest that requires them

On 04.02.2015 01:59, G. Richard Bellamy wrote:> As I mentioned, I got the instances to launch... but they're only
> taking HugePages from "Node 0", when I believe my setup should
pull
> from both nodes.
> 
> [atlas] http://sprunge.us/FSEf
> [prometheus] http://sprunge.us/PJcR
[pasting interesting nits from both XMLs]

<domain type='kvm' id='2'>
  <name>atlas</name>
  <uuid>d9991b1c-2f2d-498a-9d21-51f3cf8e6cd9</uuid>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
    <nosharepages/>
  </memoryBacking>
  <!-- no numa pining -->
</domain>

<domain type='kvm' id='3'>
  <name>prometheus</name>
  <uuid>dda7d085-701b-4d0a-96d4-584678104fb3</uuid>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='2'/>
    </hugepages>
    <nosharepages/>
  </memoryBacking>
  <!-- again no numa pining -->
</domain>

So, at start, the @nodeset attribute to <page/> element refers to guest
numa nodes, not host ones. And since you don't define any numa nodes for
your guests, it's useless. Side note - I wonder if we should make
libvirt fail explicitly in this case.

Moreover, you haven't pinned your guests onto any host numa nodes. This
means it's up to the host kernel and its scheduler where guest will take
memory from. And subsequently hugepages as well. I think you want to add:

  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>

to guest XMLs, where @nodeset refers to host numa nodes and tells where
the guest should be placed. There are other modes too so please see
documentation to tune the XML to match your use case perfectly.

Michal

G. Richard Bellamy

2015-Feb-04 20:03 UTC

head link

Re: [libvirt-users] HugePages - can't start guest that requires them

*facepalm*

Now that I'm re-reading the documentation it's obvious that
<page/>
and @nodeset are for the guest, "This tells the hypervisor that the
guest should have its memory allocated using hugepages instead of the
normal native page size." Pretty clear there.

Thank you SO much for the guidance, I'll return to my tweaking. I'll
report back here with my results.




On Wed, Feb 4, 2015 at 12:17 AM, Michal Privoznik <mprivozn@redhat.com>
wrote:> On 04.02.2015 01:59, G. Richard Bellamy wrote:
>> As I mentioned, I got the instances to launch... but they're only
>> taking HugePages from "Node 0", when I believe my setup
should pull
>> from both nodes.
>>
>> [atlas] http://sprunge.us/FSEf
>> [prometheus] http://sprunge.us/PJcR
>
> [pasting interesting nits from both XMLs]
>
> <domain type='kvm' id='2'>
>   <name>atlas</name>
>   <uuid>d9991b1c-2f2d-498a-9d21-51f3cf8e6cd9</uuid>
>   <memory unit='KiB'>16777216</memory>
>   <currentMemory unit='KiB'>16777216</currentMemory>
>   <memoryBacking>
>     <hugepages>
>       <page size='2048' unit='KiB'
nodeset='0'/>
>     </hugepages>
>     <nosharepages/>
>   </memoryBacking>
>   <!-- no numa pining -->
> </domain>
>
>
> <domain type='kvm' id='3'>
>   <name>prometheus</name>
>   <uuid>dda7d085-701b-4d0a-96d4-584678104fb3</uuid>
>   <memory unit='KiB'>16777216</memory>
>   <currentMemory unit='KiB'>16777216</currentMemory>
>   <memoryBacking>
>     <hugepages>
>       <page size='2048' unit='KiB'
nodeset='2'/>
>     </hugepages>
>     <nosharepages/>
>   </memoryBacking>
>   <!-- again no numa pining -->
> </domain>
>
> So, at start, the @nodeset attribute to <page/> element refers to
guest
> numa nodes, not host ones. And since you don't define any numa nodes
for
> your guests, it's useless. Side note - I wonder if we should make
> libvirt fail explicitly in this case.
>
> Moreover, you haven't pinned your guests onto any host numa nodes. This
> means it's up to the host kernel and its scheduler where guest will
take
> memory from. And subsequently hugepages as well. I think you want to add:
>
>   <numatune>
>     <memory mode='strict' nodeset='0'/>
>   </numatune>
>
> to guest XMLs, where @nodeset refers to host numa nodes and tells where
> the guest should be placed. There are other modes too so please see
> documentation to tune the XML to match your use case perfectly.
>
> Michal

Possibly Parallel Threads

Search for more reasonably related threads

libvirt users - Feb 2015 - Re: HugePages - can't start guest that requires them

Re: [libvirt-users] HugePages - can't start guest that requires them

Re: [libvirt-users] HugePages - can't start guest that requires them

Re: [libvirt-users] HugePages - can't start guest that requires them

Possibly Parallel Threads