libvirt users - Jan 2015 - Re: HugePages - can't start guest that requires them

Hello All,

I'm trying to enable hugepages, I've turned off THP (Transparent Huge
Pages), and enabled hugepages in memoryBacking, and set my 2MB
hugepages count via sysctl.

I'm getting "libvirtd[5788]: Failed to autostart VM 'atlas':
internal
error: Unable to find any usable hugetlbfs mount for 16777216 KiB"
where atlas is one of my guests and 16777216 KiB is the amount of
memory I'm trying to give to the guest.

Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted
via /dev/hugepages and there is a dir structure
/dev/hugepages/libvirt/qemu (it's empty).

HugePages is big enough to accommodate the 16G i'm allocating... and
changing the perms on that directory structure to 777 doesn't work
either.

Any help is much appreciated.

HOST: http://sprunge.us/SEdc
GUEST: http://sprunge.us/VCYB

Regards,
Richard

On Fri, Jan 30, 2015 at 03:33:43PM -0800, G. Richard Bellamy
wrote:
>Hello All,
>
>I'm trying to enable hugepages, I've turned off THP (Transparent
Huge
>Pages), and enabled hugepages in memoryBacking, and set my 2MB
>hugepages count via sysctl.
>
>I'm getting "libvirtd[5788]: Failed to autostart VM
'atlas': internal
>error: Unable to find any usable hugetlbfs mount for 16777216 KiB"
>where atlas is one of my guests and 16777216 KiB is the amount of
>memory I'm trying to give to the guest.
>
Looking at the XML:

  <memoryBacking>
    <hugepages>
      <page size='16777216' unit='KiB'
nodeset='0'/>

This means you want the guest's memory to be allocated from 16GiB
hugepages.  You probably wanted to put this there:

  <page size='2' unit='MiB' ...
>Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted
>via /dev/hugepages and there is a dir structure
>/dev/hugepages/libvirt/qemu (it's empty).
>
>HugePages is big enough to accommodate the 16G i'm allocating... and
>changing the perms on that directory structure to 777 doesn't work
>either.
>
>Any help is much appreciated.
>
>HOST: http://sprunge.us/SEdc
>GUEST: http://sprunge.us/VCYB
>
>Regards,
>Richard
>
>_______________________________________________
>libvirt-users mailing list
>libvirt-users@redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users

Did you create a mount for the hugepages? If you did, that's maybe the
problem. I did that also at first but with libvirt it isn't necessary and in
my case, it broke hugepages...

If I'm not mistaking, libvirt takes care of the hugepages mount.

A while ago, I've written a wiki to use hugepages in libvirt and Ubuntu.
https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepages

Maybe this helps?

________________________________________
Van: G. Richard Bellamy [rbellamy@pteradigm.com]
Verzonden: zaterdag 31 januari 2015 0:33
Aan: libvirt-users@redhat.com
Onderwerp: [libvirt-users] HugePages - can't start guest that requires them

Hello All,

I'm trying to enable hugepages, I've turned off THP (Transparent Huge
Pages), and enabled hugepages in memoryBacking, and set my 2MB
hugepages count via sysctl.

I'm getting "libvirtd[5788]: Failed to autostart VM 'atlas':
internal
error: Unable to find any usable hugetlbfs mount for 16777216 KiB"
where atlas is one of my guests and 16777216 KiB is the amount of
memory I'm trying to give to the guest.

Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted
via /dev/hugepages and there is a dir structure
/dev/hugepages/libvirt/qemu (it's empty).

HugePages is big enough to accommodate the 16G i'm allocating... and
changing the perms on that directory structure to 777 doesn't work
either.

Any help is much appreciated.

HOST: http://sprunge.us/SEdc
GUEST: http://sprunge.us/VCYB

Regards,
Richard

_______________________________________________
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users

ack. Yeah, I had seen that and thought I corrected it. Thank you very much,
the instances start.

Now I just have to figure out why I seem to be using 2x the number of
hugepages I think I should be. Numstat seems to think that now that I've
started up the two VMs, I am using twice as many hugepages as the amount I
had allocated via sysctl [1].

[1] http://sprunge.us/LLNM

On Sat, Jan 31, 2015 at 9:55 AM, Martin Kletzander <mkletzan@redhat.com>
wrote:
> On Fri, Jan 30, 2015 at 03:33:43PM -0800, G. Richard Bellamy wrote:
>
>> Hello All,
>>
>> I'm trying to enable hugepages, I've turned off THP
(Transparent Huge
>> Pages), and enabled hugepages in memoryBacking, and set my 2MB
>> hugepages count via sysctl.
>>
>> I'm getting "libvirtd[5788]: Failed to autostart VM
'atlas': internal
>> error: Unable to find any usable hugetlbfs mount for 16777216 KiB"
>> where atlas is one of my guests and 16777216 KiB is the amount of
>> memory I'm trying to give to the guest.
>>
>>
> Looking at the XML:
>
>  <memoryBacking>
>    <hugepages>
>      <page size='16777216' unit='KiB'
nodeset='0'/>
>
> This means you want the guest's memory to be allocated from 16GiB
> hugepages.  You probably wanted to put this there:
>
>  <page size='2' unit='MiB' ...
>
>  Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted
>> via /dev/hugepages and there is a dir structure
>> /dev/hugepages/libvirt/qemu (it's empty).
>>
>> HugePages is big enough to accommodate the 16G i'm allocating...
and
>> changing the perms on that directory structure to 777 doesn't work
>> either.
>>
>> Any help is much appreciated.
>>
>> HOST: http://sprunge.us/SEdc
>> GUEST: http://sprunge.us/VCYB
>>
>> Regards,
>> Richard
>>
>> _______________________________________________
>> libvirt-users mailing list
>> libvirt-users@redhat.com
>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>
>

Yeah, Dominique, your wiki was one of the many docs I read through
before/during/after starting down this primrose path... thanks for writing
it. I'm an Arch user, and I couldn't find anything to indicate qemu, as
its
compiled for Arch, will look in /etc/default/qemu-kvm. And now that I've
got the right page size, the instances are starting...

The reason I want to use the page element to the hugepages directive is
that I want to target a numa node directly - in other words, I like the
idea of one VM running on Node 0, and the other running on Node 2.

Your comment about libvirt taking care of the hugepages mount isn't
consistent with my reading or experience - on a systemd-based system,
systemd takes care of the hugetlbfs mount to /dev/hugepages, and the
libvirt builds the /dev/hugepages/qemu... directory structure. At least
that's what I've seen.

-rb

On Sat, Jan 31, 2015 at 11:43 AM, Dominique Ramaekers <
dominique.ramaekers@cometal.be> wrote:
> Did you create a mount for the hugepages? If you did, that's maybe the
> problem. I did that also at first but with libvirt it isn't necessary
and
> in my case, it broke hugepages...
>
> If I'm not mistaking, libvirt takes care of the hugepages mount.
>
> A while ago, I've written a wiki to use hugepages in libvirt and
Ubuntu.
> https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepages
>
> Maybe this helps?
>
> ________________________________________
> Van: G. Richard Bellamy [rbellamy@pteradigm.com]
> Verzonden: zaterdag 31 januari 2015 0:33
> Aan: libvirt-users@redhat.com
> Onderwerp: [libvirt-users] HugePages - can't start guest that requires
them
>
> Hello All,
>
> I'm trying to enable hugepages, I've turned off THP (Transparent
Huge
> Pages), and enabled hugepages in memoryBacking, and set my 2MB
> hugepages count via sysctl.
>
> I'm getting "libvirtd[5788]: Failed to autostart VM
'atlas': internal
> error: Unable to find any usable hugetlbfs mount for 16777216 KiB"
> where atlas is one of my guests and 16777216 KiB is the amount of
> memory I'm trying to give to the guest.
>
> Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted
> via /dev/hugepages and there is a dir structure
> /dev/hugepages/libvirt/qemu (it's empty).
>
> HugePages is big enough to accommodate the 16G i'm allocating... and
> changing the perms on that directory structure to 777 doesn't work
> either.
>
> Any help is much appreciated.
>
> HOST: http://sprunge.us/SEdc
> GUEST: http://sprunge.us/VCYB
>
> Regards,
> Richard
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>