G. Richard Bellamy
2015-Jan-31 23:01 UTC
Re: [libvirt-users] HugePages - can't start guest that requires them
Yeah, Dominique, your wiki was one of the many docs I read through before/during/after starting down this primrose path... thanks for writing it. I'm an Arch user, and I couldn't find anything to indicate qemu, as its compiled for Arch, will look in /etc/default/qemu-kvm. And now that I've got the right page size, the instances are starting... The reason I want to use the page element to the hugepages directive is that I want to target a numa node directly - in other words, I like the idea of one VM running on Node 0, and the other running on Node 2. Your comment about libvirt taking care of the hugepages mount isn't consistent with my reading or experience - on a systemd-based system, systemd takes care of the hugetlbfs mount to /dev/hugepages, and the libvirt builds the /dev/hugepages/qemu... directory structure. At least that's what I've seen. -rb On Sat, Jan 31, 2015 at 11:43 AM, Dominique Ramaekers < dominique.ramaekers@cometal.be> wrote:> Did you create a mount for the hugepages? If you did, that's maybe the > problem. I did that also at first but with libvirt it isn't necessary and > in my case, it broke hugepages... > > If I'm not mistaking, libvirt takes care of the hugepages mount. > > A while ago, I've written a wiki to use hugepages in libvirt and Ubuntu. > https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepages > > Maybe this helps? > > ________________________________________ > Van: G. Richard Bellamy [rbellamy@pteradigm.com] > Verzonden: zaterdag 31 januari 2015 0:33 > Aan: libvirt-users@redhat.com > Onderwerp: [libvirt-users] HugePages - can't start guest that requires them > > Hello All, > > I'm trying to enable hugepages, I've turned off THP (Transparent Huge > Pages), and enabled hugepages in memoryBacking, and set my 2MB > hugepages count via sysctl. > > I'm getting "libvirtd[5788]: Failed to autostart VM 'atlas': internal > error: Unable to find any usable hugetlbfs mount for 16777216 KiB" > where atlas is one of my guests and 16777216 KiB is the amount of > memory I'm trying to give to the guest. > > Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted > via /dev/hugepages and there is a dir structure > /dev/hugepages/libvirt/qemu (it's empty). > > HugePages is big enough to accommodate the 16G i'm allocating... and > changing the perms on that directory structure to 777 doesn't work > either. > > Any help is much appreciated. > > HOST: http://sprunge.us/SEdc > GUEST: http://sprunge.us/VCYB > > Regards, > Richard > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users >
Dominique Ramaekers
2015-Feb-02 06:43 UTC
[libvirt-users] HugePages - can't start guest that requires them
Regarding fine tuning my explanation about what system does de actual mounting of Hugepages, you?re probably right?. Thanks for the correction. On upstart systems (like Ubuntu) the mounting of Hugepages is done by the init script qemu-kvm.conf Van: G. Richard Bellamy [mailto:rbellamy at pteradigm.com] Verzonden: zondag 1 februari 2015 0:02 Aan: Dominique Ramaekers CC: libvirt-users at redhat.com Onderwerp: Re: [libvirt-users] HugePages - can't start guest that requires them Yeah, Dominique, your wiki was one of the many docs I read through before/during/after starting down this primrose path... thanks for writing it. I'm an Arch user, and I couldn't find anything to indicate qemu, as its compiled for Arch, will look in /etc/default/qemu-kvm. And now that I've got the right page size, the instances are starting... The reason I want to use the page element to the hugepages directive is that I want to target a numa node directly - in other words, I like the idea of one VM running on Node 0, and the other running on Node 2. Your comment about libvirt taking care of the hugepages mount isn't consistent with my reading or experience - on a systemd-based system, systemd takes care of the hugetlbfs mount to /dev/hugepages, and the libvirt builds the /dev/hugepages/qemu... directory structure. At least that's what I've seen. -rb On Sat, Jan 31, 2015 at 11:43 AM, Dominique Ramaekers <dominique.ramaekers at cometal.be<mailto:dominique.ramaekers at cometal.be>> wrote: Did you create a mount for the hugepages? If you did, that's maybe the problem. I did that also at first but with libvirt it isn't necessary and in my case, it broke hugepages... If I'm not mistaking, libvirt takes care of the hugepages mount. A while ago, I've written a wiki to use hugepages in libvirt and Ubuntu. https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepages Maybe this helps? ________________________________________ Van: G. Richard Bellamy [rbellamy at pteradigm.com<mailto:rbellamy at pteradigm.com>] Verzonden: zaterdag 31 januari 2015 0:33 Aan: libvirt-users at redhat.com<mailto:libvirt-users at redhat.com> Onderwerp: [libvirt-users] HugePages - can't start guest that requires them Hello All, I'm trying to enable hugepages, I've turned off THP (Transparent Huge Pages), and enabled hugepages in memoryBacking, and set my 2MB hugepages count via sysctl. I'm getting "libvirtd[5788]: Failed to autostart VM 'atlas': internal error: Unable to find any usable hugetlbfs mount for 16777216 KiB" where atlas is one of my guests and 16777216 KiB is the amount of memory I'm trying to give to the guest. Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted via /dev/hugepages and there is a dir structure /dev/hugepages/libvirt/qemu (it's empty). HugePages is big enough to accommodate the 16G i'm allocating... and changing the perms on that directory structure to 777 doesn't work either. Any help is much appreciated. HOST: http://sprunge.us/SEdc GUEST: http://sprunge.us/VCYB Regards, Richard _______________________________________________ libvirt-users mailing list libvirt-users at redhat.com<mailto:libvirt-users at redhat.com> https://www.redhat.com/mailman/listinfo/libvirt-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20150202/4459640f/attachment.htm>
G. Richard Bellamy
2015-Feb-04 00:59 UTC
Re: [libvirt-users] HugePages - can't start guest that requires them
As I mentioned, I got the instances to launch... but they're only
taking HugePages from "Node 0", when I believe my setup should pull
from both nodes.
[atlas] http://sprunge.us/FSEf
[prometheus] http://sprunge.us/PJcR
2015-02-03 16:51:48
root@eanna i ~ # virsh start atlas
Domain atlas started
2015-02-03 16:51:58
root@eanna i ~ # virsh start prometheus
Domain prometheus started
2015-02-03 16:52:53
root@eanna i ~ # numastat -m
Per-node system memory usage (in MBs):
Node 0 Node 2 Total
--------------- --------------- ---------------
MemTotal 32113.93 32238.27 64352.20
MemFree 7030.30 7175.87 14206.17
MemUsed 25083.63 25062.40 50146.04
Active 3737.86 4089.77 7827.63
Inactive 3423.46 2832.16 6255.61
Active(anon) 1658.46 2830.59 4489.05
Inactive(anon) 54.35 64.71 119.05
Active(file) 2079.39 1259.18 3338.57
Inactive(file) 3369.11 2767.45 6136.56
Unevictable 15.68 64.39 80.07
Mlocked 15.68 64.39 80.07
Dirty 11.45 6.98 18.43
Writeback 0.00 0.00 0.00
FilePages 5515.35 4078.96 9594.32
Mapped 396.62 336.48 733.10
AnonPages 1661.74 2906.95 4568.69
Shmem 62.90 50.22 113.12
KernelStack 12.89 9.81 22.70
PageTables 46.08 36.50 82.58
NFS_Unstable 0.00 0.00 0.00
Bounce 0.00 0.00 0.00
WritebackTmp 0.00 0.00 0.00
Slab 192.31 160.72 353.03
SReclaimable 137.67 118.47 256.14
SUnreclaim 54.64 42.25 96.89
AnonHugePages 0.00 0.00 0.00
HugePages_Total 17408.00 17408.00 34816.00
HugePages_Free 2048.00 0.00 2048.00
HugePages_Surp 0.00 0.00 0.00
2015-02-03 16:53:47
root@eanna i ~ # numastat -p qemu
Per-node process memory usage (in MBs)
PID Node 0 Node 2 Total
----------------------- --------------- --------------- ---------------
10315 (qemu-system-x86) 589.76 0.00 32391.84
10346 (qemu-system-x86) 14839.83 0.00 18128.85
----------------------- --------------- --------------- ---------------
Total 15429.59 0.00 50520.68
On Sun, Feb 1, 2015 at 10:43 PM, Dominique Ramaekers
<dominique.ramaekers@cometal.be> wrote:> Regarding fine tuning my explanation about what system does de actual
> mounting of Hugepages, you’re probably right…. Thanks for the correction.
>
> On upstart systems (like Ubuntu) the mounting of Hugepages is done by the
> init script qemu-kvm.conf
>
>
>
>
>
>
>
>
>
> Van: G. Richard Bellamy [mailto:rbellamy@pteradigm.com]
> Verzonden: zondag 1 februari 2015 0:02
> Aan: Dominique Ramaekers
> CC: libvirt-users@redhat.com
> Onderwerp: Re: [libvirt-users] HugePages - can't start guest that
requires
> them
>
>
>
> Yeah, Dominique, your wiki was one of the many docs I read through
> before/during/after starting down this primrose path... thanks for writing
> it. I'm an Arch user, and I couldn't find anything to indicate
qemu, as its
> compiled for Arch, will look in /etc/default/qemu-kvm. And now that
I've got
> the right page size, the instances are starting...
>
>
>
> The reason I want to use the page element to the hugepages directive is
that
> I want to target a numa node directly - in other words, I like the idea of
> one VM running on Node 0, and the other running on Node 2.
>
>
>
> Your comment about libvirt taking care of the hugepages mount isn't
> consistent with my reading or experience - on a systemd-based system,
> systemd takes care of the hugetlbfs mount to /dev/hugepages, and the
libvirt
> builds the /dev/hugepages/qemu... directory structure. At least that's
what
> I've seen.
>
>
>
> -rb
>
>
>
> On Sat, Jan 31, 2015 at 11:43 AM, Dominique Ramaekers
> <dominique.ramaekers@cometal.be> wrote:
>
> Did you create a mount for the hugepages? If you did, that's maybe the
> problem. I did that also at first but with libvirt it isn't necessary
and in
> my case, it broke hugepages...
>
> If I'm not mistaking, libvirt takes care of the hugepages mount.
>
> A while ago, I've written a wiki to use hugepages in libvirt and
Ubuntu.
> https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepages
>
> Maybe this helps?
>
> ________________________________________
> Van: G. Richard Bellamy [rbellamy@pteradigm.com]
> Verzonden: zaterdag 31 januari 2015 0:33
> Aan: libvirt-users@redhat.com
> Onderwerp: [libvirt-users] HugePages - can't start guest that requires
them
>
>
> Hello All,
>
> I'm trying to enable hugepages, I've turned off THP (Transparent
Huge
> Pages), and enabled hugepages in memoryBacking, and set my 2MB
> hugepages count via sysctl.
>
> I'm getting "libvirtd[5788]: Failed to autostart VM
'atlas': internal
> error: Unable to find any usable hugetlbfs mount for 16777216 KiB"
> where atlas is one of my guests and 16777216 KiB is the amount of
> memory I'm trying to give to the guest.
>
> Yes, i can see the hugepages via numastat -m and hugetlbfs is mounted
> via /dev/hugepages and there is a dir structure
> /dev/hugepages/libvirt/qemu (it's empty).
>
> HugePages is big enough to accommodate the 16G i'm allocating... and
> changing the perms on that directory structure to 777 doesn't work
> either.
>
> Any help is much appreciated.
>
> HOST: http://sprunge.us/SEdc
> GUEST: http://sprunge.us/VCYB
>
> Regards,
> Richard
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>
>