Manuel Ullmann
2016-Dec-14 13:58 UTC
Re: [libvirt-users] virsh not detecting hugepage mount; disabled by config?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Thanks a lot you two, yes hugetlbfs is not mounted, when libvirtd is started… It was a long night. Did put it to fstab and since I do not allocate them on regular boots, it won’t eat my ram (that was the main consideration for doing it like this). As a last means before going to sleep, I added <qemu:arg value='-numa'/> <qemu:arg value='node,memdev=mem42'/> <qemu:arg value='-object'/> <qemu:arg value='memory-backend-file,size=6G,mem-path=/var/lib/hugetlbfs,share=on,id=mem42'/> to the xml which worked too (surprise), but I have it working now in the intended way. Then I never got to the last part of the qemu.conf option comment so I had to do another 3 reboots. Well, reading helps. So the fstab entry reads now: hugetlbfs /var/lib/hugetlbfs/libvirt/qemu hugetlbfs relatime,rw,pagesize=1073741824 0 0 with /var/lib/hugetlbfs being the mount point in qemu.conf. At least I didn’t forget to create the mount point before rebooting. Best regards, Manuel> Hi, > > I use gentoo with desktop+systemd profile to run Windows 10 VM withpci > passthrough and hugetlbs. > > Things I had to do for hugetlbs: > 1) Made sure their support was enabled in kernel > 2) Passed parameters for their allocation to kernel. There are other > ways, but I use 1GB page size and I have to allocate them before memory > gets fragmented. Note, you do that with hugepages=3072. > 3) Passed hugetlbs settings to libvirt VM domain (XML). > > That is it. I don't mount them myself. I think (can't check now) they > are automounted to /dev/hugetlbs. By default, libvirtd runs as root, and > qemu as user qemu. Consuming hugetlb and releasing them should be > transparent. > > I suggest starting without softlevel=qemuvm into default runlevel, and > then check /proc/meminfo. Hugetlbs total and free should be > 0. Do the > same with qemuvm runlevel. Maybe this runlevel misses something. > > On 2016-12-14 00:05, Manuel Ullmann wrote: >> Hi, >> >> I’m struggling with virsh not detecting my hugepage mount. >> >> I have the following kernel command line: >> BOOT_IMAGE=/vmlinuz-4.8.13-gentoo root=/dev/mapper/gensd-gentoo ro quiet >> splash intel_iommu=on video=efifb:off,vesafb:off,simplefb:off >> splash=verbose,theme:livedvd-aurora kvm.ignore_msrs=1 >> transparent_hugepage=never hugepages=3072 softlevel=qemuvm >> >> My startup script outputs the following: >> hugetlbfs /var/lib/hugetlbfs hugetlbfs >> rw,relatime,pagesize=2097152,uid=77,gid=77,mode=0770 0 0 >> [2016.12.13 22:22:50 virsh 2808] ERROR Failed to start domain win10 >> [2016.12.13 22:22:50 virsh 2808] ERROR internal error: hugetlbfs >> filesystem is not mounted or disabled by administrator config >> virsh was unsuccessful >> >> So hugetlbfs is definitively mounted, but virsh does either not detect >> it or some magical option, that I’ve not found in a day searching for >> it, disables it (the documentation could explain that better). Since the >> Ubuntu guys refer to the KVM_HUGEPAGES environment, I tried adding it to >> start-stop-daemon environment, but it seems non-relevant. 77 is the qemu >> user id, but I’m quite sure the permission issues were excluded (tried >> root permissions as well). Besides I had permission errors occasionally >> occasionally and don’t have them anymore. If you could provide a hint to>> the magical administrator configuration option, that would behelpful. >> Tried hugeadm as well, pointing the config to the correct destination. >> >> Thanks in advance, >> Manuel >> >> PS: >> The virsh script reads like this: >> #!/bin/sh >> cmdline="$(cat /proc/cmdline)">> if [[ "${cmdline##* }" == "softlevel=qemuvm" ]]; then >> mount -orw,relatime,pagesize=2097152,uid=77,gid=77,mode=0770 -t >> hugetlbfs hugetlbfs /var/lib/hugetlbfs >> sysctl kernel.shmmax=6442450944>> cat /proc/mounts | grep hugetlb >> /var/log/virsh.log >> sed-i -e '/^hugetlb/{s/^/\#/}' /etc/libvirt/qemu.conf >> counter=0>> while [ "${counter}" -lt 10 && ! -S /var/run/libvirt/libvirt-sock]; do >> sleep 1 >> counter=$(( ++counter )) >> done >> if [ $counter -gt 9 ]; then >> echo "libvirtd socket generation timed out" >> /var/log/virsh.log >> fi >> if pidof libvirtd &>/dev/null; then >> LC_ALL=C /usr/bin/virsh -l /var/log/virsh.log start win10 >> if [ $? -gt 0 ]; then >> echo "virsh was unsuccessful" >> /var/log/virsh.log >> reboot >> fi >> else >> echo "libvirtd is not started" >> /var/log/virsh.log >> reboot >> fi >> fi >> >> reboot is more convenient, since this is a vfio-igd passthrough vm. >> >> >> _______________________________________________ >> libvirt-users mailing list >> libvirt-users@redhat.com >> https://www.redhat.com/mailman/listinfo/libvirt-users >> -----BEGIN PGP SIGNATURE----- iQQcBAEBCAAGBQJYUVAKAAoJELgK9xQKkpX6sTof/2fdLX9qjiNgecKEIzHKEjBY ln6o7E0nMY796Jf3HQ/v620uVZS3Nf4dRSFC9NVetzAgVcQizwko0ikVDsaCZlNP 5HojcRm2SK8c5CgU/ZLdLPp73PgrORtKYytYldhPQFebgOpNCJGzWAPXGnoLK78s aHsKGKb5R79+K2Bq5DAQGR5N9wiV84Oi+afNjBunjj1dT+80YPAAtXuRKkDdD3lQ hVBEdzgRLWW48FdEhIp2TlTJCyx4OwMGnIXaOPXrQSyMW6Omn49ZXYqwIgsOW45S bzFZ4BPuqhrANne4YDPjuxwtL65YUKAyHZWl/bTBmqkFrpjecxYDnlg219t2KsiM k3W9GQ0CWR74f0lJLULaV+ome13ydga5pDRe6+dyi7g0O7/4DTPuxQTRo07UiA/i dQmvtzTAbGdSpGqyFBQqIFGMG4sB8kypXp1PIK06jX1Muon6WCdzAzt1IztLeuSW qUj/srdnD9rLWWPlIV2NCZ3E5Da4szRD8gbhNIG2hg+Ss50pLdIFwlUbfksryY7B B6L6KmpOw4LrpCZVFBeFhG5lFns9YSvYNKvSBRwJaxl838j8Kr8B1bo51nMYERL1 wJXK9IyFqx8jkbmFvfwXij5lx3DFBrLEQwymGmaLmJ4FQmnA9491L2F3K+NyG8kX Yi/w+Xh9ZARoRy8/2SmVYwvB1vZkFpf04adocHxIYMT1P3ZLpCktu3P/DMdkdR26 DcnQedl7qbf7NeldxMiY0zTAti85VFuQggFQlRrB36pRdOXvFMk3mNLBiifICNvP 32sTl/9HnmscWCTSebmK/KjQjrLf9ijbc3V942WziwkEyDbOhfTUn0JfcZJl0FdM OQ0j00t5wv14KFPP/hpWqRiypwXpboHLpB4k718NZElObUOob0ziz6PJc5iCzJDW AkNbO1sEDgWN/pPorAYKPtre3TV6qk/k6A13zgNmMdZA4ZHRq6dsZLzCLnEaLwJE G5WFveDyi4COVpelysArTPrw6aL4LAzIhkRN5A5YCANgsPtnhjql6XmL3sTSN93Z R42klgPKNZVq5Bkdp2EpE70X+F/4j2jhRNznVp3YDU+6q6aNWHdqw7jCAhEji0aA 49gR3tIUgXp1883zc0vSFIMnCwYQQoCjqGpVZXWLtbxAmJrz5c4XeUCnpplfJW3Z jRS19MEy6HgKmdkq1a4bRG5XFc87Ga2JWqwSXpGb1EsMvyMU3CQq47D98WqaeC9p JTOuLMpYvlH+EEbEkL/deysLGVZ7QDTr5P4L+wOzN9jhSOrnIoNlVU5DVP7dhT2L z1d3UY3cFhoVFLKyfbLF+flK7Y5aygQ/eO/+MlXmd9TI+2qTVehNSXiYTEnnsO0=1DU1 -----END PGP SIGNATURE-----
Reasonably Related Threads
- virsh not detecting hugepage mount; disabled by config?
- about sharing the hugepage memory segment between the host and the container
- Re: HugePages - can't start guest that requires them
- Re: HugePages - can't start guest that requires them
- HugePages - can't start guest that requires them