libvirt users - Dec 2016 - virsh not detecting hugepage mount; disabled by config?

Hi,

I’m struggling with virsh not detecting my hugepage mount.

I have the following kernel command line:
BOOT_IMAGE=/vmlinuz-4.8.13-gentoo root=/dev/mapper/gensd-gentoo ro quiet
splash intel_iommu=on video=efifb:off,vesafb:off,simplefb:off
splash=verbose,theme:livedvd-aurora kvm.ignore_msrs=1
transparent_hugepage=never hugepages=3072 softlevel=qemuvm

My startup script outputs the following:
hugetlbfs /var/lib/hugetlbfs hugetlbfs
rw,relatime,pagesize=2097152,uid=77,gid=77,mode=0770 0 0
[2016.12.13 22:22:50 virsh 2808] ERROR Failed to start domain win10
[2016.12.13 22:22:50 virsh 2808] ERROR internal error: hugetlbfs
filesystem is not mounted or disabled by administrator config
virsh was unsuccessful

So hugetlbfs is definitively mounted, but virsh does either not detect
it or some magical option, that I’ve not found in a day searching for
it, disables it (the documentation could explain that better). Since the
Ubuntu guys refer to the KVM_HUGEPAGES environment, I tried adding it to
start-stop-daemon environment, but it seems non-relevant. 77 is the qemu
user id, but I’m quite sure the permission issues were excluded (tried
root permissions as well). Besides I had permission errors occasionally
occasionally and don’t have them anymore. If you could provide a hint to
the magical administrator configuration option, that would be helpful.
Tried hugeadm as well, pointing the config to the correct destination.

Thanks in advance,
Manuel

PS:
The virsh script reads like this:
#!/bin/sh
cmdline="$(cat /proc/cmdline)"
if [[ "${cmdline##* }" == "softlevel=qemuvm" ]]; then
    mount -o rw,relatime,pagesize=2097152,uid=77,gid=77,mode=0770 -t
hugetlbfs hugetlbfs /var/lib/hugetlbfs
    sysctl kernel.shmmax=6442450944
    cat /proc/mounts | grep hugetlb >> /var/log/virsh.log
    sed -i -e '/^hugetlb/{s/^/\#/}' /etc/libvirt/qemu.conf
    counter=0
    while [ "${counter}" -lt 10 && ! -S
/var/run/libvirt/libvirt-sock ]; do
    sleep 1
    counter=$(( ++counter ))
    done
    if [ $counter -gt 9 ]; then
    echo "libvirtd socket generation timed out" >>
/var/log/virsh.log
    fi
    if pidof libvirtd &>/dev/null; then
    LC_ALL=C /usr/bin/virsh -l /var/log/virsh.log start win10
    if [ $? -gt 0 ]; then
        echo "virsh was unsuccessful" >> /var/log/virsh.log
        reboot
    fi
    else
    echo "libvirtd is not started" >> /var/log/virsh.log
    reboot
    fi
fi

reboot is more convenient, since this is a vfio-igd passthrough vm.

On Tue, Dec 13, 2016 at 11:05:11PM +0100, Manuel Ullmann
wrote:
>Hi,
>
>I’m struggling with virsh not detecting my hugepage mount.
>
>I have the following kernel command line:
>BOOT_IMAGE=/vmlinuz-4.8.13-gentoo root=/dev/mapper/gensd-gentoo ro quiet
>splash intel_iommu=on video=efifb:off,vesafb:off,simplefb:off
>splash=verbose,theme:livedvd-aurora kvm.ignore_msrs=1
>transparent_hugepage=never hugepages=3072 softlevel=qemuvm
>
>My startup script outputs the following:
>hugetlbfs /var/lib/hugetlbfs hugetlbfs
>rw,relatime,pagesize=2097152,uid=77,gid=77,mode=0770 0 0
>[2016.12.13 22:22:50 virsh 2808] ERROR Failed to start domain win10
>[2016.12.13 22:22:50 virsh 2808] ERROR internal error: hugetlbfs
>filesystem is not mounted or disabled by administrator config
>virsh was unsuccessful
>
>So hugetlbfs is definitively mounted, but virsh does either not detect
>it or some magical option, that I’ve not found in a day searching for
>it, disables it (the documentation could explain that better). Since the
>Ubuntu guys refer to the KVM_HUGEPAGES environment, I tried adding it to
>start-stop-daemon environment, but it seems non-relevant. 77 is the qemu
>user id, but I’m quite sure the permission issues were excluded (tried
>root permissions as well). Besides I had permission errors occasionally
>occasionally and don’t have them anymore. If you could provide a hint to
>the magical administrator configuration option, that would be helpful.
Nothing magical about that.  If you have no hugepages configured in
qemu.conf, libvirtd will *try* to find all the mounts if running as
root.  The only thing is that in the error message "or" should be
changed to "and", I guess.
>Tried hugeadm as well, pointing the config to the correct destination.
>
>Thanks in advance,
>Manuel
>
>PS:
>The virsh script reads like this:
>#!/bin/sh
>cmdline="$(cat /proc/cmdline)"
>if [[ "${cmdline##* }" == "softlevel=qemuvm" ]]; then
>    mount -o rw,relatime,pagesize=2097152,uid=77,gid=77,mode=0770 -t
>hugetlbfs hugetlbfs /var/lib/hugetlbfs
If libvirtd starts before this ^^
>    sysctl kernel.shmmax=6442450944
>    cat /proc/mounts | grep hugetlb >> /var/log/virsh.log
>    sed -i -e '/^hugetlb/{s/^/\#/}' /etc/libvirt/qemu.conf
or this ^^ is done, then the settings are not reflected.  That's the
only thing I could glean from the quick look.

I bet adding 'service libvirtd restart' here helps.
>    counter=0
>    while [ "${counter}" -lt 10 && ! -S
/var/run/libvirt/libvirt-sock ]; do
>    sleep 1
>    counter=$(( ++counter ))
>    done
>    if [ $counter -gt 9 ]; then
>    echo "libvirtd socket generation timed out" >>
/var/log/virsh.log
>    fi
>    if pidof libvirtd &>/dev/null; then
>    LC_ALL=C /usr/bin/virsh -l /var/log/virsh.log start win10
>    if [ $? -gt 0 ]; then
>        echo "virsh was unsuccessful" >> /var/log/virsh.log
>        reboot
>    fi
>    else
>    echo "libvirtd is not started" >> /var/log/virsh.log
>    reboot
>    fi
>fi
>
>reboot is more convenient, since this is a vfio-igd passthrough vm.
>
>
>_______________________________________________
>libvirt-users mailing list
>libvirt-users@redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users