André Bauer
2016-Jun-15 09:11 UTC
[Gluster-users] Glusterfs 3.7.11 with LibGFApi in Qemu on Ubuntu Xenial does not work
Hi Lists,
i just updated on of my Ubuntu KVM Servers from 14.04 (Trusty) to 16.06
(Xenial).
I use the Glusterfs packages from the officail Ubuntu PPA and my own
Qemu packages (
https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.7 )
which have libgfapi enabled.
On Ubuntu 14.04 everything is working fine. I only had to add the
following lines to the Apparmor config in
/etc/apparmor.d/abstractions/libvirt-qemu to get it work:
# for glusterfs
/proc/sys/net/ipv4/ip_local_reserved_ports r,
/usr/lib/@{multiarch}/glusterfs/**.so mr,
/tmp/** rw,
In Ubuntu 16.04 i'm not able to start the my VMs via libvirt or to
create new images via qemu-img using libgfapi.
Mounting the volume via fuse does work without problems.
Examples:
qemu-img create gluster://storage.mydomain/vmimages/kvm2test.img 1G
Formatting 'gluster://storage.intdmz.h1.mdd/vmimages/kvm2test.img',
fmt=raw size=1073741824
[2016-06-15 08:15:26.710665] E [MSGID: 108006]
[afr-common.c:4046:afr_notify] 0-vmimages-replicate-0: All subvolumes
are down. Going offline until atleast one of them comes back up.
[2016-06-15 08:15:26.710736] E [MSGID: 108006]
[afr-common.c:4046:afr_notify] 0-vmimages-replicate-1: All subvolumes
are down. Going offline until atleast one of them comes back up.
Libvirtd log:
[2016-06-13 16:53:57.055113] E [MSGID: 104007]
[glfs-mgmt.c:637:glfs_mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch
volume file (key:vmimages) [Invalid argument]
[2016-06-13 16:53:57.055196] E [MSGID: 104024]
[glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with
remote-host: storage.intdmz.h1.mdd (Permission denied) [Permission denied]
2016-06-13T16:53:58.049945Z qemu-system-x86_64: -drive
file=gluster://storage.intdmz.h1.mdd/vmimages/checkbox.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback:
Gluster connection failed for server=storage.intdmz.h1.mdd port=0
volume=vmimages image=checkbox.qcow2 transport=tcp: Permission denied
I don't see anything in the apparmor logs when setting everything to
complain or audit.
It also seems GlusterFS servers don't get any request because brick logs
are not complaining anything.
Any hints?
--
Regards
Andr? Bauer
Prasanna Kalever
2016-Jun-15 10:09 UTC
[Gluster-users] [Gluster-devel] Glusterfs 3.7.11 with LibGFApi in Qemu on Ubuntu Xenial does not work
On Wed, Jun 15, 2016 at 2:41 PM, Andr? Bauer <abauer at magix.net> wrote:> > Hi Lists, > > i just updated on of my Ubuntu KVM Servers from 14.04 (Trusty) to 16.06 > (Xenial). > > I use the Glusterfs packages from the officail Ubuntu PPA and my own > Qemu packages ( > https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.7 ) > which have libgfapi enabled. > > On Ubuntu 14.04 everything is working fine. I only had to add the > following lines to the Apparmor config in > /etc/apparmor.d/abstractions/libvirt-qemu to get it work: > > # for glusterfs > /proc/sys/net/ipv4/ip_local_reserved_ports r, > /usr/lib/@{multiarch}/glusterfs/**.so mr, > /tmp/** rw, > > In Ubuntu 16.04 i'm not able to start the my VMs via libvirt or to > create new images via qemu-img using libgfapi. > > Mounting the volume via fuse does work without problems. > > Examples: > > qemu-img create gluster://storage.mydomain/vmimages/kvm2test.img 1G > Formatting 'gluster://storage.intdmz.h1.mdd/vmimages/kvm2test.img', > fmt=raw size=1073741824 > [2016-06-15 08:15:26.710665] E [MSGID: 108006] > [afr-common.c:4046:afr_notify] 0-vmimages-replicate-0: All subvolumes > are down. Going offline until atleast one of them comes back up. > [2016-06-15 08:15:26.710736] E [MSGID: 108006] > [afr-common.c:4046:afr_notify] 0-vmimages-replicate-1: All subvolumes > are down. Going offline until atleast one of them comes back up. > > Libvirtd log: > > [2016-06-13 16:53:57.055113] E [MSGID: 104007] > [glfs-mgmt.c:637:glfs_mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch > volume file (key:vmimages) [Invalid argument] > [2016-06-13 16:53:57.055196] E [MSGID: 104024] > [glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with > remote-host: storage.intdmz.h1.mdd (Permission denied) [Permission denied] > 2016-06-13T16:53:58.049945Z qemu-system-x86_64: -drive > file=gluster://storage.intdmz.h1.mdd/vmimages/checkbox.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback: > Gluster connection failed for server=storage.intdmz.h1.mdd port=0 > volume=vmimages image=checkbox.qcow2 transport=tcp: Permission deniedI think you have missed enabling bind insecure which is needed by libgfapi access, please try again after following below steps => edit /etc/glusterfs/glusterd.vol by add "option rpc-auth-allow-insecure on" #(on all nodes) => gluster vol set $volume server.allow-insecure on => systemctl restart glusterd #(on all nodes) In case this does not work, provide help us with the below, along with the logfiles # gluster vol info # gluster vol status # gluster peer status -- Prasanna> > I don't see anything in the apparmor logs when setting everything to > complain or audit. > > It also seems GlusterFS servers don't get any request because brick logs > are not complaining anything. > > Any hints? > > > -- > Regards > Andr? Bauer > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel