Prasanna Kalever
2016-Jun-15 10:09 UTC
[Gluster-users] [Gluster-devel] Glusterfs 3.7.11 with LibGFApi in Qemu on Ubuntu Xenial does not work
On Wed, Jun 15, 2016 at 2:41 PM, Andr? Bauer <abauer at magix.net> wrote:> > Hi Lists, > > i just updated on of my Ubuntu KVM Servers from 14.04 (Trusty) to 16.06 > (Xenial). > > I use the Glusterfs packages from the officail Ubuntu PPA and my own > Qemu packages ( > https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.7 ) > which have libgfapi enabled. > > On Ubuntu 14.04 everything is working fine. I only had to add the > following lines to the Apparmor config in > /etc/apparmor.d/abstractions/libvirt-qemu to get it work: > > # for glusterfs > /proc/sys/net/ipv4/ip_local_reserved_ports r, > /usr/lib/@{multiarch}/glusterfs/**.so mr, > /tmp/** rw, > > In Ubuntu 16.04 i'm not able to start the my VMs via libvirt or to > create new images via qemu-img using libgfapi. > > Mounting the volume via fuse does work without problems. > > Examples: > > qemu-img create gluster://storage.mydomain/vmimages/kvm2test.img 1G > Formatting 'gluster://storage.intdmz.h1.mdd/vmimages/kvm2test.img', > fmt=raw size=1073741824 > [2016-06-15 08:15:26.710665] E [MSGID: 108006] > [afr-common.c:4046:afr_notify] 0-vmimages-replicate-0: All subvolumes > are down. Going offline until atleast one of them comes back up. > [2016-06-15 08:15:26.710736] E [MSGID: 108006] > [afr-common.c:4046:afr_notify] 0-vmimages-replicate-1: All subvolumes > are down. Going offline until atleast one of them comes back up. > > Libvirtd log: > > [2016-06-13 16:53:57.055113] E [MSGID: 104007] > [glfs-mgmt.c:637:glfs_mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch > volume file (key:vmimages) [Invalid argument] > [2016-06-13 16:53:57.055196] E [MSGID: 104024] > [glfs-mgmt.c:738:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with > remote-host: storage.intdmz.h1.mdd (Permission denied) [Permission denied] > 2016-06-13T16:53:58.049945Z qemu-system-x86_64: -drive > file=gluster://storage.intdmz.h1.mdd/vmimages/checkbox.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback: > Gluster connection failed for server=storage.intdmz.h1.mdd port=0 > volume=vmimages image=checkbox.qcow2 transport=tcp: Permission deniedI think you have missed enabling bind insecure which is needed by libgfapi access, please try again after following below steps => edit /etc/glusterfs/glusterd.vol by add "option rpc-auth-allow-insecure on" #(on all nodes) => gluster vol set $volume server.allow-insecure on => systemctl restart glusterd #(on all nodes) In case this does not work, provide help us with the below, along with the logfiles # gluster vol info # gluster vol status # gluster peer status -- Prasanna> > I don't see anything in the apparmor logs when setting everything to > complain or audit. > > It also seems GlusterFS servers don't get any request because brick logs > are not complaining anything. > > Any hints? > > > -- > Regards > Andr? Bauer > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel
André Bauer
2016-Jun-15 12:07 UTC
[Gluster-users] [Gluster-devel] Glusterfs 3.7.11 with LibGFApi in Qemu on Ubuntu Xenial does not work
Hi Prasanna, Am 15.06.2016 um 12:09 schrieb Prasanna Kalever:> > I think you have missed enabling bind insecure which is needed by > libgfapi access, please try again after following below steps > > => edit /etc/glusterfs/glusterd.vol by add "option > rpc-auth-allow-insecure on" #(on all nodes) > => gluster vol set $volume server.allow-insecure on > => systemctl restart glusterd #(on all nodes) >No, thats not the case. All services are up and runnig correctly, allow-insecure is set and the volume works fine with libgfapi access from my Ubuntu 14.04 KVM/Qemu servers. Just the server which was updated to Ubuntu 16.04 can't access the volume via libgfapi anmyore (fuse mount still works). GlusterFS logs are empty when trying to access the GlusterFS nodes so i think the requests are blocked on the client side. Maybe apparmor again? Regards Andr?> > -- > Prasanna > >> >> I don't see anything in the apparmor logs when setting everything to >> complain or audit. >> >> It also seems GlusterFS servers don't get any request because brick logs >> are not complaining anything. >> >> Any hints? >> >> >> -- >> Regards >> Andr? Bauer >> >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-devel >-- Mit freundlichen Gr??en Andr? Bauer MAGIX Software GmbH Andr? Bauer Administrator August-Bebel-Stra?e 48 01219 Dresden GERMANY tel.: 0351 41884875 e-mail: abauer at magix.net abauer at magix.net <mailto:Email> www.magix.com <http://www.magix.com/> Gesch?ftsf?hrer | Managing Directors: Dr. Arnd Schr?der, Klaus Schmidt Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205 Find us on: <http://www.facebook.com/MAGIX> <http://www.twitter.com/magix_de> <http://www.youtube.com/wwwmagixcom> <http://www.magixmagazin.de> ---------------------------------------------------------------------- The information in this email is intended only for the addressee named above. Access to this email by anyone else is unauthorized. If you are not the intended recipient of this message any disclosure, copying, distribution or any action taken in reliance on it is prohibited and may be unlawful. MAGIX does not warrant that any attachments are free from viruses or other defects and accepts no liability for any losses resulting from infected email transmissions. Please note that any views expressed in this email may be those of the originator and do not necessarily represent the agenda of the company. ----------------------------------------------------------------------