Gluster users - Mar 2014 - Openstack Havana Gluster /var/lib/instances issues

I was wondering if anyone had seen similar issues before, when booting a vm with
a root volume on local disk things work fine.

When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse
I get the following: http://paste.ubuntu.com/7116594/

I have of course insured perms were correct with chown -R nova:nova
/var/lib/nova/instances, additionally app armor is completely unloaded/disabled.

Has anyone encountered this before? Gluster version is 3.4.2


-ryan

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140318/5606cd66/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140318/5606cd66/attachment.sig>

It might be more useful to look at your hypervisor log and/or client log.

On March 18, 2014 3:04:48 PM PDT, "Aydelott, Ryan M." <ryade at
mcs.anl.gov> wrote:
>I was wondering if anyone had seen similar issues before, when booting
>a vm with a root volume on local disk things work fine. 
>
>When I try to do the same with /var/lib/nova/instances mounted via
>gluster/fuse I get the following: http://paste.ubuntu.com/7116594/
>
>I have of course insured perms were correct with chown -R nova:nova
>/var/lib/nova/instances, additionally app armor is completely
>unloaded/disabled. 
>
>Has anyone encountered this before? Gluster version is 3.4.2
>
>
>-ryan
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Gluster-users mailing list
>Gluster-users at gluster.org
>http://supercolony.gluster.org/mailman/listinfo/gluster-users
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140318/d288dfaf/attachment.html>

>From the pastebin it seems like its getting a ton of I/O errors.
It would be good to look at gluster logs (/var/log/glusterfs) and whether
that carries any errors, since I/O error is a generic error that glusterfs
can return for many reasons.


On Wed, Mar 19, 2014 at 3:34 AM, Aydelott, Ryan M. <ryade at
mcs.anl.gov>wrote:
> I was wondering if anyone had seen similar issues before, when booting a
> vm with a root volume on local disk things work fine.
>
> When I try to do the same with /var/lib/nova/instances mounted via
> gluster/fuse I get the following: http://paste.ubuntu.com/7116594/
>
> I have of course insured perms were correct with chown -R nova:nova
> /var/lib/nova/instances, additionally app armor is completely
> unloaded/disabled.
>
> Has anyone encountered this before? Gluster version is 3.4.2
>
>
> -ryan
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140319/1ba7b077/attachment.html>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I have not seen that with the fuse client.  I had it happen repeatedly
when we were using gluster/NFS for /var/lib/nova/instances and/or
/var/lib/nova/instances/_base.  The access to instances-XXXXXX/disk
(or its qcow2 backing image) would hang completely or be very slow
under heavy load.  The VM interprets the delay as a problem on
/dev/vda and remounts RO.


On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote:
> I was wondering if anyone had seen similar issues before, when
> booting a vm with a root volume on local disk things work fine.
> 
> When I try to do the same with /var/lib/nova/instances mounted via
> gluster/fuse I get the following: http://paste.ubuntu.com/7116594/
> 
> I have of course insured perms were correct with chown -R nova:nova
> /var/lib/nova/instances, additionally app armor is completely
> unloaded/disabled.
> 
> Has anyone encountered this before? Gluster version is 3.4.2
> 
> 
> -ryan
> 
> 
> 
> 
> _______________________________________________ Gluster-users
> mailing list Gluster-users at gluster.org 
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
- -- 
Raymond Powell
Systems Administrator
Laboratory for Advanced Computing
University of Chicago, KCBD 10160
900 East 57th Street
Chicago, IL 60637
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9
lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy
pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B
YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm
eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg
huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/
HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD
lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa
sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA
o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF
JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe
+UhAq6+74ie1MokNssB8
=un7j
-----END PGP SIGNATURE-----

If anyone has encountered this issue - I was graciously given a solution by some
of the gluster folks.

The short of it is: gluster volume set yournovashared remote-dio on

https://ask.openstack.org/en/question/25371/gluster-volume-for-nova-instance/

On Mar 18, 2014, at 5:04 PM, Ryan Aydelott <ryade at anl.gov> wrote:
> I was wondering if anyone had seen similar issues before, when booting a vm
with a root volume on local disk things work fine.
> 
> When I try to do the same with /var/lib/nova/instances mounted via
gluster/fuse I get the following: http://paste.ubuntu.com/7116594/
> 
> I have of course insured perms were correct with chown -R nova:nova
/var/lib/nova/instances, additionally app armor is completely unloaded/disabled.
> 
> Has anyone encountered this before? Gluster version is 3.4.2
> 
> 
> -ryan
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140326/4cff9dbd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140326/4cff9dbd/attachment.sig>