Gluster users - Mar 2014 - Gluster 3.4.2 on Ubuntu 12.04 LTS Server - Upstart No Go

I thought I would share my experience in case it helps someone else avoid a
bruised forehead.

I'm running the semiosis Gluster 3.4.2 on Ubuntu 12.04 (two nodes,
replicated) as server and client on both.  Going through all the typical
configuration work, all seemed good.  The problem came when I tried to get
the boxes to start Gluster and mount the volume at boot... without user
intervention.  Thinking that the most difficult part of the installation
should be configuring Gluster (newbie to distributed file systems in
general, never used Gluster before), I foolishly wasted a pile of time
trying to fix the *simple* boot problem.  Well, I did fix it, but only by
finally disabling the upstart files bundled in the install
(glusterfs-server.conf and mounting-glusterfs.conf).

This was a clean install of Ubuntu 12.04.  Actually, two clean installs for
testing.  And then another clean install on one box to rule out the
original install!  Same behaviour on all installs.

I spent some time experimenting with different fstab entries and found I
could change the mount behaviour, but only to determine *when* the mount
would fail... it always failed!

My solution was to disable the two Gluster upstart conf files with
.override files containing 'manual'.  Then a simple script to a) start
glusterfs-server; b) sleep for a few seconds; c) mount the Gluster volume.
 The script is called from rc.local.

Hardly an elegant solution I know, but at some point *'works'* beats
*'pretty
but non-functional'*.

It's possible there is something quirky with my install that could be
responsible.  But given that other than installing and configuring Gluster,
nothing on the servers was tweaked, I'm thinking there is a problem with
the Gluster upstart conf files.  Or something in 12.04 has rendered them
inoperable. Or maybe it's as a friend in Poland says... (direct
translation) "it's the Malice of Things..."  Truly a better
description
than anything we have in English! ;-)

I'm not looking for a solution unless someone has an obvious fix that
Google couldn't locate.  I have a quick and dirty work around that works.
 I just hope to save someone else the pain.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140304/d345e109/attachment.html>

I could be completely off-base, but were you trying to mount with noatime?

I ran into a similar problem where, after upgrading to 3.4.2, all my client
mounts failed.  I discovered (by accident) that it was only due to
incompatible mount options (namely noatime).  In previous versions, the
client just gave a warning message about unsupported options, but now it
fails without mentioning the cause.  I believe there is a bug report for
this issue (that I discovered afterwards).

Anyways, I thought this had a chance of being the issue, but maybe it's
something different in your case.

Matt


On Tue, Mar 4, 2014 at 2:09 PM, Brock Nanson <brock at nanson.net> wrote:
> I thought I would share my experience in case it helps someone else avoid
> a bruised forehead.
>
> I'm running the semiosis Gluster 3.4.2 on Ubuntu 12.04 (two nodes,
> replicated) as server and client on both.  Going through all the typical
> configuration work, all seemed good.  The problem came when I tried to get
> the boxes to start Gluster and mount the volume at boot... without user
> intervention.  Thinking that the most difficult part of the installation
> should be configuring Gluster (newbie to distributed file systems in
> general, never used Gluster before), I foolishly wasted a pile of time
> trying to fix the *simple* boot problem.  Well, I did fix it, but only by
> finally disabling the upstart files bundled in the install
> (glusterfs-server.conf and mounting-glusterfs.conf).
>
> This was a clean install of Ubuntu 12.04.  Actually, two clean installs
> for testing.  And then another clean install on one box to rule out the
> original install!  Same behaviour on all installs.
>
> I spent some time experimenting with different fstab entries and found I
> could change the mount behaviour, but only to determine *when* the mount
> would fail... it always failed!
>
> My solution was to disable the two Gluster upstart conf files with
> .override files containing 'manual'.  Then a simple script to a)
start
> glusterfs-server; b) sleep for a few seconds; c) mount the Gluster volume.
>  The script is called from rc.local.
>
> Hardly an elegant solution I know, but at some point *'works'*
beats *'pretty
> but non-functional'*.
>
> It's possible there is something quirky with my install that could be
> responsible.  But given that other than installing and configuring Gluster,
> nothing on the servers was tweaked, I'm thinking there is a problem
with
> the Gluster upstart conf files.  Or something in 12.04 has rendered them
> inoperable. Or maybe it's as a friend in Poland says... (direct
> translation) "it's the Malice of Things..."  Truly a better
description
> than anything we have in English! ;-)
>
> I'm not looking for a solution unless someone has an obvious fix that
> Google couldn't locate.  I have a quick and dirty work around that
works.
>  I just hope to save someone else the pain.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140304/dd01679d/attachment.html>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I get around the problem on our gluster clusters by putting
"nobootwait" and/or "noauto" in my fstab and then having
rc.local run
the mount command for the gluster mount points. Do not know if that is
more or less elegant then your workaround.  It at least doesn't
require one to fight with upstart.

On 3/4/14 1:09 PM, Brock Nanson wrote:
> I thought I would share my experience in case it helps someone else
> avoid a bruised forehead.
> 
> I'm running the semiosis Gluster 3.4.2 on Ubuntu 12.04 (two nodes, 
> replicated) as server and client on both.  Going through all the
> typical configuration work, all seemed good.  The problem came when
> I tried to get the boxes to start Gluster and mount the volume at
> boot... without user intervention.  Thinking that the most
> difficult part of the installation should be configuring Gluster
> (newbie to distributed file systems in general, never used Gluster
> before), I foolishly wasted a pile of time trying to fix the
> *simple* boot problem.  Well, I did fix it, but only by finally
> disabling the upstart files bundled in the install 
> (glusterfs-server.conf and mounting-glusterfs.conf).
> 
> This was a clean install of Ubuntu 12.04.  Actually, two clean
> installs for testing.  And then another clean install on one box to
> rule out the original install!  Same behaviour on all installs.
> 
> I spent some time experimenting with different fstab entries and
> found I could change the mount behaviour, but only to determine
> *when* the mount would fail... it always failed!
> 
> My solution was to disable the two Gluster upstart conf files with 
> .override files containing 'manual'.  Then a simple script to a)
> start glusterfs-server; b) sleep for a few seconds; c) mount the
> Gluster volume. The script is called from rc.local.
> 
> Hardly an elegant solution I know, but at some point *'works'*
> beats *'pretty but non-functional'*.
> 
> It's possible there is something quirky with my install that could
> be responsible.  But given that other than installing and
> configuring Gluster, nothing on the servers was tweaked, I'm
> thinking there is a problem with the Gluster upstart conf files.
> Or something in 12.04 has rendered them inoperable. Or maybe it's
> as a friend in Poland says... (direct translation) "it's the
Malice
> of Things..."  Truly a better description than anything we have in
> English! ;-)
> 
> I'm not looking for a solution unless someone has an obvious fix
> that Google couldn't locate.  I have a quick and dirty work around
> that works. I just hope to save someone else the pain.
> 
> 
> 
> _______________________________________________ Gluster-users
> mailing list Gluster-users at gluster.org 
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
- -- 
Raymond Powell
Systems Administrator
Laboratory for Advanced Computing
University of Chicago, KCBD 10160
900 East 57th Street
Chicago, IL 60637
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.20 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJTGN1dAAoJEBIjAczqLCW8rU8QAKk86q7zUVZ5r/iJVM5wteGb
apfYEeiNlTGrNNZxTPTI9I8nHVVHtCit4PmOjUTWAutLtc8DHWxbLTIE0tsmSRZB
Iu6U41DMkl+wBiOFFgNsiaGQPc0gdoe3r8f8kulruA+UNQj5dGSZfkT7+wCeA8mM
xZcHCFVKMSRkJ7uLD/TWsRkRjXBYfPw3vmFC4ya1k6Uh9QSRBew8J/nh6oR1pQj4
w3ix6UtbwC7rxW2d2qHDiuNMdNrmqJE0UROWHSlDyMjKsexcuUhGMVJdNFnsR4uS
ox2d1IdX2D3bz4wY4zCn/dv9GZkuogZJGwKRnuzYUwqtL3bTWjDHhLAwm9ZpZxr5
h04NiNuxcKQiz4WY716JESK/C1/F7GebymxX1fzoejHinACb8lL5ql7BsVA7vyj4
ANgvu35HPUzHk6sIEpslqqQas+hPokLmWCvkGTmhayWNdp6hG4gUboIsVnvEUpKy
Yllnbx5Y1E1f1Fp90FZNHTM+15UDU4hZdSsMBZGLdPkVv0r5zmMQKKCadBuI8w3u
yDYBcz9c6k9XPTUsm2viaV6qkgDsS55CJc4QMX5Ic0jZN6ijiHc+FiZS8ZZ/WfdM
BzQZEkcmcN39vmEhRx4R/qlFqykhUPhhdRslgb3HxacIEotyFF3s8f8UmM5ty8lX
J37OwYbND3kM4IHpKxwb
=z/nW
-----END PGP SIGNATURE-----