Gluster users - Sep 2012 - Unexpected Gluster behavior on startup without backing stores mounted

Hi,

This may just be unexpected by me. But I'm wondering if there are safeguards
against it, either that are in 3.1.4 that I haven't set up, or in
subsequent versions. What happened:

Despite dual power supplies and UPSes, two Gluster hosts managed to get
suddenly shut down at once. On coming back up, one of the several mirrored
Gluster shares, whose backing store is normally a logical volume mounted at
/mnt/xyz on both hosts, didn't end up with that backing store mounted
because someone (me) must have been distracted the day I should have added
that to fstab.

Here's the unexpected behavior: Gluster restored the nfs export based on the
backing store. But without that backing store really mounted, it used
/mnt/xyz, which at that point was a local subdirectory of /. This is not
optimal. An error or warning or refusal to run would be preferable to
presenting the export when the "real" backing store has gone missing.

Due to particularities of the use case, the error wasn't immediately obvious
(although those who remember my string of messages on a log-filling
samba-related problem a week or so back, well, those were from this
instance, but days later).

Learn something new every day. Didn't visualize that Gluster could work at
all without having the backing stores be unique, full partitions. Yet it
mostly did (aside from the runaway log problem a week back) appear to be
exporting the share normally.

Things are back where they belong now, after I finally found the underlying
cause. But should - and does current if not 3.1.x - Gluster make the attempt
to use, and export, a share based on a backing store other than what it was
originally set up on? It seems to me there are various ways it could have
recognized there was something wrong in this case, logged that, and refused
to export the share. Would have been highly useful if startup had a sanity
check on this.

Obviously the main error was between chair and keyboard here. Still, it
seems the safety of Gluster in this sort of case can be improved, if it
hasn't been already.

Thanks,
Whit

On Thu, Sep 06, 2012 at 03:19:53PM -0400, Whit Blauvelt
wrote:
> Here's the unexpected behavior: Gluster restored the nfs export based
on the
> backing store. But without that backing store really mounted, it used
> /mnt/xyz, which at that point was a local subdirectory of /. This is not
> optimal. An error or warning or refusal to run would be preferable to
> presenting the export when the "real" backing store has gone
missing.
For a different reason, I have been using a subdirectory of the mountpoint
for the brick - e.g. if the filesystem is /mnt/xyz then I have done
mkdir /mnt/xyz/brick1 and set the brick as server1:/mnt/xyz/brick1

However I think this also fixes your problem, because if you try to mount
when the brick1 directory is not present, the glusterfsd process doesn't
start.  The client doesn't get a good error message, but buried in the
server logs you can see what happened.

https://bugzilla.redhat.com/show_bug.cgi?id=839021

[2012-07-10 18:25:57.804526] E [posix.c:3930:init] 0-single1-posix: Directory
'/disk/storage1/single1' doesn't exist, exiting.

Cheers,

Brian.