Gluster users - Nov 2010 - expected behavior when only some servers available?

I'm trying out a configuration where, for the purpose of gluster,
the universe consists of only two CentOS 5.5 hosts (outside of gluster,
this is a traditional HA cluster).

I'm seeing some hangs, and I'd like to verify my understanding of
the *expected* behavior of glusterfs.

What I would like to do is both use these nodes to serve a replicated
filesystem *and* to mount /home on those nodes on that replicated
filesystem.

So I have gluster 3.1.0 configured on both nodes.  If I have both
nodes running I can manually mount /home via glusterfs.  Fine.
(This is using the native client, not NFS.)

Now in testing I wanted to see what would happen from a cold start.
Knowing that I potentially had a chicken-and-egg problem, I changed
the glusterd init script to use:
    # chkconfig: 35 24 76
A start number of 24 was picked because NFS, gluster, and other network
filesystems get mounted via /etc/rc.d/rc3.d/S25netfs.  So fine, the
daemon is started before we try to mount the filesystem (verified
by watching the boot sequence).

I then changed /etc/fstab to mount /home on glusterfs at boot time
via the native client:

    /etc/glusterd/vols/glhome/glhome-fuse.vol      /home   glusterfs
volume-name=glhome     0 0
    
I did a controlled shut down of both nodes, and left NodeB powered
off.  I powered up NodeA and watched the boot sequence.  Glusterd
starts fine, but the system hangs when it gets to mounting the filesystem.

Reboot to single user mode, comment out the /home entry in fstab, 
reboot.  Up and running again.  Manually mount /home, and then
do a 'df /home'.  The df (or an ls; tried in another shell) hangs
and ^C can't break out of it.

So I'm making the assumption here that both the mount-at-boot problem
and the df-hang problem are due to NodeB being powered off.  The
question is, though, is this actually the "expected" behavior?
Is a two-node gluster cluster not usable by the native client
unless *both* nodes are up?  (Or that they were at least up at some
point?)  If this is not the expected behavior, what am I missing?

Devin

This behavior is fixed in the repository code and will be available in
3.1.1.

Avati

On Thu, Nov 18, 2010 at 11:06 AM, Devin Reade <gdr at gno.org> wrote:
> I'm trying out a configuration where, for the purpose of gluster,
> the universe consists of only two CentOS 5.5 hosts (outside of gluster,
> this is a traditional HA cluster).
>
> I'm seeing some hangs, and I'd like to verify my understanding of
> the *expected* behavior of glusterfs.
>
> What I would like to do is both use these nodes to serve a replicated
> filesystem *and* to mount /home on those nodes on that replicated
> filesystem.
>
> So I have gluster 3.1.0 configured on both nodes.  If I have both
> nodes running I can manually mount /home via glusterfs.  Fine.
> (This is using the native client, not NFS.)
>
> Now in testing I wanted to see what would happen from a cold start.
> Knowing that I potentially had a chicken-and-egg problem, I changed
> the glusterd init script to use:
>    # chkconfig: 35 24 76
> A start number of 24 was picked because NFS, gluster, and other network
> filesystems get mounted via /etc/rc.d/rc3.d/S25netfs.  So fine, the
> daemon is started before we try to mount the filesystem (verified
> by watching the boot sequence).
>
> I then changed /etc/fstab to mount /home on glusterfs at boot time
> via the native client:
>
>    /etc/glusterd/vols/glhome/glhome-fuse.vol      /home   glusterfs
> volume-name=glhome     0 0
>
> I did a controlled shut down of both nodes, and left NodeB powered
> off.  I powered up NodeA and watched the boot sequence.  Glusterd
> starts fine, but the system hangs when it gets to mounting the filesystem.
>
> Reboot to single user mode, comment out the /home entry in fstab,
> reboot.  Up and running again.  Manually mount /home, and then
> do a 'df /home'.  The df (or an ls; tried in another shell) hangs
> and ^C can't break out of it.
>
> So I'm making the assumption here that both the mount-at-boot problem
> and the df-hang problem are due to NodeB being powered off.  The
> question is, though, is this actually the "expected" behavior?
> Is a two-node gluster cluster not usable by the native client
> unless *both* nodes are up?  (Or that they were at least up at some
> point?)  If this is not the expected behavior, what am I missing?
>
> Devin
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>