Gluster users - Oct 2010 - Can't delete or add files when a node fails.

I am trying to track down a problem I reported on the list last week and
discovered a new problem during my testing.

If you have a four node setup with replicate/distribute and one of the nodes has
a filesystem failure, the operating system will typically remount the filesystem
read only.   When this happens, the glusterfsd is still running on the failed
machine, but i doesn't seem to recognize that there is a problem.   If you
try to create new files from a client and do an ls you will see that some of the
files don't appear.  Conversely if you remove files from the client they
will still be there along with their content.

This is trivial to reproduce by remounting the filesystem readonly on one of the
bricks.   If you are on a typical linux install and the gluster export directory
is part of the root filesystem, you would only need to 'mount -o
remount,abort /'

Considering that this is a very typical path for failure, I would expect gluster
to handle this properly.

Regards,

Brian Hirt

bump

Begin forwarded message:
> From: Brian Hirt <bhirt at me.com>
> Date: October 8, 2010 8:01:58 AM MDT
> To: gluster-users at gluster.org
> Subject: Can't delete or add files when a node fails.
> 
> I am trying to track down a problem I reported on the list last week and
discovered a new problem during my testing.
> 
> If you have a four node setup with replicate/distribute and one of the
nodes has a filesystem failure, the operating system will typically remount the
filesystem read only.   When this happens, the glusterfsd is still running on
the failed machine, but i doesn't seem to recognize that there is a problem.
If you try to create new files from a client and do an ls you will see that some
of the files don't appear.  Conversely if you remove files from the client
they will still be there along with their content.
> 
> This is trivial to reproduce by remounting the filesystem readonly on one
of the bricks.   If you are on a typical linux install and the gluster export
directory is part of the root filesystem, you would only need to 'mount -o
remount,abort /'
> 
> Considering that this is a very typical path for failure, I would expect
gluster to handle this properly.
> 
> Regards,
> 
> Brian Hirt
>

Brian - 
This is to be expected. If the filesystem `disappears` from under Gluster,
Gluster will need to be restarted in order to reconnect to it. This appears to
be a FUSE limitation.



Thanks, 

Craig 

-- 
Craig Carl 













Senior Systems Engineer; Gluster, Inc. 
Cell - ( 408) 829-9953 (California, USA) 
Office - ( 408) 770-1884 
Gtalk - craig.carl at gmail.com 
Twitter - @gluster 
Installing Gluster Storage Platform, the movie! 
http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ 



From: "Brian Hirt" <bhirt at me.com> 
To: gluster-users at gluster.org 
Sent: Friday, October 8, 2010 7:01:58 AM 
Subject: [Gluster-users] Can't delete or add files when a node fails. 

I am trying to track down a problem I reported on the list last week and
discovered a new problem during my testing.

If you have a four node setup with replicate/distribute and one of the nodes has
a filesystem failure, the operating system will typically remount the filesystem
read only. When this happens, the glusterfsd is still running on the failed
machine, but i doesn't seem to recognize that there is a problem. If you try
to create new files from a client and do an ls you will see that some of the
files don't appear. Conversely if you remove files from the client they will
still be there along with their content.

This is trivial to reproduce by remounting the filesystem readonly on one of the
bricks. If you are on a typical linux install and the gluster export directory
is part of the root filesystem, you would only need to 'mount -o
remount,abort /'

Considering that this is a very typical path for failure, I would expect gluster
to handle this properly.

Regards, 

Brian Hirt 


_______________________________________________ 
Gluster-users mailing list 
Gluster-users at gluster.org 
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users