Gluster users - Sep 2012 - A problem with gluster 3.3.0 and Sun Grid Engine

Hi,

	
	We got a huge problem on our sun grid engine cluster with glusterfs
3.3.0. Could somebody help me?


	Based on my understanding, if a folder is removed and recreated on
other client node, a program that tries to create a new file under the
folder fails very often.


	We partially fixed this problem by "ls" the folder before doing
anything in our command, however, Sun Grid Engine tries to create a new
log file  before executing our script. We often get the error message
"no such file or directory" in SGE log and cannot do anything about
it.


	flush-behind is already turned off. What's the next thing we should
try? 


Error log on client is 

[2012-09-10 11:18:48.129102] W
[client3_1-fops.c:2630:client3_1_lookup_cbk] 0-home-client-0: remote
operation failed: Stale NFS file handle. Path: XXXXXXXXXXXXXX
(488a7270-1039-473b-9122-ae07b9e2c617)
[2012-09-10 11:18:48.129162] W
[client3_1-fops.c:2630:client3_1_lookup_cbk] 0-home-client-1: remote
operation failed: Stale NFS file handle. Path: XXXXXXXXXXXXXX
(488a7270-1039-473b-9122-ae07b9e2c617)


Error log on two brick severs are


[2012-09-10 11:18:48.046828] I [server3_1-fops.c:1707:server_stat_cbk]
0-home-server: 3531033: STAT (null) (--) ==> -1 (No such file or
directory)
[2012-09-10 11:18:48.047176] I [server3_1-fops.c:1707:server_stat_cbk]
0-home-server: 3531035: STAT (null) (--) ==> -1 (No such file or
directory)


[2012-09-10 11:18:48.054315] I [server3_1-fops.c:1707:server_stat_cbk]
0-home-server: 5347741: STAT (null) (--) ==> -1 (No such file or
directory)
[2012-09-10 11:18:48.054544] I [server3_1-fops.c:1707:server_stat_cbk]
0-home-server: 5347742: STAT (null) (--) ==> -1 (No such file or
directory)
[2012-09-10 11:18:48.056036] I [server3_1-fops.c:1707:server_stat_cbk]
0-home-server: 5347746: STAT (null) (--) ==> -1 (No such file or
directory)



Best,
Manhong
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120910/8e763302/attachment.html>

On Mon, Sep 10, 2012 at 8:30 AM, Manhong Dai <daimh at umich.edu> wrote:
> **
> Hi,
>
>
> We got a huge problem on our sun grid engine cluster with glusterfs 3.3.0.
> Could somebody help me?
>
>
> Based on my understanding, if a folder is removed and recreated on other
> client node, a program that tries to create a new file under the folder
> fails very often.
>

Is the directory deleted and recreated by another client/mount while the
application which attempts to create the file stays cd'ed inside the
directory? Can you try to confirm if this is the pattern?

Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120910/94d98e69/attachment.html>