Gluster users - Apr 2012 - files on stripe not visible with ls or stat, but files visible at the filesystem layer can still be cat'd / viewed through the gluster fuse layer (you just have to know the file name)

Hi,

I've got a strange issue inside of a directory on a 5 brick stripe. ls
or stat doesn't show any of the files in the directory through the
gluster fuse mount and I get this error in the client logs :

[2012-04-05 16:19:58.327166] W [fuse-bridge.c:2095:fuse_readdir_cbk]
0-glusterfs-fuse: 1417: READDIR => -1 (No such file or directory)
[2012-04-05 16:21:32.647667] E
[client3_1-fops.c:411:client3_1_stat_cbk] 0-pirstripe-client-2: remote
operation failed: No such file or directory
[2012-04-05 16:21:32.647925] E
[client3_1-fops.c:411:client3_1_stat_cbk] 0-pirstripe-client-3: remote
operation failed: No such file or directory
[2012-04-05 16:21:32.647981] E
[client3_1-fops.c:411:client3_1_stat_cbk] 0-pirstripe-client-1: remote
operation failed: No such file or directory
[2012-04-05 16:21:32.648034] E
[client3_1-fops.c:411:client3_1_stat_cbk] 0-pirstripe-client-4: remote
operation failed: No such file or directory

However, if I cat the file through the fuse layer the contents show
up. I can also see that the files exist in the filesystem layer on all
the bricks. Files which are smaller than the stripe size exist on one
of the nodes and all the other files are 0 byted, which looks correct.
Large files are properly split across the bricks. I've tried
restarting glusterd and glusterfsd on the bricks, unmounting and
remounting the fuse mount point from several clients, but the files
still don't want to show up. The server logs don't show any errors
whenever I try an ls in the directory, the READDIR error only shows up
in the client log.

How can I get the files to show up in ls?

Thanks,
Sabuj

Also if I try to rm -rf the directory from the fuse mount, it'll tell
me that it has files in it and won't let me do it.

On Thu, Apr 5, 2012 at 4:30 PM, Sabuj Pattanayek <sabujp at gmail.com>
wrote:
> Hi,
>
> I've got a strange issue inside of a directory on a 5 brick stripe. ls
> or stat doesn't show any of the files in the directory through the
> gluster fuse mount and I get this error in the client logs :
>
> [2012-04-05 16:19:58.327166] W [fuse-bridge.c:2095:fuse_readdir_cbk]
> 0-glusterfs-fuse: 1417: READDIR => -1 (No such file or directory)
> [2012-04-05 16:21:32.647667] E
> [client3_1-fops.c:411:client3_1_stat_cbk] 0-pirstripe-client-2: remote
> operation failed: No such file or directory
> [2012-04-05 16:21:32.647925] E
> [client3_1-fops.c:411:client3_1_stat_cbk] 0-pirstripe-client-3: remote
> operation failed: No such file or directory
> [2012-04-05 16:21:32.647981] E
> [client3_1-fops.c:411:client3_1_stat_cbk] 0-pirstripe-client-1: remote
> operation failed: No such file or directory
> [2012-04-05 16:21:32.648034] E
> [client3_1-fops.c:411:client3_1_stat_cbk] 0-pirstripe-client-4: remote
> operation failed: No such file or directory
>
> However, if I cat the file through the fuse layer the contents show
> up. I can also see that the files exist in the filesystem layer on all
> the bricks. Files which are smaller than the stripe size exist on one
> of the nodes and all the other files are 0 byted, which looks correct.
> Large files are properly split across the bricks. I've tried
> restarting glusterd and glusterfsd on the bricks, unmounting and
> remounting the fuse mount point from several clients, but the files
> still don't want to show up. The server logs don't show any errors
> whenever I try an ls in the directory, the READDIR error only shows up
> in the client log.
>
> How can I get the files to show up in ls?
>
> Thanks,
> Sabuj