Gluster users - Nov 2010 - Problems with Gluster 3.1 and replicate/mirror

Hi :)

In another thread, I had problems with the quick-read translator that
was fixed on 3.1.1. Since I'm using 3.1.0 I disabled the translator
and the updates on small files began to work fine.

Now, I'm having another problem. I'm using 2 servers in
replicate/mirror mode. I can't always reproduce the problem, but it's
happening some times at random. For example, there's a file named
tags.txt on the gluster filesystem, accessed on both clients using
fuse. Both servers can read and update it.

Sometimes, when I update the file on one of the clients, it breaks its
access in the other client:

server1$ md5sum tags.txt
5c6a268f03c8d6b94dc1c3d0bbd3396a
server1$ cat tags.txt
[... full contents of the file ...]

server2$ md5sum tags.txt
5c6a268f03c8d6b94dc1c3d0bbd3396a
server2$ cat tags.txt
cat: tags.txt: No such file or directory

Using vim to edit the file also gives me a "Permission denied". I can
read the directory contents, and even get a md5 checksum of the file,
but when trying to access the file, it fails :( The problem is fixed
when I remount the gluster mount point at the client 2.

I think there's some split brain ocurring. Log messages includes some of
those:

W [fuse-bridge.c:2075:fuse_readdir_cbk] glusterfs-fuse: 1074214:
READDIR => -1 (File descriptor in bad state)
I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] blogs-mirror:
entry self-heal triggered. path: /upload/19/files, reason: check
sums of directory differ, forced merge option set
E [afr-common.c:110:afr_set_split_brain] blogs-mirror: invalid argument: inode

W [fuse-bridge.c:570:fuse_fd_cbk] glusterfs-fuse: 1065386: OPEN()
/tags.txt => -1 (No such file or directory)

There's lot of these, saying that an entry self-heal is triggered.

Maybe a good option to solve this is re-syncing (with rsync, for
example) the server 2 with the server 1. But is this a known bug or
something? I remember this happening when I was messing with read-only
options and translators.

Thanks,

-- 
[]'s
Hugo
www.devin.com.br

Hugo -
     How did you disable the quick-read translator?

Thanks,

Craig

-->
Craig Carl
Senior Systems Engineer
Gluster

On 11/22/2010 06:48 AM, Hugo Cisneiros (Eitch) wrote:
> Hi :)
>
> In another thread, I had problems with the quick-read translator that
> was fixed on 3.1.1. Since I'm using 3.1.0 I disabled the translator
> and the updates on small files began to work fine.
>
> Now, I'm having another problem. I'm using 2 servers in
> replicate/mirror mode. I can't always reproduce the problem, but
it's
> happening some times at random. For example, there's a file named
> tags.txt on the gluster filesystem, accessed on both clients using
> fuse. Both servers can read and update it.
>
> Sometimes, when I update the file on one of the clients, it breaks its
> access in the other client:
>
> server1$ md5sum tags.txt
> 5c6a268f03c8d6b94dc1c3d0bbd3396a
> server1$ cat tags.txt
> [... full contents of the file ...]
>
> server2$ md5sum tags.txt
> 5c6a268f03c8d6b94dc1c3d0bbd3396a
> server2$ cat tags.txt
> cat: tags.txt: No such file or directory
>
> Using vim to edit the file also gives me a "Permission denied". I
can
> read the directory contents, and even get a md5 checksum of the file,
> but when trying to access the file, it fails :( The problem is fixed
> when I remount the gluster mount point at the client 2.
>
> I think there's some split brain ocurring. Log messages includes some
of those:
>
> W [fuse-bridge.c:2075:fuse_readdir_cbk] glusterfs-fuse: 1074214:
> READDIR =>  -1 (File descriptor in bad state)
> I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] blogs-mirror:
> entry self-heal triggered. path: /upload/19/files, reason: check
> sums of directory differ, forced merge option set
> E [afr-common.c:110:afr_set_split_brain] blogs-mirror: invalid argument:
inode
>
> W [fuse-bridge.c:570:fuse_fd_cbk] glusterfs-fuse: 1065386: OPEN()
> /tags.txt =>  -1 (No such file or directory)
>
> There's lot of these, saying that an entry self-heal is triggered.
>
> Maybe a good option to solve this is re-syncing (with rsync, for
> example) the server 2 with the server 1. But is this a known bug or
> something? I remember this happening when I was messing with read-only
> options and translators.
>
> Thanks,
>

This bug was also fixed post 3.1.0 in commit
413b2a5f9b77fd3d7f3b26c848482ec7b914102f

Avati

On Mon, Nov 22, 2010 at 8:18 PM, Hugo Cisneiros (Eitch) <
hugo.cisneiros at gmail.com> wrote:
> Hi :)
>
> In another thread, I had problems with the quick-read translator that
> was fixed on 3.1.1. Since I'm using 3.1.0 I disabled the translator
> and the updates on small files began to work fine.
>
> Now, I'm having another problem. I'm using 2 servers in
> replicate/mirror mode. I can't always reproduce the problem, but
it's
> happening some times at random. For example, there's a file named
> tags.txt on the gluster filesystem, accessed on both clients using
> fuse. Both servers can read and update it.
>
> Sometimes, when I update the file on one of the clients, it breaks its
> access in the other client:
>
> server1$ md5sum tags.txt
> 5c6a268f03c8d6b94dc1c3d0bbd3396a
> server1$ cat tags.txt
> [... full contents of the file ...]
>
> server2$ md5sum tags.txt
> 5c6a268f03c8d6b94dc1c3d0bbd3396a
> server2$ cat tags.txt
> cat: tags.txt: No such file or directory
>
> Using vim to edit the file also gives me a "Permission denied". I
can
> read the directory contents, and even get a md5 checksum of the file,
> but when trying to access the file, it fails :( The problem is fixed
> when I remount the gluster mount point at the client 2.
>
> I think there's some split brain ocurring. Log messages includes some
of
> those:
>
> W [fuse-bridge.c:2075:fuse_readdir_cbk] glusterfs-fuse: 1074214:
> READDIR => -1 (File descriptor in bad state)
> I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk] blogs-mirror:
> entry self-heal triggered. path: /upload/19/files, reason: check
> sums of directory differ, forced merge option set
> E [afr-common.c:110:afr_set_split_brain] blogs-mirror: invalid argument:
> inode
>
> W [fuse-bridge.c:570:fuse_fd_cbk] glusterfs-fuse: 1065386: OPEN()
> /tags.txt => -1 (No such file or directory)
>
> There's lot of these, saying that an entry self-heal is triggered.
>
> Maybe a good option to solve this is re-syncing (with rsync, for
> example) the server 2 with the server 1. But is this a known bug or
> something? I remember this happening when I was messing with read-only
> options and translators.
>
> Thanks,
>
> --
> []'s
> Hugo
> www.devin.com.br
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>