Gluster users - Nov 2009 - Replication does not replicate the reference node on crash

Hi ! I'm continuing to test glusterfs,

I've a strange problem us replication in fact. My setup is prety easy,
i've
3 nodes, and I want to make replication through this 3 nodes

Basic configuration:

volume mega-replicator
       type       cluster/replicate
       subvolumes zbrick1 zbrick2 zbrick3
end-volume

The scenario:

1) I copy data to the gluster mount point: -> Data is copied on the 3 nodes
[OK]
2) I kill zbrick2 glusterfs, I erase data on zbrick2, I remount glusterfs
mount point, do a ls -Rl on the mount point, data is replicated to zbrick2
[OK]
3) I do exactly the same thing with zbrick1, when I mount again the zbrick1,
I do a ls -Rl on the mount point and there is nothing, the directory is
empty.
    So I do a ls -Rl on the zbrick2 and on zbrick3, and I get the same
result, nothing on the mount point.
    Why ?


Note: that I use the latest version from the git repository.

Joel

On Thu, Nov 05, 2009 at 01:51:20PM +0100, joel vennin
wrote:
> Basic configuration:
> 
> volume mega-replicator
>        type       cluster/replicate
>        subvolumes zbrick1 zbrick2 zbrick3
> end-volume
> 
> The scenario:
> 
> 1) I copy data to the gluster mount point: -> Data is copied on the 3
nodes
> [OK]
> 2) I kill zbrick2 glusterfs, I erase data on zbrick2, I remount glusterfs
> mount point, do a ls -Rl on the mount point, data is replicated to zbrick2
> [OK]
> 3) I do exactly the same thing with zbrick1, when I mount again the
zbrick1,
> I do a ls -Rl on the mount point and there is nothing, the directory is
> empty.
Could you please be more precise: do you mount zbrick1 only (expected to
be empty) or the whole mega-replicator volume? Was glusterfsd restarted
on the node serving zbrick1, or was it all done on a "living body"?
These questions are possibly naive or even stupid, but many troubles result
from simple omissions...
>     So I do a ls -Rl on the zbrick2 and on zbrick3, and I get the same
> result, nothing on the mount point.
Does it mean that zbrick2 and zbrick3 became empty _after_ rebuilding
zbrick1 with ls -R? Data deleted, because empty zbrick1 was "newer"?
Krzysztof

On Thu, 5 Nov 2009 13:51:20 +0100
joel vennin <joel.vennin at gmail.com> wrote:
> Hi ! I'm continuing to test glusterfs,
> 
> I've a strange problem us replication in fact. My setup is prety easy,
i've
> 3 nodes, and I want to make replication through this 3 nodes
> 
> Basic configuration:
> 
> volume mega-replicator
>        type       cluster/replicate
>        subvolumes zbrick1 zbrick2 zbrick3
> end-volume
> 
> The scenario:
> 
> 1) I copy data to the gluster mount point: -> Data is copied on the 3
nodes
> [OK]
> 2) I kill zbrick2 glusterfs, I erase data on zbrick2, I remount glusterfs
> mount point, do a ls -Rl on the mount point, data is replicated to zbrick2
> [OK]
> 3) I do exactly the same thing with zbrick1, when I mount again the
zbrick1,
> I do a ls -Rl on the mount point and there is nothing, the directory is
> empty.
>     So I do a ls -Rl on the zbrick2 and on zbrick3, and I get the same
> result, nothing on the mount point.
>     Why ?
This sounds like the classical case of not being able to heal brick1 without
knowing the file names. Please try to create a file list on brick 2 with find
and then ls the files in the list on the mountpoint. You should see healing of
brick1. The docs talk explicitly about this point.

> 
> 
> Note: that I use the latest version from the git repository.
> 
> Joel
> 

-- 
Regards,
Stephan

joel vennin wrote:
> Hi ! I'm continuing to test glusterfs,
>
> I've a strange problem us replication in fact. My setup is prety easy,
i've
> 3 nodes, and I want to make replication through this 3 nodes
>
> Basic configuration:
>
> volume mega-replicator
>        type       cluster/replicate
>        subvolumes zbrick1 zbrick2 zbrick3
> end-volume
>
> The scenario:
>
> 1) I copy data to the gluster mount point: -> Data is copied on the 3
nodes
> [OK]
> 2) I kill zbrick2 glusterfs, I erase data on zbrick2, I remount glusterfs
> mount point, do a ls -Rl on the mount point, data is replicated to zbrick2
> [OK]
> 3) I do exactly the same thing with zbrick1, when I mount again the
zbrick1,
> I do a ls -Rl on the mount point and there is nothing, the directory is
> empty.
>     So I do a ls -Rl on the zbrick2 and on zbrick3, and I get the same
> result, nothing on the mount point.
>     Why ?
>
>
>   
This is a known issue for now:
http://gluster.com/community/documentation/index.php/Understanding_AFR_Translator#Self-heal_of_a_file_that_does_not_exist_on_the_first_subvolume

Vikas