Gluster users - Sep 2009 - self-heal and file deletion problem

I attempted to simulate a node failure with complete loss of data the the
exported filesystem and have notiiced that after bringing the node back
online, the files that it used to contain do not get re-created on the node
and in fact they get deleted from the /gluster filesystem.

Here's the situation:

2 pairs of 2 nodes with cluster/replicate
the 2 pairs then are distributed.

I created zero-byte files on the glusterfs and noted how they were
distributed between pairs and replicated between nodes. I then shutdown
glusterfsd and the glusterfs mount point on node 3. Then I blew away the
data on the underlying filesystem and rebooted. When the machine came back
up the data (which is still contained on node 4, its replication partner)
was deleted from the glusterfs and was not self-healed.

Am I missing something here?

On Fri, 4 Sep 2009 15:30:49 -0600
mike foster <mfosterm at gmail.com> wrote:
> I attempted to simulate a node failure with complete loss of data the the
> exported filesystem and have notiiced that after bringing the node back
> online, the files that it used to contain do not get re-created on the node
> and in fact they get deleted from the /gluster filesystem.
> 
> Here's the situation:
> 
> 2 pairs of 2 nodes with cluster/replicate
> the 2 pairs then are distributed.
> 
> I created zero-byte files on the glusterfs and noted how they were
> distributed between pairs and replicated between nodes. I then shutdown
> glusterfsd and the glusterfs mount point on node 3. Then I blew away the
> data on the underlying filesystem and rebooted. When the machine came back
> up the data (which is still contained on node 4, its replication partner)
> was deleted from the glusterfs and was not self-healed.
> 
> Am I missing something here?
This sounds like node 3 is a primary node of a replicate set. If you shoot
down your nodes  1 and 3 you will notice this problem. If you shoot down node
2 and 4 you will see the files being healed.
BUT you should nevertheless be able to heal the data by only stat'ing the
missing files on the clients - even if you cannot see them first, they should
reappear after stat.

-- 
Regards,
Stephan