Gluster users - Oct 2014 - Initial sync

Hi,

I'm curious about how GlusterFS manages to sync the bricks in the initial
phase, when
the volume is created or
extended.

I first create a volume consisting of only one brick, which clients will start
to
read and write.
After a while I add a second brick to the volume to create a replicated volume.

If this new brick is empty, I guess that files will be copied from the first
brick to
get the bricks in sync, right?

However, if the second brick is not empty but rather contains a subset of the
files
on the first brick I don't see
how GlusterFS will solve the problem of syncing the bricks.

I guess that all files which lack extended attributes could be removed in this
scenario, because they were created
when the disk was not part of a GlusterFS volume. However, in case the brick was
used
in the volume previously,
for instance before that server restarted, there will be extended attributes for
the
files on the second brick which
weren't updated during the downtime (when the volume consisted of only one
brick).
There could be multiple
changes to the files during this time. In this case I don't understand how
the
extended attributes could be used to
determine which of the bricks contains the most recent file.

Can anyone explain how this works? Is it only allowed to add empty bricks to a
volume?

 

-- 
Regards
Andreas

On 10/28/2014 03:58 PM, Andreas Hollaus wrote:
> Hi,
>
> I'm curious about how GlusterFS manages to sync the bricks in the
initial phase, when
> the volume is created or
> extended.
>
> I first create a volume consisting of only one brick, which clients will
start to
> read and write.
> After a while I add a second brick to the volume to create a replicated
volume.
>
> If this new brick is empty, I guess that files will be copied from the
first brick to
> get the bricks in sync, right?
>
> However, if the second brick is not empty but rather contains a subset of
the files
> on the first brick I don't see
> how GlusterFS will solve the problem of syncing the bricks.
>
> I guess that all files which lack extended attributes could be removed in
this
> scenario, because they were created
> when the disk was not part of a GlusterFS volume. However, in case the
brick was used
> in the volume previously,
> for instance before that server restarted, there will be extended
attributes for the
> files on the second brick which
> weren't updated during the downtime (when the volume consisted of only
one brick).
> There could be multiple
> changes to the files during this time. In this case I don't understand
how the
> extended attributes could be used to
> determine which of the bricks contains the most recent file.
>
> Can anyone explain how this works? Is it only allowed to add empty bricks
to a volume?
>
>   
>
It is allowed to add only empty bricks to the volume. Writing directly 
to bricks is not supported. One needs to access the volume only from a 
mount point or using libgfapi.
After adding a brick to increase the distribute count, you need to run 
the volume rebalance command so that the some of the existing files are 
hashed (moved) to this newly added brick.
After adding a brick to increase the replica count, you need to run the 
volume heal full command to sync the files from the other replica into 
the newly added brick.
https://github.com/gluster/glusterfs/blob/master/doc/features/afr-v1.md 
will give you an idea of how the replicate translator uses xattrs to 
keep files in sync.

HTH,
Ravi