Gluster users - Oct 2011 - Vol full of .*.gfs* after migrate-data

Hello All,
I have been testing rebalance...migrate-data in GlusterFS version 3.2.3, 
following add-brick and fix-layout.  After migrate-data the the volume 
is 97% full with some bricks being 100% full.  I have not added any 
files to the volume so there should be an amount of free space at least 
as big as the new bricks that were added.  However, it seems as if all 
the extra space has been taken up with files matching the pattern 
.*.gfs*.  I presume these are temporary files used for the transfer real 
files, which should have been renamed once the transfers were completed 
and verified, and the original versions deleted.  The new bricks contain 
mostly these temporary files, and zero byte link files pointing to the 
corresponding real files on other bricks.  An example of such a pair is 
shown below.

---------T 1 root root 0 Sep 30 03:14 
/mnt/local/glusterfs/root/backup/behemoth_system/bin/df
-rwxr-xr-x 1 root root 60416 Sep 30 18:20 
/mnt/local/glusterfs/root/backup/behemoth_system/bin/.df.gfs60416

Is this a known bug, and is there a work-around?  If not, is it safe to 
delete the .*.gfs* files so I can at least use the volume?

Regards
Dan Bretherton

Dan,

Answer inline.

On 02-Oct-2011, at 1:26 AM, Dan Bretherton <d.a.bretherton at
reading.ac.uk> wrote:
> Hello All,
> I have been testing rebalance...migrate-data in GlusterFS version 3.2.3,
following add-brick and fix-layout.  After migrate-data the the volume is 97%
full with some bricks being 100% full.  I have not added any files to the volume
so there should be an amount of free space at least as big as the new bricks
that were added.  However, it seems as if all the extra space has been taken up
with files matching the pattern .*.gfs*.  I presume these are temporary files
used for the transfer real files, which should have been renamed once the
transfers were completed and verified, and the original versions deleted.  The
new bricks contain mostly these temporary files, and zero byte link files
pointing to the corresponding real files on other bricks.  An example of such a
pair is shown below.
> 
> ---------T 1 root root 0 Sep 30 03:14
/mnt/local/glusterfs/root/backup/behemoth_system/bin
> -rwxr-xr-x 1 root root 60416 Sep 30 18:20
/mnt/local/glusterfs/root/backup/behemoth_system/bin/.df.gfs60416
> 
> Is this a known bug, and is there a work-around?  If not, is it safe to
delete the .*.gfs* files so I can at least use the volume?
> 
This is not a known issue but surely seems like a bug. If the source file is
intact you can delete the temp file to get the space back. Also if md5sum is
same, you can rename temp file to original, so you get space in existing bricks.

Regards,
Amar

> Regards
> Dan Bretherton
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Hi,
> I have been testing rebalance...migrate-data in GlusterFS version 3.2.3, 
> following add-brick and fix-layout.  After migrate-data the the volume 
> is 97% full with some bricks being 100% full.  I have not added any 
> files to the volume so there should be an amount of free space at least 
> as big as the new bricks that were added.  However, it seems as if all 
> the extra space has been taken up with files matching the pattern 
> .*.gfs*.  I presume these are temporary files used for the transfer real 
> files, which should have been renamed once the transfers were completed 
> and verified, and the original versions deleted.  The new bricks contain 
> mostly these temporary files, and zero byte link files pointing to the 
> corresponding real files on other bricks.  An example of such a pair is 
> shown below.
> 
> ---------T 1 root root 0 Sep 30 03:14 
> /mnt/local/glusterfs/root/backup/behemoth_system/bin/df
> -rwxr-xr-x 1 root root 60416 Sep 30 18:20 
> /mnt/local/glusterfs/root/backup/behemoth_system/bin/.df.gfs60416
> 
> Is this a known bug, and is there a work-around?  If not, is it safe to 
> delete the .*.gfs* files so I can at least use the volume?
I have had the same exact behaviour with the latest package, on Ubuntu
11.0. The log shows a file size error after rebalance; while both file
are identical. I presume that may be due to some caching done in the
OS and the new file is not yet properly fulshed when it is being
stat'ed for it's size.

Best regards,

Olivier
> Regards
> Dan Bretherton
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>

Hi Amar,
>       > Is this a known bug, and is there a work-around? ?If not, is
>       it safe to
>       > delete the .*.gfs* files so I can at least use the volume?
> 
> I have had the same exact behaviour with the latest package, on Ubuntu
> 11.0. The log shows a file size error after rebalance; while both file
> are identical. I presume that may be due to some caching done in the
> OS and the new file is not yet properly fulshed when it is being
> stat'ed for it's size.
> 
> 
> A patch is sent ?to remove the temporary file if any error happens while
> copying the data. (ref: ?http://review.gluster.com/552)?
Thank you for the patch. I have moved to test on CentOS where there is no 
such behaviour.
> Mean time, if you send me the exact log of 'rebalance process'
(found at
> /var/log/glusterfs/etc-glusterd-mount-<volname>.log) it will help to
corner
> the issue faster. Anyways, We will look into possibilities of the
> mismatching file size errors.
I attached a bzip'ed/tar'ed file that contains the log of 
etc-glusterd-mount-test-vol.log and nfs.log (that shows the error).

Best regards,

Olivier
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gluster.tar.bz2
Type: application/octet-stream
Size: 170691 bytes
Desc: 
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111003/04d3768a/attachment.obj>