Gluster users - Sep 2014 - Migrating data from a failing filesystem

I inherited a non-replicated gluster system based on antique hardware.

One of the brick filesystems is flaking out, and remounts read-only.  I
repair it and remount it, but this is only postponing the inevitable.

How can I migrate files off a failing brick that intermittently turns
read-only?  I have enough space, thanks to a catastrophic failure on
another brick; I don't want to present people with another one.  But if I
understand migration correctly references have to be deleted, which isn't
possible if the filesystem turns read-only.

What I want to do is migrate the files off, remove it from gluster,
rebuild the array, rebuild the filesystem, and then add it back as a
brick.  (Actually what I'd really like is to hear that the students are
all done with the system and I can turn the whole thing off, but theses
aren't complete yet.)

Any advice or words of warning will be appreciated.

James Bellinger

On 09/23/2014 08:56 PM, james.bellinger at icecube.wisc.edu
wrote:
> I inherited a non-replicated gluster system based on antique hardware.
>
> One of the brick filesystems is flaking out, and remounts read-only.  I
> repair it and remount it, but this is only postponing the inevitable.
>
> How can I migrate files off a failing brick that intermittently turns
> read-only?  I have enough space, thanks to a catastrophic failure on
> another brick; I don't want to present people with another one.  But if
I
> understand migration correctly references have to be deleted, which
isn't
> possible if the filesystem turns read-only.
What you could do is initiate the  migration  with `remove-brick start' 
and monitor the progress with 'remove-brick status`. Irrespective of 
whether the rebalance  completes or fails (due to the brick turning 
read-only), you could anyway update the volume configuration with 
'remove-brick commit`. Now if the brick still has files left, mount the 
gluster volume on that node and copy the files from the brick to the 
volume via the mount.  You can then safely rebuild the array/ add a 
different brick or whatever.
> What I want to do is migrate the files off, remove it from gluster,
> rebuild the array, rebuild the filesystem, and then add it back as a
> brick.  (Actually what I'd really like is to hear that the students are
> all done with the system and I can turn the whole thing off, but theses
> aren't complete yet.)
>
> Any advice or words of warning will be appreciated.
Looks like your bricks are in trouble for over a year now 
(http://gluster.org/pipermail/gluster-users.old/2013-September/014319.html). 
Better get them fixed sooner than later! :-)
HTH,
Ravi
> James Bellinger
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users