On 03/31/2015 05:12 AM, Lilley, John F. wrote:> Hi, > > I'd like to shrink my aws/ebs-based *distribute-only* gluster file > system by migrating the data to other already existing, active and > partially utilized bricks but found the 'replace-brick start' > mentioned in the documentation is now deprecated. I see that there > has been some back and forth on the mailing list regarding migrating > data using self-heal on a replicated system but not so much on a > distribute-only file system. Can anyone tell me the blessed way of > doing this in 3.6.2? Is there one? > > To be clear, all of the ebs-based bricks are partially utilized at > this point so I'd need a method to migrate the data first. >If I understand you correctly, you want to replace a brick in a distribute volume with one of lesser capacity. You could first add a new brick and then remove the existing brick with remove-brick start/status/commit sequence. Something like this: ------------------------------------------------ [root at tuxpad ~]# gluster volume info testvol Volume Name: testvol Type: Distribute Volume ID: a89aa154-885c-4e14-8d3a-b555733b11f1 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 127.0.0.2:/home/ravi/bricks/brick1 Brick2: 127.0.0.2:/home/ravi/bricks/brick2 Brick3: 127.0.0.2:/home/ravi/bricks/brick3 [root at tuxpad ~]# [root at tuxpad ~]# [root at tuxpad ~]# gluster volume add-brick testvol 127.0.0.2:/home/ravi/bricks/brick{4..6} volume add-brick: success [root at tuxpad ~]# [root at tuxpad ~]# gluster volume info testvol Volume Name: testvol Type: Distribute Volume ID: a89aa154-885c-4e14-8d3a-b555733b11f1 Status: Started Number of Bricks: 6 Transport-type: tcp Bricks: Brick1: 127.0.0.2:/home/ravi/bricks/brick1 Brick2: 127.0.0.2:/home/ravi/bricks/brick2 Brick3: 127.0.0.2:/home/ravi/bricks/brick3 Brick4: 127.0.0.2:/home/ravi/bricks/brick4 Brick5: 127.0.0.2:/home/ravi/bricks/brick5 Brick6: 127.0.0.2:/home/ravi/bricks/brick6 [root at tuxpad ~]# [root at tuxpad ~]# [root at tuxpad ~]# [root at tuxpad ~]# gluster v remove-brick testvol 127.0.0.2:/home/ravi/bricks/brick{1..3} start volume remove-brick start: success ID: d535675e-8362-4a44-a291-1e567a77531e [root at tuxpad ~]# gluster v remove-brick testvol 127.0.0.2:/home/ravi/bricks/brick{1..3} status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 10 0Bytes 20 0 0 *completed * 0.00 [root at tuxpad ~]# [root at tuxpad ~]# gluster v remove-brick testvol 127.0.0.2:/home/ravi/bricks/brick{1..3} commit Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y volume remove-brick commit: success *Check the removed bricks to ensure all files are migrated.** **If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. * [root at tuxpad ~]# [root at tuxpad ~]# gluster volume info testvol Volume Name: testvol Type: Distribute Volume ID: a89aa154-885c-4e14-8d3a-b555733b11f1 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 127.0.0.2:/home/ravi/bricks/brick4 Brick2: 127.0.0.2:/home/ravi/bricks/brick5 Brick3: 127.0.0.2:/home/ravi/bricks/brick6 [root at tuxpad ~]# ------------------------------------------------ Hope this helps. Ravi> Thank You, > John > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150401/02fdb403/attachment.html>
Lilley, John F.
2015-Apr-02 21:52 UTC
[Gluster-users] Shrinking gluster filesystem in 3.6.2
Hi Ravi, Thanks for the clarification. For one reason or another I thought there was no longer a way to shrink a gluster filesystem but it appears from testing you are correct. I generated 100 test files on a 2node/4 ebs brick distributed gluster volume, removed one brick, committed and ended up having those files successfully migrated to the remaining 3 bricks. Volume Name: volume1 Type: Distribute Volume ID: 1180f5ec-af82-4322-8fc2-1add01161442 Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: gluster1.local:/export/xvdf/brick Brick2: gluster1.local:/export/xvdg/brick Brick3: gluster2.local:/export/xvdf/brick Brick4: gluster2.local:/export/xvdg/brick [root at ip-172-31-19-77 ~]# Gluster1 [root at ip-172-31-19-77 ~]# ls /export/xvdf/brick/ 1 13 24 26 33 47 48 5 51 53 54 58 61 62 64 65 70 72 73 8 9 91 94 96 97 98 [root at ip-172-31-19-77 ~]# ls /export/xvdg/brick/ 11 18 19 20 23 27 28 32 35 38 44 46 49 63 68 69 7 78 79 82 85 86 88 93 99 [root at ip-172-31-19-77 ~]# ls /export/xvdf/brick/ | wc -l 26 [root at ip-172-31-19-77 ~]# ls /export/xvdg/brick/ | wc -l 25 Gluster2 [ec2-user at ip-172-31-19-78 ~]$ ls /export/xvdf/brick/ 16 22 29 30 34 37 4 41 43 56 57 60 71 74 75 76 80 83 89 90 92 [ec2-user at ip-172-31-19-78 ~]$ ls /export/xvdg/brick/ 10 12 15 2 25 31 39 42 50 55 6 67 81 87 100 14 17 21 3 36 40 45 52 59 66 77 84 95 [ec2-user at ip-172-31-19-78 ~]$ ls /export/xvdg/brick/ | wc -l 28 [ec2-user at ip-172-31-19-78 ~]$ ls /export/xvdf/brick/ | wc -l 21 Remove Brick Operation [root at ip-172-31-19-77 ~]# gluster volume remove-brick volume1 gluster1.local:/export/xvdf/brick start volume remove-brick start: success ID: b0a6decd-9130-4773-97f9-af108ff0bd77 [root at ip-172-31-19-77 ~]# gluster volume remove-brick volume1 gluster1.local:/export/xvdf/brick status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 41 0Bytes 141 0 0 completed 1.00 [root at ip-172-31-19-77 ~]# gluster volume remove-brick volume1 gluster1.local:/export/xvdf/brick commit Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y volume remove-brick commit: success [root at ip-172-31-19-77 ~]# gluster volume info volume1 Volume Name: volume1 Type: Distribute Volume ID: 1180f5ec-af82-4322-8fc2-1add01161442 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: gluster1.local:/export/xvdg/brick Brick2: gluster2.local:/export/xvdf/brick Brick3: gluster2.local:/export/xvdg/brick I checked the brick mount points and the overall gluster mount point and found that it did indeed migrate the data from the removed brick to the others and all files were accounted for. I checked the most recent documentation on github here https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_managing_volumes.md#shrinking-volumes and found only examples for 'remove-brick force' (which obviously doesn't migrate any data) and 'replace-brick' (which is now deprecated but isn't noted as such in the docs). I'm left wondering why they don't have a non-forceful remove-brick operation documented for people that simply want to remove a brick while migrating data to remaining bricks. Am I missing something or this common use case undocumented? John From: Ravishankar N Date: Wednesday, April 1, 2015 at 9:38 AM To: John Lilley, "gluster-users at gluster.org<mailto:gluster-users at gluster.org>" Subject: Re: [Gluster-users] Shrinking gluster filesystem in 3.6.2 On 03/31/2015 05:12 AM, Lilley, John F. wrote: Hi, I'd like to shrink my aws/ebs-based *distribute-only* gluster file system by migrating the data to other already existing, active and partially utilized bricks but found the 'replace-brick start' mentioned in the documentation is now deprecated. I see that there has been some back and forth on the mailing list regarding migrating data using self-heal on a replicated system but not so much on a distribute-only file system. Can anyone tell me the blessed way of doing this in 3.6.2? Is there one? To be clear, all of the ebs-based bricks are partially utilized at this point so I'd need a method to migrate the data first. If I understand you correctly, you want to replace a brick in a distribute volume with one of lesser capacity. You could first add a new brick and then remove the existing brick with remove-brick start/status/commit sequence. Something like this: ------------------------------------------------ [root at tuxpad ~]# gluster volume info testvol Volume Name: testvol Type: Distribute Volume ID: a89aa154-885c-4e14-8d3a-b555733b11f1 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 127.0.0.2:/home/ravi/bricks/brick1 Brick2: 127.0.0.2:/home/ravi/bricks/brick2 Brick3: 127.0.0.2:/home/ravi/bricks/brick3 [root at tuxpad ~]# [root at tuxpad ~]# [root at tuxpad ~]# gluster volume add-brick testvol 127.0.0.2:/home/ravi/bricks/brick{4..6} volume add-brick: success [root at tuxpad ~]# [root at tuxpad ~]# gluster volume info testvol Volume Name: testvol Type: Distribute Volume ID: a89aa154-885c-4e14-8d3a-b555733b11f1 Status: Started Number of Bricks: 6 Transport-type: tcp Bricks: Brick1: 127.0.0.2:/home/ravi/bricks/brick1 Brick2: 127.0.0.2:/home/ravi/bricks/brick2 Brick3: 127.0.0.2:/home/ravi/bricks/brick3 Brick4: 127.0.0.2:/home/ravi/bricks/brick4 Brick5: 127.0.0.2:/home/ravi/bricks/brick5 Brick6: 127.0.0.2:/home/ravi/bricks/brick6 [root at tuxpad ~]# [root at tuxpad ~]# [root at tuxpad ~]# [root at tuxpad ~]# gluster v remove-brick testvol 127.0.0.2:/home/ravi/bricks/brick{1..3} start volume remove-brick start: success ID: d535675e-8362-4a44-a291-1e567a77531e [root at tuxpad ~]# gluster v remove-brick testvol 127.0.0.2:/home/ravi/bricks/brick{1..3} status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 10 0Bytes 20 0 0 completed 0.00 [root at tuxpad ~]# [root at tuxpad ~]# gluster v remove-brick testvol 127.0.0.2:/home/ravi/bricks/brick{1..3} commit Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y volume remove-brick commit: success Check the removed bricks to ensure all files are migrated. If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. [root at tuxpad ~]# [root at tuxpad ~]# gluster volume info testvol Volume Name: testvol Type: Distribute Volume ID: a89aa154-885c-4e14-8d3a-b555733b11f1 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 127.0.0.2:/home/ravi/bricks/brick4 Brick2: 127.0.0.2:/home/ravi/bricks/brick5 Brick3: 127.0.0.2:/home/ravi/bricks/brick6 [root at tuxpad ~]# ------------------------------------------------ Hope this helps. Ravi Thank You, John _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150402/0c930b81/attachment.html>