Just started trying gluster, to decide if we want to put it into production.
Running version 3.7.11-1
Replicated, distributed volume, two servers, 20 bricks per server:
[root at storinator1 ~]# gluster volume status gv0
Status of volume: gv0
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick storinator1:/export/brick1/gv0 49153 0 Y
2554
Brick storinator2:/export/brick1/gv0 49153 0 Y
9686
Brick storinator1:/export/brick2/gv0 49154 0 Y
2562
Brick storinator2:/export/brick2/gv0 49154 0 Y
9708
Brick storinator1:/export/brick3/gv0 49155 0 Y
2568
Brick storinator2:/export/brick3/gv0 49155 0 Y
9692
Brick storinator1:/export/brick4/gv0 49156 0 Y
2574
Brick storinator2:/export/brick4/gv0 49156 0 Y
9765
Brick storinator1:/export/brick5/gv0 49173 0 Y
16901
Brick storinator2:/export/brick5/gv0 49173 0 Y
9727
Brick storinator1:/export/brick6/gv0 49174 0 Y
16920
Brick storinator2:/export/brick6/gv0 49174 0 Y
9733
Brick storinator1:/export/brick7/gv0 49175 0 Y
16939
Brick storinator2:/export/brick7/gv0 49175 0 Y
9739
Brick storinator1:/export/brick8/gv0 49176 0 Y
16958
Brick storinator2:/export/brick8/gv0 49176 0 Y
9703
Brick storinator1:/export/brick9/gv0 49177 0 Y
16977
Brick storinator2:/export/brick9/gv0 49177 0 Y
9713
Brick storinator1:/export/brick10/gv0 49178 0 Y
16996
Brick storinator2:/export/brick10/gv0 49178 0 Y
9718
Brick storinator1:/export/brick11/gv0 49179 0 Y
17015
Brick storinator2:/export/brick11/gv0 49179 0 Y
9746
Brick storinator1:/export/brick12/gv0 49180 0 Y
17034
Brick storinator2:/export/brick12/gv0 49180 0 Y
9792
Brick storinator1:/export/brick13/gv0 49181 0 Y
17053
Brick storinator2:/export/brick13/gv0 49181 0 Y
9755
Brick storinator1:/export/brick14/gv0 49182 0 Y
17072
Brick storinator2:/export/brick14/gv0 49182 0 Y
9767
Brick storinator1:/export/brick15/gv0 49183 0 Y
17091
Brick storinator2:/export/brick15/gv0 N/A N/A N
N/A
Brick storinator1:/export/brick16/gv0 49184 0 Y
17110
Brick storinator2:/export/brick16/gv0 49184 0 Y
9791
Brick storinator1:/export/brick17/gv0 49185 0 Y
17129
Brick storinator2:/export/brick17/gv0 49185 0 Y
9756
Brick storinator1:/export/brick18/gv0 49186 0 Y
17148
Brick storinator2:/export/brick18/gv0 49186 0 Y
9766
Brick storinator1:/export/brick19/gv0 49187 0 Y
17167
Brick storinator2:/export/brick19/gv0 49187 0 Y
9745
Brick storinator1:/export/brick20/gv0 49188 0 Y
17186
Brick storinator2:/export/brick20/gv0 49188 0 Y
9783
NFS Server on localhost 2049 0 Y
17206
Self-heal Daemon on localhost N/A N/A Y
17214
NFS Server on storinator2 2049 0 Y
9657
Self-heal Daemon on storinator2 N/A N/A Y
9677
Task Status of Volume gv0
------------------------------------------------------------------------------
Task : Rebalance
ID : 28c733e9-d618-44fc-873f-405d3b29a609
Status : completed
Wouldn't you know it, within a week or two of pulling the hardware together
and getting gluster installed and configured, a disk dies. Note the dead
process for brick15 on server storinator2.
I would like to remove (not replace) the failed brick (and its replica).
(I don't have a spare disk handy, and there's plenty of room on the
other
bricks.) But gluster doesn't seem to want to remove a brick if the brick
is dead:
[root at storinator1 ~]# gluster volume remove-brick gv0
storinator{1..2}:/export/brick15/gv0 start
volume remove-brick start: failed: Staging failed on storinator2. Error:
Found stopped brick storinator2:/export/brick15/gv0
So what do I do? I can't remove the brick while the brick is bad, but I
want to remove the brick *because* the brick is bad. Bit of a Catch-22.
Thanks in advance for any help you can give.
phil
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160609/3a24e46a/attachment.html>
Nithya Balachandran
2016-Jun-15 04:49 UTC
[Gluster-users] Disk failed, how do I remove brick?
On Fri, Jun 10, 2016 at 1:25 AM, Phil Dumont <phil at solidstatescientific.com> wrote:> Just started trying gluster, to decide if we want to put it into > production. > > Running version 3.7.11-1 > > Replicated, distributed volume, two servers, 20 bricks per server: > > [root at storinator1 ~]# gluster volume status gv0 > Status of volume: gv0 > Gluster process TCP Port RDMA Port Online > Pid > > ------------------------------------------------------------------------------ > Brick storinator1:/export/brick1/gv0 49153 0 Y > 2554 > Brick storinator2:/export/brick1/gv0 49153 0 Y > 9686 > Brick storinator1:/export/brick2/gv0 49154 0 Y > 2562 > Brick storinator2:/export/brick2/gv0 49154 0 Y > 9708 > Brick storinator1:/export/brick3/gv0 49155 0 Y > 2568 > Brick storinator2:/export/brick3/gv0 49155 0 Y > 9692 > Brick storinator1:/export/brick4/gv0 49156 0 Y > 2574 > Brick storinator2:/export/brick4/gv0 49156 0 Y > 9765 > Brick storinator1:/export/brick5/gv0 49173 0 Y > 16901 > Brick storinator2:/export/brick5/gv0 49173 0 Y > 9727 > Brick storinator1:/export/brick6/gv0 49174 0 Y > 16920 > Brick storinator2:/export/brick6/gv0 49174 0 Y > 9733 > Brick storinator1:/export/brick7/gv0 49175 0 Y > 16939 > Brick storinator2:/export/brick7/gv0 49175 0 Y > 9739 > Brick storinator1:/export/brick8/gv0 49176 0 Y > 16958 > Brick storinator2:/export/brick8/gv0 49176 0 Y > 9703 > Brick storinator1:/export/brick9/gv0 49177 0 Y > 16977 > Brick storinator2:/export/brick9/gv0 49177 0 Y > 9713 > Brick storinator1:/export/brick10/gv0 49178 0 Y > 16996 > Brick storinator2:/export/brick10/gv0 49178 0 Y > 9718 > Brick storinator1:/export/brick11/gv0 49179 0 Y > 17015 > Brick storinator2:/export/brick11/gv0 49179 0 Y > 9746 > Brick storinator1:/export/brick12/gv0 49180 0 Y > 17034 > Brick storinator2:/export/brick12/gv0 49180 0 Y > 9792 > Brick storinator1:/export/brick13/gv0 49181 0 Y > 17053 > Brick storinator2:/export/brick13/gv0 49181 0 Y > 9755 > Brick storinator1:/export/brick14/gv0 49182 0 Y > 17072 > Brick storinator2:/export/brick14/gv0 49182 0 Y > 9767 > Brick storinator1:/export/brick15/gv0 49183 0 Y > 17091 > Brick storinator2:/export/brick15/gv0 N/A N/A N > N/A > Brick storinator1:/export/brick16/gv0 49184 0 Y > 17110 > Brick storinator2:/export/brick16/gv0 49184 0 Y > 9791 > Brick storinator1:/export/brick17/gv0 49185 0 Y > 17129 > Brick storinator2:/export/brick17/gv0 49185 0 Y > 9756 > Brick storinator1:/export/brick18/gv0 49186 0 Y > 17148 > Brick storinator2:/export/brick18/gv0 49186 0 Y > 9766 > Brick storinator1:/export/brick19/gv0 49187 0 Y > 17167 > Brick storinator2:/export/brick19/gv0 49187 0 Y > 9745 > Brick storinator1:/export/brick20/gv0 49188 0 Y > 17186 > Brick storinator2:/export/brick20/gv0 49188 0 Y > 9783 > NFS Server on localhost 2049 0 Y > 17206 > Self-heal Daemon on localhost N/A N/A Y > 17214 > NFS Server on storinator2 2049 0 Y > 9657 > Self-heal Daemon on storinator2 N/A N/A Y > 9677 > > Task Status of Volume gv0 > > ------------------------------------------------------------------------------ > Task : Rebalance > ID : 28c733e9-d618-44fc-873f-405d3b29a609 > Status : completed > > > Wouldn't you know it, within a week or two of pulling the hardware > together and getting gluster installed and configured, a disk dies. Note > the dead process for brick15 on server storinator2. > > I would like to remove (not replace) the failed brick (and its replica). > (I don't have a spare disk handy, and there's plenty of room on the other > bricks.) But gluster doesn't seem to want to remove a brick if the brick > is dead: > > [root at storinator1 ~]# gluster volume remove-brick gv0 > storinator{1..2}:/export/brick15/gv0 start > volume remove-brick start: failed: Staging failed on storinator2. Error: > Found stopped brick storinator2:/export/brick15/gv0 > > > So what do I do? I can't remove the brick while the brick is bad, but I > want to remove the brick *because* the brick is bad. Bit of a Catch-22. >Hi Phil, This looks like a bug - please open a new BZ for this issue. You can remove these bricks using gluster v remove-brick <vol> <bricks> force However, that will not copy the data from the bricks that have been removed. You will need to copy the files manually from the brick to the gluster volume mount point. Thanks, Nithya> Thanks in advance for any help you can give. > > phil > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160615/a0a30c63/attachment.html>