Hi Nithya
Thanks for replying so quickly. It is very much appreciated.
There are lots if " [No space left on device] " errors which I can
not
understand as there are much space on all of the nodes.
A little bit of background will be useful in this case. I had cluster of
seven nodes of varying capacity(73, 73, 73, 46, 46, 46,46 TB) . The
cluster was almost 90% full so every node has almost 8 to 15 TB free
space. I added two new nodes with 100TB each and ran fix-layout which
completed successfully.
After that I started remove-brick operation. I don't think that any point
, any of the nodes were 100% full. Looking at my ganglia graph, there is
minimum 5TB always available at every node.
I was keeping an eye on remove-brick status and for very long time there
was no failures and then at some point these 17000 failures appeared and it
stayed like that.
Thanks
Kashif
Let me explain a little bit of background.
On Mon, Feb 4, 2019 at 5:09 AM Nithya Balachandran <nbalacha at
redhat.com>
wrote:
> Hi,
>
> The status shows quite a few failures. Please check the rebalance logs to
> see why that happened. We can decide what to do based on the errors.
> Once you run a commit, the brick will no longer be part of the volume and
> you will not be able to access those files via the client.
> Do you have sufficient space on the remaining bricks for the files on the
> removed brick?
>
> Regards,
> Nithya
>
> On Mon, 4 Feb 2019 at 03:50, mohammad kashif <kashif.alig at
gmail.com>
> wrote:
>
>> Hi
>>
>> I have a pure distributed gluster volume with nine nodes and trying to
>> remove one node, I ran
>> gluster volume remove-brick atlasglust
>> nodename:/glusteratlas/brick007/gv0 start
>>
>> It completed but with around 17000 failures
>>
>> Node Rebalanced-files size scanned failures
>> skipped status run time in h:m:s
>> --------- ----------- -----------
>> ----------- ----------- ----------- ------------
>> --------------
>> nodename 4185858 27.5TB 6746030
>> 17488 0 completed 405:15:34
>>
>> I can see that there is still 1.5 TB of data on the node which I was
>> trying to remove.
>>
>> I am not sure what to do now? Should I run remove-brick command again
so
>> the files which has been failed can be tried again?
>>
>> or should I run commit first and then try to remove node again?
>>
>> Please advise as I don't want to remove files.
>>
>> Thanks
>>
>> Kashif
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190204/2ca11f7b/attachment.html>