Hi,
On Mon, 4 Feb 2019 at 16:39, mohammad kashif <kashif.alig at gmail.com>
wrote:
> Hi Nithya
>
> Thanks for replying so quickly. It is very much appreciated.
>
> There are lots if " [No space left on device] " errors which I
can not
> understand as there are much space on all of the nodes.
>
This means that Gluster could not find sufficient space for the file. Would
you be willing to share your rebalance log file?
Please provide the following information:
- The gluster version
- The gluster volume info for the volume
- How full are the individual bricks for the volume?
> A little bit of background will be useful in this case. I had cluster of
> seven nodes of varying capacity(73, 73, 73, 46, 46, 46,46 TB) . The
> cluster was almost 90% full so every node has almost 8 to 15 TB free
> space. I added two new nodes with 100TB each and ran fix-layout which
> completed successfully.
>
> After that I started remove-brick operation. I don't think that any
point
> , any of the nodes were 100% full. Looking at my ganglia graph, there is
> minimum 5TB always available at every node.
>
> I was keeping an eye on remove-brick status and for very long time there
> was no failures and then at some point these 17000 failures appeared and it
> stayed like that.
>
> Thanks
>
> Kashif
>
>
>
>
>
> Let me explain a little bit of background.
>
>
> On Mon, Feb 4, 2019 at 5:09 AM Nithya Balachandran <nbalacha at
redhat.com>
> wrote:
>
>> Hi,
>>
>> The status shows quite a few failures. Please check the rebalance logs
to
>> see why that happened. We can decide what to do based on the errors.
>> Once you run a commit, the brick will no longer be part of the volume
and
>> you will not be able to access those files via the client.
>> Do you have sufficient space on the remaining bricks for the files on
the
>> removed brick?
>>
>> Regards,
>> Nithya
>>
>> On Mon, 4 Feb 2019 at 03:50, mohammad kashif <kashif.alig at
gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I have a pure distributed gluster volume with nine nodes and trying
to
>>> remove one node, I ran
>>> gluster volume remove-brick atlasglust
>>> nodename:/glusteratlas/brick007/gv0 start
>>>
>>> It completed but with around 17000 failures
>>>
>>> Node Rebalanced-files size scanned
failures
>>> skipped status run time in h:m:s
>>> --------- -----------
-----------
>>> ----------- ----------- ----------- ------------
>>> --------------
>>> nodename 4185858 27.5TB 6746030
>>> 17488 0 completed 405:15:34
>>>
>>> I can see that there is still 1.5 TB of data on the node which I
was
>>> trying to remove.
>>>
>>> I am not sure what to do now? Should I run remove-brick command
again
>>> so the files which has been failed can be tried again?
>>>
>>> or should I run commit first and then try to remove node again?
>>>
>>> Please advise as I don't want to remove files.
>>>
>>> Thanks
>>>
>>> Kashif
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190204/b819b55e/attachment.html>