So b53c8e46-068b-4286-94a6-7cf54f711983 is not a gfid? What else is it?
Mathias
On 08.08.20 09:00, Strahil Nikolov wrote:> In glusterfs the long string is called "gfid" and does not
represent the name.
>
> Best Regards,
> Strahil Nikolov
>
>
>
>
>
>
> ? ?????, 7 ?????? 2020 ?., 21:40:11 ???????+3, Mathias Waack
<mathias.waack at seim-partner.de> ??????:
>
>
>
>
>
> Hi Strahil,
>
> but I cannot find these files in the heal info:
>
> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
> ...
> 7443397? 132463 -rw-------?? 1 999????? docker?? 1073741824 Aug? 3 10:35
> /zbrick/.glusterfs/b5/3c/b53c8e46-068b-4286-94a6-7cf54f711983
>
> Now looking for this file in the heal infos:
>
> gluster volume heal gvol info | grep b53c8e46-068b-4286-94a6-7cf54f711983
>
> shows nothing.
>
> So I do not know, what I have to heal...
>
> Mathias
>
> On 07.08.20 14:32, Strahil Nikolov wrote:
>> Have you tried to gluster heal and check if the files are back into
their place?
>>
>> I always thought that those hard links are used? by the healing
mechanism? and if that is true - gluster should restore the files to their
original location and then wiping the correct files from FUSE will be easy.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ?? 7 ?????? 2020 ?. 10:24:38 GMT+03:00, Mathias Waack <mathias.waack
at seim-partner.de> ??????:
>>> Hi all,
>>>
>>> maybe I should add some more information:
>>>
>>> The container which filled up the space was running on node x,
which
>>> still shows a nearly filled fs:
>>>
>>> 192.168.1.x:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>>
>>> nearly the same situation on the underlying brick partition on node
x:
>>>
>>> zdata/brick???? 2.6T? 2.4T? 176G? 94% /zbrick
>>>
>>> On node y the network card crashed, glusterfs shows the same
values:
>>>
>>> 192.168.1.y:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>>
>>> but different values on the brick:
>>>
>>> zdata/brick???? 2.9T? 1.6T? 1.4T? 54% /zbrick
>>>
>>> I think this happened because glusterfs still has hardlinks to the
>>> deleted files on node x? So I can find these files with:
>>>
>>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>>>
>>> But now I am lost. How can I verify these files really belongs to
the
>>> right container? Or can I just delete this files because there is
no
>>> way
>>> to access it? Or offers glusterfs a way to solve this situation?
>>>
>>> Mathias
>>>
>>> On 05.08.20 15:48, Mathias Waack wrote:
>>>> Hi all,
>>>>
>>>> we are running a gluster setup with two nodes:
>>>>
>>>> Status of volume: gvol
>>>> Gluster process???????????????????????????? TCP Port? RDMA Port
>>>> Online? Pid
>>>>
>>>
------------------------------------------------------------------------------
>>>
>>>> Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y 13350
>>>> Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y 5965
>>>> Self-heal Daemon on localhost?????????????? N/A?????? N/A Y
14188
>>>> Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y
6003
>>>>
>>>> Task Status of Volume gvol
>>>>
>>>
------------------------------------------------------------------------------
>>>
>>>> There are no active volume tasks
>>>>
>>>> The glusterfs hosts a bunch of containers with its data
volumes. The
>>>> underlying fs is zfs. Few days ago one of the containers
created a
>>> lot
>>>> of files in one of its data volumes, and at the end it
completely
>>>> filled up the space of the glusterfs volume. But this happened
only
>>> on
>>>> one host, on the other host there was still enough space. We
finally
>>>> were able to identify this container and found out, the sizes
of the
>>>> data on /zbrick were different on both hosts for this
container. Now
>>>> we made the big mistake to delete these files on both hosts in
the
>>>> /zbrick volume, not on the mounted glusterfs volume.
>>>>
>>>> Later we found the reason for this behavior: the network driver
on
>>> the
>>>> second node partially crashed (which means we ware able to
login on
>>>> the node, so we assumed the network was running, but the card
was
>>>> already dropping packets at this time) at the same time, as the
>>> failed
>>>> container started to fill up the gluster volume. After
rebooting the
>>>> second node? the gluster became available again.
>>>>
>>>> Now the glusterfs volume is running again- but it is still
(nearly)
>>>> full: the files created by the container are not visible, but
they
>>>> still count into amount of free space. How can we fix this?
>>>>
>>>> In addition there are some files which are no longer accessible
since
>>>> this accident:
>>>>
>>>> tail access.log.old
>>>> tail: cannot open 'access.log.old' for reading:
Input/output error
>>>>
>>>> Looks like affected by this error are files which have been
changed
>>>> during the accident. Is there a way to fix this too?
>>>>
>>>> Thanks
>>>> ? ??? Mathias
>>>>
>>>>
>>>> ________
>>>>
>>>>
>>>>
>>>> Community Meeting Calendar:
>>>>
>>>> Schedule -
>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> Bridge: https://bluejeans.com/441850968
>>>>
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users