Gluster users - Jan 2024 - Filled up file system and glusterd not working anymore. To probe or not to probe?

Dear all,

we have a replica 3 configuration an issue after that the file system 
filled up. The gluster daemon is not starting anymore on the affected 
node and the inconsistency that we noticed is that the 
/var/lib/glusterd/peers does contain only one "good" node and the
other
one is missing.

Now what we would like to try is to probe from the affected peer the 
missing one. We also thought about just copying the missing node config 
file /var/lib/glusterd/peers here but we think this might be more 
inconsistent because perhaps of the time passed between finding out the 
issue and when the fill up had occurred (if this has relevance). We 
assume probing will fill that missing entry in the affected gluster and 
will sync to it. What we want to avoid is that this happens the other 
way around.

So in brief the question is if in our situation probing is too risky and 
we should try other methods (removing-adding brick) or anything that you 
might want to suggest from your experience.

Many thanks in advance

Ilias

-- 
?forumZFD
Entschieden f?r Frieden | Committed to Peace

Ilias Chasapakis
Referent IT | IT Consultant

Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am K?lner Brett 8 | 50825 K?ln | Germany

Tel 0221 91273243 | Fax 0221 91273299 | http://www.forumZFD.de

Vorstand nach ? 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht K?ln

Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00   BIC GENODEM1GLS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 665 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20240108/438db5e2/attachment.sig>

Dear all,

first of all thanks to Stefan (Kania) for the feedback. So detaching the 
peer, removing brick and downgrading to replica 2 until you add a new 
brick as you would do with any otherwise faulty node.

So here is how we proceeded:

  * check mounts where the faulty (ex-filled-up) node is present and
    remove it from there
  * detach "bad" peer with:? gluster peer detach
<your_faulty_gluster_node>
  * remove brick: gluster volume remove-brick <vol> replica 2
    <your_faulty_gluster_node>:<the_brick_path> force

So in our case we are ready to add a new brick to restore our replica 3.

This should be safer (and in the end as simple) than just copying the 
missing files probably bringing along meta-data inconsistencies. Which 
was also the doubt expressed on this post.

Am 08.01.24 um 11:13 schrieb Ilias Chasapakis forumZFD:
> Dear all,
>
> we have a replica 3 configuration an issue after that the file system 
> filled up. The gluster daemon is not starting anymore on the affected 
> node and the inconsistency that we noticed is that the 
> /var/lib/glusterd/peers does contain only one "good" node and the
> other one is missing.
>
> Now what we would like to try is to probe from the affected peer the 
> missing one. We also thought about just copying the missing node 
> config file /var/lib/glusterd/peers here but we think this might be 
> more inconsistent because perhaps of the time passed between finding 
> out the issue and when the fill up had occurred (if this has 
> relevance). We assume probing will fill that missing entry in the 
> affected gluster and will sync to it. What we want to avoid is that 
> this happens the other way around.
>
> So in brief the question is if in our situation probing is too risky 
> and we should try other methods (removing-adding brick) or anything 
> that you might want to suggest from your experience.
>
> Many thanks in advance
>
> Ilias
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge:https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-- 
?forumZFD
Entschieden f?r Frieden | Committed to Peace

Ilias Chasapakis
Referent IT | IT Consultant

Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am K?lner Brett 8 | 50825 K?ln | Germany

Tel 0221 91273243 | Fax 0221 91273299 |http://www.forumZFD.de

Vorstand nach ? 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht K?ln

Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00   BIC GENODEM1GLS

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20240109/159de7a9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 665 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20240109/159de7a9/attachment.sig>