Shreyansh Shah
2025-Jan-28 14:29 UTC
[Gluster-users] Glusterfs mounts on clients with few bricks missing
Hi, We are facing a weird scenario where on the clients the glusterfs volume mounts successfully but the size on the client is less than the total volume size. A few bricks are missing and hence the size as seen on the client is incorrect and we face issues of missing files. This happens randomly and we are unable to reproduce it. The gluster volume status shows all bricks up and running. We are running distributed glusterfs version 9.3 with 13 nodes having 50 bricks divided amongst them. We are mounting it on the client with the below entry in fstab "10.132.2.101:/datavol /data glusterfs ro,_netdev 0 0" where 10.132.2.101 is one of the nodes and datavol is the volume name. Any help would be appreciated. In the client glusterfs log we see the below prints in case these help to figure out the issue. [2025-01-28 11:00:11.404092 +0000] W [MSGID: 109222]> [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}] > [2025-01-28 11:00:11.405029 +0000] W [dict.c:1532:dict_get_with_ref] > (-->/usr/lib64/glusterfs/9.3/xlator/cluster/distribute.so(+0x556ea) > [0x7f63cdf176ea] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3f) [0x7f63 > dd12ffaf] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x8e) > [0x7f63dd12f29e] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash) > is NULL [Invalid argument] > [2025-01-28 11:00:11.406595 +0000] I [MSGID: 109061] > [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies > [{path=/code}, {gfid=949ca970-0b14-4fed-af73-d9ea216905f1}, {holes=1}, > {overlaps=0}] > [2025-01-28 11:00:11.406731 +0000] W [MSGID: 109222] > [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > selfheal failed [{path=/code}, {subvol-down=1}, {Not-fixing}, > {gfid=949ca970-0b14-4fed-a > f73-d9ea216905f1}] > [2025-01-28 11:00:11.408356 +0000] W [dict.c:1532:dict_get_with_ref] > (-->/usr/lib64/glusterfs/9.3/xlator/cluster/distribute.so(+0x556ea) > [0x7f63cdf176ea] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3f) [0x7f63 > dd12ffaf] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x8e) > [0x7f63dd12f29e] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash) > is NULL [Invalid argument] > [2025-01-28 11:00:11.409546 +0000] I [MSGID: 109061] > [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies > [{path=/code/summary}, {gfid=c96f0e78-dd0e-4c61-89a3-f94ca1b502c4}, > {holes=1}, {overlap > s=0}] > [2025-01-28 11:00:11.409605 +0000] W [MSGID: 109222] > [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > selfheal failed [{path=/code/summary}, {subvol-down=1}, {Not-fixing}, > {gfid=c96f0e78-dd0 > e-4c61-89a3-f94ca1b502c4}] > [2025-01-28 11:00:28.910500 +0000] W [MSGID: 109222] > [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}] > [2025-01-28 11:00:28.911125 +0000] I [MSGID: 109061] > [dht-layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies > [{path=/}, {gfid=00000000-0000-0000-0000-000000000001}, {holes=1}, > {overlaps=0}] > [2025-01-28 11:00:28.911167 +0000] W [MSGID: 109222] > [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}] > [2025-01-28 11:00:40.359587 +0000] W [MSGID: 109222] > [dht-selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}] > [2025-01-28 11:00:40.375728 +0000] W [MSGID: 109011] > [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed > subvolume [{hash-value=0x5877673a}] > [2025-01-28 11:00:40.377646 +0000] W [MSGID: 109011] > [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed > subvolume [{hash-value=0x5877673a}] > [2025-01-28 11:00:40.428513 +0000] W [MSGID: 109011] > [dht-layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed > subvolume [{hash-value=0x599f6a37}]-- Regards, Shreyansh Shah AlphaGrep* Securities Pvt. Ltd. * -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20250128/754b57ed/attachment.html>
Andreas Schwibbe
2025-Jan-28 14:39 UTC
[Gluster-users] Glusterfs mounts on clients with few bricks missing
Hey Shreyansh, is this eventually happening if one node is going down/gluster service restart while the client has the volume mounted? I think I am experiencing something similar for my VM disks, when I restart a node/glusterd. I only can fix it by re-mounting the client. I believe the change of brick-ports of the newley restarted node/service is not being populated to the client, but all other nodes, thus the nodes don't show any problems and the client keeps connected to the bricks it knows. A. Am Dienstag, dem 28.01.2025 um 19:59 +0530 schrieb Shreyansh Shah:> Hi, > We are facing a weird scenario where on the clients the glusterfs > volume?mounts successfully but the size on the client is less than > the?total volume?size. A few?bricks are missing and hence the size as > seen on the client is incorrect and we face issues of missing files. > This happens randomly and we are unable to reproduce it. The gluster > volume status shows all bricks up and running. > We are running distributed glusterfs version 9.3 with 13 nodes having > 50 bricks divided amongst them. We are mounting it on the client with > the?below entry in fstab > "10.132.2.101:/datavol /data glusterfs ro,_netdev 0 0" where > 10.132.2.101 is one of the nodes and datavol is the?volume name. > > Any help would be appreciated. In the client glusterfs log we see > the?below prints in case these help to figure out the?issue.? > > > [2025-01-28 11:00:11.404092 +0000] W [MSGID: 109222] [dht- > > selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > > selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}] > > [2025-01-28 11:00:11.405029 +0000] W > > [dict.c:1532:dict_get_with_ref] (-- > > >/usr/lib64/glusterfs/9.3/xlator/cluster/distribute.so(+0x556ea) > > [0x7f63cdf176ea] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3f) > > [0x7f63 > > dd12ffaf] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x8e) > > [0x7f63dd12f29e] ) 0-dict: dict OR key > > (trusted.glusterfs.dht.commithash) is NULL [Invalid argument] > > [2025-01-28 11:00:11.406595 +0000] I [MSGID: 109061] [dht- > > layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies > > [{path=/code}, {gfid=949ca970-0b14-4fed-af73-d9ea216905f1}, > > {holes=1}, {overlaps=0}] > > [2025-01-28 11:00:11.406731 +0000] W [MSGID: 109222] [dht- > > selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > > selfheal failed [{path=/code}, {subvol-down=1}, {Not-fixing}, > > {gfid=949ca970-0b14-4fed-a > > f73-d9ea216905f1}] > > [2025-01-28 11:00:11.408356 +0000] W > > [dict.c:1532:dict_get_with_ref] (-- > > >/usr/lib64/glusterfs/9.3/xlator/cluster/distribute.so(+0x556ea) > > [0x7f63cdf176ea] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3f) > > [0x7f63 > > dd12ffaf] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x8e) > > [0x7f63dd12f29e] ) 0-dict: dict OR key > > (trusted.glusterfs.dht.commithash) is NULL [Invalid argument] > > [2025-01-28 11:00:11.409546 +0000] I [MSGID: 109061] [dht- > > layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies > > [{path=/code/summary}, {gfid=c96f0e78-dd0e-4c61-89a3-f94ca1b502c4}, > > {holes=1}, {overlap > > s=0}] > > [2025-01-28 11:00:11.409605 +0000] W [MSGID: 109222] [dht- > > selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > > selfheal failed [{path=/code/summary}, {subvol-down=1}, {Not- > > fixing}, {gfid=c96f0e78-dd0 > > e-4c61-89a3-f94ca1b502c4}] > > [2025-01-28 11:00:28.910500 +0000] W [MSGID: 109222] [dht- > > selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > > selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}] > > [2025-01-28 11:00:28.911125 +0000] I [MSGID: 109061] [dht- > > layout.c:641:dht_layout_normalize] 0-datavol-dht: Found anomalies > > [{path=/}, {gfid=00000000-0000-0000-0000-000000000001}, {holes=1}, > > {overlaps=0}] > > [2025-01-28 11:00:28.911167 +0000] W [MSGID: 109222] [dht- > > selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > > selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}] > > [2025-01-28 11:00:40.359587 +0000] W [MSGID: 109222] [dht- > > selfheal.c:2024:dht_selfheal_directory] 0-datavol-dht: Directory > > selfheal failed [{path=/}, {subvol-down=1}, {Not-fixing}, {gfid=}] > > [2025-01-28 11:00:40.375728 +0000] W [MSGID: 109011] [dht- > > layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed > > subvolume [{hash-value=0x5877673a}] > > [2025-01-28 11:00:40.377646 +0000] W [MSGID: 109011] [dht- > > layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed > > subvolume [{hash-value=0x5877673a}] > > [2025-01-28 11:00:40.428513 +0000] W [MSGID: 109011] [dht- > > layout.c:148:dht_layout_search] 0-datavol-dht: Failed to get hashed > > subvolume [{hash-value=0x599f6a37}] > > > ? > > -- > Regards, > Shreyansh Shah > AlphaGrep?Securities Pvt. Ltd.? > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20250128/afba93f9/attachment.html>