search for: remove_me

...le back where one of the server's crashed, we got the server back up and running and ensured that all healing entries cleared, and also increased the server spec (CPU/Mem) as this seemed to be the potential cause. Since then however, we've seen some strange behaviour, whereby a lot of 'remove_me' files are building up under `/data/glusterfs/gv1/brick2/brick/.shard/.remove_me/` and `/data/glusterfs/gv1/brick3/brick/.shard/.remove_me/`. This is causing the arbiter to run out of space on brick2 and brick3, as the remove_me files are constantly increasing. brick1 appears to be fine, the d...

...ile back where one of the server's crashed, we got the server back up and running and ensured that all healing entries cleared, and also increased the server spec (CPU/Mem) as this seemed to be the potential cause. Since then however, we've seen some strange behaviour, whereby a lot of 'remove_me' files are building up under `/data/glusterfs/gv1/brick2/brick/.shard/.remove_me/` and `/data/glusterfs/gv1/brick3/brick/.shard/.remove_me/`.?This is causing the arbiter to run out of space on brick2 and brick3, as the remove_me files are constantly increasing. brick1 appears to be fine, the di...

...ck2/brick/.glusterfs-anonymous-inode-d3d1fdec-7df9-4f71-b9fc-660d12c2a046 12G /data/glusterfs/gv1/brick2/brick Also, this is the du -sh output for the specific directory that appears to be taking up space on brick 3: root at uk3-prod-gfs-arb-01:~# du -sh /data/glusterfs/gv1/brick3/brick/.shard/.remove_me/ 10G /data/glusterfs/gv1/brick3/brick/.shard/.remove_me/ The gluster package version on all the servers is 11.0, and I believe they were upgraded from 7.2>8.6>11.0; the op.version was only changed after the 11.0 upgrade. The archival job deletes the files, files shouldn't be overwri...

...? ? ? ? ? ? ?6.1M ?3.0M ?3.1M ? 49% /data/glusterfs/gv1/brick2 So the inode usage appears to be fine, but we're seeing that the actual disk usage keeps increasing on the bricks despite it being the arbiter. The actual issue appears to be that files under /data/glusterfs/gv1/brick3/brick/.shard/.remove_me/? and/data/glusterfs/gv1/brick2/brick/.shard/.remove_me/??are being retained, even when the original files are deleted from the data nodes. For reference, I've attached disk usage graphs for brick 3 over the past two weeks; one is a graph from a data node, the other from the arb. As you can see...

...6.1M 3.0M 3.1M 49% /data/glusterfs/gv1/brick2 So the inode usage appears to be fine, but we're seeing that the actual disk usage keeps increasing on the bricks despite it being the arbiter. The actual issue appears to be that files under /data/glusterfs/gv1/brick3/brick/.shard/.remove_me/? and /data/glusterfs/gv1/brick2/brick/.shard/.remove_me/? are being retained, even when the original files are deleted from the data nodes. For reference, I've attached disk usage graphs for brick 3 over the past two weeks; one is a graph from a data node, the other from the arb. As you can...

...11-b76c-6ff66f52e6dd]<https://www.ek.co/> ________________________________ From: Strahil Nikolov <hunter86_bg at yahoo.com> Sent: 03 July 2023 18:28 To: Liam Smith <liam.smith at ek.co>; gluster-users at gluster.org <gluster-users at gluster.org> Subject: Re: [Gluster-users] remove_me files building up CAUTION: This e-mail originates from outside of Ekco. Do not click links or attachments unless you recognise the sender. ________________________________ Hi, you mentioned that the arbiter bricks run out of inodes. Are you using XFS ? Can you provide the xfs_info of each brick...

...? | | Linux?Systems?Support?Engineer,?Scholar | | | | ? | | | | | | From: Strahil Nikolov <hunter86_bg at yahoo.com> Sent: 03 July 2023 18:28 To: Liam Smith <liam.smith at ek.co>; gluster-users at gluster.org <gluster-users at gluster.org> Subject: Re: [Gluster-users] remove_me files building up? | CAUTION:?This e-mail originates from outside of Ekco. Do not click links or attachments unless you recognise the sender. | Hi, you mentioned that the arbiter bricks run out of inodes.Are you using XFS ?Can you provide the xfs_info of each brick ? Best Regards,Strahil Nikol...