similar to: Qustionmark in permission and Owner

Seen something similar when FUSE client died, but it marked the whole mountpoint, not just some files. Might be a desync or communication loss between the nodes? Diego Il 05/06/2023 11:23, Stefan Kania ha scritto: > Hello, > > I have a strange problem on a gluster volume > > If I do an "ls -l" in a directory insight a mountet gluster-volume I > see, only for some

Theoretically it might help.If possible, try to resolve any pending heals. Best Regards,Strahil Nikolov? On Thu, Mar 16, 2023 at 15:29, Diego Zuccato<diego.zuccato at unibo.it> wrote: In Debian stopping glusterd does not stop brick processes: to stop everything (and free the memory) I have to systemctl stop glusterd ? killall glusterfs{,d} ? killall glfsheal ? systemctl start

I have no clue. Have you checked for errors in the logs ? Maybe you might find something useful. Best Regards,Strahil Nikolov? On Tue, Mar 21, 2023 at 9:56, Diego Zuccato<diego.zuccato at unibo.it> wrote: Killed glfsheal, after a day there were 218 processes, then they got killed by OOM during the weekend. Now there are no processes active. Trying to run "heal info" reports

Can you restart glusterd service (first check that it was not modified to kill the bricks)? Best Regards,Strahil Nikolov? On Thu, Mar 16, 2023 at 8:26, Diego Zuccato<diego.zuccato at unibo.it> wrote: OOM is just just a matter of time. Today mem use is up to 177G/187 and: # ps aux|grep glfsheal|wc -l 551 (well, one is actually the grep process, so "only" 550 glfsheal

Killed glfsheal, after a day there were 218 processes, then they got killed by OOM during the weekend. Now there are no processes active. Trying to run "heal info" reports lots of files quite quickly but does not spawn any glfsheal process. And neither does restarting glusterd. Is there some way to selectively run glfsheal to fix one brick at a time? Diego Il 21/03/2023 01:21,

Do you use brick multiplexing ? Best Regards,Strahil Nikolov? On Tue, Mar 14, 2023 at 16:44, Diego Zuccato<diego.zuccato at unibo.it> wrote: Hello all. Our Gluster 9.6 cluster is showing increasing problems. Currently it's composed of 3 servers (2x Intel Xeon 4210 [20 cores dual thread, total 40 threads], 192GB RAM, 30x HGST HUH721212AL5200 [12TB]), configured in replica 3

In glfsheal-Connection.log I see many lines like: [2023-03-13 23:04:40.241481 +0000] E [MSGID: 104021] [glfs-mgmt.c:586:glfs_mgmt_getspec_cbk] 0-gfapi: failed to get the volume file [{from server}, {errno=2}, {error=File o directory non esistente}] And *lots* of gfid-mismatch errors in glustershd.log . Couldn't find anything that would prevent heal to start. :( Diego Il 21/03/2023

If you don't experience any OOM , you can focus on the heals. 284 processes of glfsheal seems odd. Can you check the ppid for 2-3 randomly picked ?ps -o ppid= <pid> Best Regards,Strahil Nikolov? On Wed, Mar 15, 2023 at 9:54, Diego Zuccato<diego.zuccato at unibo.it> wrote: I enabled it yesterday and that greatly reduced memory pressure. Current volume info: -8<-- Volume

Can you check your volume file contents?Maybe it really can't find (or access) a specific volfile ? Best Regards,Strahil Nikolov? On Fri, Mar 24, 2023 at 8:07, Diego Zuccato<diego.zuccato at unibo.it> wrote: In glfsheal-Connection.log I see many lines like: [2023-03-13 23:04:40.241481 +0000] E [MSGID: 104021] [glfs-mgmt.c:586:glfs_mgmt_getspec_cbk] 0-gfapi: failed to get the

There are 285 files in /var/lib/glusterd/vols/cluster_data ... including many files with names related to quorum bricks already moved to a different path (like cluster_data.client.clustor02.srv-quorum-00-d.vol that should already have been replaced by cluster_data.clustor02.srv-bricks-00-q.vol -- and both vol files exist). Is there something I should check inside the volfiles? Diego Il

OOM is just just a matter of time. Today mem use is up to 177G/187 and: # ps aux|grep glfsheal|wc -l 551 (well, one is actually the grep process, so "only" 550 glfsheal processes. I'll take the last 5: root 3266352 0.5 0.0 600292 93044 ? Sl 06:55 0:07 /usr/libexec/glusterfs/glfsheal cluster_data info-summary --xml root 3267220 0.7 0.0 600292 91964 ?

I enabled it yesterday and that greatly reduced memory pressure. Current volume info: -8<-- Volume Name: cluster_data Type: Distributed-Replicate Volume ID: a8caaa90-d161-45bb-a68c-278263a8531a Status: Started Snapshot Count: 0 Number of Bricks: 45 x (2 + 1) = 135 Transport-type: tcp Bricks: Brick1: clustor00:/srv/bricks/00/d Brick2: clustor01:/srv/bricks/00/d Brick3: clustor02:/srv/bricks/00/q

On December 17, 2023 5:40:52 AM PST, Diego Zuccato <diego.zuccato at unibo.it> wrote: >Il 14/12/2023 16:08, Joe Julian ha scritto: > >> With ceph, if the placement database is corrupted, all your data is lost (happened to my employer, once, losing 5PB of customer data). > >From what I've been told (by experts) it's really hard to make it happen. More if proper

good evening, i am new to glusterFS and read about tiering option, BUT cannot find it on current v10 ubuntu 22 lts version, my fault? thank you. https://www.gluster.org/automated-tiering-in-gluster-2/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240221/2e1fc4ab/attachment.html>

Hello all. Our Gluster 9.6 cluster is showing increasing problems. Currently it's composed of 3 servers (2x Intel Xeon 4210 [20 cores dual thread, total 40 threads], 192GB RAM, 30x HGST HUH721212AL5200 [12TB]), configured in replica 3 arbiter 1. Using Debian packages from Gluster 9.x latest repository. Seems 192G RAM are not enough to handle 30 data bricks + 15 arbiters and I often had

Hi Diego, I have had a look at BeeGFS and is seems more similar to ceph then to gluster. It requires extra management nodes similar to ceph, right? Second of all there are no snapshots in BeeGFS, as I understand it. I know ceph has snapshots so for us this seems a better alternative. What is your experience of ceph? I am sorry to hear about your problems with gluster, from my experience we had

After a lot of tests and unsuccessful searching, I decided to start from scratch: I'm going to ditch the old volume and create a new one. I have 3 servers with 30 12TB disks each. Since I'm going to start a new volume, could it be better to group disks in 10 3-disk (or 6 5-disk) RAID-0 volumes to reduce the number of bricks? Redundancy would be given by replica 2 (still undecided

Maybe a bit OT... I'm no expert on either, but the concepts are quite similar. Both require "extra" nodes (metadata and monitor), but those can be virtual machines or you can host the services on OSD machines. We don't use snapshots, so I can't comment on that. My experience with Ceph is limited to having it working on Proxmox. No experience yet with CephFS. BeeGFS is

Are you able to set the logs to debug level ?It might provide a clue what it is going on. Best Regards,Strahil Nikolov On Thu, Jan 18, 2024 at 13:08, Diego Zuccato<diego.zuccato at unibo.it> wrote: That's the same kind of errors I keep seeing on my 2 clusters, regenerated some months ago. Seems a pseudo-split-brain that should be impossible on a replica 3 cluster but keeps

were you able to solve the problem? Can it be treated like a "normal" split brain? 'gluster peer status' and 'gluster volume status' are ok, so kinda looks like "pseudo"... hubert Am Do., 18. Jan. 2024 um 08:28 Uhr schrieb Diego Zuccato <diego.zuccato at unibo.it>: > > That's the same kind of errors I keep seeing on my 2 clusters, >