similar to: Heal Info Shows Split Brain, but "file not in split brain" when attempted heal

Hello, I''m trying to build a replica volume, on two servers. The servers are: blade6 and blade7. (another blade1 in the peer, but with no volumes) The volume seems ok, but I cannot mount it from NFS. Here are some logs: [root@blade6 stor1]# df -h /dev/mapper/gluster_stor1 882G 200M 837G 1% /gluster/stor1 [root@blade7 stor1]# df -h /dev/mapper/gluster_fast

On Thu, Sep 28, 2017 at 12:11 PM, Zhou, Cynthia (NSB - CN/Hangzhou) < cynthia.zhou at nokia-sbell.com> wrote: > > > The version I am using is glusterfs 3.6.9 > This is a very old version which is EOL. If you can upgrade to any of the supported version (3.10 or 3.12) would be great. They have many new features, bug fixes & performance improvements. If you can try to reproduce

The sizes of the files are extremely varied, there are millions of small (<1 MB) files and thousands of files larger than 1 GB. Attached is the tier log for gluster1 and gluster2. These are full of "demotion failed" messages, which is also shown in the status: [root at pod-sjc1-gluster1 gv0]# gluster volume tier gv0 status Node Promoted files Demoted files

Hello, Is it possible to clear the heal-failed and split-brain status in a nice way? I would personally like if gluster would automatically remove failed states when they are resolved ( if future reference is needed you can always look at the logs) I'm asking because I'd like to monitor these for issues. The monitoring script would be trivial to setup if the failed status is / can be

forgot to mention Gluster version 3.10.6 On 18 October 2017 at 13:26, Alastair Neil <ajneil.tech at gmail.com> wrote: > a short while ago I experimented with tiering on one of my volumes. I > decided it was not working out so I removed the tier. I now have spam in > the glusterd.log evert 7 seconds: > > [2017-10-18 17:17:29.578327] W [socket.c:3207:socket_connect] 0-tierd:

a short while ago I experimented with tiering on one of my volumes. I decided it was not working out so I removed the tier. I now have spam in the glusterd.log evert 7 seconds: [2017-10-18 17:17:29.578327] W [socket.c:3207:socket_connect] 0-tierd: Ignore failed connection attempt on /var/run/gluster/2e3df1c501d0a19e5076304179d1e43e.socket, (No such file or directory) [2017-10-18

Hello, I'm trying to fix an issue with a Directory Split on a gluster 3.10.3. The effect consist of a specific file in this splitted directory to randomly be unavailable on some clients. I have gathered all the informations on this gist: https://gist.githubusercontent.com/lucagervasi/534e0024d349933eef44615fa8a5c374/raw/52ff8dd6a9cc8ba09b7f258aa85743d2854f9acc/splitinfo.txt I discovered the

On 04/09/2018 05:40 PM, mabi wrote: > Again thanks that worked and I have now no more unsynched files. > > You mentioned that this bug has been fixed in 3.13, would it be possible to backport it to 3.12? I am asking because 3.13 is not a long-term release and as such I would not like to have to upgrade to 3.13. I don't think there will be another 3.12 release. Adding Karthik to see

The version I am using is glusterfs 3.6.9 Best regards, Cynthia ???? MBB SM HETRAN SW3 MATRIX Storage Mobile: +86 (0)18657188311 From: Karthik Subrahmanya [mailto:ksubrahm at redhat.com] Sent: Thursday, September 28, 2017 2:37 PM To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com> Cc: Gluster-users at gluster.org; gluster-devel at gluster.org Subject: Re: [Gluster-users]

Hi, All, We create a GlusterFS cluster with tiers. The hot tier is distributed-replicated SSDs. The cold tier is a n*(6+2) disperse volume. When copy millions of files to the cluster, we find these logs: W [socket.c:3292:socket_connect] 0-tierd: Ignore failed connection attempt on /var/run/gluster/39668fb028de4b1bb6f4880e7450c064.socket, (No such file or directory) W

Here would be also the corresponding log entries on a gluster node brick log file: [2018-04-09 06:58:47.363536] W [MSGID: 113093] [posix-gfid-path.c:84:posix_remove_gfid2path_xattr] 0-myvol-private-posix: removing gfid2path xattr failed on /data/myvol-private/brick/.glusterfs/12/67/126759f6-8364-453c-9a9c-d9ed39198b7a: key = trusted.gfid2path.2529bb66b56be110 [No data available] [2018-04-09

There are several messages "no space left on device". I would check first that free disk space is available for the volume. On Oct 22, 2017 18:42, "Milind Changire" <mchangir at redhat.com> wrote: > Herb, > What are the high and low watermarks for the tier set at ? > > # gluster volume get <vol> cluster.watermark-hi > > # gluster volume get

Herb, I'm trying to weed out issues here. So, I can see quota turned *on* and would like you to check the quota settings and test to see system behavior *if quota is turned off*. Although the file size that failed migration was 29K, I'm being a bit paranoid while weeding out issues. Are you still facing tiering errors ? I can see your response to Alex with the disk space consumption and

Hi, We create a GlusterFS cluster with tiers. The hot tier is distributed-replicated SSDs. The cold tier is a n*(6+2) disperse volume. When copy millions of files to the cluster, we find these logs: W [socket.c:3292:socket_connect] 0-tierd: Ignore failed connection attempt on /var/run/gluster/39668fb028de4b1bb6f4880e7450c064.socket, (No such file or directory) W [socket.c:3292:socket_connect]

Hello, Last Friday I upgraded my GlusterFS 3.10.7 3-way replica (with arbitrer) cluster to 3.12.7 and this morning I got a warning that 9 files on one of my volumes are not synced. Ineeded checking that volume with a "volume heal info" shows that the third node (the arbitrer node) has 9 files to be healed but are not being healed automatically. All nodes were always online and there

Milind - Thank you for the response.. >> What are the high and low watermarks for the tier set at ? # gluster volume get <vol> cluster.watermark-hi Option Value ------ ----- cluster.watermark-hi 90 # gluster volume get <vol> cluster.watermark-low Option

Herb, What are the high and low watermarks for the tier set at ? # gluster volume get <vol> cluster.watermark-hi # gluster volume get <vol> cluster.watermark-low What is the size of the file that failed to migrate as per the following tierd log: [2017-10-19 17:52:07.519614] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for

On 04/09/2018 04:36 PM, mabi wrote: > As I was suggested in the past by this mailing list a now ran a stat and getfattr on one of the problematic files on all nodes and at the end a stat on the fuse mount directly. The output is below: > > NODE1: > > STAT: > File:

Hey Henrik, Good to know that the issue got resolved. I will try to answer some of the questions you have. - The time taken to heal the file depends on its size. That's why you were seeing some delay in getting everything back to normal in the heal info output. - You did not hit the split-brain situation. In split-brain all the bricks will be blaming the other bricks. But in your case the

On 04/09/2018 05:09 PM, mabi wrote: > Thanks Ravi for your answer. > > Stupid question but how do I delete the trusted.afr xattrs on this brick? > > And when you say "this brick", do you mean the brick on the arbitrer node (node 3 in my case)? Sorry I should have been clearer. Yes the brick on the 3rd node. `setfattr -x trusted.afr.myvol-private-client-0