Ulrich Pötter
2020-Feb-06 11:27 UTC
[Gluster-users] Healing entries get healed but there are constantly new entries appearing
Dear Gluster Users, we are running the following Gluster setup: Replica 3 on 3 servers. Two are CentOs 7.6 with Gluster 6.5 and one was upgraded to Centos 7.7 with Gluster 6.7. Since the upgrade to gluster 6.7 on one of the servers, we encountered the following issue: New healing entries appear and get healed, but soon afterwards new healing entries appear. The abovementioned problem started after we upgraded the server. The healing issues do not only appear on the upgraded server, but on all three. This does not seem to be a split brain issue as the output of the command "gluster volume head <vol> info split-brain" is "number of entries in split-brain: 0" Has anyone else observed such behavior with different Gluster versions in one replica setup? We hesitate with updating the other nodes, as we do not know if this standard Gluster behaviour or if there is more to this problem. Can you help us? Thanks in advance, Ulrich
Karthik Subrahmanya
2020-Feb-06 11:51 UTC
[Gluster-users] Healing entries get healed but there are constantly new entries appearing
Hi Ulrich,>From the problem statement, seems like the client(s) have lost connectionwith brick. Can you give the following information? - How many clients are there for this volume and which version they are in? - gluster volume info <vol-name> & gluster volume status <vol-name> outputs - Check whether all the clients are connected to all the bricks. If you are using the fuse clients give the output of the following from all the clients cat <mount-path>/.meta/graphs/active/<vol-name>-client-*/private | egrep -i 'connected' -If you are using non fuse clients generate the statedumps ( https://docs.gluster.org/en/latest/Troubleshooting/statedump/) of each clients and give the output of grep -A 2 "xlator.protocol.client" /var/run/gluster/<dump-file> (If you have changed the statedump-path replace the path in the above command) Regards, Karthik On Thu, Feb 6, 2020 at 5:06 PM Ulrich P?tter <ulrich.poetter at menzel-it.net> wrote:> Dear Gluster Users, > > we are running the following Gluster setup: > Replica 3 on 3 servers. Two are CentOs 7.6 with Gluster 6.5 and one was > upgraded to Centos 7.7 with Gluster 6.7. > > Since the upgrade to gluster 6.7 on one of the servers, we encountered > the following issue: > New healing entries appear and get healed, but soon afterwards new > healing entries appear. > The abovementioned problem started after we upgraded the server. > The healing issues do not only appear on the upgraded server, but on all > three. > > This does not seem to be a split brain issue as the output of the > command "gluster volume head <vol> info split-brain" is "number of > entries in split-brain: 0" > > Has anyone else observed such behavior with different Gluster versions > in one replica setup? > > We hesitate with updating the other nodes, as we do not know if this > standard Gluster behaviour or if there is more to this problem. > > Can you help us? > > Thanks in advance, > Ulrich > > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/441850968 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200206/6b501870/attachment.html>