Lindsay, Did you see any problems in the setup before you set those options? Also, could you please share glusterd and glfsheal logs before you revert to 3.7.11, so that it can be analyzed? ----- Original Message -----> From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com> > To: "gluster-users" <gluster-users at gluster.org> > Sent: Wednesday, June 29, 2016 2:07:30 PM > Subject: Re: [Gluster-users] 3.7.12 disaster > > On 29 June 2016 at 18:30, Lindsay Mathieson <lindsay.mathieson at gmail.com> > wrote: > > Same problem again. VM froze and heal info timed out with "Not able to > > fetch volfile from glusterd". I'm going to have to revert to 3.7.11 > > > Heal process seems to be stuck at the following: > > gluster v heal datastore4 info > Brick vnb.proxmox.softlog:/tank/vmdata/datastore4 > Status: Connected > Number of entries: 0 > > Brick vng.proxmox.softlog:/tank/vmdata/datastore4 > <gfid:be318638-e8a0-4c6d-977d-7a937aa84806> - Possibly undergoing heal > > Status: Connected > Number of entries: 1 > > Brick vna.proxmox.softlog:/tank/vmdata/datastore4 > <gfid:be318638-e8a0-4c6d-977d-7a937aa84806> - Possibly undergoing heal > > Status: Connected > Number of entries: 1 > > I'm on my home now, will be offline for a couple of hours. But for now > my cluster is offline. > > -- > Lindsay > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-- Thanks, Anuradha.
No i didn?t, but I set those options and rebooted to upgrade the client at the same time. Will get the logs Sent from my Windows 10 phone From: Anuradha Talur -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160629/06e05a31/attachment.html>
On 29/06/2016 7:05 PM, Anuradha Talur wrote:> Also, could you please share glusterd and glfsheal logs before you revert to 3.7.11, > so that it can be analyzed?All the logs including the bricks :)> Can you share the glusterd log and the glfsheal log for the volume > from the system on which you ran the heal command?I fairly sure it was the vng node. I'll double check that tomorrow. The brick logs for vnb & vng are exceptionally large - seems to have a lot of lock errors and are to large to email. I> This is the part that confuses me. You can *not* set 3.7.12 option > when 3.7.11 clients are still in play. Something is a miss. Can you > give me 'ls -l <path/to/glfsheal> please?Not entirely sure if I'm interpreting your request correctly: ls -l /usr/sbin/glfsheal -rwxr-xr-x 1 root root 30912 Jun 28 03:19 /usr/sbin/glfsheal I checked all the apt versions on all 3 nodes, they all seem to be 3.7.12 When I refer to "client" I mean the qemu/gfapi client, not the fuse client. Thanks, -- Lindsay Mathieson -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160629/577a74bd/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: logs.zip Type: application/x-zip-compressed Size: 4684551 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160629/577a74bd/attachment-0001.bin>
When I try to start a VM the brick log is spammed with the following: [2016-06-29 12:29:17.844704] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.832 failed [File exists] [2016-06-29 12:29:18.030093] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.835 failed [File exists] [2016-06-29 12:29:19.276670] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.478 failed [File exists] [2016-06-29 12:29:19.915686] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.813 failed [File exists] [2016-06-29 12:29:20.270403] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.476 failed [File exists] [2016-06-29 12:29:20.750933] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.505 failed [File exists] [2016-06-29 12:29:21.175397] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.159 failed [File exists] [2016-06-29 12:29:21.366887] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.220 failed [File exists] [2016-06-29 12:29:21.827546] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.640 failed [File exists] [2016-06-29 12:29:22.329726] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.509 failed [File exists] [2016-06-29 12:29:22.790829] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.119 failed [File exists] [2016-06-29 12:29:24.180752] E [MSGID: 113022] [posix.c:1244:posix_mknod] 0-datastore4-posix: mknod on /tank/vmdata/datastore4/.shard/b2996a69-f629-4425-9098-e62c25d9f033.508 failed [File exists] -- Lindsay Mathieson