Hello list, I have an issue on my Gluster cluster. It is composed of two data nodes and an arbiter for all my volumes. After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is what I get : ??? - on node 1, volumes won't start, and glusterd.log shows a lot of : ??? ??? [2018-05-02 09:46:06.267817] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549) [0x7f0047ae2549] -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0) [0x7f0047aebdf0] -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371) [0x7f0047b98371] ) 0-management: Lock for vol thedude not held ??? ??? The message "W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for rom" repeated 3 times between [2018-05-02 09:45:57.262321] and [2018-05-02 09:46:06.267804] ??? ??? [2018-05-02 09:46:06.267826] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for thedude ??? - on node 2, volume are up but don't seem to be willing to correctly heal. The logs show a lot of : ??? ??? [2018-05-02 09:23:01.054196] I [MSGID: 108026] [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-thedude-replicate-0: performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3 ??? ??? [2018-05-02 09:23:01.222596] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No space left on device] ??? - on arbiter, glustershd.log shows a lot of : ??? ??? [2018-05-02 09:44:54.619476] I [MSGID: 108026] [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0: performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667 ??? ??? [2018-05-02 09:44:54.640276] E [MSGID: 114031] [client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No space left on device] ??? ??? [2018-05-02 09:44:54.657045] I [MSGID: 108026] [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0: performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389 ??? ??? [2018-05-02 09:47:09.121060] W [MSGID: 101088] [common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to save the backtrace. The clients connecting to the cluster experience problems, such as Gluster refusing to create files, etc. I'm lost here, where should I start ? ??? Thanks for your help ! ??? ??? Hoggins! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180502/15abeafc/attachment.sig>
Oh, and *there is* space on the device where the brick's data is located. ??? /dev/mapper/fedora-home?? 942G??? 868G?? 74G? 93% /export Le 02/05/2018 ? 11:49, Hoggins! a ?crit?:> Hello list, > > I have an issue on my Gluster cluster. It is composed of two data nodes > and an arbiter for all my volumes. > > After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is > what I get : > > ??? - on node 1, volumes won't start, and glusterd.log shows a lot of : > ??? ??? [2018-05-02 09:46:06.267817] W > [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] > (-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549) > [0x7f0047ae2549] > -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0) > [0x7f0047aebdf0] > -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371) > [0x7f0047b98371] ) 0-management: Lock for vol thedude not held > ??? ??? The message "W [MSGID: 106118] > [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock > not released for rom" repeated 3 times between [2018-05-02 > 09:45:57.262321] and [2018-05-02 09:46:06.267804] > ??? ??? [2018-05-02 09:46:06.267826] W [MSGID: 106118] > [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock > not released for thedude > > > ??? - on node 2, volume are up but don't seem to be willing to correctly > heal. The logs show a lot of : > ??? ??? [2018-05-02 09:23:01.054196] I [MSGID: 108026] > [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-thedude-replicate-0: > performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3 > ??? ??? [2018-05-02 09:23:01.222596] E [MSGID: 114031] > [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote > operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No > space left on device] > > > ??? - on arbiter, glustershd.log shows a lot of : > ??? ??? [2018-05-02 09:44:54.619476] I [MSGID: 108026] > [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0: > performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667 > ??? ??? [2018-05-02 09:44:54.640276] E [MSGID: 114031] > [client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote > operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No > space left on device] > ??? ??? [2018-05-02 09:44:54.657045] I [MSGID: 108026] > [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0: > performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389 > ??? ??? [2018-05-02 09:47:09.121060] W [MSGID: 101088] > [common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to > save the backtrace. > > > The clients connecting to the cluster experience problems, such as > Gluster refusing to create files, etc. > > I'm lost here, where should I start ? > > ??? Thanks for your help ! > > ??? ??? Hoggins! > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180502/096d531a/attachment.sig>
There are also free inodes on the disks of all the machines... don't where to look to solve this. Any idea ? Le 02/05/2018 ? 12:39, Hoggins! a ?crit?:> Oh, and *there is* space on the device where the brick's data is located. > > ??? /dev/mapper/fedora-home?? 942G??? 868G?? 74G? 93% /export > > Le 02/05/2018 ? 11:49, Hoggins! a ?crit?: >> Hello list, >> >> I have an issue on my Gluster cluster. It is composed of two data nodes >> and an arbiter for all my volumes. >> >> After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is >> what I get : >> >> ??? - on node 1, volumes won't start, and glusterd.log shows a lot of : >> ??? ??? [2018-05-02 09:46:06.267817] W >> [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] >> (-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549) >> [0x7f0047ae2549] >> -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0) >> [0x7f0047aebdf0] >> -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371) >> [0x7f0047b98371] ) 0-management: Lock for vol thedude not held >> ??? ??? The message "W [MSGID: 106118] >> [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock >> not released for rom" repeated 3 times between [2018-05-02 >> 09:45:57.262321] and [2018-05-02 09:46:06.267804] >> ??? ??? [2018-05-02 09:46:06.267826] W [MSGID: 106118] >> [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock >> not released for thedude >> >> >> ??? - on node 2, volume are up but don't seem to be willing to correctly >> heal. The logs show a lot of : >> ??? ??? [2018-05-02 09:23:01.054196] I [MSGID: 108026] >> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-thedude-replicate-0: >> performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3 >> ??? ??? [2018-05-02 09:23:01.222596] E [MSGID: 114031] >> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote >> operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No >> space left on device] >> >> >> ??? - on arbiter, glustershd.log shows a lot of : >> ??? ??? [2018-05-02 09:44:54.619476] I [MSGID: 108026] >> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0: >> performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667 >> ??? ??? [2018-05-02 09:44:54.640276] E [MSGID: 114031] >> [client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote >> operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No >> space left on device] >> ??? ??? [2018-05-02 09:44:54.657045] I [MSGID: 108026] >> [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0: >> performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389 >> ??? ??? [2018-05-02 09:47:09.121060] W [MSGID: 101088] >> [common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to >> save the backtrace. >> >> >> The clients connecting to the cluster experience problems, such as >> Gluster refusing to create files, etc. >> >> I'm lost here, where should I start ? >> >> ??? Thanks for your help ! >> >> ??? ??? Hoggins! >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180503/22f83307/attachment.sig>
Nithya Balachandran
2018-May-03 10:50 UTC
[Gluster-users] Healing : No space left on device
Hi, We need some more information in order to debug this. The version of Gluster you were running before the upgrade The output of gluster volume info <volname> The brick logs for the volume when the operation is performed. Regards, Nithya On 2 May 2018 at 15:19, Hoggins! <fuckspam at wheres5.com> wrote:> Hello list, > > I have an issue on my Gluster cluster. It is composed of two data nodes > and an arbiter for all my volumes. > > After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is > what I get : > > - on node 1, volumes won't start, and glusterd.log shows a lot of : > [2018-05-02 09:46:06.267817] W > [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] > (-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549) > [0x7f0047ae2549] > -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0) > [0x7f0047aebdf0] > -->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371) > [0x7f0047b98371] ) 0-management: Lock for vol thedude not held > The message "W [MSGID: 106118] > [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock > not released for rom" repeated 3 times between [2018-05-02 > 09:45:57.262321] and [2018-05-02 09:46:06.267804] > [2018-05-02 09:46:06.267826] W [MSGID: 106118] > [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock > not released for thedude > > > - on node 2, volume are up but don't seem to be willing to correctly > heal. The logs show a lot of : > [2018-05-02 09:23:01.054196] I [MSGID: 108026] > [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-thedude-replicate-0: > performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3 > [2018-05-02 09:23:01.222596] E [MSGID: 114031] > [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote > operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No > space left on device] > > > - on arbiter, glustershd.log shows a lot of : > [2018-05-02 09:44:54.619476] I [MSGID: 108026] > [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0: > performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667 > [2018-05-02 09:44:54.640276] E [MSGID: 114031] > [client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote > operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No > space left on device] > [2018-05-02 09:44:54.657045] I [MSGID: 108026] > [afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0: > performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389 > [2018-05-02 09:47:09.121060] W [MSGID: 101088] > [common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to > save the backtrace. > > > The clients connecting to the cluster experience problems, such as > Gluster refusing to create files, etc. > > I'm lost here, where should I start ? > > Thanks for your help ! > > Hoggins! > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180503/1abb0433/attachment.html>