Pranith Kumar Karampuri
2015-May-29 08:50 UTC
[Gluster-users] Volume heal info not reporting files in split brain and core dumping, after upgrading to 3.7.0
On 05/29/2015 02:18 PM, Pranith Kumar Karampuri wrote:> > > On 05/29/2015 02:13 PM, Alessandro De Salvo wrote: >> Hi, >> I'm facing a strange issue with split brain reporting. >> I have upgraded to 3.7.0, after stopping all gluster processes as >> described in the twiki, on all servers hosting the volumes. The >> upgrade and the restart was fine, and the volumes are accessible. >> However I had two files in split brain that I did not heal before >> upgrading, so I tried a full heal with 3.7.0. The heal was launched >> correctly, but when I now perform an heal info there is no output, >> while the heal statistics says there are actually 2 files in split >> brain. In the logs I see something like this: >> >> glustershd.log: >> [2015-05-29 08:28:43.008373] I >> [afr-self-heal-entry.c:558:afr_selfheal_entry_do] >> 0-adsnet-gluster-01-replicate-0: performing entry selfheal on >> 7fd1262d-949b-402e-96c2-ae487c8d4e27 >> [2015-05-29 08:28:43.012690] W >> [client-rpc-fops.c:241:client3_3_mknod_cbk] >> 0-adsnet-gluster-01-client-1: remote operation failed: Invalid >> argument. Path: (null) > Hey could you let us know "gluster volume info" output? Please let us > know the backtrace printed by > /var/log/glusterfs/glfsheal-<volname>.log as well.Please attach /var/log/glusterfs/glfsheal-<volname>.log file to this thread so that I can take a look. Pranith> > Pranith >> >> >> So, it seems like the files to be healed are not correctly >> identified, or at least their path is null. >> Also, every time I issue a "gluster volume heal <volname> info" a >> core dump is generated in the log area. >> All servers are using the latest CentOS 7. >> Any idea why this might be happening and how to solve it? >> Thanks, >> >> Alessandro >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Alessandro De Salvo
2015-May-29 09:07 UTC
[Gluster-users] Volume heal info not reporting files in split brain and core dumping, after upgrading to 3.7.0
Hi Pranith, many thanks for the help! The volume info of the problematic volume is the following: # gluster volume info adsnet-vm-01 Volume Name: adsnet-vm-01 Type: Replicate Volume ID: f8f615df-3dde-4ea6-9bdb-29a1706e864c Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gwads02.sta.adsnet.it:/gluster/vm01/data Brick2: gwads03.sta.adsnet.it:/gluster/vm01/data Options Reconfigured: nfs.disable: true features.barrier: disable features.file-snapshot: on server.allow-insecure: on The log is in attachment. I just wanted to add that the heal info command works fine on other volumes hosted by the same machines, so it?s just this volume which is causing problems. Thanks, Alessandro> Il giorno 29/mag/2015, alle ore 10:50, Pranith Kumar Karampuri <pkarampu at redhat.com> ha scritto: > > > > On 05/29/2015 02:18 PM, Pranith Kumar Karampuri wrote: >> >> >> On 05/29/2015 02:13 PM, Alessandro De Salvo wrote: >>> Hi, >>> I'm facing a strange issue with split brain reporting. >>> I have upgraded to 3.7.0, after stopping all gluster processes as described in the twiki, on all servers hosting the volumes. The upgrade and the restart was fine, and the volumes are accessible. >>> However I had two files in split brain that I did not heal before upgrading, so I tried a full heal with 3.7.0. The heal was launched correctly, but when I now perform an heal info there is no output, while the heal statistics says there are actually 2 files in split brain. In the logs I see something like this: >>> >>> glustershd.log: >>> [2015-05-29 08:28:43.008373] I [afr-self-heal-entry.c:558:afr_selfheal_entry_do] 0-adsnet-gluster-01-replicate-0: performing entry selfheal on 7fd1262d-949b-402e-96c2-ae487c8d4e27 >>> [2015-05-29 08:28:43.012690] W [client-rpc-fops.c:241:client3_3_mknod_cbk] 0-adsnet-gluster-01-client-1: remote operation failed: Invalid argument. Path: (null) >> Hey could you let us know "gluster volume info" output? Please let us know the backtrace printed by /var/log/glusterfs/glfsheal-<volname>.log as well. > Please attach /var/log/glusterfs/glfsheal-<volname>.log file to this thread so that I can take a look. > > Pranith >> >> Pranith >>> >>> >>> So, it seems like the files to be healed are not correctly identified, or at least their path is null. >>> Also, every time I issue a "gluster volume heal <volname> info" a core dump is generated in the log area. >>> All servers are using the latest CentOS 7. >>> Any idea why this might be happening and how to solve it? >>> Thanks, >>> >>> Alessandro >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>> http://www.gluster.org/mailman/listinfo/gluster-users <http://www.gluster.org/mailman/listinfo/gluster-users> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> http://www.gluster.org/mailman/listinfo/gluster-users <http://www.gluster.org/mailman/listinfo/gluster-users>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150529/20e5f7ab/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glfsheal-adsnet-vm-01.log.gz Type: application/x-gzip Size: 2470 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150529/20e5f7ab/attachment.gz> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150529/20e5f7ab/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1770 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150529/20e5f7ab/attachment.p7s>