On Fri, Jan 29, 2016 at 10:46:14AM +0000, Ronny Adsetts wrote:> Follow up for the archives. I fixed this by 'touch'ing all the files with zero links: > > Testing: > > # ls -al /stor/software/win_patches/IE11-Windows6.1-x64-en-us.exe > -rwxr--r-- 0 ainet Domain Admins 55915216 Oct 15 2013 IE11-Windows6.1-x64-en-us.exe > > # touch -a /stor/software/win_patches/IE11-Windows6.1-x64-en-us.exe > > # ls -al /stor/software/win_patches/IE11-Windows6.1-x64-en-us.exe > -rwxr--r-- 1 ainet Domain Admins 55915216 Oct 15 2013 IE11-Windows6.1-x64-en-us.exe > > and then: > > # find /stor/software -links 0 -print0 | xargs -0 touch -a > > No idea how it came about in the first place. Suspect it happened during either an operating system upgrade or upgrading glusterfs packages to those from gluster.org.If you're still willing to debug, please send across brick log file of the concerned node.> > Anyway, thanks to Joe for the pointers. > > Ronny > -- > Ronny Adsetts > Technical Director > Amazing Internet Ltd, London > t: +44 20 8977 8943 > w: www.amazinginternet.com > > Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ > Registered in England. Company No. 4042957 > >> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Venky Shankar wrote on 29/01/2016 11:09:> On Fri, Jan 29, 2016 at 10:46:14AM +0000, Ronny Adsetts wrote: >> >> No idea how it came about in the first place. Suspect it happened >> during either an operating system upgrade or upgrading glusterfs >> packages to those from gluster.org. > > If you're still willing to debug, please send across brick log file > of the concerned node.Of course. Attached is the current brick log. My previous comments about having fixed the problem were a little premature. I now have most of the files that were showing with zero links now in the 'failed-heal' state: # gluster volume heal software statistics [...] Starting time of crawl: Fri Jan 29 11:15:53 2016 Ending time of crawl: Fri Jan 29 11:15:58 2016 Type of crawl: INDEX No. of entries healed: 0 No. of entries in split-brain: 0 No. of heal failed entries: 1976 The volume log file shows all the details but the "E" error entries look like this: [2016-01-29 11:12:00.479724] E [client-rpc-fops.c:914:client3_3_flush_cbk] 0-software-client-1: remote operation failed: Operation not permitted And warnings like this: [2016-01-29 11:11:33.747428] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote operation failed: No such file or directory. Path: /win_patches/NDP20SP2-KB2446704-v2-x86.exe (9aa6b79a-b18f-42f2-ae9a-a886c523a0e7) [2016-01-29 11:11:33.749192] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote operation failed: No such file or directory. Path: <gfid:9aa6b79a-b18f-42f2-ae9a-a886c523a0e7> (9aa6b79a-b18f-42f2-ae9a-a886c523a0e7) Ronny -- Ronny Adsetts Technical Director Amazing Internet Ltd, London t: +44 20 8977 8943 w: www.amazinginternet.com Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ Registered in England. Company No. 4042957 -------------- next part -------------- A non-text attachment was scrubbed... Name: data-glusterfs-software-brick1-brick.log.gz Type: application/gzip Size: 987772 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160129/1d71201f/attachment-0001.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: OpenPGP digital signature URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160129/1d71201f/attachment-0001.sig>