Hi, we need some help...we are running a 6 node distributed-replicated master-volume which needs to be geo-replicated to another equal configured volume. recently both locations were updated from 3.4.7 to 3.5.6 to 3.6.7...running on ubuntu 14.04 lts. since update to 3.5.6 the geo-replication remains in status 'hybrid crawl' as shown below. [ 16:31:46 ] - root at gluster-ger-ber-09 /home/admin $gluster volume geo-replication ger-ber-01 gluster-wien-02::aut-wien-01 status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE STATUS CHECKPOINT STATUS CRAWL STATUS FILES SYNCD FILES PENDING BYTES PENDING DELETES PENDING FILES SKIPPED --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- gluster-ger-ber-09 ger-ber-01 /gluster-export gluster-wien-05-int::aut-wien-01 Active N/A Changelog Crawl 214645 4 0 2 11 gluster-ger-ber-12 ger-ber-01 /gluster-export gluster-wien-06-int::aut-wien-01 Passive N/A N/A 0 0 0 0 0 gluster-ger-ber-11 ger-ber-01 /gluster-export gluster-wien-03-int::aut-wien-01 Active N/A Hybrid Crawl 2831091 0 0 0 491 gluster-ger-ber-10 ger-ber-01 /gluster-export gluster-wien-02-int::aut-wien-01 Passive N/A N/A 0 0 0 0 0 gluster-ger-ber-07 ger-ber-01 /gluster-export gluster-wien-07-int::aut-wien-01 Active N/A Hybrid Crawl 2886662 0 0 0 474 gluster-ger-ber-08 ger-ber-01 /gluster-export gluster-wien-04-int::aut-wien-01 Passive N/A N/A 0 0 0 0 0 [ 16:31:51 ] - root at gluster-ger-ber-09 /home/admin $ on a slave we found a lot of repeating messages like : root at gluster-wien-07 /var/log/glusterfs/geo-replication-slaves $tail -f 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671\:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-01.gluster.log ... [2015-12-01 23:30:19.782137] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-aut-wien-01-client-2: remote operation failed: File exists. Path: <gfid:135991e9-5591-4217-b932-04cce065982f>/rfvg2cmFNJ8Xt9HG.png [2015-12-01 23:30:19.782178] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-aut-wien-01-client-3: remote operation failed: File exists. Path: <gfid:135991e9-5591-4217-b932-04cce065982f>/rfvg2cmFNJ8Xt9HG.png [2015-12-01 23:30:19.782573] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 258115: SETXATTR() /.gfid/135991e9-5591-4217-b932-04cce065982f => -1 (File exists) ... all mentioned files have the same structure on the master volume...they appear on two sub-volumes. as you can see below, on one subvolume the mentioned file was created on the 8th jun. on that day we had a rebalance on the master-volume. I'm not really sure if this dht link file should have 0 bytes on this subvolume ? i already read this but i believe our situation is a bit more complicated... https://www.gluster.org/pipermail/gluster-users/2015-February/020608.html is it safe to delete those link files and the corresponding gfid on the affected bricks ? best regards dietmar tron at dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png master_gfid_file.txt e95a43c2-4969-4662-8ea9-72e02e82df90 thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png tron at dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-ger-ber-* ls-lisa-gluster.gluster-ger-ber-07-int.out:121247406422 612 -rw-rw-rw- 2 root root 619294 Okt 18 2014 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-ger-ber-08-int.out:121289252163 612 -rw-rw-rw- 2 root root 619294 Okt 18 2014 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-ger-ber-11-int.out:43282956834 4 ---------T 2 root root 619294 Jun 8 09:52 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-ger-ber-12-int.out:65271376731 4 ---------T 2 root root 619294 Jun 8 09:52 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png tron at dp-server:~/gluster-9$ grep rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-wien-0* ls-lisa-gluster.gluster-wien-04.out:82283414551 612 -rw-rw-rw- 2 root root 619294 Oct 18 2014 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png ls-lisa-gluster.gluster-wien-05.out:82278502871 612 -rw-rw-rw- 2 root root 619294 Oct 18 2014 /gluster-export/thumbs/2014/2485/272648/rfvg2cmFNJ8Xt9HG.png tron at dp-server:~/gluster-9$ -- Dietmar Putz 3Q Medien GmbH Wetzlarer Str. 86 D-14482 Potsdam Telefax: +49 (0)331 / 2797 866 - 1 Telefon: +49 (0)331 / 2792 866 - 8 Mobile: +49 171 / 90 160 39 Mail: putz at 3qmedien.net