Dietmar Putz
2016-Jan-20 17:53 UTC
[Gluster-users] geo-replication 3.6.7 - no transition from hybrid to changelog crawl
Hi Milind, thank you for your reply... meanwhile i realized that the setfattr command don't work so i decided to delete the affected files and directories but without stopping the geo-replication...i did it before i red you mail. the affected folders are already replicated with the same gfid like on the master...so this is solved for the moment. afterwards i did not receive the 'errcode: 23' messages on the masters and the 'Operation not permitted' messages on the slaves for 2 1/2 days but the geo-replication restarted all the time about every 2 - 4 hours on each active master / slave with below shown "OSError: [Errno 16] Device or resource busy" message on master and slave. anytime the geo-replication restarts i saw following line embedded in the event of restart (as shown far below) : I [dht-layout.c:663:dht_layout_normalize] 0-aut-wien-vol-01-dht: Found anomalies in /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 i have found 24 of these hidden .dst* folders. All of them are stored in the same 2 different subfolders on the master and slave but the shown .dstXXb70G3x is the only one that just exists on the slave volume. I checked that folder and deleted it on the slave since it was empty. i believe that such folders somehow belongs to the geo-replication process but i have no details. however, about one day after deletion this folder was recreated on the slave again but since deletion there are no more 'Found anomalies' messages. currently the geo-replication still restarts frequently with the far below shown message and unfortunately some 'Operation not permitted' messages appears again but for different files than before. I already checked all folders on master/slave for different gfid's but there are no more different gfid's. i guess there is no way around to compare all gfid's of all files on master and slave...since it is a dist-repl. volume there are several million lines. if geo-replication then still not works i will start the suggested strace of the gsyncd. in regard to the strace i have two questions... do i need to start the strace on all active masters / slaves or is it sufficient to trace one active master and the corresponding active slave ? should i try to capture a geo-rep restart by the strace or is it sufficient to let it run one minute randomly ? maybe this is of interest to solve the problem...on the active masters there are lines like : ... [2016-01-19 18:34:58.441606] W [master(/gluster-export):1015:process] _GMaster: incomplete sync, retrying changelogs: XSYNC-CHANGELOG.1453225971 [2016-01-19 18:36:27.313515] W [master(/gluster-export):1015:process] _GMaster: incomplete sync, retrying changelogs: XSYNC-CHANGELOG.1453225971 [2016-01-19 18:37:56.337660] W [master(/gluster-export):996:process] _GMaster: changelogs XSYNC-CHANGELOG.1453225971 could not be processed - moving on... [2016-01-19 18:37:56.339124] W [master(/gluster-export):1000:process] _GMaster: SKIPPED GFID = 5a47cc07-f32f-4685-ae8e-4969995f3f1c,<huge list with gfid's> they end up in a huge list of comma separated gfid's. is there any hint to get benefit out of this xsync-changelogs, a way to find out what was incomplete ? one more thing that concerns me...i'm trying to understand the distributed geo-replication. the master volume is a living object, accessed by some clients who are upload, delete and recreate files and folders. not very frequent but i observed the mentioned two folders with different gfid's on master / slave and now they are deleted by any client on the master volume. The geo-replication is still in hybrid crawl and afaik can not delete or rename files and folders on the slave volume until changelog mode is reached. When now a client recreates the same folders on the master again they get a new gfid assigned which are different from the still existing gfid's on the slave i believe...so geo-replication should get in conflict again because of existing folders on the slave with the same path but a different gfid than on the master...like for any other files which are deleted and later recreated while geo-rep is in hybrid crawl. is that right ? if so it will be difficult to reach the changelog modus on large gluster volumes because in our case the initial hybrid crawl took some days for about 45 TB...or r/w access needs to be stopped for that time ? thanks in advance and best regards dietmar Am 19.01.2016 um 10:53 schrieb Milind Changire:> Hi Dietmar, > After discussion with Aravinda we realized that unfortunately the suggestion to: > setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR> > setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <file-path> > > won't work with 3.6.7, since provision for that workaround was added after 3.6.7. > > There's an alternative way to achieve the geo-replication: > 1. stop geo-replication > 2. delete files and directories with conflicting gfid on SLAVE > 3. use the "touch" command to touch files and directories with conflicting gfid > on MASTER > 4. start geo-replication > > This *should* get things correctly replicated to SLAVE. > Geo-replication should start with hybrid-crawl and trigger the replication to SLAVE. > > If not, then there's more to look at. > You could then send us output of the strace command for the gsyncd process, while > geo-replication is running: > # strace -ff -p <gsyncd-pid> -o gsyncd-strace > > You could terminate strace after about one minute and send us all the gsyncd-strace.<pid> > files which will help us debug the issue if its not resolved by the alternative > mechanism mentioned above. > > Also, crawl status Hybrid Crawl is not an entirely bad thing. It just could mean > that there's a lot of entries that are being processed. However, if things don't > return back to the normal state after trying out the alternative suggestion, we > could take a look at the strace output and get some clues. > > -- > Milind > > ----- Original Message ----- > From: "Dietmar Putz" <putz at 3qmedien.net> > To: "Aravinda" <avishwan at redhat.com>, gluster-users at gluster.org, "Milind Changire" <mchangir at redhat.com> > Sent: Thursday, January 14, 2016 11:07:55 PM > Subject: Re: [Gluster-users] geo-replication 3.6.7 - no transition from hybrid to changelog crawl > > Hello all, > > after some days of inactivity i started another attempt to solve this > geo-replication issue...step by step. > it looks like that some of the directories on the slave volume does not > have the same gfid like the corresponding directory on the master volume. > > for example : > > on a master-node i can see a lot of 'errcode: 23' lines like : > [2016-01-14 09:58:36.96585] W [master(/gluster-export):301:regjob] > _GMaster: Rsync: .gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50 [errcode: 23] > > on the corresponding slave the corresponding message : > [2016-01-14 09:57:06.070452] W [fuse-bridge.c:1967:fuse_create_cbk] > 0-glusterfs-fuse: 1185648: /.gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50 > => -1 (Operation not permitted) > > This is the file on the master, the file is still not replicated to the > slave. > > 120533444364 97332 -rw-r--r-- 2 2001 2001 99662854 Jan 8 13:40 > /gluster-export/3912/uploads/BSZ-2015/Z_002895D0-C832-4698-84E6-89F34CDEC2AE_20144555_ST_1.mp4 > 120533444364 97332 -rw-r--r-- 2 2001 2001 99662854 Jan 8 13:40 > /gluster-export/.glusterfs/a8/d0/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50 > > The directory on the slave already contain some files, all of them are > not available on the master anymore, obviously deleted in meantime on > the master by a client. > i have deleted and recreated this file on the master and observed the > logs for recurrence of the newly created gfid of this file...same as before. > > in http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20703 > a user reports a geo-replication problem which is possibly caused by > different gfid's of underlying directories. > and yes, the directory of this file-example above shows that the gfid of > the underlying directory differs from the gfid on the master while the > most other directories have the same gfid. > > master : > ... > # file: gluster-export/3912/uploads/BSP-2012 > trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350 > -------------- > # file: gluster-export/3912/uploads/BSZ-2003 > trusted.gfid=0xe80adc088e604234b778997d8e8c2018 > -------------- > # file: gluster-export/3912/uploads/BSZ-2004 > trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced > -------------- > # file: gluster-export/3912/uploads/BSZ-2010 > trusted.gfid=0x8044e436407d4ed3a67c81df8a7ad47f ### > -------------- > # file: gluster-export/3912/uploads/BSZ-2015 > trusted.gfid=0x0c30f50480204e02b65d4716a048b029 ### > > slave : > ... > # file: gluster-export/3912/uploads/BSP-2012 > trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350 > -------------- > # file: gluster-export/3912/uploads/BSZ-2003 > trusted.gfid=0xe80adc088e604234b778997d8e8c2018 > -------------- > # file: gluster-export/3912/uploads/BSZ-2004 > trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced > -------------- > # file: gluster-export/3912/uploads/BSZ-2010 > trusted.gfid=0xd83e8fb568c74e33a2091c547512a6ce ### > -------------- > # file: gluster-export/3912/uploads/BSZ-2015 > trusted.gfid=0xa406e1bec7f3454d8f2ce9c5f9c70eb3 ### > > > now the question...how to fix this..? > in the thread above Aravinda wrote : > > ... > To fix the issue, > ----------------- > Find the parent directory of "main.mdb", > Get the GFID of that directory, using getfattr > Check the GFID of the same directory in Slave(To confirm GFIDs are different) > To fix the issue, Delete that directory in Slave. > Set virtual xattr for that directory and all the files inside that directory. > setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR> > setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <file-path> > > Geo-rep will recreate the directory with Proper GFID and starts sync. > > deletion of the affected slave directory might be helpful... > but do i have to execute above shown setfattr commands on the master or > do they just speed up synchronization ? > usually sync should start automatically or could there be a problem > because crawl status is still in 'hybrid crawl'...? > > thanks in advance... > best regards > dietmar > > > > > On 04.01.2016 12:08, Dietmar Putz wrote: >> Hello Aravinda, >> >> thank you for your reply. >> i just made a 'find /gluster-export -type f -exec ls -lisa {} \; > >> ls-lisa-gluster-export-`hostname`.out' on each brick and checked the >> output for files with less than 2 link counts. >> i found nothing...all files on each brick have exact 2 links. >> >> the entire output for all bricks contain more than 7 million lines >> including .glusterfs but without non relevant directories and files.. >> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >> 'indices|landfill|changelogs|health_check' | wc -l >> 7007316 >> >> link count is on $4 : >> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >> 'indices|landfill|changelogs|health_check' | awk '{if($4=="2")print}' >> | tail -1 >> 62648153697 4 -rw-rw-rw- 2 root root 1713 Jan 4 01:44 >> /gluster-export/3500/files/16/01/387233/3500-6dqMmBcVby97PQtR.ism >> >> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >> 'indices|landfill|changelogs|health_check' | awk '{if($4=="1")print}' >> tron at dp-server:~/geo_rep_3$ >> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >> 'indices|landfill|changelogs|health_check' | awk '{if($4!="2")print}' >> tron at dp-server:~/geo_rep_3$ >> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >> 'indices|landfill|changelogs|health_check' | awk '{print $4}' | sort | >> uniq -c >> 7007316 2 >> tron at dp-server:~/geo_rep_3$ >> >> If i understood you right this can not be the reason for the problem. >> is there any other hint which i can check on the master or slave to >> analyse the problem....? >> >> Any help would be very appreciated >> best regards >> dietmar >> >> >> >> Am 04.01.2016 um 07:14 schrieb Aravinda: >>> Hi, >>> >>> Looks like issue with Geo-rep due to race between Create and Rename. >>> Geo-replication uses gfid-access (Mount Volume with aux-gfid-mount) >>> to create and rename files. If Create and Rename replayed more than >>> once then Geo-rep creates a two files with same GFID(not hardlink). >>> This causes one file without backend GFID link. >>> >>> Milind is working on the patch to disallow the creation of second >>> file with same GFID. >>> @Milind, Please provide more update about your patch. >>> >>> As a workaround, identify all the files in Slave volume which do not >>> have backend links and delete those files(Only in Slaves, keep backup >>> if required) >>> >>> In Brick backend, Crawl and look for files with link count less than >>> 2. (Exclude .glusterfs and .trashcan directory) >>> >>> regards >>> Aravinda >>> >>> On 01/02/2016 09:56 PM, Dietmar Putz wrote: >>>> Hello all, >>>> >>>> one more time i need some help with a geo-replication problem. >>>> recently i started a new geo-replication. the master volume contains >>>> about 45 TB data and the slave volume was new created before >>>> geo-replication setup was done. >>>> master and slave is a 6 node distributed replicated volume running >>>> glusterfs-server 3.6.7-ubuntu1~trusty1. >>>> geo-rep was starting without problems. since few days the slave >>>> volume contains about 200 GB more data than the master volume and i >>>> expected that the crawl status changes from 'hybrid crawl' to >>>> 'changelog crawl' but it remains in 'hybrid crawl'. >>>> the 'status detail' output far below shows more than 10 million >>>> synced files while the entire master volume contains just about 2 >>>> million files. some tests show that files are not deleted on the >>>> slave volume. >>>> as far as i know the hybrid crawl has the limitation of not >>>> replicating deletes and renames to the slave thus the geo-rep needs >>>> to achieve the 'changelog crawl' status after initial sync... >>>> usually this should happen more or less automatically, is this right ? >>>> >>>> the geo-rep frequently fails with below shown "OSError: [Errno 16] >>>> Device or resource busy", this error appears about every 3-4 hours >>>> on each active master node. >>>> i guess the frequent appearance of this error prevent geo-rep from >>>> changing to 'changelog crawl', does somebody experienced such >>>> problem, is this the cause of the problem ? >>>> >>>> i found some similar reports on gluster.org for gfs 3.5, 3.6 and 3.7 >>>> but none of them point me to a solution... >>>> does anybody know a solution or is there a workaround to achieve the >>>> changelog crawl status...? >>>> >>>> Any help would be very appreciated >>>> best regards >>>> dietmar >>>> >>>> >>>> >>>> >>>> Master gluster-ger-ber-07: >>>> ----------------------------- >>>> >>>> [2016-01-02 11:39:48.122546] I [master(/gluster-export):1343:crawl] >>>> _GMaster: processing xsync changelog >>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a >>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724692 >>>> [2016-01-02 11:42:55.182342] I [master(/gluster-export):1343:crawl] >>>> _GMaster: processing xsync changelog >>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a >>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724751 >>>> [2016-01-02 11:44:11.168962] I [master(/gluster-export):1340:crawl] >>>> _GMaster: finished hybrid crawl syncing, stime: (-1, 0) >>>> [2016-01-02 11:44:11.246845] I >>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master >>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ... >>>> [2016-01-02 11:44:11.265209] I >>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 3 >>>> seconds >>>> [2016-01-02 11:44:11.896940] I [master(/gluster-export):1192:crawl] >>>> _GMaster: slave's time: (-1, 0) >>>> [2016-01-02 11:44:12.171761] E [repce(/gluster-export):207:__call__] >>>> RepceClient: call 18897:139899553576768:1451735052.09 (entry_ops) >>>> failed on peer with OSError >>>> [2016-01-02 11:44:12.172101] E >>>> [syncdutils(/gluster-export):270:log_raise_exception] <top>: FAIL: >>>> Traceback (most recent call last): >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", >>>> line 164, in main >>>> main_i() >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", >>>> line 643, in main_i >>>> local.service_loop(*[r for r in [remote] if r]) >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", >>>> line 1344, in service_loop >>>> g2.crawlwrap() >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", >>>> line 539, in crawlwrap >>>> self.crawl(no_stime_update=no_stime_update) >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", >>>> line 1204, in crawl >>>> self.process(changes) >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", >>>> line 956, in process >>>> self.process_change(change, done, retry) >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", >>>> line 920, in process_change >>>> self.slave.server.entry_ops(entries) >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", >>>> line 226, in __call__ >>>> return self.ins(self.meth, *a) >>>> File >>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", >>>> line 208, in __call__ >>>> raise res >>>> OSError: [Errno 16] Device or resource busy >>>> [2016-01-02 11:44:12.258982] I >>>> [syncdutils(/gluster-export):214:finalize] <top>: exiting. >>>> [2016-01-02 11:44:12.321808] I [repce(agent):92:service_loop] >>>> RepceServer: terminating on reaching EOF. >>>> [2016-01-02 11:44:12.349766] I [syncdutils(agent):214:finalize] >>>> <top>: exiting. >>>> [2016-01-02 11:44:12.435992] I [monitor(monitor):141:set_state] >>>> Monitor: new state: faulty >>>> [2016-01-02 11:44:23.164284] I [monitor(monitor):215:monitor] >>>> Monitor: ------------------------------------------------------------ >>>> [2016-01-02 11:44:23.169981] I [monitor(monitor):216:monitor] >>>> Monitor: starting gsyncd worker >>>> [2016-01-02 11:44:23.216662] I [changelogagent(agent):72:__init__] >>>> ChangelogAgent: Agent listining... >>>> [2016-01-02 11:44:23.239778] I [gsyncd(/gluster-export):633:main_i] >>>> <top>: syncing: gluster://localhost:ger-ber-01 -> >>>> ssh://root at gluster-wien-07-int:gluster://localhost:aut-wien-vol-01 >>>> [2016-01-02 11:44:26.358613] I >>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up xsync >>>> change detection mode >>>> [2016-01-02 11:44:26.358983] I >>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as >>>> the sync engine >>>> [2016-01-02 11:44:26.359985] I >>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up >>>> changelog change detection mode >>>> [2016-01-02 11:44:26.360243] I >>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as >>>> the sync engine >>>> [2016-01-02 11:44:26.361159] I >>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up >>>> changeloghistory change detection mode >>>> [2016-01-02 11:44:26.361377] I >>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as >>>> the sync engine >>>> [2016-01-02 11:44:26.402601] I >>>> [master(/gluster-export):1311:register] _GMaster: xsync temp >>>> directory: >>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6e >>>> 41d8d6e56984/xsync >>>> [2016-01-02 11:44:26.402848] I >>>> [resource(/gluster-export):1318:service_loop] GLUSTER: Register >>>> time: 1451735066 >>>> [2016-01-02 11:44:27.26012] I >>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master >>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ... >>>> [2016-01-02 11:44:27.31605] I >>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 1 >>>> seconds >>>> [2016-01-02 11:44:27.66868] I [master(/gluster-export):1226:crawl] >>>> _GMaster: starting history crawl... turns: 1, stime: (-1, 0) >>>> [2016-01-02 11:44:27.67043] I [master(/gluster-export):1229:crawl] >>>> _GMaster: stime not available, abandoning history crawl >>>> [2016-01-02 11:44:27.112426] I >>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master >>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ... >>>> [2016-01-02 11:44:27.117506] I >>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 60 >>>> seconds >>>> [2016-01-02 11:44:27.140610] I [master(/gluster-export):1333:crawl] >>>> _GMaster: starting hybrid crawl..., stime: (-1, 0) >>>> [2016-01-02 11:45:23.417233] I [monitor(monitor):141:set_state] >>>> Monitor: new state: Stable >>>> [2016-01-02 11:45:48.225915] I [master(/gluster-export):1343:crawl] >>>> _GMaster: processing xsync changelog >>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a >>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735067 >>>> [2016-01-02 11:47:08.65231] I [master(/gluster-export):1343:crawl] >>>> _GMaster: processing xsync changelog >>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6 >>>> e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735148 >>>> ... >>>> >>>> >>>> slave gluster-wien-07 : >>>> ------------------------ >>>> >>>> [2016-01-02 11:44:12.007744] W [fuse-bridge.c:1261:fuse_err_cbk] >>>> 0-glusterfs-fuse: 1959820: SETXATTR() >>>> /.gfid/5e436e5b-086b-4720-9e70-0e49c8e09698 => -1 (File exists) >>>> [2016-01-02 11:44:12.010970] W >>>> [client-rpc-fops.c:240:client3_3_mknod_cbk] >>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists. >>>> Path: >>>> <gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm >>>> [2016-01-02 11:44:12.011327] W >>>> [client-rpc-fops.c:240:client3_3_mknod_cbk] >>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists. >>>> Path: >>>> <gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm >>>> [2016-01-02 11:44:12.012054] W [fuse-bridge.c:1261:fuse_err_cbk] >>>> 0-glusterfs-fuse: 1959822: SETXATTR() >>>> /.gfid/666bceac-7c14-4efd-81fe-8185458fcf1f => -1 (File exists) >>>> [2016-01-02 11:44:12.024743] W >>>> [client-rpc-fops.c:240:client3_3_mknod_cbk] >>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists. >>>> Path: <gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png >>>> [2016-01-02 11:44:12.024970] W >>>> [client-rpc-fops.c:240:client3_3_mknod_cbk] >>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists. >>>> Path: <gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png >>>> [2016-01-02 11:44:12.025601] W [fuse-bridge.c:1261:fuse_err_cbk] >>>> 0-glusterfs-fuse: 1959823: SETXATTR() >>>> /.gfid/5bfd6f99-07e8-4b2f-844b-aa0b6535c055 => -1 (File exists) >>>> [2016-01-02 11:44:12.100688] I >>>> [dht-selfheal.c:1065:dht_selfheal_layout_new_directory] >>>> 0-aut-wien-vol-01-dht: chunk size = 0xffffffff / 57217563 = 0x4b >>>> [2016-01-02 11:44:12.100765] I >>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>> aut-wien-vol-01-replicate-0 >>>> [2016-01-02 11:44:12.100785] I >>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>> aut-wien-vol-01-replicate-1 >>>> [2016-01-02 11:44:12.100800] I >>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>> aut-wien-vol-01-replicate-2 >>>> [2016-01-02 11:44:12.100839] I [MSGID: 109036] >>>> [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal] >>>> 0-aut-wien-vol-01-dht: Setting layout of >>>> <gfid:d4815ee4-3348-4105-9136-d0219d956ed8>/.dstXXX0HUpRD with >>>> [Subvol_name: aut-wien-vol-01-re >>>> plicate-0, Err: -1 , Start: 0 , Stop: 1430439074 ], [Subvol_name: >>>> aut-wien-vol-01-replicate-1, Err: -1 , Start: 1430439075 , Stop: >>>> 2860878149 ], [Subvol_name: aut-wien-vol-01-replicate-2, Err: -1 , >>>> Start: 2860878150 , Stop: 4294967295 ], >>>> [2016-01-02 11:44:12.114192] W >>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk] >>>> 0-aut-wien-vol-01-client-2: remote operation failed: File exists. >>>> Path: <gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x >>>> [2016-01-02 11:44:12.114275] W >>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk] >>>> 0-aut-wien-vol-01-client-3: remote operation failed: File exists. >>>> Path: <gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x >>>> [2016-01-02 11:44:12.114879] W [fuse-bridge.c:1261:fuse_err_cbk] >>>> 0-glusterfs-fuse: 1959831: SETXATTR() >>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2 => -1 (File exists) >>>> [2016-01-02 11:44:12.118473] I >>>> [dht-layout.c:663:dht_layout_normalize] 0-aut-wien-vol-01-dht: Found >>>> anomalies in >>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x (gfid >>>> 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 >>>> [2016-01-02 11:44:12.118537] I >>>> [dht-selfheal.c:1065:dht_selfheal_layout_new_directory] >>>> 0-aut-wien-vol-01-dht: chunk size = 0xffffffff / 57217563 = 0x4b >>>> [2016-01-02 11:44:12.118562] I >>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>> aut-wien-vol-01-replicate-2 >>>> [2016-01-02 11:44:12.118579] I >>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>> aut-wien-vol-01-replicate-0 >>>> [2016-01-02 11:44:12.118613] I >>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>> aut-wien-vol-01-replicate-1 >>>> [2016-01-02 11:44:12.120352] I [MSGID: 109036] >>>> [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal] >>>> 0-aut-wien-vol-01-dht: Setting layout of >>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x with >>>> [Subvol_name: aut-wien-vol-01-rep >>>> licate-0, Err: -1 , Start: 1430439075 , Stop: 2860878149 ], >>>> [Subvol_name: aut-wien-vol-01-replicate-1, Err: -1 , Start: >>>> 2860878150 , Stop: 4294967295 ], [Subvol_name: >>>> aut-wien-vol-01-replicate-2, Err: -1 , Start: 0 , Stop: 1430439074 ], >>>> [2016-01-02 11:44:12.630949] I [fuse-bridge.c:4927:fuse_thread_proc] >>>> 0-fuse: unmounting /tmp/gsyncd-aux-mount-tOUOsz >>>> [2016-01-02 11:44:12.633952] W [glusterfsd.c:1211:cleanup_and_exit] >>>> (--> 0-: received signum (15), shutting down >>>> [2016-01-02 11:44:12.633964] I [fuse-bridge.c:5607:fini] 0-fuse: >>>> Unmounting '/tmp/gsyncd-aux-mount-tOUOsz'. >>>> [2016-01-02 11:44:23.946702] I [MSGID: 100030] >>>> [glusterfsd.c:2035:main] 0-/usr/sbin/glusterfs: Started running >>>> /usr/sbin/glusterfs version 3.6.7 (args: /usr/sbin/glusterfs >>>> --aux-gfid-mount --log-file=/var/log/glusterfs/geo-replication-slav >>>> es/6a071cfa-b150-4f0b-b1ed-96ab5d4bd671:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log >>>> --volfile-server=localhost --volfile-id=aut-wien-vol-01 >>>> --client-pid=-1 /tmp/gsyncd-aux-mount-otU3wS) >>>> [2016-01-02 11:44:24.042128] I [dht-shared.c:337:dht_init_regex] >>>> 0-aut-wien-vol-01-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >>>> [2016-01-02 11:44:24.046315] I [client.c:2268:notify] >>>> 0-aut-wien-vol-01-client-0: parent translators are ready, attempting >>>> connect on transport >>>> [2016-01-02 11:44:24.046532] I [client.c:2268:notify] >>>> 0-aut-wien-vol-01-client-1: parent translators are ready, attempting >>>> connect on transport >>>> [2016-01-02 11:44:24.046664] I [client.c:2268:notify] >>>> 0-aut-wien-vol-01-client-2: parent translators are ready, attempting >>>> connect on transport >>>> [2016-01-02 11:44:24.046806] I [client.c:2268:notify] >>>> 0-aut-wien-vol-01-client-3: parent translators are ready, attempting >>>> connect on transport >>>> [2016-01-02 11:44:24.046940] I [client.c:2268:notify] >>>> 0-aut-wien-vol-01-client-4: parent translators are ready, attempting >>>> connect on transport >>>> [2016-01-02 11:44:24.047070] I [client.c:2268:notify] >>>> 0-aut-wien-vol-01-client-5: parent translators are ready, attempting >>>> connect on transport >>>> Final graph: >>>> +------------------------------------------------------------------------------+ >>>> >>>> 1: volume aut-wien-vol-01-client-0 >>>> 2: type protocol/client >>>> 3: option ping-timeout 10 >>>> 4: option remote-host gluster-wien-02-int >>>> 5: option remote-subvolume /gluster-export >>>> 6: option transport-type socket >>>> 7: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>> 8: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>> 9: option send-gids true >>>> 10: end-volume >>>> 11: >>>> 12: volume aut-wien-vol-01-client-1 >>>> 13: type protocol/client >>>> 14: option ping-timeout 10 >>>> 15: option remote-host gluster-wien-03-int >>>> 16: option remote-subvolume /gluster-export >>>> 17: option transport-type socket >>>> 18: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>> 19: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>> 20: option send-gids true >>>> 21: end-volume >>>> 22: >>>> 23: volume aut-wien-vol-01-replicate-0 >>>> 24: type cluster/replicate >>>> 25: subvolumes aut-wien-vol-01-client-0 aut-wien-vol-01-client-1 >>>> 26: end-volume >>>> 27: >>>> 28: volume aut-wien-vol-01-client-2 >>>> 29: type protocol/client >>>> 30: option ping-timeout 10 >>>> 31: option remote-host gluster-wien-04-int >>>> 32: option remote-subvolume /gluster-export >>>> 33: option transport-type socket >>>> 34: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>> 35: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>> 36: option send-gids true >>>> 37: end-volume >>>> 38: >>>> 39: volume aut-wien-vol-01-client-3 >>>> 40: type protocol/client >>>> 41: option ping-timeout 10 >>>> 42: option remote-host gluster-wien-05-int >>>> 43: option remote-subvolume /gluster-export >>>> 44: option transport-type socket >>>> 45: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>> 46: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>> 47: option send-gids true >>>> 48: end-volume >>>> 49: >>>> 50: volume aut-wien-vol-01-replicate-1 >>>> 51: type cluster/replicate >>>> 52: subvolumes aut-wien-vol-01-client-2 aut-wien-vol-01-client-3 >>>> 53: end-volume >>>> 54: >>>> 55: volume aut-wien-vol-01-client-4 >>>> 56: type protocol/client >>>> 57: option ping-timeout 10 >>>> 58: option remote-host gluster-wien-06-int >>>> 59: option remote-subvolume /gluster-export >>>> 60: option transport-type socket >>>> 61: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>> 62: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>> 63: option send-gids true >>>> 64: end-volume >>>> 65: >>>> 66: volume aut-wien-vol-01-client-5 >>>> 67: type protocol/client >>>> 68: option ping-timeout 10 >>>> 69: option remote-host gluster-wien-07-int >>>> 70: option remote-subvolume /gluster-export >>>> 71: option transport-type socket >>>> 72: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>> 73: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>> 74: option send-gids true >>>> 75: end-volume >>>> 76: >>>> 77: volume aut-wien-vol-01-replicate-2 >>>> 78: type cluster/replicate >>>> 79: subvolumes aut-wien-vol-01-client-4 aut-wien-vol-01-client-5 >>>> 80: end-volume >>>> 81: >>>> 82: volume aut-wien-vol-01-dht >>>> 83: type cluster/distribute >>>> 84: subvolumes aut-wien-vol-01-replicate-0 >>>> aut-wien-vol-01-replicate-1 aut-wien-vol-01-replicate-2 >>>> 85: end-volume >>>> 86: >>>> 87: volume aut-wien-vol-01-write-behind >>>> 88: type performance/write-behind >>>> 89: subvolumes aut-wien-vol-01-dht >>>> 90: end-volume >>>> 91: >>>> 92: volume aut-wien-vol-01-read-ahead >>>> 93: type performance/read-ahead >>>> 94: subvolumes aut-wien-vol-01-write-behind >>>> 95: end-volume >>>> 96: >>>> 97: volume aut-wien-vol-01-io-cache >>>> 98: type performance/io-cache >>>> 99: option min-file-size 0 >>>> 100: option cache-timeout 2 >>>> 101: option cache-size 1024MB >>>> 102: subvolumes aut-wien-vol-01-read-ahead >>>> 103: end-volume >>>> 104: >>>> 105: volume aut-wien-vol-01-quick-read >>>> 106: type performance/quick-read >>>> 107: option cache-size 1024MB >>>> 108: subvolumes aut-wien-vol-01-io-cache >>>> 109: end-volume >>>> 110: >>>> 111: volume aut-wien-vol-01-open-behind >>>> 112: type performance/open-behind >>>> 113: subvolumes aut-wien-vol-01-quick-read >>>> 114: end-volume >>>> 115: >>>> 116: volume aut-wien-vol-01-md-cache >>>> 117: type performance/md-cache >>>> 118: subvolumes aut-wien-vol-01-open-behind >>>> 119: end-volume >>>> 120: >>>> 121: volume aut-wien-vol-01 >>>> 122: type debug/io-stats >>>> 123: option latency-measurement off >>>> 124: option count-fop-hits off >>>> 125: subvolumes aut-wien-vol-01-md-cache >>>> 126: end-volume >>>> 127: >>>> 128: volume gfid-access-autoload >>>> 129: type features/gfid-access >>>> 130: subvolumes aut-wien-vol-01 >>>> 131: end-volume >>>> 132: >>>> 133: volume meta-autoload >>>> 134: type meta >>>> 135: subvolumes gfid-access-autoload >>>> 136: end-volume >>>> 137: >>>> +------------------------------------------------------------------------------+ >>>> >>>> [2016-01-02 11:44:24.047642] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>> 0-aut-wien-vol-01-client-5: changing port to 49153 (from 0) >>>> [2016-01-02 11:44:24.047927] I >>>> [client-handshake.c:1413:select_server_supported_programs] >>>> 0-aut-wien-vol-01-client-5: Using Program GlusterFS 3.3, Num >>>> (1298437), Version (330) >>>> [2016-01-02 11:44:24.048044] I >>>> [client-handshake.c:1200:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-5: Connected to aut-wien-vol-01-client-5, >>>> attached to remote volume '/gluster-export'. >>>> [2016-01-02 11:44:24.048050] I >>>> [client-handshake.c:1210:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-5: Server and Client lk-version numbers are >>>> not same, reopening the fds >>>> [2016-01-02 11:44:24.048088] I [MSGID: 108005] >>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-2: >>>> Subvolume 'aut-wien-vol-01-client-5' came back up; going online. >>>> [2016-01-02 11:44:24.048114] I >>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>> 0-aut-wien-vol-01-client-5: Server lk version = 1 >>>> [2016-01-02 11:44:24.048124] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>> 0-aut-wien-vol-01-client-0: changing port to 49153 (from 0) >>>> [2016-01-02 11:44:24.048132] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>> 0-aut-wien-vol-01-client-1: changing port to 49153 (from 0) >>>> [2016-01-02 11:44:24.048138] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>> 0-aut-wien-vol-01-client-2: changing port to 49153 (from 0) >>>> [2016-01-02 11:44:24.048146] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>> 0-aut-wien-vol-01-client-3: changing port to 49153 (from 0) >>>> [2016-01-02 11:44:24.048153] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>> 0-aut-wien-vol-01-client-4: changing port to 49153 (from 0) >>>> [2016-01-02 11:44:24.049070] I >>>> [client-handshake.c:1413:select_server_supported_programs] >>>> 0-aut-wien-vol-01-client-0: Using Program GlusterFS 3.3, Num >>>> (1298437), Version (330) >>>> [2016-01-02 11:44:24.049094] I >>>> [client-handshake.c:1413:select_server_supported_programs] >>>> 0-aut-wien-vol-01-client-3: Using Program GlusterFS 3.3, Num >>>> (1298437), Version (330) >>>> [2016-01-02 11:44:24.049113] I >>>> [client-handshake.c:1413:select_server_supported_programs] >>>> 0-aut-wien-vol-01-client-2: Using Program GlusterFS 3.3, Num >>>> (1298437), Version (330) >>>> [2016-01-02 11:44:24.049131] I >>>> [client-handshake.c:1413:select_server_supported_programs] >>>> 0-aut-wien-vol-01-client-1: Using Program GlusterFS 3.3, Num >>>> (1298437), Version (330) >>>> [2016-01-02 11:44:24.049224] I >>>> [client-handshake.c:1413:select_server_supported_programs] >>>> 0-aut-wien-vol-01-client-4: Using Program GlusterFS 3.3, Num >>>> (1298437), Version (330) >>>> [2016-01-02 11:44:24.049307] I >>>> [client-handshake.c:1200:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-0: Connected to aut-wien-vol-01-client-0, >>>> attached to remote volume '/gluster-export'. >>>> [2016-01-02 11:44:24.049312] I >>>> [client-handshake.c:1210:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-0: Server and Client lk-version numbers are >>>> not same, reopening the fds >>>> [2016-01-02 11:44:24.049324] I [MSGID: 108005] >>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-0: >>>> Subvolume 'aut-wien-vol-01-client-0' came back up; going online. >>>> [2016-01-02 11:44:24.049384] I >>>> [client-handshake.c:1200:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-3: Connected to aut-wien-vol-01-client-3, >>>> attached to remote volume '/gluster-export'. >>>> [2016-01-02 11:44:24.049389] I >>>> [client-handshake.c:1210:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-3: Server and Client lk-version numbers are >>>> not same, reopening the fds >>>> [2016-01-02 11:44:24.049400] I [MSGID: 108005] >>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-1: >>>> Subvolume 'aut-wien-vol-01-client-3' came back up; going online. >>>> [2016-01-02 11:44:24.049418] I >>>> [client-handshake.c:1200:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-2: Connected to aut-wien-vol-01-client-2, >>>> attached to remote volume '/gluster-export'. >>>> [2016-01-02 11:44:24.049422] I >>>> [client-handshake.c:1210:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-2: Server and Client lk-version numbers are >>>> not same, reopening the fds >>>> [2016-01-02 11:44:24.049460] I >>>> [client-handshake.c:1200:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-1: Connected to aut-wien-vol-01-client-1, >>>> attached to remote volume '/gluster-export'. >>>> [2016-01-02 11:44:24.049465] I >>>> [client-handshake.c:1210:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-1: Server and Client lk-version numbers are >>>> not same, reopening the fds >>>> [2016-01-02 11:44:24.049493] I >>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>> 0-aut-wien-vol-01-client-0: Server lk version = 1 >>>> [2016-01-02 11:44:24.049567] I >>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>> 0-aut-wien-vol-01-client-3: Server lk version = 1 >>>> [2016-01-02 11:44:24.049632] I >>>> [client-handshake.c:1200:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-4: Connected to aut-wien-vol-01-client-4, >>>> attached to remote volume '/gluster-export'. >>>> [2016-01-02 11:44:24.049638] I >>>> [client-handshake.c:1210:client_setvolume_cbk] >>>> 0-aut-wien-vol-01-client-4: Server and Client lk-version numbers are >>>> not same, reopening the fds >>>> [2016-01-02 11:44:24.052103] I [fuse-bridge.c:5086:fuse_graph_setup] >>>> 0-fuse: switched to graph 0 >>>> [2016-01-02 11:44:24.052150] I >>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>> 0-aut-wien-vol-01-client-2: Server lk version = 1 >>>> [2016-01-02 11:44:24.052163] I >>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>> 0-aut-wien-vol-01-client-4: Server lk version = 1 >>>> [2016-01-02 11:44:24.052192] I >>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>> 0-aut-wien-vol-01-client-1: Server lk version = 1 >>>> [2016-01-02 11:44:24.052204] I [fuse-bridge.c:4015:fuse_init] >>>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 >>>> kernel 7.20 >>>> [2016-01-02 11:44:24.053991] I >>>> [afr-common.c:1491:afr_local_discovery_cbk] >>>> 0-aut-wien-vol-01-replicate-2: selecting local read_child >>>> aut-wien-vol-01-client-5 >>>> [2016-01-02 11:45:48.613563] W >>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk] >>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists. >>>> Path: /keys >>>> [2016-01-02 11:45:48.614131] W >>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk] >>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists. >>>> Path: /keys >>>> [2016-01-02 11:45:48.614436] W [fuse-bridge.c:1261:fuse_err_cbk] >>>> 0-glusterfs-fuse: 12: SETXATTR() >>>> /.gfid/00000000-0000-0000-0000-000000000001 => -1 (File exists) >>>> ... >>>> >>>> >>>> [ 13:41:40 ] - root at gluster-ger-ber-07 >>>> /var/log/glusterfs/geo-replication/ger-ber-01 $gluster volume >>>> geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 status >>>> detail >>>> >>>> MASTER NODE MASTER VOL MASTER BRICK >>>> SLAVE STATUS CHECKPOINT >>>> STATUS CRAWL STATUS FILES SYNCD FILES PENDING BYTES >>>> PENDING DELETES PENDING FILES SKIPPED >>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>> >>>> gluster-ger-ber-07 ger-ber-01 /gluster-export >>>> gluster-wien-07-int::aut-wien-vol-01 Active N/A >>>> Hybrid Crawl 10743644 8192 0 0 0 >>>> gluster-ger-ber-11 ger-ber-01 /gluster-export >>>> gluster-wien-03-int::aut-wien-vol-01 Active N/A >>>> Hybrid Crawl 16037091 8192 0 0 0 >>>> gluster-ger-ber-10 ger-ber-01 /gluster-export >>>> gluster-wien-02-int::aut-wien-vol-01 Passive N/A >>>> N/A 0 0 0 0 0 >>>> gluster-ger-ber-12 ger-ber-01 /gluster-export >>>> gluster-wien-06-int::aut-wien-vol-01 Passive N/A >>>> N/A 0 0 0 0 0 >>>> gluster-ger-ber-09 ger-ber-01 /gluster-export >>>> gluster-wien-05-int::aut-wien-vol-01 Active N/A >>>> Hybrid Crawl 16180514 8192 0 0 0 >>>> gluster-ger-ber-08 ger-ber-01 /gluster-export >>>> gluster-wien-04-int::aut-wien-vol-01 Passive N/A >>>> N/A 0 0 0 0 0 >>>> >>>> >>>> [ 13:41:55 ] - root at gluster-ger-ber-07 >>>> /var/log/glusterfs/geo-replication/ger-ber-01 $gluster volume >>>> geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 config >>>> special_sync_mode: partial >>>> state_socket_unencoded: >>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.socket >>>> gluster_log_file: >>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log >>>> ssh_command: ssh -p 2503 -oPasswordAuthentication=no >>>> -oStrictHostKeyChecking=no -i >>>> /var/lib/glusterd/geo-replication/secret.pem >>>> ignore_deletes: true >>>> change_detector: changelog >>>> ssh_command_tar: ssh -p 2503 -oPasswordAuthentication=no >>>> -oStrictHostKeyChecking=no -i >>>> /var/lib/glusterd/geo-replication/tar_ssh.pem >>>> state_file: >>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.status >>>> remote_gsyncd: /nonexistent/gsyncd >>>> log_file: >>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.log >>>> changelog_log_file: >>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-changes.log >>>> socketdir: /var/run >>>> working_dir: >>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01 >>>> state_detail_file: >>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-detail.status >>>> session_owner: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 >>>> gluster_command_dir: /usr/sbin/ >>>> pid_file: >>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.pid >>>> georep_session_working_dir: >>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ >>>> gluster_params: aux-gfid-mount >>>> volume_id: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 >>>> [ 13:42:11 ] - root at gluster-ger-ber-07 >>>> /var/log/glusterfs/geo-replication/ger-ber-01 $ >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-users >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users
Milind Changire
2016-Jan-20 18:28 UTC
[Gluster-users] geo-replication 3.6.7 - no transition from hybrid to changelog crawl
Dietmar, I just looked at your very first post describing the problem and I found ignore_deletes: true in the geo-replication config command output. If you'd like the slave volume to replicate file deletion as well, then the "ignore_deletes" should be set to "false" That should help resolve the CREATE + DELETE + CREATE issue. If this doesn't help, then strace output for gsyncd could be the savior. ----- To add further ... Lately we've stumbled across another issue with the CREATE + RENAME + CREATE sequence on geo-replication restart. Two fixes have been posted and are available for review upstream. geo-rep: avoid creating multiple entries with same gfid -- http://review.gluster.org/13186 geo-rep: hard-link rename issues on changelog replay -- http://review.gluster.org/13189 I'll post info about the fix propagation plan for the 3.6.x series later. -- Milind On Wed, Jan 20, 2016 at 11:23 PM, Dietmar Putz <putz at 3qmedien.net> wrote:> Hi Milind, > > thank you for your reply... > meanwhile i realized that the setfattr command don't work so i decided to > delete the affected files and directories but without stopping the > geo-replication...i did it before i red you mail. > the affected folders are already replicated with the same gfid like on the > master...so this is solved for the moment. > afterwards i did not receive the 'errcode: 23' messages on the masters and > the 'Operation not permitted' messages on the slaves for 2 1/2 days but the > geo-replication restarted all the time about every 2 - 4 hours on each > active master / slave with below shown "OSError: [Errno 16] Device or > resource busy" message on master and slave. > anytime the geo-replication restarts i saw following line embedded in the > event of restart (as shown far below) : > > I [dht-layout.c:663:dht_layout_normalize] 0-aut-wien-vol-01-dht: Found > anomalies in /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x (gfid > = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 > > i have found 24 of these hidden .dst* folders. All of them are stored in > the same 2 different subfolders on the master and slave but the shown > .dstXXb70G3x is the only one that just exists on the slave volume. I > checked that folder and deleted it on the slave since it was empty. i > believe that such folders somehow belongs to the geo-replication process > but i have no details. > however, about one day after deletion this folder was recreated on the > slave again but since deletion there are no more 'Found anomalies' messages. > currently the geo-replication still restarts frequently with the far below > shown message and unfortunately some 'Operation not permitted' messages > appears again but for different files than before. > I already checked all folders on master/slave for different gfid's but > there are no more different gfid's. i guess there is no way around to > compare all gfid's of all files on master and slave...since it is a > dist-repl. volume there are several million lines. > if geo-replication then still not works i will start the suggested strace > of the gsyncd. in regard to the strace i have two questions... > > do i need to start the strace on all active masters / slaves or is it > sufficient to trace one active master and the corresponding active slave ? > should i try to capture a geo-rep restart by the strace or is it > sufficient to let it run one minute randomly ? > > > maybe this is of interest to solve the problem...on the active masters > there are lines like : > ... > [2016-01-19 18:34:58.441606] W [master(/gluster-export):1015:process] > _GMaster: incomplete sync, retrying changelogs: XSYNC-CHANGELOG.1453225971 > [2016-01-19 18:36:27.313515] W [master(/gluster-export):1015:process] > _GMaster: incomplete sync, retrying changelogs: XSYNC-CHANGELOG.1453225971 > [2016-01-19 18:37:56.337660] W [master(/gluster-export):996:process] > _GMaster: changelogs XSYNC-CHANGELOG.1453225971 could not be processed - > moving on... > [2016-01-19 18:37:56.339124] W [master(/gluster-export):1000:process] > _GMaster: SKIPPED GFID = 5a47cc07-f32f-4685-ae8e-4969995f3f1c,<huge list > with gfid's> > > they end up in a huge list of comma separated gfid's. is there any hint to > get benefit out of this xsync-changelogs, a way to find out what was > incomplete ? > > one more thing that concerns me...i'm trying to understand the distributed > geo-replication. > the master volume is a living object, accessed by some clients who are > upload, delete and recreate files and folders. not very frequent but i > observed the mentioned two folders with different gfid's on master / slave > and now they are deleted by any client on the master volume. The > geo-replication is still in hybrid crawl and afaik can not delete or rename > files and folders on the slave volume until changelog mode is reached. When > now a client recreates the same folders on the master again they get a new > gfid assigned which are different from the still existing gfid's on the > slave i believe...so geo-replication should get in conflict again because > of existing folders on the slave with the same path but a different gfid > than on the master...like for any other files which are deleted and later > recreated while geo-rep is in hybrid crawl. is that right ? > if so it will be difficult to reach the changelog modus on large gluster > volumes because in our case the initial hybrid crawl took some days for > about 45 TB...or r/w access needs to be stopped for that time ? > > thanks in advance and > best regards > dietmar > > > > > Am 19.01.2016 um 10:53 schrieb Milind Changire: > >> Hi Dietmar, >> After discussion with Aravinda we realized that unfortunately the >> suggestion to: >> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR> >> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <file-path> >> >> won't work with 3.6.7, since provision for that workaround was added >> after 3.6.7. >> >> There's an alternative way to achieve the geo-replication: >> 1. stop geo-replication >> 2. delete files and directories with conflicting gfid on SLAVE >> 3. use the "touch" command to touch files and directories with >> conflicting gfid >> on MASTER >> 4. start geo-replication >> >> This *should* get things correctly replicated to SLAVE. >> Geo-replication should start with hybrid-crawl and trigger the >> replication to SLAVE. >> >> If not, then there's more to look at. >> You could then send us output of the strace command for the gsyncd >> process, while >> geo-replication is running: >> # strace -ff -p <gsyncd-pid> -o gsyncd-strace >> >> You could terminate strace after about one minute and send us all the >> gsyncd-strace.<pid> >> files which will help us debug the issue if its not resolved by the >> alternative >> mechanism mentioned above. >> >> Also, crawl status Hybrid Crawl is not an entirely bad thing. It just >> could mean >> that there's a lot of entries that are being processed. However, if >> things don't >> return back to the normal state after trying out the alternative >> suggestion, we >> could take a look at the strace output and get some clues. >> >> -- >> Milind >> >> ----- Original Message ----- >> From: "Dietmar Putz" <putz at 3qmedien.net> >> To: "Aravinda" <avishwan at redhat.com>, gluster-users at gluster.org, "Milind >> Changire" <mchangir at redhat.com> >> Sent: Thursday, January 14, 2016 11:07:55 PM >> Subject: Re: [Gluster-users] geo-replication 3.6.7 - no transition from >> hybrid to changelog crawl >> >> Hello all, >> >> after some days of inactivity i started another attempt to solve this >> geo-replication issue...step by step. >> it looks like that some of the directories on the slave volume does not >> have the same gfid like the corresponding directory on the master volume. >> >> for example : >> >> on a master-node i can see a lot of 'errcode: 23' lines like : >> [2016-01-14 09:58:36.96585] W [master(/gluster-export):301:regjob] >> _GMaster: Rsync: .gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50 [errcode: 23] >> >> on the corresponding slave the corresponding message : >> [2016-01-14 09:57:06.070452] W [fuse-bridge.c:1967:fuse_create_cbk] >> 0-glusterfs-fuse: 1185648: /.gfid/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50 >> => -1 (Operation not permitted) >> >> This is the file on the master, the file is still not replicated to the >> slave. >> >> 120533444364 97332 -rw-r--r-- 2 2001 2001 99662854 Jan 8 13:40 >> >> /gluster-export/3912/uploads/BSZ-2015/Z_002895D0-C832-4698-84E6-89F34CDEC2AE_20144555_ST_1.mp4 >> 120533444364 97332 -rw-r--r-- 2 2001 2001 99662854 Jan 8 13:40 >> /gluster-export/.glusterfs/a8/d0/a8d0387d-c5ad-4eeb-9fc6-637fb8299a50 >> >> The directory on the slave already contain some files, all of them are >> not available on the master anymore, obviously deleted in meantime on >> the master by a client. >> i have deleted and recreated this file on the master and observed the >> logs for recurrence of the newly created gfid of this file...same as >> before. >> >> in http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20703 >> a user reports a geo-replication problem which is possibly caused by >> different gfid's of underlying directories. >> and yes, the directory of this file-example above shows that the gfid of >> the underlying directory differs from the gfid on the master while the >> most other directories have the same gfid. >> >> master : >> ... >> # file: gluster-export/3912/uploads/BSP-2012 >> trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350 >> -------------- >> # file: gluster-export/3912/uploads/BSZ-2003 >> trusted.gfid=0xe80adc088e604234b778997d8e8c2018 >> -------------- >> # file: gluster-export/3912/uploads/BSZ-2004 >> trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced >> -------------- >> # file: gluster-export/3912/uploads/BSZ-2010 >> trusted.gfid=0x8044e436407d4ed3a67c81df8a7ad47f ### >> -------------- >> # file: gluster-export/3912/uploads/BSZ-2015 >> trusted.gfid=0x0c30f50480204e02b65d4716a048b029 ### >> >> slave : >> ... >> # file: gluster-export/3912/uploads/BSP-2012 >> trusted.gfid=0x8f1d480351bb455b9adde190f2c2b350 >> -------------- >> # file: gluster-export/3912/uploads/BSZ-2003 >> trusted.gfid=0xe80adc088e604234b778997d8e8c2018 >> -------------- >> # file: gluster-export/3912/uploads/BSZ-2004 >> trusted.gfid=0xfe417dd16bbe4ae4a6a1936cfee7aced >> -------------- >> # file: gluster-export/3912/uploads/BSZ-2010 >> trusted.gfid=0xd83e8fb568c74e33a2091c547512a6ce ### >> -------------- >> # file: gluster-export/3912/uploads/BSZ-2015 >> trusted.gfid=0xa406e1bec7f3454d8f2ce9c5f9c70eb3 ### >> >> >> now the question...how to fix this..? >> in the thread above Aravinda wrote : >> >> ... >> To fix the issue, >> ----------------- >> Find the parent directory of "main.mdb", >> Get the GFID of that directory, using getfattr >> Check the GFID of the same directory in Slave(To confirm GFIDs are >> different) >> To fix the issue, Delete that directory in Slave. >> Set virtual xattr for that directory and all the files inside that >> directory. >> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <DIR> >> setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <file-path> >> >> Geo-rep will recreate the directory with Proper GFID and starts sync. >> >> deletion of the affected slave directory might be helpful... >> but do i have to execute above shown setfattr commands on the master or >> do they just speed up synchronization ? >> usually sync should start automatically or could there be a problem >> because crawl status is still in 'hybrid crawl'...? >> >> thanks in advance... >> best regards >> dietmar >> >> >> >> >> On 04.01.2016 12:08, Dietmar Putz wrote: >> >>> Hello Aravinda, >>> >>> thank you for your reply. >>> i just made a 'find /gluster-export -type f -exec ls -lisa {} \; > >>> ls-lisa-gluster-export-`hostname`.out' on each brick and checked the >>> output for files with less than 2 link counts. >>> i found nothing...all files on each brick have exact 2 links. >>> >>> the entire output for all bricks contain more than 7 million lines >>> including .glusterfs but without non relevant directories and files.. >>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >>> 'indices|landfill|changelogs|health_check' | wc -l >>> 7007316 >>> >>> link count is on $4 : >>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >>> 'indices|landfill|changelogs|health_check' | awk '{if($4=="2")print}' >>> | tail -1 >>> 62648153697 4 -rw-rw-rw- 2 root root 1713 Jan 4 01:44 >>> /gluster-export/3500/files/16/01/387233/3500-6dqMmBcVby97PQtR.ism >>> >>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >>> 'indices|landfill|changelogs|health_check' | awk '{if($4=="1")print}' >>> tron at dp-server:~/geo_rep_3$ >>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >>> 'indices|landfill|changelogs|health_check' | awk '{if($4!="2")print}' >>> tron at dp-server:~/geo_rep_3$ >>> tron at dp-server:~/geo_rep_3$ cat ls-lisa-gluster-wien-0* | egrep -v >>> 'indices|landfill|changelogs|health_check' | awk '{print $4}' | sort | >>> uniq -c >>> 7007316 2 >>> tron at dp-server:~/geo_rep_3$ >>> >>> If i understood you right this can not be the reason for the problem. >>> is there any other hint which i can check on the master or slave to >>> analyse the problem....? >>> >>> Any help would be very appreciated >>> best regards >>> dietmar >>> >>> >>> >>> Am 04.01.2016 um 07:14 schrieb Aravinda: >>> >>>> Hi, >>>> >>>> Looks like issue with Geo-rep due to race between Create and Rename. >>>> Geo-replication uses gfid-access (Mount Volume with aux-gfid-mount) >>>> to create and rename files. If Create and Rename replayed more than >>>> once then Geo-rep creates a two files with same GFID(not hardlink). >>>> This causes one file without backend GFID link. >>>> >>>> Milind is working on the patch to disallow the creation of second >>>> file with same GFID. >>>> @Milind, Please provide more update about your patch. >>>> >>>> As a workaround, identify all the files in Slave volume which do not >>>> have backend links and delete those files(Only in Slaves, keep backup >>>> if required) >>>> >>>> In Brick backend, Crawl and look for files with link count less than >>>> 2. (Exclude .glusterfs and .trashcan directory) >>>> >>>> regards >>>> Aravinda >>>> >>>> On 01/02/2016 09:56 PM, Dietmar Putz wrote: >>>> >>>>> Hello all, >>>>> >>>>> one more time i need some help with a geo-replication problem. >>>>> recently i started a new geo-replication. the master volume contains >>>>> about 45 TB data and the slave volume was new created before >>>>> geo-replication setup was done. >>>>> master and slave is a 6 node distributed replicated volume running >>>>> glusterfs-server 3.6.7-ubuntu1~trusty1. >>>>> geo-rep was starting without problems. since few days the slave >>>>> volume contains about 200 GB more data than the master volume and i >>>>> expected that the crawl status changes from 'hybrid crawl' to >>>>> 'changelog crawl' but it remains in 'hybrid crawl'. >>>>> the 'status detail' output far below shows more than 10 million >>>>> synced files while the entire master volume contains just about 2 >>>>> million files. some tests show that files are not deleted on the >>>>> slave volume. >>>>> as far as i know the hybrid crawl has the limitation of not >>>>> replicating deletes and renames to the slave thus the geo-rep needs >>>>> to achieve the 'changelog crawl' status after initial sync... >>>>> usually this should happen more or less automatically, is this right ? >>>>> >>>>> the geo-rep frequently fails with below shown "OSError: [Errno 16] >>>>> Device or resource busy", this error appears about every 3-4 hours >>>>> on each active master node. >>>>> i guess the frequent appearance of this error prevent geo-rep from >>>>> changing to 'changelog crawl', does somebody experienced such >>>>> problem, is this the cause of the problem ? >>>>> >>>>> i found some similar reports on gluster.org for gfs 3.5, 3.6 and 3.7 >>>>> but none of them point me to a solution... >>>>> does anybody know a solution or is there a workaround to achieve the >>>>> changelog crawl status...? >>>>> >>>>> Any help would be very appreciated >>>>> best regards >>>>> dietmar >>>>> >>>>> >>>>> >>>>> >>>>> Master gluster-ger-ber-07: >>>>> ----------------------------- >>>>> >>>>> [2016-01-02 11:39:48.122546] I [master(/gluster-export):1343:crawl] >>>>> _GMaster: processing xsync changelog >>>>> >>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a >>>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724692 >>>>> [2016-01-02 11:42:55.182342] I [master(/gluster-export):1343:crawl] >>>>> _GMaster: processing xsync changelog >>>>> >>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a >>>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451724751 >>>>> [2016-01-02 11:44:11.168962] I [master(/gluster-export):1340:crawl] >>>>> _GMaster: finished hybrid crawl syncing, stime: (-1, 0) >>>>> [2016-01-02 11:44:11.246845] I >>>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master >>>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ... >>>>> [2016-01-02 11:44:11.265209] I >>>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 3 >>>>> seconds >>>>> [2016-01-02 11:44:11.896940] I [master(/gluster-export):1192:crawl] >>>>> _GMaster: slave's time: (-1, 0) >>>>> [2016-01-02 11:44:12.171761] E [repce(/gluster-export):207:__call__] >>>>> RepceClient: call 18897:139899553576768:1451735052.09 (entry_ops) >>>>> failed on peer with OSError >>>>> [2016-01-02 11:44:12.172101] E >>>>> [syncdutils(/gluster-export):270:log_raise_exception] <top>: FAIL: >>>>> Traceback (most recent call last): >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", >>>>> line 164, in main >>>>> main_i() >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py", >>>>> line 643, in main_i >>>>> local.service_loop(*[r for r in [remote] if r]) >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", >>>>> line 1344, in service_loop >>>>> g2.crawlwrap() >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", >>>>> line 539, in crawlwrap >>>>> self.crawl(no_stime_update=no_stime_update) >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", >>>>> line 1204, in crawl >>>>> self.process(changes) >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", >>>>> line 956, in process >>>>> self.process_change(change, done, retry) >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py", >>>>> line 920, in process_change >>>>> self.slave.server.entry_ops(entries) >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", >>>>> line 226, in __call__ >>>>> return self.ins(self.meth, *a) >>>>> File >>>>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", >>>>> line 208, in __call__ >>>>> raise res >>>>> OSError: [Errno 16] Device or resource busy >>>>> [2016-01-02 11:44:12.258982] I >>>>> [syncdutils(/gluster-export):214:finalize] <top>: exiting. >>>>> [2016-01-02 11:44:12.321808] I [repce(agent):92:service_loop] >>>>> RepceServer: terminating on reaching EOF. >>>>> [2016-01-02 11:44:12.349766] I [syncdutils(agent):214:finalize] >>>>> <top>: exiting. >>>>> [2016-01-02 11:44:12.435992] I [monitor(monitor):141:set_state] >>>>> Monitor: new state: faulty >>>>> [2016-01-02 11:44:23.164284] I [monitor(monitor):215:monitor] >>>>> Monitor: ------------------------------------------------------------ >>>>> [2016-01-02 11:44:23.169981] I [monitor(monitor):216:monitor] >>>>> Monitor: starting gsyncd worker >>>>> [2016-01-02 11:44:23.216662] I [changelogagent(agent):72:__init__] >>>>> ChangelogAgent: Agent listining... >>>>> [2016-01-02 11:44:23.239778] I [gsyncd(/gluster-export):633:main_i] >>>>> <top>: syncing: gluster://localhost:ger-ber-01 -> >>>>> ssh://root at gluster-wien-07-int:gluster://localhost:aut-wien-vol-01 >>>>> [2016-01-02 11:44:26.358613] I >>>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up xsync >>>>> change detection mode >>>>> [2016-01-02 11:44:26.358983] I >>>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as >>>>> the sync engine >>>>> [2016-01-02 11:44:26.359985] I >>>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up >>>>> changelog change detection mode >>>>> [2016-01-02 11:44:26.360243] I >>>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as >>>>> the sync engine >>>>> [2016-01-02 11:44:26.361159] I >>>>> [master(/gluster-export):75:gmaster_builder] <top>: setting up >>>>> changeloghistory change detection mode >>>>> [2016-01-02 11:44:26.361377] I >>>>> [master(/gluster-export):413:__init__] _GMaster: using 'rsync' as >>>>> the sync engine >>>>> [2016-01-02 11:44:26.402601] I >>>>> [master(/gluster-export):1311:register] _GMaster: xsync temp >>>>> directory: >>>>> >>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6e >>>>> 41d8d6e56984/xsync >>>>> [2016-01-02 11:44:26.402848] I >>>>> [resource(/gluster-export):1318:service_loop] GLUSTER: Register >>>>> time: 1451735066 >>>>> [2016-01-02 11:44:27.26012] I >>>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master >>>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ... >>>>> [2016-01-02 11:44:27.31605] I >>>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 1 >>>>> seconds >>>>> [2016-01-02 11:44:27.66868] I [master(/gluster-export):1226:crawl] >>>>> _GMaster: starting history crawl... turns: 1, stime: (-1, 0) >>>>> [2016-01-02 11:44:27.67043] I [master(/gluster-export):1229:crawl] >>>>> _GMaster: stime not available, abandoning history crawl >>>>> [2016-01-02 11:44:27.112426] I >>>>> [master(/gluster-export):490:crawlwrap] _GMaster: primary master >>>>> with volume id 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 ... >>>>> [2016-01-02 11:44:27.117506] I >>>>> [master(/gluster-export):501:crawlwrap] _GMaster: crawl interval: 60 >>>>> seconds >>>>> [2016-01-02 11:44:27.140610] I [master(/gluster-export):1333:crawl] >>>>> _GMaster: starting hybrid crawl..., stime: (-1, 0) >>>>> [2016-01-02 11:45:23.417233] I [monitor(monitor):141:set_state] >>>>> Monitor: new state: Stable >>>>> [2016-01-02 11:45:48.225915] I [master(/gluster-export):1343:crawl] >>>>> _GMaster: processing xsync changelog >>>>> >>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a >>>>> 6e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735067 >>>>> [2016-01-02 11:47:08.65231] I [master(/gluster-export):1343:crawl] >>>>> _GMaster: processing xsync changelog >>>>> >>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01/9d7139ecf10a6fc33a6 >>>>> e41d8d6e56984/xsync/XSYNC-CHANGELOG.1451735148 >>>>> ... >>>>> >>>>> >>>>> slave gluster-wien-07 : >>>>> ------------------------ >>>>> >>>>> [2016-01-02 11:44:12.007744] W [fuse-bridge.c:1261:fuse_err_cbk] >>>>> 0-glusterfs-fuse: 1959820: SETXATTR() >>>>> /.gfid/5e436e5b-086b-4720-9e70-0e49c8e09698 => -1 (File exists) >>>>> [2016-01-02 11:44:12.010970] W >>>>> [client-rpc-fops.c:240:client3_3_mknod_cbk] >>>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists. >>>>> Path: >>>>> <gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm >>>>> [2016-01-02 11:44:12.011327] W >>>>> [client-rpc-fops.c:240:client3_3_mknod_cbk] >>>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists. >>>>> Path: >>>>> <gfid:666bceac-7c14-4efd-81fe-8185458fcf1f>/11-kxyrM3NgdtBWPFv4.webm >>>>> [2016-01-02 11:44:12.012054] W [fuse-bridge.c:1261:fuse_err_cbk] >>>>> 0-glusterfs-fuse: 1959822: SETXATTR() >>>>> /.gfid/666bceac-7c14-4efd-81fe-8185458fcf1f => -1 (File exists) >>>>> [2016-01-02 11:44:12.024743] W >>>>> [client-rpc-fops.c:240:client3_3_mknod_cbk] >>>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists. >>>>> Path: <gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png >>>>> [2016-01-02 11:44:12.024970] W >>>>> [client-rpc-fops.c:240:client3_3_mknod_cbk] >>>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists. >>>>> Path: <gfid:5bfd6f99-07e8-4b2f-844b-aa0b6535c055>/Gf4FYbpDTC7yK2mv.png >>>>> [2016-01-02 11:44:12.025601] W [fuse-bridge.c:1261:fuse_err_cbk] >>>>> 0-glusterfs-fuse: 1959823: SETXATTR() >>>>> /.gfid/5bfd6f99-07e8-4b2f-844b-aa0b6535c055 => -1 (File exists) >>>>> [2016-01-02 11:44:12.100688] I >>>>> [dht-selfheal.c:1065:dht_selfheal_layout_new_directory] >>>>> 0-aut-wien-vol-01-dht: chunk size = 0xffffffff / 57217563 = 0x4b >>>>> [2016-01-02 11:44:12.100765] I >>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>>> aut-wien-vol-01-replicate-0 >>>>> [2016-01-02 11:44:12.100785] I >>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>>> aut-wien-vol-01-replicate-1 >>>>> [2016-01-02 11:44:12.100800] I >>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>>> aut-wien-vol-01-replicate-2 >>>>> [2016-01-02 11:44:12.100839] I [MSGID: 109036] >>>>> [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal] >>>>> 0-aut-wien-vol-01-dht: Setting layout of >>>>> <gfid:d4815ee4-3348-4105-9136-d0219d956ed8>/.dstXXX0HUpRD with >>>>> [Subvol_name: aut-wien-vol-01-re >>>>> plicate-0, Err: -1 , Start: 0 , Stop: 1430439074 ], [Subvol_name: >>>>> aut-wien-vol-01-replicate-1, Err: -1 , Start: 1430439075 , Stop: >>>>> 2860878149 ], [Subvol_name: aut-wien-vol-01-replicate-2, Err: -1 , >>>>> Start: 2860878150 , Stop: 4294967295 ], >>>>> [2016-01-02 11:44:12.114192] W >>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk] >>>>> 0-aut-wien-vol-01-client-2: remote operation failed: File exists. >>>>> Path: <gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x >>>>> [2016-01-02 11:44:12.114275] W >>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk] >>>>> 0-aut-wien-vol-01-client-3: remote operation failed: File exists. >>>>> Path: <gfid:cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2>/.dstXXb70G3x >>>>> [2016-01-02 11:44:12.114879] W [fuse-bridge.c:1261:fuse_err_cbk] >>>>> 0-glusterfs-fuse: 1959831: SETXATTR() >>>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2 => -1 (File exists) >>>>> [2016-01-02 11:44:12.118473] I >>>>> [dht-layout.c:663:dht_layout_normalize] 0-aut-wien-vol-01-dht: Found >>>>> anomalies in >>>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x (gfid >>>>> 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 >>>>> [2016-01-02 11:44:12.118537] I >>>>> [dht-selfheal.c:1065:dht_selfheal_layout_new_directory] >>>>> 0-aut-wien-vol-01-dht: chunk size = 0xffffffff / 57217563 = 0x4b >>>>> [2016-01-02 11:44:12.118562] I >>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>>> aut-wien-vol-01-replicate-2 >>>>> [2016-01-02 11:44:12.118579] I >>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>>> aut-wien-vol-01-replicate-0 >>>>> [2016-01-02 11:44:12.118613] I >>>>> [dht-selfheal.c:1103:dht_selfheal_layout_new_directory] >>>>> 0-aut-wien-vol-01-dht: assigning range size 0x5542c4a3 to >>>>> aut-wien-vol-01-replicate-1 >>>>> [2016-01-02 11:44:12.120352] I [MSGID: 109036] >>>>> [dht-common.c:6296:dht_log_new_layout_for_dir_selfheal] >>>>> 0-aut-wien-vol-01-dht: Setting layout of >>>>> /.gfid/cd3fd9ba-34b8-4c6b-ba72-4796b80b0ff2/.dstXXb70G3x with >>>>> [Subvol_name: aut-wien-vol-01-rep >>>>> licate-0, Err: -1 , Start: 1430439075 , Stop: 2860878149 ], >>>>> [Subvol_name: aut-wien-vol-01-replicate-1, Err: -1 , Start: >>>>> 2860878150 , Stop: 4294967295 ], [Subvol_name: >>>>> aut-wien-vol-01-replicate-2, Err: -1 , Start: 0 , Stop: 1430439074 ], >>>>> [2016-01-02 11:44:12.630949] I [fuse-bridge.c:4927:fuse_thread_proc] >>>>> 0-fuse: unmounting /tmp/gsyncd-aux-mount-tOUOsz >>>>> [2016-01-02 11:44:12.633952] W [glusterfsd.c:1211:cleanup_and_exit] >>>>> (--> 0-: received signum (15), shutting down >>>>> [2016-01-02 11:44:12.633964] I [fuse-bridge.c:5607:fini] 0-fuse: >>>>> Unmounting '/tmp/gsyncd-aux-mount-tOUOsz'. >>>>> [2016-01-02 11:44:23.946702] I [MSGID: 100030] >>>>> [glusterfsd.c:2035:main] 0-/usr/sbin/glusterfs: Started running >>>>> /usr/sbin/glusterfs version 3.6.7 (args: /usr/sbin/glusterfs >>>>> --aux-gfid-mount --log-file=/var/log/glusterfs/geo-replication-slav >>>>> >>>>> es/6a071cfa-b150-4f0b-b1ed-96ab5d4bd671:gluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log >>>>> --volfile-server=localhost --volfile-id=aut-wien-vol-01 >>>>> --client-pid=-1 /tmp/gsyncd-aux-mount-otU3wS) >>>>> [2016-01-02 11:44:24.042128] I [dht-shared.c:337:dht_init_regex] >>>>> 0-aut-wien-vol-01-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >>>>> [2016-01-02 11:44:24.046315] I [client.c:2268:notify] >>>>> 0-aut-wien-vol-01-client-0: parent translators are ready, attempting >>>>> connect on transport >>>>> [2016-01-02 11:44:24.046532] I [client.c:2268:notify] >>>>> 0-aut-wien-vol-01-client-1: parent translators are ready, attempting >>>>> connect on transport >>>>> [2016-01-02 11:44:24.046664] I [client.c:2268:notify] >>>>> 0-aut-wien-vol-01-client-2: parent translators are ready, attempting >>>>> connect on transport >>>>> [2016-01-02 11:44:24.046806] I [client.c:2268:notify] >>>>> 0-aut-wien-vol-01-client-3: parent translators are ready, attempting >>>>> connect on transport >>>>> [2016-01-02 11:44:24.046940] I [client.c:2268:notify] >>>>> 0-aut-wien-vol-01-client-4: parent translators are ready, attempting >>>>> connect on transport >>>>> [2016-01-02 11:44:24.047070] I [client.c:2268:notify] >>>>> 0-aut-wien-vol-01-client-5: parent translators are ready, attempting >>>>> connect on transport >>>>> Final graph: >>>>> >>>>> +------------------------------------------------------------------------------+ >>>>> >>>>> 1: volume aut-wien-vol-01-client-0 >>>>> 2: type protocol/client >>>>> 3: option ping-timeout 10 >>>>> 4: option remote-host gluster-wien-02-int >>>>> 5: option remote-subvolume /gluster-export >>>>> 6: option transport-type socket >>>>> 7: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>>> 8: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>>> 9: option send-gids true >>>>> 10: end-volume >>>>> 11: >>>>> 12: volume aut-wien-vol-01-client-1 >>>>> 13: type protocol/client >>>>> 14: option ping-timeout 10 >>>>> 15: option remote-host gluster-wien-03-int >>>>> 16: option remote-subvolume /gluster-export >>>>> 17: option transport-type socket >>>>> 18: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>>> 19: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>>> 20: option send-gids true >>>>> 21: end-volume >>>>> 22: >>>>> 23: volume aut-wien-vol-01-replicate-0 >>>>> 24: type cluster/replicate >>>>> 25: subvolumes aut-wien-vol-01-client-0 aut-wien-vol-01-client-1 >>>>> 26: end-volume >>>>> 27: >>>>> 28: volume aut-wien-vol-01-client-2 >>>>> 29: type protocol/client >>>>> 30: option ping-timeout 10 >>>>> 31: option remote-host gluster-wien-04-int >>>>> 32: option remote-subvolume /gluster-export >>>>> 33: option transport-type socket >>>>> 34: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>>> 35: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>>> 36: option send-gids true >>>>> 37: end-volume >>>>> 38: >>>>> 39: volume aut-wien-vol-01-client-3 >>>>> 40: type protocol/client >>>>> 41: option ping-timeout 10 >>>>> 42: option remote-host gluster-wien-05-int >>>>> 43: option remote-subvolume /gluster-export >>>>> 44: option transport-type socket >>>>> 45: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>>> 46: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>>> 47: option send-gids true >>>>> 48: end-volume >>>>> 49: >>>>> 50: volume aut-wien-vol-01-replicate-1 >>>>> 51: type cluster/replicate >>>>> 52: subvolumes aut-wien-vol-01-client-2 aut-wien-vol-01-client-3 >>>>> 53: end-volume >>>>> 54: >>>>> 55: volume aut-wien-vol-01-client-4 >>>>> 56: type protocol/client >>>>> 57: option ping-timeout 10 >>>>> 58: option remote-host gluster-wien-06-int >>>>> 59: option remote-subvolume /gluster-export >>>>> 60: option transport-type socket >>>>> 61: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>>> 62: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>>> 63: option send-gids true >>>>> 64: end-volume >>>>> 65: >>>>> 66: volume aut-wien-vol-01-client-5 >>>>> 67: type protocol/client >>>>> 68: option ping-timeout 10 >>>>> 69: option remote-host gluster-wien-07-int >>>>> 70: option remote-subvolume /gluster-export >>>>> 71: option transport-type socket >>>>> 72: option username 6b3d1fae-fa3e-4305-a4a0-fd27c7ac9929 >>>>> 73: option password 8777e154-476c-449a-89b2-3199872e4a1f >>>>> 74: option send-gids true >>>>> 75: end-volume >>>>> 76: >>>>> 77: volume aut-wien-vol-01-replicate-2 >>>>> 78: type cluster/replicate >>>>> 79: subvolumes aut-wien-vol-01-client-4 aut-wien-vol-01-client-5 >>>>> 80: end-volume >>>>> 81: >>>>> 82: volume aut-wien-vol-01-dht >>>>> 83: type cluster/distribute >>>>> 84: subvolumes aut-wien-vol-01-replicate-0 >>>>> aut-wien-vol-01-replicate-1 aut-wien-vol-01-replicate-2 >>>>> 85: end-volume >>>>> 86: >>>>> 87: volume aut-wien-vol-01-write-behind >>>>> 88: type performance/write-behind >>>>> 89: subvolumes aut-wien-vol-01-dht >>>>> 90: end-volume >>>>> 91: >>>>> 92: volume aut-wien-vol-01-read-ahead >>>>> 93: type performance/read-ahead >>>>> 94: subvolumes aut-wien-vol-01-write-behind >>>>> 95: end-volume >>>>> 96: >>>>> 97: volume aut-wien-vol-01-io-cache >>>>> 98: type performance/io-cache >>>>> 99: option min-file-size 0 >>>>> 100: option cache-timeout 2 >>>>> 101: option cache-size 1024MB >>>>> 102: subvolumes aut-wien-vol-01-read-ahead >>>>> 103: end-volume >>>>> 104: >>>>> 105: volume aut-wien-vol-01-quick-read >>>>> 106: type performance/quick-read >>>>> 107: option cache-size 1024MB >>>>> 108: subvolumes aut-wien-vol-01-io-cache >>>>> 109: end-volume >>>>> 110: >>>>> 111: volume aut-wien-vol-01-open-behind >>>>> 112: type performance/open-behind >>>>> 113: subvolumes aut-wien-vol-01-quick-read >>>>> 114: end-volume >>>>> 115: >>>>> 116: volume aut-wien-vol-01-md-cache >>>>> 117: type performance/md-cache >>>>> 118: subvolumes aut-wien-vol-01-open-behind >>>>> 119: end-volume >>>>> 120: >>>>> 121: volume aut-wien-vol-01 >>>>> 122: type debug/io-stats >>>>> 123: option latency-measurement off >>>>> 124: option count-fop-hits off >>>>> 125: subvolumes aut-wien-vol-01-md-cache >>>>> 126: end-volume >>>>> 127: >>>>> 128: volume gfid-access-autoload >>>>> 129: type features/gfid-access >>>>> 130: subvolumes aut-wien-vol-01 >>>>> 131: end-volume >>>>> 132: >>>>> 133: volume meta-autoload >>>>> 134: type meta >>>>> 135: subvolumes gfid-access-autoload >>>>> 136: end-volume >>>>> 137: >>>>> >>>>> +------------------------------------------------------------------------------+ >>>>> >>>>> [2016-01-02 11:44:24.047642] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>>> 0-aut-wien-vol-01-client-5: changing port to 49153 (from 0) >>>>> [2016-01-02 11:44:24.047927] I >>>>> [client-handshake.c:1413:select_server_supported_programs] >>>>> 0-aut-wien-vol-01-client-5: Using Program GlusterFS 3.3, Num >>>>> (1298437), Version (330) >>>>> [2016-01-02 11:44:24.048044] I >>>>> [client-handshake.c:1200:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-5: Connected to aut-wien-vol-01-client-5, >>>>> attached to remote volume '/gluster-export'. >>>>> [2016-01-02 11:44:24.048050] I >>>>> [client-handshake.c:1210:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-5: Server and Client lk-version numbers are >>>>> not same, reopening the fds >>>>> [2016-01-02 11:44:24.048088] I [MSGID: 108005] >>>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-2: >>>>> Subvolume 'aut-wien-vol-01-client-5' came back up; going online. >>>>> [2016-01-02 11:44:24.048114] I >>>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>>> 0-aut-wien-vol-01-client-5: Server lk version = 1 >>>>> [2016-01-02 11:44:24.048124] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>>> 0-aut-wien-vol-01-client-0: changing port to 49153 (from 0) >>>>> [2016-01-02 11:44:24.048132] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>>> 0-aut-wien-vol-01-client-1: changing port to 49153 (from 0) >>>>> [2016-01-02 11:44:24.048138] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>>> 0-aut-wien-vol-01-client-2: changing port to 49153 (from 0) >>>>> [2016-01-02 11:44:24.048146] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>>> 0-aut-wien-vol-01-client-3: changing port to 49153 (from 0) >>>>> [2016-01-02 11:44:24.048153] I [rpc-clnt.c:1761:rpc_clnt_reconfig] >>>>> 0-aut-wien-vol-01-client-4: changing port to 49153 (from 0) >>>>> [2016-01-02 11:44:24.049070] I >>>>> [client-handshake.c:1413:select_server_supported_programs] >>>>> 0-aut-wien-vol-01-client-0: Using Program GlusterFS 3.3, Num >>>>> (1298437), Version (330) >>>>> [2016-01-02 11:44:24.049094] I >>>>> [client-handshake.c:1413:select_server_supported_programs] >>>>> 0-aut-wien-vol-01-client-3: Using Program GlusterFS 3.3, Num >>>>> (1298437), Version (330) >>>>> [2016-01-02 11:44:24.049113] I >>>>> [client-handshake.c:1413:select_server_supported_programs] >>>>> 0-aut-wien-vol-01-client-2: Using Program GlusterFS 3.3, Num >>>>> (1298437), Version (330) >>>>> [2016-01-02 11:44:24.049131] I >>>>> [client-handshake.c:1413:select_server_supported_programs] >>>>> 0-aut-wien-vol-01-client-1: Using Program GlusterFS 3.3, Num >>>>> (1298437), Version (330) >>>>> [2016-01-02 11:44:24.049224] I >>>>> [client-handshake.c:1413:select_server_supported_programs] >>>>> 0-aut-wien-vol-01-client-4: Using Program GlusterFS 3.3, Num >>>>> (1298437), Version (330) >>>>> [2016-01-02 11:44:24.049307] I >>>>> [client-handshake.c:1200:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-0: Connected to aut-wien-vol-01-client-0, >>>>> attached to remote volume '/gluster-export'. >>>>> [2016-01-02 11:44:24.049312] I >>>>> [client-handshake.c:1210:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-0: Server and Client lk-version numbers are >>>>> not same, reopening the fds >>>>> [2016-01-02 11:44:24.049324] I [MSGID: 108005] >>>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-0: >>>>> Subvolume 'aut-wien-vol-01-client-0' came back up; going online. >>>>> [2016-01-02 11:44:24.049384] I >>>>> [client-handshake.c:1200:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-3: Connected to aut-wien-vol-01-client-3, >>>>> attached to remote volume '/gluster-export'. >>>>> [2016-01-02 11:44:24.049389] I >>>>> [client-handshake.c:1210:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-3: Server and Client lk-version numbers are >>>>> not same, reopening the fds >>>>> [2016-01-02 11:44:24.049400] I [MSGID: 108005] >>>>> [afr-common.c:3684:afr_notify] 0-aut-wien-vol-01-replicate-1: >>>>> Subvolume 'aut-wien-vol-01-client-3' came back up; going online. >>>>> [2016-01-02 11:44:24.049418] I >>>>> [client-handshake.c:1200:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-2: Connected to aut-wien-vol-01-client-2, >>>>> attached to remote volume '/gluster-export'. >>>>> [2016-01-02 11:44:24.049422] I >>>>> [client-handshake.c:1210:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-2: Server and Client lk-version numbers are >>>>> not same, reopening the fds >>>>> [2016-01-02 11:44:24.049460] I >>>>> [client-handshake.c:1200:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-1: Connected to aut-wien-vol-01-client-1, >>>>> attached to remote volume '/gluster-export'. >>>>> [2016-01-02 11:44:24.049465] I >>>>> [client-handshake.c:1210:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-1: Server and Client lk-version numbers are >>>>> not same, reopening the fds >>>>> [2016-01-02 11:44:24.049493] I >>>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>>> 0-aut-wien-vol-01-client-0: Server lk version = 1 >>>>> [2016-01-02 11:44:24.049567] I >>>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>>> 0-aut-wien-vol-01-client-3: Server lk version = 1 >>>>> [2016-01-02 11:44:24.049632] I >>>>> [client-handshake.c:1200:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-4: Connected to aut-wien-vol-01-client-4, >>>>> attached to remote volume '/gluster-export'. >>>>> [2016-01-02 11:44:24.049638] I >>>>> [client-handshake.c:1210:client_setvolume_cbk] >>>>> 0-aut-wien-vol-01-client-4: Server and Client lk-version numbers are >>>>> not same, reopening the fds >>>>> [2016-01-02 11:44:24.052103] I [fuse-bridge.c:5086:fuse_graph_setup] >>>>> 0-fuse: switched to graph 0 >>>>> [2016-01-02 11:44:24.052150] I >>>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>>> 0-aut-wien-vol-01-client-2: Server lk version = 1 >>>>> [2016-01-02 11:44:24.052163] I >>>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>>> 0-aut-wien-vol-01-client-4: Server lk version = 1 >>>>> [2016-01-02 11:44:24.052192] I >>>>> [client-handshake.c:188:client_set_lk_version_cbk] >>>>> 0-aut-wien-vol-01-client-1: Server lk version = 1 >>>>> [2016-01-02 11:44:24.052204] I [fuse-bridge.c:4015:fuse_init] >>>>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 >>>>> kernel 7.20 >>>>> [2016-01-02 11:44:24.053991] I >>>>> [afr-common.c:1491:afr_local_discovery_cbk] >>>>> 0-aut-wien-vol-01-replicate-2: selecting local read_child >>>>> aut-wien-vol-01-client-5 >>>>> [2016-01-02 11:45:48.613563] W >>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk] >>>>> 0-aut-wien-vol-01-client-5: remote operation failed: File exists. >>>>> Path: /keys >>>>> [2016-01-02 11:45:48.614131] W >>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk] >>>>> 0-aut-wien-vol-01-client-4: remote operation failed: File exists. >>>>> Path: /keys >>>>> [2016-01-02 11:45:48.614436] W [fuse-bridge.c:1261:fuse_err_cbk] >>>>> 0-glusterfs-fuse: 12: SETXATTR() >>>>> /.gfid/00000000-0000-0000-0000-000000000001 => -1 (File exists) >>>>> ... >>>>> >>>>> >>>>> [ 13:41:40 ] - root at gluster-ger-ber-07 >>>>> /var/log/glusterfs/geo-replication/ger-ber-01 $gluster volume >>>>> geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 status >>>>> detail >>>>> >>>>> MASTER NODE MASTER VOL MASTER BRICK >>>>> SLAVE STATUS CHECKPOINT >>>>> STATUS CRAWL STATUS FILES SYNCD FILES PENDING BYTES >>>>> PENDING DELETES PENDING FILES SKIPPED >>>>> >>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>> >>>>> gluster-ger-ber-07 ger-ber-01 /gluster-export >>>>> gluster-wien-07-int::aut-wien-vol-01 Active N/A >>>>> Hybrid Crawl 10743644 8192 0 0 0 >>>>> gluster-ger-ber-11 ger-ber-01 /gluster-export >>>>> gluster-wien-03-int::aut-wien-vol-01 Active N/A >>>>> Hybrid Crawl 16037091 8192 0 0 0 >>>>> gluster-ger-ber-10 ger-ber-01 /gluster-export >>>>> gluster-wien-02-int::aut-wien-vol-01 Passive N/A >>>>> N/A 0 0 0 0 0 >>>>> gluster-ger-ber-12 ger-ber-01 /gluster-export >>>>> gluster-wien-06-int::aut-wien-vol-01 Passive N/A >>>>> N/A 0 0 0 0 0 >>>>> gluster-ger-ber-09 ger-ber-01 /gluster-export >>>>> gluster-wien-05-int::aut-wien-vol-01 Active N/A >>>>> Hybrid Crawl 16180514 8192 0 0 0 >>>>> gluster-ger-ber-08 ger-ber-01 /gluster-export >>>>> gluster-wien-04-int::aut-wien-vol-01 Passive N/A >>>>> N/A 0 0 0 0 0 >>>>> >>>>> >>>>> [ 13:41:55 ] - root at gluster-ger-ber-07 >>>>> /var/log/glusterfs/geo-replication/ger-ber-01 $gluster volume >>>>> geo-replication ger-ber-01 gluster-wien-07::aut-wien-vol-01 config >>>>> special_sync_mode: partial >>>>> state_socket_unencoded: >>>>> >>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.socket >>>>> gluster_log_file: >>>>> >>>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.gluster.log >>>>> ssh_command: ssh -p 2503 -oPasswordAuthentication=no >>>>> -oStrictHostKeyChecking=no -i >>>>> /var/lib/glusterd/geo-replication/secret.pem >>>>> ignore_deletes: true >>>>> change_detector: changelog >>>>> ssh_command_tar: ssh -p 2503 -oPasswordAuthentication=no >>>>> -oStrictHostKeyChecking=no -i >>>>> /var/lib/glusterd/geo-replication/tar_ssh.pem >>>>> state_file: >>>>> >>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.status >>>>> remote_gsyncd: /nonexistent/gsyncd >>>>> log_file: >>>>> >>>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.log >>>>> changelog_log_file: >>>>> >>>>> /var/log/glusterfs/geo-replication/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-changes.log >>>>> socketdir: /var/run >>>>> working_dir: >>>>> >>>>> /var/lib/misc/glusterfsd/ger-ber-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01 >>>>> state_detail_file: >>>>> >>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01-detail.status >>>>> session_owner: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 >>>>> gluster_command_dir: /usr/sbin/ >>>>> pid_file: >>>>> >>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ssh%3A%2F%2Froot%4082.199.131.132%3Agluster%3A%2F%2F127.0.0.1%3Aaut-wien-vol-01.pid >>>>> georep_session_working_dir: >>>>> >>>>> /var/lib/glusterd/geo-replication/ger-ber-01_gluster-wien-07_aut-wien-vol-01/ >>>>> gluster_params: aux-gfid-mount >>>>> volume_id: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671 >>>>> [ 13:42:11 ] - root at gluster-ger-ber-07 >>>>> /var/log/glusterfs/geo-replication/ger-ber-01 $ >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160120/320bc02d/attachment.html>