Kris Laib
2016-Feb-03 01:46 UTC
[Gluster-users] geo-replication appears to be working, but "failures" are logged for each file transferred.
I recently setup geo-replication between two sites, and everything appears to be working, however, if I do a "gluster volume geo-replication SourceVol DestVol status detail", I see that the "failures" column seems to be incrementing for each file it transfers. I've looked at the logs in /var/log/glusterfs/geo-replication/SourceVol/* , but nothing stood out to me. I've verified that the MD5 sum matches on the files at the destination after it's been replicated, so I'm unsure what is triggering the failure. Aside from the incrementing failure column, it all looks good to me. All boxes in both clusters are CentOS 6.7, running glusterfs 3.7.5 My questions: 1) What's causing the failures to be logged (please let me know what additional logs to provide)? 2) How do I reset the failures counter? It stays at the current value even once the files are deleted from the source 3) Is there documentation somewhere I missed that explains what Entry, Data, Meta, and Failures are in the status detail view for geo-replication? 4) Should "gluster volume geo-replication status" show me anything on the destination? Mine simply outputs "No active geo-replication sessions", though the command on the source works fine. Thanks, Kris ----------- Here's an example of what I'm seeing: [root at gfs-prd02.mgmt temp]$ mount | grep "gfs-prd.mgmt:/prd_items" gfs-prd.mgmt:/prd_items on /mnt/temp type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) [root at gfs-prd02.mgmt temp]$ pwd /mnt/temp [root at gfs-prd02.mgmt temp]$ gluster volume geo-replication prd_items georep at gfs-slc01.mgmt::dr_items status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- gfs-prd02.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep at gfs-slc01.mgmt::dr_items gfs-slc03.mgmt Active Changelog Crawl 2016-02-03 01:28:26 0 0 0 2 N/A N/A N/A gfs-prd03.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep at gfs-slc01.mgmt::dr_items gfs-slc02.mgmt Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A [root at gfs-prd02.mgmt temp]$ dd if=/dev/urandom of=test01 bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.179957 s, 5.8 MB/s [root at gfs-prd02.mgmt temp]$ dd if=/dev/urandom of=test02 bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.181704 s, 5.8 MB/s [root at gfs-prd02.mgmt temp]$ dd if=/dev/urandom of=test03 bs=1M count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.181575 s, 5.8 MB/s [root at gfs-prd02.mgmt temp]$ gluster volume geo-replication prd_items georep at gfs-slc01.mgmt::dr_items status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- gfs-prd02.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep at gfs-slc01.mgmt::dr_items gfs-slc03.mgmt Active Changelog Crawl 2016-02-03 01:29:11 0 0 0 5 N/A N/A N/A gfs-prd03.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep at gfs-slc01.mgmt::dr_items gfs-slc02.mgmt Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.log ============= [2016-02-03 01:26:14.702147] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns [2016-02-03 01:26:14.718751] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462711, 0) [2016-02-03 01:27:15.371404] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns [2016-02-03 01:27:15.379936] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462771, 0) [2016-02-03 01:27:58.15177] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462831, 0) [2016-02-03 01:27:59.54204] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:00.101433] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:01.104391] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:02.312911] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:03.356967] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:04.364518] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:05.565407] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:06.610732] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:07.620142] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 [2016-02-03 01:28:08.818785] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462877 could not be processed - moving on... [2016-02-03 01:28:08.826875] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = f5a44b23-cdd8-4278-92d1-1b2a16e8c9d5 [2016-02-03 01:28:15.94074] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462876, 0) [2016-02-03 01:28:18.669955] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 17 crawls, 3 turns [2016-02-03 01:28:27.720731] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462891, 0) [2016-02-03 01:28:58.387859] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462906, 0) [2016-02-03 01:28:59.360420] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:00.430566] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:01.610398] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:02.667408] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:03.692369] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:04.947613] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:05.932200] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:06.947702] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:08.186101] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 [2016-02-03 01:29:09.203042] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462937 could not be processed - moving on... [2016-02-03 01:29:09.205854] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = c99a05c9-9e7d-41cf-b148-9bec91f527c8 [2016-02-03 01:29:15.432744] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462936, 0) [2016-02-03 01:29:16.743381] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:17.944102] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:18.970493] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:20.77699] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:21.197900] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:22.259825] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:23.349780] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:24.469357] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:25.526793] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 [2016-02-03 01:29:26.612977] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462952 could not be processed - moving on... [2016-02-03 01:29:26.615757] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = a7dc6c28-9239-44af-909f-a2938bb387b6,550ffdd3-4636-45b8-b278-fbfe120b9d85 [2016-02-03 01:29:29.828999] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 16 crawls, 3 turns [2016-02-03 01:30:15.98556] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462951, 0) [2016-02-03 01:30:30.709733] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns [2016-02-03 01:31:12.918898] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454463011, 0) [2016-02-03 01:31:31.423204] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1-changes.log ==============[2016-02-03 01:26:12.472345] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:26:12.472396] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1721, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:26:12.472415] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:26:12.472426] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:26:12.472440] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2478 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462772.472206), (vec: 1, len: 4104) [2016-02-03 01:26:12.472467] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:26:12.472481] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:26:12.472502] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6b9, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:26:14.952267] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462772 to processed directory [2016-02-03 01:27:12.531138] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:27:12.531177] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1722, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:27:12.531195] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:27:12.531207] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:27:12.531220] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2479 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462832.531035), (vec: 1, len: 4104) [2016-02-03 01:27:12.531241] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:27:12.531256] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:27:12.531274] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6ba, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:27:15.660249] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462832 to processed directory [2016-02-03 01:27:57.576147] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:27:57.576198] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1723, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:27:57.576217] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:27:57.576230] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:27:57.576246] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2480 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462877.576035), (vec: 1, len: 4104) [2016-02-03 01:27:57.576282] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:27:57.576297] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:27:57.576327] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bb, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:28:08.833040] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462877 to processed directory [2016-02-03 01:28:12.589377] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:28:12.589420] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1724, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:28:12.589439] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:28:12.589451] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:28:12.589466] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2481 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462892.589270), (vec: 1, len: 4104) [2016-02-03 01:28:12.589487] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:28:12.589500] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:28:12.589521] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bc, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:28:15.398144] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462892 to processed directory [2016-02-03 01:28:27.604453] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:28:27.604510] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1725, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:28:27.604530] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:28:27.604542] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:28:27.604557] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2482 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462907.604293), (vec: 1, len: 4104) [2016-02-03 01:28:27.604580] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:28:27.604594] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:28:27.604615] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bd, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:28:28.066455] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462907 to processed directory [2016-02-03 01:28:57.634864] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:28:57.634938] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1726, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:28:57.635006] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:28:57.635028] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:28:57.635049] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2483 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462937.634652), (vec: 1, len: 4104) [2016-02-03 01:28:57.635079] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:28:57.635098] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:28:57.635127] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6be, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:29:09.211888] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462937 to processed directory [2016-02-03 01:29:12.650651] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:29:12.650705] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1727, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:29:12.650726] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:29:12.650739] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:29:12.650755] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2484 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462952.650534), (vec: 1, len: 4104) [2016-02-03 01:29:12.650787] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:29:12.650802] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:29:12.650824] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bf, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:29:26.622432] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462952 to processed directory [2016-02-03 01:30:12.693458] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:30:12.693499] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1728, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:30:12.693528] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:30:12.693544] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:30:12.693562] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2485 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463012.693353), (vec: 1, len: 4104) [2016-02-03 01:30:12.693587] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:30:12.693610] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:30:12.693638] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c0, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:30:15.432599] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463012 to processed directory [2016-02-03 01:31:12.754211] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:31:12.754243] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1729, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:31:12.754260] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:31:12.754273] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:31:12.754294] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2486 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463072.754125), (vec: 1, len: 4104) [2016-02-03 01:31:12.754323] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:31:12.754338] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:31:12.754357] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c1, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:31:13.156812] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463072 to processed directory [2016-02-03 01:32:12.816173] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:32:12.816219] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1730, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:32:12.816236] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:32:12.816248] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:32:12.816261] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2487 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463132.816061), (vec: 1, len: 4104) [2016-02-03 01:32:12.816298] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:32:12.816319] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:32:12.816361] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c2, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:32:13.864741] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463132 to processed directory [2016-02-03 01:33:12.877209] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2016-02-03 01:33:12.877247] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1731, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) [2016-02-03 01:33:12.877276] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2016-02-03 01:33:12.877308] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for [2016-02-03 01:33:12.877332] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2488 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463192.877131), (vec: 1, len: 4104) [2016-02-03 01:33:12.877360] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 [2016-02-03 01:33:12.877380] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 [2016-02-03 01:33:12.877408] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c3, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) [2016-02-03 01:33:14.577963] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463192 to processed directory Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1.gluster.log ============[2016-02-01 20:45:23.591390] I [MSGID: 108031] [afr-common.c:1783:afr_local_discovery_cbk] 0-prd_items-replicate-0: selecting local read_child prd_items-client-0 [2016-02-03 01:27:58.464278] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage [2016-02-03 01:29:03.558047] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage [2016-02-03 01:29:25.386117] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160203/14ee6bfc/attachment.html>
Aravinda
2016-Feb-03 05:47 UTC
[Gluster-users] geo-replication appears to be working, but "failures" are logged for each file transferred.
Hi Kris, These failures are mostly while transferring data from Master to Slave. Geo-replication is not intelligent to find exact number and reason for failures. I sent patch to effectively handle the rsync/tar failures. This patch is expected to be available with 3.7.8 release. http://review.gluster.org/#/c/12856/ Above patch identifies the reason for failure and logs in /var/log/glusterfs/geo-replication/<MASTERVOL>/*.log. Answers inline. regards Aravinda On 02/03/2016 07:16 AM, Kris Laib wrote:> I recently setup geo-replication between two sites, and everything appears to be working, however, if I do a "gluster volume geo-replication SourceVol DestVol status detail", I see that the "failures" column seems to be incrementing for each file it transfers. I've looked at the logs in /var/log/glusterfs/geo-replication/SourceVol/* , but nothing stood out to me. > > I've verified that the MD5 sum matches on the files at the destination after it's been replicated, so I'm unsure what is triggering the failure. Aside from the incrementing failure column, it all looks good to me.Looks like these issues can be ignored. Errors may be due to xattrs syncing. With the above patch, logging is improved to log the failure reasons.> > All boxes in both clusters are CentOS 6.7, running glusterfs 3.7.5 > > > My questions: > > 1) What's causing the failures to be logged (please let me know what additional logs to provide)?Possible reasons for Sync failures, - May be failure to sync xattrs/acls to Slave. - I/O error on Slave - Files unlinked in Slave - GFID Conflict> 2) How do I reset the failures counter? It stays at the current value even once the files are deleted from the sourceWill fix this issue in upcoming releases.> 3) Is there documentation somewhere I missed that explains what Entry, Data, Meta, and Failures are in the status detail view for geo-replication?Entry - Pending Entry operations - Entry operations like CREATE, RENAME, UNLINK etc, which are executed as RPC calls. Data - Pending Data operations - Rsync/Tar is used to transfer data from Master to Slave. Meta - Pending Meta operations - Changing ownership, permissions are executed as RPC calls. Failures - Going to be obsolete since Errors will be logged in log files.> 4) Should "gluster volume geo-replication status" show me anything on the destination? Mine simply outputs "No active geo-replication sessions", though the command on the source works fine.Geo-replication status only available from Master.> > Thanks, > Kris > > ----------- > > Here's an example of what I'm seeing: > > > [root at gfs-prd02.mgmt temp]$ mount | grep "gfs-prd.mgmt:/prd_items" > gfs-prd.mgmt:/prd_items on /mnt/temp type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) > [root at gfs-prd02.mgmt temp]$ pwd > /mnt/temp > [root at gfs-prd02.mgmt temp]$ gluster volume geo-replication prd_items georep at gfs-slc01.mgmt::dr_items status detail > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > gfs-prd02.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep at gfs-slc01.mgmt::dr_items gfs-slc03.mgmt Active Changelog Crawl 2016-02-03 01:28:26 0 0 0 2 N/A N/A N/A > gfs-prd03.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep at gfs-slc01.mgmt::dr_items gfs-slc02.mgmt Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A > [root at gfs-prd02.mgmt temp]$ dd if=/dev/urandom of=test01 bs=1M count=1 > 1+0 records in > 1+0 records out > 1048576 bytes (1.0 MB) copied, 0.179957 s, 5.8 MB/s > [root at gfs-prd02.mgmt temp]$ dd if=/dev/urandom of=test02 bs=1M count=1 > 1+0 records in > 1+0 records out > 1048576 bytes (1.0 MB) copied, 0.181704 s, 5.8 MB/s > [root at gfs-prd02.mgmt temp]$ dd if=/dev/urandom of=test03 bs=1M count=1 > 1+0 records in > 1+0 records out > 1048576 bytes (1.0 MB) copied, 0.181575 s, 5.8 MB/s > [root at gfs-prd02.mgmt temp]$ gluster volume geo-replication prd_items georep at gfs-slc01.mgmt::dr_items status detail > MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > gfs-prd02.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep at gfs-slc01.mgmt::dr_items gfs-slc03.mgmt Active Changelog Crawl 2016-02-03 01:29:11 0 0 0 5 N/A N/A N/A > gfs-prd03.mgmt prd_items /data/glusterfs/prd_items_brick1/brick1 georep georep at gfs-slc01.mgmt::dr_items gfs-slc02.mgmt Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A > > > > > Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.log > =============> > [2016-02-03 01:26:14.702147] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns > [2016-02-03 01:26:14.718751] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462711, 0) > [2016-02-03 01:27:15.371404] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns > [2016-02-03 01:27:15.379936] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462771, 0) > [2016-02-03 01:27:58.15177] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462831, 0) > [2016-02-03 01:27:59.54204] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:00.101433] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:01.104391] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:02.312911] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:03.356967] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:04.364518] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:05.565407] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:06.610732] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:07.620142] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462877 > [2016-02-03 01:28:08.818785] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462877 could not be processed - moving on... > [2016-02-03 01:28:08.826875] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = f5a44b23-cdd8-4278-92d1-1b2a16e8c9d5 > [2016-02-03 01:28:15.94074] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462876, 0) > [2016-02-03 01:28:18.669955] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 17 crawls, 3 turns > [2016-02-03 01:28:27.720731] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462891, 0) > [2016-02-03 01:28:58.387859] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462906, 0) > [2016-02-03 01:28:59.360420] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:00.430566] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:01.610398] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:02.667408] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:03.692369] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:04.947613] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:05.932200] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:06.947702] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:08.186101] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462937 > [2016-02-03 01:29:09.203042] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462937 could not be processed - moving on... > [2016-02-03 01:29:09.205854] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = c99a05c9-9e7d-41cf-b148-9bec91f527c8 > [2016-02-03 01:29:15.432744] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462936, 0) > [2016-02-03 01:29:16.743381] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:17.944102] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:18.970493] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:20.77699] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:21.197900] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:22.259825] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:23.349780] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:24.469357] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:25.526793] W [master(/data/glusterfs/prd_items_brick1/brick1):1041:process] _GMaster: incomplete sync, retrying changelogs: CHANGELOG.1454462952 > [2016-02-03 01:29:26.612977] W [master(/data/glusterfs/prd_items_brick1/brick1):1010:process] _GMaster: changelogs CHANGELOG.1454462952 could not be processed - moving on... > [2016-02-03 01:29:26.615757] W [master(/data/glusterfs/prd_items_brick1/brick1):1014:process] _GMaster: SKIPPED GFID = a7dc6c28-9239-44af-909f-a2938bb387b6,550ffdd3-4636-45b8-b278-fbfe120b9d85 > [2016-02-03 01:29:29.828999] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 16 crawls, 3 turns > [2016-02-03 01:30:15.98556] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454462951, 0) > [2016-02-03 01:30:30.709733] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns > [2016-02-03 01:31:12.918898] I [master(/data/glusterfs/prd_items_brick1/brick1):1095:crawl] _GMaster: slave's time: (1454463011, 0) > [2016-02-03 01:31:31.423204] I [master(/data/glusterfs/prd_items_brick1/brick1):552:crawlwrap] _GMaster: 20 crawls, 1 turns > > > Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1-changes.log > ==============> [2016-02-03 01:26:12.472345] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:26:12.472396] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1721, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:26:12.472415] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:26:12.472426] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:26:12.472440] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2478 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462772.472206), (vec: 1, len: 4104) > [2016-02-03 01:26:12.472467] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:26:12.472481] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:26:12.472502] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6b9, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:26:14.952267] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462772 to processed directory > [2016-02-03 01:27:12.531138] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:27:12.531177] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1722, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:27:12.531195] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:27:12.531207] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:27:12.531220] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2479 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462832.531035), (vec: 1, len: 4104) > [2016-02-03 01:27:12.531241] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:27:12.531256] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:27:12.531274] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6ba, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:27:15.660249] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462832 to processed directory > [2016-02-03 01:27:57.576147] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:27:57.576198] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1723, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:27:57.576217] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:27:57.576230] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:27:57.576246] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2480 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462877.576035), (vec: 1, len: 4104) > [2016-02-03 01:27:57.576282] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:27:57.576297] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:27:57.576327] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bb, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:28:08.833040] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462877 to processed directory > [2016-02-03 01:28:12.589377] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:28:12.589420] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1724, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:28:12.589439] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:28:12.589451] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:28:12.589466] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2481 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462892.589270), (vec: 1, len: 4104) > [2016-02-03 01:28:12.589487] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:28:12.589500] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:28:12.589521] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bc, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:28:15.398144] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462892 to processed directory > [2016-02-03 01:28:27.604453] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:28:27.604510] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1725, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:28:27.604530] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:28:27.604542] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:28:27.604557] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2482 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462907.604293), (vec: 1, len: 4104) > [2016-02-03 01:28:27.604580] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:28:27.604594] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:28:27.604615] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bd, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:28:28.066455] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462907 to processed directory > [2016-02-03 01:28:57.634864] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:28:57.634938] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1726, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:28:57.635006] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:28:57.635028] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:28:57.635049] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2483 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462937.634652), (vec: 1, len: 4104) > [2016-02-03 01:28:57.635079] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:28:57.635098] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:28:57.635127] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6be, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:29:09.211888] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462937 to processed directory > [2016-02-03 01:29:12.650651] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:29:12.650705] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1727, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:29:12.650726] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:29:12.650739] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:29:12.650755] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2484 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454462952.650534), (vec: 1, len: 4104) > [2016-02-03 01:29:12.650787] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:29:12.650802] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:29:12.650824] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6bf, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:29:26.622432] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454462952 to processed directory > [2016-02-03 01:30:12.693458] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:30:12.693499] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1728, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:30:12.693528] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:30:12.693544] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:30:12.693562] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2485 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463012.693353), (vec: 1, len: 4104) > [2016-02-03 01:30:12.693587] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:30:12.693610] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:30:12.693638] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c0, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:30:15.432599] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463012 to processed directory > [2016-02-03 01:31:12.754211] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:31:12.754243] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1729, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:31:12.754260] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:31:12.754273] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:31:12.754294] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2486 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463072.754125), (vec: 1, len: 4104) > [2016-02-03 01:31:12.754323] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:31:12.754338] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:31:12.754357] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c1, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:31:13.156812] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463072 to processed directory > [2016-02-03 01:32:12.816173] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:32:12.816219] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1730, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:32:12.816236] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:32:12.816248] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:32:12.816261] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2487 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463132.816061), (vec: 1, len: 4104) > [2016-02-03 01:32:12.816298] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:32:12.816319] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:32:12.816361] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c2, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:32:13.864741] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463132 to processed directory > [2016-02-03 01:33:12.877209] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2016-02-03 01:33:12.877247] T [rpcsvc.c:495:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1731, Ver: 2, Program: 1886350951, ProgVers: 1, Proc: 1) from rpc-transport (socket.gfchangelog) > [2016-02-03 01:33:12.877276] T [auth-glusterfs.c:242:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2016-02-03 01:33:12.877308] T [rpcsvc.c:316:rpcsvc_program_actor] 0-rpc-service: Actor found: LIBGFCHANGELOG REBORP - CHANGELOG EVENT HANDLER for > [2016-02-03 01:33:12.877332] D [gf-changelog-reborp.c:406:gf_changelog_event_handler] 0-gfchangelog: seq: 2488 [/data/glusterfs/prd_items_brick1/brick1] (time: 1454463192.877131), (vec: 1, len: 4104) > [2016-02-03 01:33:12.877360] T [rpcsvc.c:1273:rpcsvc_submit_generic] 0-rpc-service: Tx message: 8 > [2016-02-03 01:33:12.877380] T [rpcsvc.c:863:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 32, payload: 8, rpc hdr: 24 > [2016-02-03 01:33:12.877408] T [rpcsvc.c:1321:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x6c3, Program: LIBGFCHANGELOG REBORP, ProgVers: 1, Proc: 1) to rpc-transport (socket.gfchangelog) > [2016-02-03 01:33:14.577963] D [gf-changelog-api.c:53:gf_changelog_done] 0-gfchangelog: moving /var/lib/misc/glusterfsd/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items/ffdfc13dfaa3d74827a7b7cb855c6ce0/.processing/CHANGELOG.1454463192 to processed directory > > > Snippet from /var/log/glusterfs/geo-replication/prd_items/ssh%3A%2F%2Fgeorep%4010.124.0.82%3Agluster%3A%2F%2F127.0.0.1%3Adr_items.%2Fdata%2Fglusterfs%2Fprd_items_brick1%2Fbrick1.gluster.log > ============> [2016-02-01 20:45:23.591390] I [MSGID: 108031] [afr-common.c:1783:afr_local_discovery_cbk] 0-prd_items-replicate-0: selecting local read_child prd_items-client-0 > [2016-02-03 01:27:58.464278] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage > [2016-02-03 01:29:03.558047] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage > [2016-02-03 01:29:25.386117] E [fuse-bridge.c:3336:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160203/0f26bfa9/attachment.html>