Strahil Nikolov
2020-Oct-19 04:01 UTC
[Gluster-users] Gluster7 GeoReplication Operation not permitted and incomplete sync
>[2020-10-16 20:30:25.039659] E [MSGID: 109009] [dht-helper.c:1384:dht_migration_complete_check_task] 0-pcic-backup-dht: 24bf0575-6ab0-4613-b42a-3b63b3c00165: gfid different on the target file on pcic-backup-readdir-ahead-0 >[2020-10-16 20:30:25.039695] E [MSGID: 148002] [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-pcic-backup-utime: dict set of key for set-ctime-mdata failed [Input/output error]I would start by finding that gfid on the source and then identify the gfid of the file in the geo-rep volume. Also, it seems that you got some acl/extended attributes issues on the geo-rep destination - so take a look. Best Regards, Strahil Nikolov
Matthew Benstead
2020-Oct-19 16:56 UTC
[Gluster-users] Gluster7 GeoReplication Operation not permitted and incomplete sync
Thanks - I found the file path from the GFID - but I don't see any weird xattr's: [root at storage01 ~]# mkdir /mnt/storage2-gfid [root at storage01 ~]# mount -t glusterfs -o aux-gfid-mount 10.0.231.91:/storage /mnt/storage2-gfid [root at storage01 ~]# getfattr -n trusted.glusterfs.pathinfo -e text /mnt/storage2-gfid/.gfid/24bf0575-6ab0-4613-b42a-3b63b3c00165 getfattr: Removing leading '/' from absolute path names # file: mnt/storage2-gfid/.gfid/24bf0575-6ab0-4613-b42a-3b63b3c00165 trusted.glusterfs.pathinfo="(<DISTRIBUTE:storage-dht> (<REPLICATE:storage-replicate-1> <POSIX(/data/storage_b/storage):storage03.pcic.uvic.ca:/data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h> <POSIX(/data/storage_c/storage):storage01.pcic.uvic.ca:/data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h> <POSIX(/data/storage_a/storage):storage02.pcic.uvic.ca:/data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h>))" [root at storage03 ~]# ls -lh /data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h -rw-rw-r--. 4 USER ftp 108K May 25? 2018 /data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h [root at storage03 ~]# getfattr -d -m. -e hex /data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h getfattr: Removing leading '/' from absolute path names # file: data/storage_b/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x24bf05756ab04613b42a3b63b3c00165 trusted.gfid2path.12529fafe3b56cc3=0x33346136613135632d643031662d343230302d626164652d3738303136633634323638352f636169726f2e68 trusted.gfid2path.e445cee32fde338d=0x62376364343961322d303531392d346333342d626436342d3936616334346432353835302f636169726f2e68 trusted.gfid2path.fbe2d5aaa0195e94=0x62626138356435362d623330302d343566322d393863312d6233303233626635333762362f636169726f2e68 trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x5f8a063a0001f5c1 trusted.glusterfs.mdata=0x010000000000000000000000005ee25c8f000000002d2a4984000000005b08129b0000000000000000000000005ed26b83000000001f9b00c1 trusted.glusterfs.quota.34a6a15c-d01f-4200-bade-78016c642685.contri.1=0x000000000001ae000000000000000001 trusted.glusterfs.quota.b7cd49a2-0519-4c34-bd64-96ac44d25850.contri.1=0x000000000001ae000000000000000001 trusted.glusterfs.quota.bba85d56-b300-45f2-98c1-b3023bf537b6.contri.1=0x000000000001ae000000000000000001 trusted.pgfid.34a6a15c-d01f-4200-bade-78016c642685=0x00000001 trusted.pgfid.b7cd49a2-0519-4c34-bd64-96ac44d25850=0x00000001 trusted.pgfid.bba85d56-b300-45f2-98c1-b3023bf537b6=0x00000001 [root at storage02 ~]# ls -lh /data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h -rw-rw-r--. 4 USER ftp 108K May 25? 2018 /data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h [root at storage02 ~]# getfattr -d -m . -e hex /data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h getfattr: Removing leading '/' from absolute path names # file: data/storage_a/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x24bf05756ab04613b42a3b63b3c00165 trusted.gfid2path.12529fafe3b56cc3=0x33346136613135632d643031662d343230302d626164652d3738303136633634323638352f636169726f2e68 trusted.gfid2path.e445cee32fde338d=0x62376364343961322d303531392d346333342d626436342d3936616334346432353835302f636169726f2e68 trusted.gfid2path.fbe2d5aaa0195e94=0x62626138356435362d623330302d343566322d393863312d6233303233626635333762362f636169726f2e68 trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x5f8a063a0001e6bc trusted.glusterfs.mdata=0x010000000000000000000000005ee25c8f000000002d2a4984000000005b08129b0000000000000000000000005ed26b83000000001f9b00c1 trusted.glusterfs.quota.34a6a15c-d01f-4200-bade-78016c642685.contri.1=0x000000000001ae000000000000000001 trusted.glusterfs.quota.b7cd49a2-0519-4c34-bd64-96ac44d25850.contri.1=0x000000000001ae000000000000000001 trusted.glusterfs.quota.bba85d56-b300-45f2-98c1-b3023bf537b6.contri.1=0x000000000001ae000000000000000001 trusted.pgfid.34a6a15c-d01f-4200-bade-78016c642685=0x00000001 trusted.pgfid.b7cd49a2-0519-4c34-bd64-96ac44d25850=0x00000001 trusted.pgfid.bba85d56-b300-45f2-98c1-b3023bf537b6=0x00000001 [root at storage01 ~]# ls -lh /data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h -rw-rw-r--. 4 USER ftp 0 May 25? 2018 /data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h [root at storage01 ~]# getfattr -d -m. -e hex /data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h getfattr: Removing leading '/' from absolute path names # file: data/storage_c/storage/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x24bf05756ab04613b42a3b63b3c00165 trusted.gfid2path.12529fafe3b56cc3=0x33346136613135632d643031662d343230302d626164652d3738303136633634323638352f636169726f2e68 trusted.gfid2path.e445cee32fde338d=0x62376364343961322d303531392d346333342d626436342d3936616334346432353835302f636169726f2e68 trusted.gfid2path.fbe2d5aaa0195e94=0x62626138356435362d623330302d343566322d393863312d6233303233626635333762362f636169726f2e68 trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x5f8a063a00023a3a trusted.glusterfs.mdata=0x010000000000000000000000005ee25c8f000000002d2a4984000000005b08129b0000000000000000000000005ed26b83000000001f9b00c1 trusted.glusterfs.quota.34a6a15c-d01f-4200-bade-78016c642685.contri.1=0x00000000000000000000000000000001 trusted.glusterfs.quota.b7cd49a2-0519-4c34-bd64-96ac44d25850.contri.1=0x00000000000000000000000000000001 trusted.glusterfs.quota.bba85d56-b300-45f2-98c1-b3023bf537b6.contri.1=0x00000000000000000000000000000001 trusted.pgfid.34a6a15c-d01f-4200-bade-78016c642685=0x00000001 trusted.pgfid.b7cd49a2-0519-4c34-bd64-96ac44d25850=0x00000001 trusted.pgfid.bba85d56-b300-45f2-98c1-b3023bf537b6=0x00000001 And nothing weird with ACLs on the file in the mount: [root at storage01 ~]# getfacl /storage2/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h getfacl: Removing leading '/' from absolute path names # file: storage2/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h # owner: USER # group: ftp user::rw- group::rw- other::r-- The file doesn't exist on the geo-rep destination, but other files in the same directory do, and don't seem to have any problems with similar xattrs: [root at pcic-backup01 ~]# ls -lh /pcic-backup/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h ls: cannot access /pcic-backup/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo.h: No such file or directory [root at pcic-backup01 ~]# getfattr -d -m. -e hex /data/brick/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo-svg.h getfattr: Removing leading '/' from absolute path names # file: data/brick/home/USER/anaconda3/pkgs/cairo-1.14.12-h8948797_3/include/cairo/cairo-svg.h security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000 trusted.gfid=0x324a54f85b084a3091749c1f93ead51b trusted.gfid2path.aabb9241b9daeeb1=0x33346136613135632d643031662d343230302d626164652d3738303136633634323638352f636169726f2d7376672e68 trusted.gfid2path.c8c216a2fda72d32=0x62626138356435362d623330302d343566322d393863312d6233303233626635333762362f636169726f2d7376672e68 trusted.gfid2path.e21e4ba36199a097=0x62376364343961322d303531392d346333342d626436342d3936616334346432353835302f636169726f2d7376672e68 trusted.glusterfs.mdata=0x010000000000000000000000005f7e60a9000000002ff17959000000005b08129b0000000000000000000000005f7e40260000000028b06e11 trusted.glusterfs.quota.34a6a15c-d01f-4200-bade-78016c642685.contri.1=0x0000000000000c000000000000000001 trusted.glusterfs.quota.bba85d56-b300-45f2-98c1-b3023bf537b6.contri.1=0x0000000000000c000000000000000001 trusted.pgfid.34a6a15c-d01f-4200-bade-78016c642685=0x00000001 trusted.pgfid.b7cd49a2-0519-4c34-bd64-96ac44d25850=0x00000001 trusted.pgfid.bba85d56-b300-45f2-98c1-b3023bf537b6=0x00000001 Anything else I can check? Thanks, ?-Matthew -- Matthew Benstead System Administrator Pacific Climate Impacts Consortium <https://pacificclimate.org/> University of Victoria, UH1 PO Box 1800, STN CSC Victoria, BC, V8W 2Y2 Phone: +1-250-721-8432 Email: matthewb at uvic.ca On 10/18/20 9:01 PM, Strahil Nikolov wrote:>> [2020-10-16 20:30:25.039659] E [MSGID: 109009] [dht-helper.c:1384:dht_migration_complete_check_task] 0-pcic-backup-dht: 24bf0575-6ab0-4613-b42a-3b63b3c00165: gfid different on the target file on pcic-backup-readdir-ahead-0 >> [2020-10-16 20:30:25.039695] E [MSGID: 148002] [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-pcic-backup-utime: dict set of key for set-ctime-mdata failed [Input/output error] > > I would start by finding that gfid on the source and then identify the gfid of the file in the geo-rep volume. > Also, it seems that you got some acl/extended attributes issues on the geo-rep destination - so take a look. > > Best Regards, > Strahil Nikolov-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20201019/87b2befd/attachment.html>