Aravinda
2024-Feb-09 17:06 UTC
[Gluster-users] __Geo-replication status is getting Faulty after few seconds
Based on the xattrs the file name is?app_docmerge12monthsfixedCUSTODIAL_2024_1_22_15_3_24_648.doc and it was created under the directory GFID?e8aff729-a310-4d21-a64b-d8cc7cb1a828.? In .glusterfs directory directory GFID is symlinked to the actual directory. You can check that files exists or not in that directory cd .glusterfs/e8/af/e8aff729-a310-4d21-a64b-d8cc7cb1a828 ls?app_docmerge12monthsfixedCUSTODIAL_2024_1_22_15_3_24_648.doc Changelog files may give an idea about what happened to this file. Since it is showing link count as 1 for this file, the file may have deleted but stale GFID link exists or recreated with different GFID.? Moving this file to a backup location may help Geo-rep to continue. -- Aravinda Kadalu Technologies ---- On Fri, 09 Feb 2024 22:24:44 +0530 Anant Saraswat <anant.saraswat at techblue.co.uk> wrote --- Thanks mailto:aravinda at kadalu.tech, Please check the output of 'getfattr' command. [root at master1 .glusterfs]# getfattr -d -m . ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 # file: d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 trusted.afr.dirty=0sAAAAAAAAAAAAAAAA trusted.gfid=0s1T+tj4TpSySesMy8vcS6qA= trusted.gfid2path.5007d273788799dd="e8aff729-a310-4d21-a64b-d8cc7cb1a828/app_docmerge12monthsfixedCUSTODIAL_2024_1_22_15_3_24_648.doc" trusted.glusterfs.93c45c14-f700-4d50-962b-7653be471e27.xtime=0sZa6EgQAFa7I trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGWuhIEAAAAADGLMhwAAAABlroSBAAAAAAxizIcAAAAAZa6EgAAAAAAq348C I will try to move the file and start geo-replication in an hour and let you know how it goes. I am also searching for the change logs as you mentioned, but it may take time to grep. I truly appreciate everyone's input so far. Kind regards, Anant From:?Aravinda <mailto:aravinda at kadalu.tech> Sent:?09 February 2024 4:39 PM To:?Anant Saraswat <mailto:anant.saraswat at techblue.co.uk> Cc:?Diego Zuccato <mailto:diego.zuccato at unibo.it>; mailto:gluster-users at gluster.org <mailto:gluster-users at gluster.org>; Strahil Nikolov <mailto:hunter86_bg at yahoo.com> Subject:?Re: [Gluster-users] __Geo-replication status is getting Faulty after few seconds ? EXTERNAL:?Do not click links or open attachments if you do not recognize the sender. If the backend GFID file only exists in Master 1, then it looks like stale GFID exists due to some failure. Run getfattr on that file to see if it shows any file path in gfid2path xattr. Please share the output of the following command. getfattr -d -m . ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 Meanwhile move this file to a backup location and see if Geo-rep succeeds. We can introspect the details about the GFID?d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8. Please share the matching changelog files from? grep -R "d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8" .glusterfs/changelogs/* -- Aravinda Kadalu Technologies ---- On Fri, 09 Feb 2024 16:23:45 +0530 Anant Saraswat <mailto:anant.saraswat at techblue.co.uk>?wrote --- Hi mailto:hunter86_bg at yahoo.com, As mentioned earlier this gfid has "1" in 2nd?column of 'ls' command output, which means this gfid doesn't?have any other hardlink. [root at master1 .glusterfs]# ls -l ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 -rw-r--r-- 1 root root 174592 Jan 22 15:06 ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 Still, I have searched in "/opt/tier1data2019/brick/.glusterfs" and there is only one file comes with samefile option, which is gfid itself, please check the following output. [root at master1 .glusterfs]# find . -samefile ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 You have mentioned the other option to link it somewhere on the real filesystem and then sync it to the other brick. Here, what do you mean by the real filesystem? Do you want me to link this GFID with an existing file on the brick? Or do you want the GFID to be linked with some temporary file, say, on "/tmp/test_file," and then sync "/tmp/test_file" and "/opt/tier1data2019/brick/.glusterfs/d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8" to master2 and master3 nodes? Many thanks, Anant From:?Strahil Nikolov <mailto:hunter86_bg at yahoo.com> Sent:?09 February 2024 5:28 AM To:?Anant Saraswat <mailto:anant.saraswat at techblue.co.uk>; Diego Zuccato <mailto:diego.zuccato at unibo.it>; mailto:gluster-users at gluster.org?<mailto:gluster-users at gluster.org>; Aravinda Vishwanathapura <mailto:aravinda at kadalu.tech> Subject:?Re: [Gluster-users] __Geo-replication status is getting Faulty after few seconds ? EXTERNAL:?Do not click links or open attachments if you do not recognize the sender. The other option around (if indeed is the only in .glusterfs) is to have it link somewhere on the real filesystem structure and then sync it to the other bricks. If georep works again - just remove it from the fuse and check if it will get deleted or not. Best Regards, Strahil Nikolov On Fri, Feb 9, 2024 at 7:25, Strahil Nikolov <mailto:hunter86_bg at yahoo.com> wrote: It's a hard link, so use find's '-samefile' option to see if it's the last one or not. If you really want to delete it, have a backup and then delete both the gfid and any other hard links. Best Regards, Strahil Nikolov DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation. DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240209/d9300436/attachment.html>
Anant Saraswat
2024-Feb-09 17:17 UTC
[Gluster-users] __Geo-replication status is getting Faulty after few seconds
I have checked the path ".glusterfs/e8/af/e8aff729-a310-4d21-a64b-d8cc7cb1a828", and there is no file named "app_docmerge12monthsfixedCUSTODIAL_2024_1_22_15_3_24_648.doc". I also attempted to match the time stamp of the GFID in question with the files under the ".glusterfs/e8/af/e8aff729-a310-4d21-a64b-d8cc7cb1a828" directory, and it appears there are three files with the same time stamp, but they are named differently. -rw-r--r-- 2 root root 40212 Jan 22 15:06 b6b6b824-303d-4932-9797-20674f92e902 -rw-r--r-- 2 root root 40212 Jan 22 15:06 56c42bea-fad4-4dee-9561-bbc57e4aaf10 -rw-r--r-- 2 root root 40212 Jan 22 15:06 1bdb4239-5823-415b-951f-ce315a45492b Please check the attached CHANGELOG file for the given gfid. Many thanks, Anant ________________________________ From: Aravinda <aravinda at kadalu.tech> Sent: 09 February 2024 5:06 PM To: Anant Saraswat <anant.saraswat at techblue.co.uk> Cc: Diego Zuccato <diego.zuccato at unibo.it>; gluster-users at gluster.org <gluster-users at gluster.org>; Strahil Nikolov <hunter86_bg at yahoo.com> Subject: Re: [Gluster-users] __Geo-replication status is getting Faulty after few seconds EXTERNAL: Do not click links or open attachments if you do not recognize the sender. Based on the xattrs the file name is app_docmerge12monthsfixedCUSTODIAL_2024_1_22_15_3_24_648.doc and it was created under the directory GFID e8aff729-a310-4d21-a64b-d8cc7cb1a828. In .glusterfs directory directory GFID is symlinked to the actual directory. You can check that files exists or not in that directory cd .glusterfs/e8/af/e8aff729-a310-4d21-a64b-d8cc7cb1a828 ls app_docmerge12monthsfixedCUSTODIAL_2024_1_22_15_3_24_648.doc Changelog files may give an idea about what happened to this file. Since it is showing link count as 1 for this file, the file may have deleted but stale GFID link exists or recreated with different GFID. Moving this file to a backup location may help Geo-rep to continue. -- Aravinda Kadalu Technologies ---- On Fri, 09 Feb 2024 22:24:44 +0530 Anant Saraswat <anant.saraswat at techblue.co.uk> wrote --- Thanks @Aravinda<mailto:aravinda at kadalu.tech>, Please check the output of 'getfattr' command. [root at master1 .glusterfs]# getfattr -d -m . ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 # file: d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 trusted.afr.dirty=0sAAAAAAAAAAAAAAAA trusted.gfid=0s1T+tj4TpSySesMy8vcS6qA=trusted.gfid2path.5007d273788799dd="e8aff729-a310-4d21-a64b-d8cc7cb1a828/app_docmerge12monthsfixedCUSTODIAL_2024_1_22_15_3_24_648.doc" trusted.glusterfs.93c45c14-f700-4d50-962b-7653be471e27.xtime=0sZa6EgQAFa7Itrusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGWuhIEAAAAADGLMhwAAAABlroSBAAAAAAxizIcAAAAAZa6EgAAAAAAq348C I will try to move the file and start geo-replication in an hour and let you know how it goes. I am also searching for the change logs as you mentioned, but it may take time to grep. I truly appreciate everyone's input so far. Kind regards, Anant ________________________________ From: Aravinda <aravinda at kadalu.tech<mailto:aravinda at kadalu.tech>> Sent: 09 February 2024 4:39 PM To: Anant Saraswat <anant.saraswat at techblue.co.uk<mailto:anant.saraswat at techblue.co.uk>> Cc: Diego Zuccato <diego.zuccato at unibo.it<mailto:diego.zuccato at unibo.it>>; gluster-users at gluster.org<mailto:gluster-users at gluster.org> <gluster-users at gluster.org<mailto:gluster-users at gluster.org>>; Strahil Nikolov <hunter86_bg at yahoo.com<mailto:hunter86_bg at yahoo.com>> Subject: Re: [Gluster-users] __Geo-replication status is getting Faulty after few seconds EXTERNAL: Do not click links or open attachments if you do not recognize the sender. If the backend GFID file only exists in Master 1, then it looks like stale GFID exists due to some failure. Run getfattr on that file to see if it shows any file path in gfid2path xattr. Please share the output of the following command. getfattr -d -m . ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 Meanwhile move this file to a backup location and see if Geo-rep succeeds. We can introspect the details about the GFID d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8. Please share the matching changelog files from grep -R "d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8" .glusterfs/changelogs/* -- Aravinda Kadalu Technologies ---- On Fri, 09 Feb 2024 16:23:45 +0530 Anant Saraswat <anant.saraswat at techblue.co.uk<mailto:anant.saraswat at techblue.co.uk>> wrote --- Hi @Strahil Nikolov<mailto:hunter86_bg at yahoo.com>, As mentioned earlier this gfid has "1" in 2nd column of 'ls' command output, which means this gfid doesn't have any other hardlink. [root at master1 .glusterfs]# ls -l ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 -rw-r--r-- 1 root root 174592 Jan 22 15:06 ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 Still, I have searched in "/opt/tier1data2019/brick/.glusterfs" and there is only one file comes with samefile option, which is gfid itself, please check the following output. [root at master1 .glusterfs]# find . -samefile ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 ./d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8 You have mentioned the other option to link it somewhere on the real filesystem and then sync it to the other brick. Here, what do you mean by the real filesystem? Do you want me to link this GFID with an existing file on the brick? Or do you want the GFID to be linked with some temporary file, say, on "/tmp/test_file," and then sync "/tmp/test_file" and "/opt/tier1data2019/brick/.glusterfs/d5/3f/d53fad8f-84e9-4b24-9eb0-ccbcbdc4baa8" to master2 and master3 nodes? Many thanks, Anant ________________________________ From: Strahil Nikolov <hunter86_bg at yahoo.com<mailto:hunter86_bg at yahoo.com>> Sent: 09 February 2024 5:28 AM To: Anant Saraswat <anant.saraswat at techblue.co.uk<mailto:anant.saraswat at techblue.co.uk>>; Diego Zuccato <diego.zuccato at unibo.it<mailto:diego.zuccato at unibo.it>>; gluster-users at gluster.org<mailto:gluster-users at gluster.org> <gluster-users at gluster.org<mailto:gluster-users at gluster.org>>; Aravinda Vishwanathapura <aravinda at kadalu.tech<mailto:aravinda at kadalu.tech>> Subject: Re: [Gluster-users] __Geo-replication status is getting Faulty after few seconds EXTERNAL: Do not click links or open attachments if you do not recognize the sender. The other option around (if indeed is the only in .glusterfs) is to have it link somewhere on the real filesystem structure and then sync it to the other bricks. If georep works again - just remove it from the fuse and check if it will get deleted or not. Best Regards, Strahil Nikolov On Fri, Feb 9, 2024 at 7:25, Strahil Nikolov <hunter86_bg at yahoo.com<mailto:hunter86_bg at yahoo.com>> wrote: It's a hard link, so use find's '-samefile' option to see if it's the last one or not. If you really want to delete it, have a backup and then delete both the gfid and any other hard links. Best Regards, Strahil Nikolov DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation. DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation. DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240209/4d7e7d00/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: CHANGELOG.1705936007 Type: application/octet-stream Size: 5672 bytes Desc: CHANGELOG.1705936007 URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240209/4d7e7d00/attachment.obj>