Ravishankar N
2018-Nov-16 04:14 UTC
[Gluster-users] Self-healing not healing 27k files on GlusterFS 4.1.5 3 nodes replica
On 11/15/2018 09:17 PM, mabi wrote:> ??????? Original Message ??????? > On Thursday, November 15, 2018 1:41 PM, Ravishankar N <ravishankar at redhat.com> wrote: > >> Thanks, noted. One more query. Are there files inside each of these >> directories? Or is it just empty directories? > You will find below the content of each of these 3 directories taken the brick on node 1: > > i)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10 > > drwxr-xr-x 4 www-data www-data 4 Nov 5 14:19 . > drwxr-xr-x 31 www-data www-data 31 Nov 5 14:23 .. > drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 dir11 > drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 another_dir > > ii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/ > drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 . > drwxr-xr-x 4 www-data www-data 4 Nov 5 14:19 .. > drwxr-xr-x 2 www-data www-data 4 Nov 5 14:19 oc_dir > > iii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir > > drwxr-xr-x 2 www-data www-data 4 Nov 5 14:19 . > drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 .. > -rw-r--r-- 2 www-data www-data 32 Nov 5 14:19 fileKey > -rw-r--r-- 2 www-data www-data 512 Nov 5 14:19 username.shareKey > > so as you see from the output above only the "oc_dir" directory has two files inside. >Okay, I'm assuming the list of files+dirs are the same on nodes 2 and 3 as well. Correct me if that isn't the case.>> symlinks are only for dirs. For files, they would be hard links to the >> actual files. So if stat >> ../brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf gives you >> a file, then you can use find -samefile to get the other hardlinks like so: >> #cd /brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf >> #find /brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf >> >> If it is a hardlink, then you can do a getfattr on >> /brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf itself. >> -Ravi > Thank you for explaining this important part. So yes with your help I could find the filenames associated to these 2 GFIDs and guess what? they are the 2 files which listed in the output above of the "oc_dir" directory. Have a look at this: > > # find /data/myvol-pro/brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf > /data/myvol-pro/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf > /data/myvol-pro/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey > > # find /data/myvol-pro/brick -samefile 3c92459b-8fa1-4669-9a3d-b38b8d41c360 > /data/myvol-pro/brick/.glusterfs/3c/92/3c92459b-8fa1-4669-9a3d-b38b8d41c360 > /data/myvol-pro/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKeyOkay, as asked in the previous mail, please share the getfattr output from all bricks for these 2 files. I think once we have this, we can try either 'adjusting' the the gfid and symlinks on node 2 for dir11 and oc_dir or see if we can set afr xattrs on dir10 for self-heal to purge everything under it on node 2 and recreate it using the other 2 nodes. -Ravi> > I hope that helps the debug further else let me know if you need anything else.
mabi
2018-Nov-16 15:37 UTC
[Gluster-users] Self-healing not healing 27k files on GlusterFS 4.1.5 3 nodes replica
??????? Original Message ??????? On Friday, November 16, 2018 5:14 AM, Ravishankar N <ravishankar at redhat.com> wrote:> Okay, as asked in the previous mail, please share the getfattr output > from all bricks for these 2 files. I think once we have this, we can try > either 'adjusting' the the gfid and symlinks on node 2 for dir11 and > oc_dir or see if we can set afr xattrs on dir10 for self-heal to purge > everything under it on node 2 and recreate it using the other 2 nodes.And finally here is the output of a getfattr from both files from the 3 nodes: FILE 1: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey NODE 1: trusted.afr.dirty=0x000000000000000000000000 trusted.afr.myvol-pro-client-1=0x000000020000000100000000 trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579 NODE 2: trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x48ccb52b788f4361b33fad43157b8ea8 trusted.gfid2path.32a8dc56983f7b8f=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f66696c654b6579 NODE 3: trusted.afr.dirty=0x000000000000000000000000 trusted.afr.myvol-pro-client-1=0x000000020000000100000000 trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579 FILE 2: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey NODE 1: trusted.afr.dirty=0x000000000000000000000000 trusted.afr.myvol-pro-client-1=0x000000020000000100000000 trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360 trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579 NODE 2: trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0xae880a4f19824bc6a3baabe2e3c62ace trusted.gfid2path.0c0f97b97351b4af=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f6a6d406d616765726c2e63682e73686172654b6579 NODE 3: trusted.afr.dirty=0x000000000000000000000000 trusted.afr.myvol-pro-client-1=0x000000020000000100000000 trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360 trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579 Thanks again in advance for your answer.