Ravishankar N
2016-Jul-15 15:01 UTC
[Gluster-users] lingering <gfid:*> entries in volume heal, gluster 3.6.3
On 07/15/2016 06:49 PM, Kingsley wrote:> On Fri, 2016-07-15 at 18:38 +0530, Ravishankar N wrote: >> On 07/15/2016 06:05 PM, Kingsley wrote: >>> chomp (my @output=`getfattr $path`); >> >> Could you try with `getfattr -d -m. -e hex $path` ? > Sure. I'm not really sure what I should be seeing, so I've uploaded the > full output here as it's rather large for a group mailing list IMO: > > http://gluster.dogwind.com/files/gfid-getfattr.txt > > The output is the same for each of the stayed-up bricks.If it is the same in all 3 bricks, the the xattrs of the file indicate heal is pending on the 3rd brick (trusted.afr.callrec-client-2) which I guess was the one that was down. Selfheal daemon should heal them ideally. But you're saying there are no messages in the shd logs corresponding to these gfids. Could you check the hardlink count (star -c%h) of these gfids (the ones corresponding to files, not directories)? It must be > 1. -Ravi> > FWIW, many of the entries have the same values: > > [root at gluster2b-1 ~]# sort gfids | perl gfid-to-fattr.pl 2> /dev/null | grep trusted.afr | sort | uniq -c > 73 trusted.afr.callrec-client-0=0x000000000000000000000000 > 73 trusted.afr.callrec-client-1=0x000000000000000000000000 > 25 trusted.afr.callrec-client-2=0x000000020000000300000000 > 5 trusted.afr.callrec-client-2=0x000000040000000300000000 > 33 trusted.afr.callrec-client-2=0x000000050000000300000000 > 1 trusted.afr.callrec-client-2=0x00000005000000ea00000000 > 1 trusted.afr.callrec-client-2=0x00000005000001f900000000 > 1 trusted.afr.callrec-client-2=0x000000050000021400000000 > 3 trusted.afr.callrec-client-2=0x000000060000000300000000 > 2 trusted.afr.callrec-client-2=0x000000080000000300000000 > 1 trusted.afr.callrec-client-2=0x000000090000000300000000 > 1 trusted.afr.callrec-client-2=0x0000000e0000000300000000 > 73 trusted.afr.callrec-client-3=0x000000000000000000000000 > 73 trusted.afr.dirty=0x000000000000000000000000 > > Cheers, > Kingsley. >
Kingsley
2016-Jul-15 15:18 UTC
[Gluster-users] lingering <gfid:*> entries in volume heal, gluster 3.6.3
On Fri, 2016-07-15 at 20:31 +0530, Ravishankar N wrote:> On 07/15/2016 06:49 PM, Kingsley wrote: > > On Fri, 2016-07-15 at 18:38 +0530, Ravishankar N wrote: > >> On 07/15/2016 06:05 PM, Kingsley wrote: > >>> chomp (my @output=`getfattr $path`); > >> > >> Could you try with `getfattr -d -m. -e hex $path` ? > > Sure. I'm not really sure what I should be seeing, so I've uploaded the > > full output here as it's rather large for a group mailing list IMO: > > > > http://gluster.dogwind.com/files/gfid-getfattr.txt > > > > The output is the same for each of the stayed-up bricks. > If it is the same in all 3 bricks, the the xattrs of the file indicate > heal is pending on the 3rd brick (trusted.afr.callrec-client-2) which I > guess was the one that was down. Selfheal daemon should heal them > ideally. But you're saying there are no messages in the shd logs > corresponding to these gfids. > Could you check the hardlink count (star -c%h) of these gfids (the ones > corresponding to files, not directories)? It must be > 1.Hi, I don't have star installed so I used ls, but yes they all have 2 links to them (see below). BTW, I noticed that the entries my script said didn't exist are actually symlinks to other gfid entries that are directories. Most of these target directories have 2 links, but one has 3 and one has 64. Anyway, the files: -rw-r--r--. 2 root root 850 Jun 1 01:13 /data/brick/callrec/.glusterfs/09/87/09872041-bc68-4eee-9c4d-ef11e4137f66 -rw-r--r--. 2 root root 2361324 May 31 19:11 /data/brick/callrec/.glusterfs/0d/16/0d1671b4-e31f-4c81-8600-fe63ffc84272 -rw-r--r--. 2 root root 35564 May 31 21:01 /data/brick/callrec/.glusterfs/0d/44/0d4431f1-10b1-4339-96a9-d8370bbd77a7 -rw-r--r--. 2 root root 104684 May 31 19:10 /data/brick/callrec/.glusterfs/16/fd/16fd1032-3bd5-474a-b30e-85971d69aaa9 -rw-r--r--. 2 root root 356204 May 31 19:11 /data/brick/callrec/.glusterfs/18/e2/18e2f17d-801f-428d-98c1-c35f7ac8a68d -rw-r--r--. 2 root root 947884 May 31 19:11 /data/brick/callrec/.glusterfs/19/29/19291239-e44c-4025-9f74-af7431aac6b9 -rw-r--r--. 2 root root 54444 May 31 20:57 /data/brick/callrec/.glusterfs/1b/a8/1ba8d90a-5f0c-4387-a79a-592427f3d1c5 -rw-r--r--. 2 root root 35564 May 31 21:38 /data/brick/callrec/.glusterfs/23/e7/23e75764-9794-4efd-b319-6eac717e6f28 -rw-r--r--. 2 root root 778284 May 31 19:11 /data/brick/callrec/.glusterfs/25/af/25af9c71-aab5-4f39-bd4a-bb4f0dab9342 -rw-r--r--. 2 root root 576364 May 31 19:11 /data/brick/callrec/.glusterfs/2b/bb/2bbb8529-4874-440d-959b-6a0745fdfda9 -rw-r--r--. 2 root root 1232 Jun 1 01:15 /data/brick/callrec/.glusterfs/33/8b/338baad1-6191-47f3-9737-dca2daf79fd8 -rw-r--r--. 2 root root 74924 May 31 19:09 /data/brick/callrec/.glusterfs/3a/50/3a50792e-0ce9-49a2-99d5-d59d1fd7e1a9 -rw-r--r--. 2 root root 6124 May 31 19:09 /data/brick/callrec/.glusterfs/3b/c9/3bc95362-be4f-4561-b95f-4cd1260c1781 -rw-r--r--. 2 root root 1113644 May 31 19:11 /data/brick/callrec/.glusterfs/3d/f8/3df80623-6bf4-47e3-a379-4e5605d0eda6 -rw-r--r--. 2 root root 451564 May 31 19:10 /data/brick/callrec/.glusterfs/3f/9f/3f9fcead-ffbe-4dd0-9dcf-8e815dfdc5b4 -rw-r--r--. 2 root root 199724 May 31 19:10 /data/brick/callrec/.glusterfs/40/3c/403c0233-432f-42d8-8ed0-eeca68c4b3f1 -rw-r--r--. 2 root root 550124 May 31 19:09 /data/brick/callrec/.glusterfs/41/97/4197cd59-1cec-452e-8c3c-c212ea9f17f6 -rw-r--r--. 2 root root 35564 May 31 20:48 /data/brick/callrec/.glusterfs/45/5c/455c2911-7a9d-401f-9ded-4380c6cec405 -rw-r--r--. 2 root root 203564 May 31 19:10 /data/brick/callrec/.glusterfs/45/93/459313b5-7ac8-43c6-a3b8-3a8e80afcebd -rw-r--r--. 2 root root 80044 May 31 21:01 /data/brick/callrec/.glusterfs/45/ae/45aed765-2ae2-4fc8-9982-8e1b5d6c19d4 -rw-r--r--. 2 root root 80044 May 31 20:57 /data/brick/callrec/.glusterfs/4e/86/4e86088b-f975-4228-ad5a-8d08fbd456fe -rw-r--r--. 2 root root 82284 May 31 19:11 /data/brick/callrec/.glusterfs/50/78/50787eb7-90ad-41fb-bd5e-3ba058b69c32 -rw-r--r--. 2 root root 663084 May 31 19:11 /data/brick/callrec/.glusterfs/53/75/5375e1e8-287f-4fc5-8b01-ec2f856eebf7 -rw-r--r--. 2 root root 35564 May 31 20:57 /data/brick/callrec/.glusterfs/5e/e3/5ee303e4-44fe-44a1-83b5-5cd6a91bc76a -rw-r--r--. 2 root root 291 May 31 21:01 /data/brick/callrec/.glusterfs/61/bf/61bf5407-c250-4448-a68f-d3bd82821260 -rw-r--r--. 2 root root 388204 May 31 19:10 /data/brick/callrec/.glusterfs/65/39/6539988f-2fe6-4500-aa7c-a7cd7cfcc5df -rw-r--r--. 2 root root 16684 May 31 19:10 /data/brick/callrec/.glusterfs/66/4f/664f979e-5acc-42f1-93be-9a46d783d430 -rw-r--r--. 2 root root 57644 May 31 19:09 /data/brick/callrec/.glusterfs/66/a4/66a4001b-9456-44e2-b547-5036383500d8 -rw-r--r--. 2 root root 80044 May 31 21:39 /data/brick/callrec/.glusterfs/68/12/68127ab7-807d-4a99-a609-6569d366d3aa -rw-r--r--. 2 root root 868844 May 31 19:10 /data/brick/callrec/.glusterfs/70/0e/700e2b0a-e6c0-4888-a5ea-f52cac55e770 -rw-r--r--. 2 root root 14764 May 31 21:01 /data/brick/callrec/.glusterfs/70/b7/70b7db21-b70f-4ae0-8660-921c3194f209 -rw-r--r--. 2 root root 59884 May 31 19:11 /data/brick/callrec/.glusterfs/70/ea/70eab476-d53d-4f2b-b56f-13cf12821a24 -rw-r--r--. 2 root root 915564 May 31 19:10 /data/brick/callrec/.glusterfs/75/c6/75c6e98b-7700-46f0-8be4-e897f969a5df -rw-r--r--. 2 root root 956204 May 31 19:10 /data/brick/callrec/.glusterfs/79/3a/793a81de-73e8-4d84-8ac1-be03c5ac2c47 -rw-r--r--. 2 root root 55084 May 31 19:10 /data/brick/callrec/.glusterfs/7a/d4/7ad41386-9d57-4a93-9fc2-3354e40d9927 -rw-r--r--. 2 root root 486764 May 31 19:09 /data/brick/callrec/.glusterfs/7c/e8/7ce8062a-37fd-4388-aa71-42e3f1125d20 -rw-r--r--. 2 root root 713324 May 31 19:11 /data/brick/callrec/.glusterfs/84/d3/84d3674c-6ec4-43b7-b590-8a125cbfbe43 -rw-r--r--. 2 root root 13484 May 31 19:09 /data/brick/callrec/.glusterfs/87/81/8781eb74-7409-4b1b-aa69-e30bf9c2387d -rw-r--r--. 2 root root 390444 May 31 19:09 /data/brick/callrec/.glusterfs/88/0c/880cd05b-46e5-4cf4-b042-bdb729acdd56 -rw-r--r--. 2 root root 628844 May 31 19:10 /data/brick/callrec/.glusterfs/8d/f8/8df8af4d-38cf-4cbd-b68e-4a6f2d41b308 -rw-r--r--. 2 root root 388524 May 31 19:11 /data/brick/callrec/.glusterfs/8f/3a/8f3a7de0-173a-4507-9b46-bc9db0a6bc41 -rw-r--r--. 2 root root 344684 May 31 19:11 /data/brick/callrec/.glusterfs/94/b5/94b5924f-65b9-45d6-8a03-b7084f8c4bdb -rw-r--r--. 2 root root 179564 May 31 19:11 /data/brick/callrec/.glusterfs/9c/62/9c6254df-e8dd-40a3-a7d1-760f6c19027a -rw-r--r--. 2 root root 276844 May 31 19:10 /data/brick/callrec/.glusterfs/9c/f4/9cf45591-c304-4348-b6ce-9dc9f8f335b1 -rw-r--r--. 2 root root 657324 May 31 19:09 /data/brick/callrec/.glusterfs/9d/80/9d80ae3c-833d-4a7c-81fc-b4ba9e645d5f -rw-r--r--. 2 root root 2367084 May 31 19:11 /data/brick/callrec/.glusterfs/9e/a0/9ea01327-d5ed-48bd-8049-cc552555b774 -rw-r--r--. 2 root root 40684 May 31 19:09 /data/brick/callrec/.glusterfs/a1/94/a194006c-b0c6-49e9-a2ec-3613403f869f -rw-r--r--. 2 root root 2217324 May 31 19:10 /data/brick/callrec/.glusterfs/a2/9c/a29c32b6-e9aa-4e51-8b07-311bb7512d89 -rw-r--r--. 2 root root 1308524 May 31 19:11 /data/brick/callrec/.glusterfs/a4/74/a4745c5f-a3ad-4bed-a504-5ae5b31200a3 -rw-r--r--. 2 root root 80044 May 31 21:39 /data/brick/callrec/.glusterfs/a6/cb/a6cb8d18-f2b3-48d4-ac37-c022603b8f8e -rw-r--r--. 2 root root 23724 May 31 20:49 /data/brick/callrec/.glusterfs/a9/e1/a9e1cad9-cd31-48f0-b9f8-322e1f602401 -rw-r--r--. 2 root root 626924 May 31 19:10 /data/brick/callrec/.glusterfs/ac/75/ac75a8e5-0906-40ea-a2ab-145f3cfcce2e -rw-r--r--. 2 root root 145004 May 31 19:10 /data/brick/callrec/.glusterfs/ae/44/ae447d7b-04b9-414e-ada6-d1880a5c6555 -rw-r--r--. 2 root root 293 May 31 21:38 /data/brick/callrec/.glusterfs/b0/f4/b0f4e951-63b3-4152-b0b0-7aad8e6ed729 -rw-r--r--. 2 root root 40684 May 31 19:11 /data/brick/callrec/.glusterfs/b1/13/b1138fe2-34cf-4d0d-af99-3b9d9aec0317 -rw-r--r--. 2 root root 65644 May 31 19:10 /data/brick/callrec/.glusterfs/b1/db/b1db87aa-c1fa-409b-aa10-429860c20dbe -rw-r--r--. 2 root root 117164 May 31 19:09 /data/brick/callrec/.glusterfs/b2/3a/b23a64dd-4522-408c-8465-4096c2977da4 -rw-r--r--. 2 root root 1595244 May 31 19:11 /data/brick/callrec/.glusterfs/c3/41/c3419ad6-3e32-4fc5-93da-2b04a6090cfa -rw-r--r--. 2 root root 4725 Jun 1 01:15 /data/brick/callrec/.glusterfs/c4/a2/c4a200b9-bb31-46cb-92c6-551fd6ad9ec3 -rw-r--r--. 2 root root 41004 May 31 21:39 /data/brick/callrec/.glusterfs/da/d2/dad27361-d38f-46b3-a77c-b6ac5df054d9 -rw-r--r--. 2 root root 987244 May 31 19:10 /data/brick/callrec/.glusterfs/de/26/de268a4d-06fd-479d-8efc-954002ff7222 -rw-r--r--. 2 root root 80044 May 31 21:39 /data/brick/callrec/.glusterfs/e3/56/e356342a-3dab-4049-b4bc-ea0de4a8ee87 -rw-r--r--. 2 root root 80684 May 31 19:09 /data/brick/callrec/.glusterfs/e9/7c/e97cb502-924d-4ea8-9730-f84eda0b69fd -rw-r--r--. 2 root root 147244 May 31 19:10 /data/brick/callrec/.glusterfs/eb/87/eb87548b-a90a-458d-b215-939ad59f5ec0 -rw-r--r--. 2 root root 510124 May 31 19:11 /data/brick/callrec/.glusterfs/ef/58/ef583ff4-72b3-408d-bf34-88ca5534c71e -rw-r--r--. 2 root root 80044 May 31 20:48 /data/brick/callrec/.glusterfs/ef/bd/efbd6028-b9f8-45de-a653-b27f75570b81 -rw-r--r--. 2 root root 1103084 May 31 19:10 /data/brick/callrec/.glusterfs/ef/bf/efbfc625-c42c-4d5e-a22a-173f84362f24 -rw-r--r--. 2 root root 292 May 31 20:48 /data/brick/callrec/.glusterfs/f1/2b/f12b4d8e-b5e6-4f71-a06c-0c0d067a7eb9 -rw-r--r--. 2 root root 1084204 May 31 19:11 /data/brick/callrec/.glusterfs/f3/4d/f34d746e-1555-4b54-9ebd-2bfe7c96b0ef -rw-r--r--. 2 root root 67564 May 31 19:10 /data/brick/callrec/.glusterfs/f4/8b/f48beb09-975f-4ff6-843e-d8906d3b21b3 -rw-r--r--. 2 root root 1148524 May 31 19:09 /data/brick/callrec/.glusterfs/f6/c7/f6c76172-c73b-4cf0-ad8a-00a6f9c6d7d2 -rw-r--r--. 2 root root 80044 May 31 20:49 /data/brick/callrec/.glusterfs/f7/0b/f70b50fb-3061-4eb9-94f0-85fb2d789d27 -rw-r--r--. 2 root root 122604 May 31 19:09 /data/brick/callrec/.glusterfs/f7/d3/f7d3ef0f-3c01-412e-93fa-704171267d9e Cheers, Kingsley.> -Ravi > > > > > > > FWIW, many of the entries have the same values: > > > > [root at gluster2b-1 ~]# sort gfids | perl gfid-to-fattr.pl 2> /dev/null | grep trusted.afr | sort | uniq -c > > 73 trusted.afr.callrec-client-0=0x000000000000000000000000 > > 73 trusted.afr.callrec-client-1=0x000000000000000000000000 > > 25 trusted.afr.callrec-client-2=0x000000020000000300000000 > > 5 trusted.afr.callrec-client-2=0x000000040000000300000000 > > 33 trusted.afr.callrec-client-2=0x000000050000000300000000 > > 1 trusted.afr.callrec-client-2=0x00000005000000ea00000000 > > 1 trusted.afr.callrec-client-2=0x00000005000001f900000000 > > 1 trusted.afr.callrec-client-2=0x000000050000021400000000 > > 3 trusted.afr.callrec-client-2=0x000000060000000300000000 > > 2 trusted.afr.callrec-client-2=0x000000080000000300000000 > > 1 trusted.afr.callrec-client-2=0x000000090000000300000000 > > 1 trusted.afr.callrec-client-2=0x0000000e0000000300000000 > > 73 trusted.afr.callrec-client-3=0x000000000000000000000000 > > 73 trusted.afr.dirty=0x000000000000000000000000 > > > > Cheers, > > Kingsley. > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >