Kingsley
2016-Jul-15 12:35 UTC
[Gluster-users] lingering <gfid:*> entries in volume heal, gluster 3.6.3
Hi Ravi, thanks for replying.
I've checked all bricks for their respective gfid files but either the
files don't exist or getfattr produces no output.
What I've also found is that the gfid list shown for the 3 bricks that
stayed up contains the same list of entries, albeit not all in the same
order. On top of that, the list of files that are present or not is the
same on the 3 bricks that remained up.
I created a "gfids" file in /root on each brick, which contained that
brick's <gfid:*> lines reported by gluster volume callrec heal info.
The output of "md5sum gfids" differed between the 3 "up"
bricks, but the
output of "sort gfids | md5sum" was the same.
I then wrote this Perl script:
#!/usr/bin/perl
use strict;
use warnings;
while(<>=~/gfid:((\w\w)(\w\w)[\w\-]+)/)
{
my $path="/data/brick/callrec/.glusterfs/$2/$3/$1";
print "$path\t: ";
if (-f $path)
{
chomp (my @output=`getfattr $path`);
@output = 'no xattrs' unless @output;
print map { "$_\n" } @output;
}
else
{ print "file not found\n"; }
}
Then, the following command produced the same output on each brick:
sort gfids | perl gfid-to-fattr.pl | md5sum
The full output:
[root at gluster2b-1 ~]# perl gfid-to-fattr.pl < gfids
/data/brick/callrec/.glusterfs/d6/e9/d6e91662-a395-42de-8af3-ba6164eb0f90
: file not found
/data/brick/callrec/.glusterfs/09/87/09872041-bc68-4eee-9c4d-ef11e4137f66
: no xattrs
/data/brick/callrec/.glusterfs/48/85/4885e073-75fd-461a-a70e-1e9578ef99cd
: file not found
/data/brick/callrec/.glusterfs/0d/44/0d4431f1-10b1-4339-96a9-d8370bbd77a7
: no xattrs
/data/brick/callrec/.glusterfs/8d/05/8d059d2b-9e98-482b-aeac-e5decd126fe7
: file not found
/data/brick/callrec/.glusterfs/b2/3a/b23a64dd-4522-408c-8465-4096c2977da4
: no xattrs
/data/brick/callrec/.glusterfs/27/63/27632fb9-ab59-45cb-bb7a-5c78697619ea
: file not found
/data/brick/callrec/.glusterfs/8d/f8/8df8af4d-38cf-4cbd-b68e-4a6f2d41b308
: no xattrs
/data/brick/callrec/.glusterfs/5c/b5/5cb576bb-8389-4f34-b812-5a770e4c0013
: file not found
/data/brick/callrec/.glusterfs/65/39/6539988f-2fe6-4500-aa7c-a7cd7cfcc5df
: no xattrs
/data/brick/callrec/.glusterfs/3d/cc/3dcce45c-8c06-45f6-8101-a2e9737441c5
: file not found
/data/brick/callrec/.glusterfs/de/26/de268a4d-06fd-479d-8efc-954002ff7222
: no xattrs
/data/brick/callrec/.glusterfs/c2/37/c2370b73-7c1c-4f94-afcf-74755b9a68a0
: file not found
/data/brick/callrec/.glusterfs/fa/7b/fa7bbd79-3394-49b1-96c1-0bf7d3b3138d
: file not found
/data/brick/callrec/.glusterfs/88/0c/880cd05b-46e5-4cf4-b042-bdb729acdd56
: no xattrs
/data/brick/callrec/.glusterfs/1b/ad/1badcc6d-52e5-42ca-a42a-3877415cf21e
: file not found
/data/brick/callrec/.glusterfs/45/93/459313b5-7ac8-43c6-a3b8-3a8e80afcebd
: no xattrs
/data/brick/callrec/.glusterfs/db/53/db53921d-ca03-4c97-95cd-9fa4b514d585
: file not found
/data/brick/callrec/.glusterfs/87/81/8781eb74-7409-4b1b-aa69-e30bf9c2387d
: no xattrs
/data/brick/callrec/.glusterfs/21/c1/21c1bcc5-4c49-4010-9b4f-e920634764be
: file not found
/data/brick/callrec/.glusterfs/70/0e/700e2b0a-e6c0-4888-a5ea-f52cac55e770
: no xattrs
/data/brick/callrec/.glusterfs/7a/1b/7a1b72f5-ca99-4ede-aacd-ba1b867959c0
: file not found
/data/brick/callrec/.glusterfs/ac/75/ac75a8e5-0906-40ea-a2ab-145f3cfcce2e
: no xattrs
/data/brick/callrec/.glusterfs/c6/03/c603b955-305b-421e-9056-6a8256c25f88
: file not found
/data/brick/callrec/.glusterfs/81/e9/81e9ab2f-3f36-4a1f-ba2a-86577501c5db
: file not found
/data/brick/callrec/.glusterfs/3f/9f/3f9fcead-ffbe-4dd0-9dcf-8e815dfdc5b4
: no xattrs
/data/brick/callrec/.glusterfs/09/12/0912f760-51ce-48bd-93b1-fffbebd965a1
: file not found
/data/brick/callrec/.glusterfs/66/a4/66a4001b-9456-44e2-b547-5036383500d8
: no xattrs
/data/brick/callrec/.glusterfs/8b/ce/8bcece86-49af-4d88-83f8-8672dbdc5ed1
: file not found
/data/brick/callrec/.glusterfs/f6/c7/f6c76172-c73b-4cf0-ad8a-00a6f9c6d7d2
: no xattrs
/data/brick/callrec/.glusterfs/b1/db/b1db87aa-c1fa-409b-aa10-429860c20dbe
: no xattrs
/data/brick/callrec/.glusterfs/e9/7c/e97cb502-924d-4ea8-9730-f84eda0b69fd
: no xattrs
/data/brick/callrec/.glusterfs/e7/21/e721ce14-9201-4ec4-a027-bbeef43ab401
: file not found
/data/brick/callrec/.glusterfs/a6/02/a6024ac9-fdde-4517-87e8-ede5845f3bb3
: file not found
/data/brick/callrec/.glusterfs/bc/3c/bc3c227c-a68e-4b9e-a81f-ce14aa30a504
: file not found
/data/brick/callrec/.glusterfs/a1/94/a194006c-b0c6-49e9-a2ec-3613403f869f
: no xattrs
/data/brick/callrec/.glusterfs/c6/10/c610a4ae-1bbb-464d-8ef3-4c50a64f1110
: file not found
/data/brick/callrec/.glusterfs/66/4f/664f979e-5acc-42f1-93be-9a46d783d430
: no xattrs
/data/brick/callrec/.glusterfs/41/97/4197cd59-1cec-452e-8c3c-c212ea9f17f6
: no xattrs
/data/brick/callrec/.glusterfs/1e/2b/1e2b6e9e-8d19-48d8-9488-4d3f78e38213
: file not found
/data/brick/callrec/.glusterfs/0b/95/0b95c327-d880-4e93-9432-3fd205b46ca4
: file not found
/data/brick/callrec/.glusterfs/ae/44/ae447d7b-04b9-414e-ada6-d1880a5c6555
: no xattrs
/data/brick/callrec/.glusterfs/a2/9c/a29c32b6-e9aa-4e51-8b07-311bb7512d89
: no xattrs
/data/brick/callrec/.glusterfs/a1/a4/a1a4c18d-8807-4a43-b9d2-99cf33a02c03
: file not found
/data/brick/callrec/.glusterfs/93/dc/93dcfe76-318c-43c3-908c-0332201387f4
: file not found
/data/brick/callrec/.glusterfs/4d/6b/4d6be1c5-2282-441f-97e1-71c8c1e42aa1
: file not found
/data/brick/callrec/.glusterfs/e4/ca/e4ca593f-73f5-4d9f-b84f-16168b4f84f1
: file not found
/data/brick/callrec/.glusterfs/9d/80/9d80ae3c-833d-4a7c-81fc-b4ba9e645d5f
: no xattrs
/data/brick/callrec/.glusterfs/c7/82/c78213fd-a60f-41fe-8471-6c8a92fdd873
: file not found
/data/brick/callrec/.glusterfs/7c/e8/7ce8062a-37fd-4388-aa71-42e3f1125d20
: no xattrs
/data/brick/callrec/.glusterfs/85/ed/85ed9df0-64e1-4d93-8572-a5506c2c2d01
: file not found
/data/brick/callrec/.glusterfs/3b/c9/3bc95362-be4f-4561-b95f-4cd1260c1781
: no xattrs
/data/brick/callrec/.glusterfs/18/e2/18e2f17d-801f-428d-98c1-c35f7ac8a68d
: no xattrs
/data/brick/callrec/.glusterfs/70/ea/70eab476-d53d-4f2b-b56f-13cf12821a24
: no xattrs
/data/brick/callrec/.glusterfs/0b/07/0b07c592-f784-4d08-a3a2-3a0de0df666e
: file not found
/data/brick/callrec/.glusterfs/1c/62/1c623eb2-0594-48bf-accb-0c7b0a3a530a
: file not found
/data/brick/callrec/.glusterfs/18/81/1881cc27-09ab-453e-b89c-afcce7aab5ea
: file not found
/data/brick/callrec/.glusterfs/3a/50/3a50792e-0ce9-49a2-99d5-d59d1fd7e1a9
: no xattrs
/data/brick/callrec/.glusterfs/9c/f4/9cf45591-c304-4348-b6ce-9dc9f8f335b1
: no xattrs
/data/brick/callrec/.glusterfs/f3/4d/f34d746e-1555-4b54-9ebd-2bfe7c96b0ef
: no xattrs
/data/brick/callrec/.glusterfs/84/d3/84d3674c-6ec4-43b7-b590-8a125cbfbe43
: no xattrs
/data/brick/callrec/.glusterfs/21/f7/21f767b1-b5f3-4441-8f0e-ac852d2cdd25
: file not found
/data/brick/callrec/.glusterfs/f7/d3/f7d3ef0f-3c01-412e-93fa-704171267d9e
: no xattrs
/data/brick/callrec/.glusterfs/33/10/331058cb-2fcf-4198-910f-5c7b78100807
: file not found
/data/brick/callrec/.glusterfs/2f/ea/2feaebdb-6eba-4e01-80ae-0812e5c770c6
: file not found
/data/brick/callrec/.glusterfs/fa/68/fa685010-3cd1-4b18-bc37-7db6c2e543ee
: file not found
/data/brick/callrec/.glusterfs/40/3c/403c0233-432f-42d8-8ed0-eeca68c4b3f1
: no xattrs
/data/brick/callrec/.glusterfs/9e/6c/9e6c8760-48e1-4b84-9ad4-e6a33c881b6c
: file not found
/data/brick/callrec/.glusterfs/ef/58/ef583ff4-72b3-408d-bf34-88ca5534c71e
: no xattrs
/data/brick/callrec/.glusterfs/c3/41/c3419ad6-3e32-4fc5-93da-2b04a6090cfa
: no xattrs
/data/brick/callrec/.glusterfs/ea/a4/eaa43674-b1a3-4833-a946-de7b7121bb88
: file not found
/data/brick/callrec/.glusterfs/79/3a/793a81de-73e8-4d84-8ac1-be03c5ac2c47
: no xattrs
/data/brick/callrec/.glusterfs/e1/d9/e1d92bdc-0fe9-4531-9fc0-0c9e227721c0
: file not found
/data/brick/callrec/.glusterfs/d6/c6/d6c6e6b4-d476-4b9b-990d-b79e66c490bf
: file not found
/data/brick/callrec/.glusterfs/25/af/25af9c71-aab5-4f39-bd4a-bb4f0dab9342
: no xattrs
/data/brick/callrec/.glusterfs/0d/16/0d1671b4-e31f-4c81-8600-fe63ffc84272
: no xattrs
/data/brick/callrec/.glusterfs/eb/87/eb87548b-a90a-458d-b215-939ad59f5ec0
: no xattrs
/data/brick/callrec/.glusterfs/9a/2c/9a2c64de-a948-4a45-9466-58a0fb556ba4
: file not found
/data/brick/callrec/.glusterfs/26/be/26bef74c-9675-4c74-96d0-327310cb0983
: file not found
/data/brick/callrec/.glusterfs/9c/62/9c6254df-e8dd-40a3-a7d1-760f6c19027a
: no xattrs
/data/brick/callrec/.glusterfs/b1/13/b1138fe2-34cf-4d0d-af99-3b9d9aec0317
: no xattrs
/data/brick/callrec/.glusterfs/52/2b/522b0117-7d22-4e94-b767-d62a5c914a62
: file not found
/data/brick/callrec/.glusterfs/4a/3e/4a3ed044-845f-438f-8d8a-cac1c0d853f0
: file not found
/data/brick/callrec/.glusterfs/2b/bb/2bbb8529-4874-440d-959b-6a0745fdfda9
: no xattrs
/data/brick/callrec/.glusterfs/94/b5/94b5924f-65b9-45d6-8a03-b7084f8c4bdb
: no xattrs
/data/brick/callrec/.glusterfs/16/fd/16fd1032-3bd5-474a-b30e-85971d69aaa9
: no xattrs
/data/brick/callrec/.glusterfs/ef/bf/efbfc625-c42c-4d5e-a22a-173f84362f24
: no xattrs
/data/brick/callrec/.glusterfs/55/4c/554ce007-a0bd-437b-8004-d70d110b5acc
: file not found
/data/brick/callrec/.glusterfs/7a/d4/7ad41386-9d57-4a93-9fc2-3354e40d9927
: no xattrs
/data/brick/callrec/.glusterfs/32/2c/322c7748-459c-437e-bfba-6f9096a938c5
: file not found
/data/brick/callrec/.glusterfs/f4/8b/f48beb09-975f-4ff6-843e-d8906d3b21b3
: no xattrs
/data/brick/callrec/.glusterfs/5c/f3/5cf36470-8d7e-4e26-904c-39a15d773b43
: file not found
/data/brick/callrec/.glusterfs/75/c6/75c6e98b-7700-46f0-8be4-e897f969a5df
: no xattrs
/data/brick/callrec/.glusterfs/d3/f5/d3f58efb-d1ac-42ab-957c-d7b684bf0972
: file not found
/data/brick/callrec/.glusterfs/8f/3a/8f3a7de0-173a-4507-9b46-bc9db0a6bc41
: no xattrs
/data/brick/callrec/.glusterfs/ca/5b/ca5b027b-dc2f-468f-8cb0-131f1a9099d8
: file not found
/data/brick/callrec/.glusterfs/19/29/19291239-e44c-4025-9f74-af7431aac6b9
: no xattrs
/data/brick/callrec/.glusterfs/28/07/2807b990-2d3c-4f82-b89b-36637cc7c181
: file not found
/data/brick/callrec/.glusterfs/3d/f8/3df80623-6bf4-47e3-a379-4e5605d0eda6
: no xattrs
/data/brick/callrec/.glusterfs/24/72/247283c9-f427-43ce-869a-895e31a9e891
: file not found
/data/brick/callrec/.glusterfs/a4/74/a4745c5f-a3ad-4bed-a504-5ae5b31200a3
: no xattrs
/data/brick/callrec/.glusterfs/d1/4c/d14c8834-a624-4288-8200-5b42fa165043
: file not found
/data/brick/callrec/.glusterfs/53/75/5375e1e8-287f-4fc5-8b01-ec2f856eebf7
: no xattrs
/data/brick/callrec/.glusterfs/fe/81/fe814468-5a2c-469c-89de-9a96b9aacbc1
: file not found
/data/brick/callrec/.glusterfs/50/78/50787eb7-90ad-41fb-bd5e-3ba058b69c32
: no xattrs
/data/brick/callrec/.glusterfs/34/49/3449d319-e2d9-4444-8823-8f7351c1d45f
: file not found
/data/brick/callrec/.glusterfs/9e/a0/9ea01327-d5ed-48bd-8049-cc552555b774
: no xattrs
/data/brick/callrec/.glusterfs/45/ae/45aed765-2ae2-4fc8-9982-8e1b5d6c19d4
: no xattrs
/data/brick/callrec/.glusterfs/70/b7/70b7db21-b70f-4ae0-8660-921c3194f209
: no xattrs
/data/brick/callrec/.glusterfs/9f/ce/9fce8690-a9f7-4cbb-b13a-ba149af652b4
: file not found
/data/brick/callrec/.glusterfs/f1/2b/f12b4d8e-b5e6-4f71-a06c-0c0d067a7eb9
: no xattrs
/data/brick/callrec/.glusterfs/45/5c/455c2911-7a9d-401f-9ded-4380c6cec405
: no xattrs
/data/brick/callrec/.glusterfs/ef/bd/efbd6028-b9f8-45de-a653-b27f75570b81
: no xattrs
/data/brick/callrec/.glusterfs/f7/0b/f70b50fb-3061-4eb9-94f0-85fb2d789d27
: no xattrs
/data/brick/callrec/.glusterfs/33/8b/338baad1-6191-47f3-9737-dca2daf79fd8
: no xattrs
/data/brick/callrec/.glusterfs/a9/e1/a9e1cad9-cd31-48f0-b9f8-322e1f602401
: no xattrs
/data/brick/callrec/.glusterfs/7d/52/7d522a0b-75fa-4b82-a0a2-4a81c98eea03
: file not found
/data/brick/callrec/.glusterfs/1b/a8/1ba8d90a-5f0c-4387-a79a-592427f3d1c5
: no xattrs
/data/brick/callrec/.glusterfs/61/bf/61bf5407-c250-4448-a68f-d3bd82821260
: no xattrs
/data/brick/callrec/.glusterfs/5e/e3/5ee303e4-44fe-44a1-83b5-5cd6a91bc76a
: no xattrs
/data/brick/callrec/.glusterfs/68/12/68127ab7-807d-4a99-a609-6569d366d3aa
: no xattrs
/data/brick/callrec/.glusterfs/a6/cb/a6cb8d18-f2b3-48d4-ac37-c022603b8f8e
: no xattrs
/data/brick/callrec/.glusterfs/4e/86/4e86088b-f975-4228-ad5a-8d08fbd456fe
: no xattrs
/data/brick/callrec/.glusterfs/c4/a2/c4a200b9-bb31-46cb-92c6-551fd6ad9ec3
: no xattrs
/data/brick/callrec/.glusterfs/e3/56/e356342a-3dab-4049-b4bc-ea0de4a8ee87
: no xattrs
/data/brick/callrec/.glusterfs/da/d2/dad27361-d38f-46b3-a77c-b6ac5df054d9
: no xattrs
/data/brick/callrec/.glusterfs/a5/be/a5be4523-2d2c-4894-b332-a7e3370658d3
: file not found
/data/brick/callrec/.glusterfs/b0/f4/b0f4e951-63b3-4152-b0b0-7aad8e6ed729
: no xattrs
/data/brick/callrec/.glusterfs/23/e7/23e75764-9794-4efd-b319-6eac717e6f28
: no xattrs
Where do I go from here?
Cheers,
Kingsley.
On Fri, 2016-07-15 at 17:08 +0530, Ravishankar N wrote:> Can you check the getfattr output of a few of those 129 entries from all
> bricks? You basically need to see if there are non zero afr-xattrs for
> the files in question which would indicate a pending heal.
>
> -Ravi
>
> On 07/08/2016 03:12 PM, Kingsley wrote:
> > Further to this, I've noticed something which might have been a
bit of a
> > red herring in my previous post.
> >
> > We have 3 volumes - gv0, voicemail and callrec. callrec is the only
one
> > showing self heal entries, yet all of the "No such file or
directory"
> > errors in glustershd.log appear to refer to gv0. gv0 has no self heal
> > entries shown by "gluster volume heal gv0 info", and no
split brain
> > entries either.
> >
> > If I de-dupe those log entries, I just get these:
> >
> > [root at gluster1a-1 glusterfs]# grep gfid: glustershd.log | awk -F\]
'{print $3}' | sort | uniq
> > 0-gv0-client-0: remote operation failed: No such file or directory.
Path: <gfid:08713e43-7bcb-43f3-818a-7b062abd6e95>
(08713e43-7bcb-43f3-818a-7b062abd6e95)
> > 0-gv0-client-0: remote operation failed: No such file or directory.
Path: <gfid:436dcbec-a12a-4df9-b8ef-bae977c98537>
(436dcbec-a12a-4df9-b8ef-bae977c98537)
> > 0-gv0-client-0: remote operation failed: No such file or directory.
Path: <gfid:81dc9194-2379-40b5-a949-f7550433b2e0>
(81dc9194-2379-40b5-a949-f7550433b2e0)
> > 0-gv0-client-0: remote operation failed: No such file or directory.
Path: <gfid:b1e273ad-9eb1-4f97-a41c-39eecb149bd6>
(b1e273ad-9eb1-4f97-a41c-39eecb149bd6)
> > 0-gv0-client-1: remote operation failed: No such file or directory.
Path: <gfid:08713e43-7bcb-43f3-818a-7b062abd6e95>
(08713e43-7bcb-43f3-818a-7b062abd6e95)
> > 0-gv0-client-1: remote operation failed: No such file or directory.
Path: <gfid:436dcbec-a12a-4df9-b8ef-bae977c98537>
(436dcbec-a12a-4df9-b8ef-bae977c98537)
> > 0-gv0-client-1: remote operation failed: No such file or directory.
Path: <gfid:81dc9194-2379-40b5-a949-f7550433b2e0>
(81dc9194-2379-40b5-a949-f7550433b2e0)
> > 0-gv0-client-3: remote operation failed: No such file or directory.
Path: <gfid:08713e43-7bcb-43f3-818a-7b062abd6e95>
(08713e43-7bcb-43f3-818a-7b062abd6e95)
> > 0-gv0-client-3: remote operation failed: No such file or directory.
Path: <gfid:81dc9194-2379-40b5-a949-f7550433b2e0>
(81dc9194-2379-40b5-a949-f7550433b2e0)
> >
> >
> > There doesn't seem anything obvious to me in glustershd.log about
the
> > callrec volume. On one of the bricks that stayed up:
> >
> > [root at gluster1a-1 glusterfs]# grep callrec glustershd.log
> > [2016-07-08 08:54:03.424446] I [graph.c:269:gf_add_cmdline_options]
0-callrec-replicate-0: adding option 'node-uuid' for volume
'callrec-replicate-0' with value
'b9d3b1a2-3214-41ba-a1c9-9c7d4b18ff5d'
> > [2016-07-08 08:54:03.429663] I [client.c:2280:notify]
0-callrec-client-0: parent translators are ready, attempting connect on
transport
> > [2016-07-08 08:54:03.432198] I [client.c:2280:notify]
0-callrec-client-1: parent translators are ready, attempting connect on
transport
> > [2016-07-08 08:54:03.434375] I [client.c:2280:notify]
0-callrec-client-2: parent translators are ready, attempting connect on
transport
> > [2016-07-08 08:54:03.436521] I [client.c:2280:notify]
0-callrec-client-3: parent translators are ready, attempting connect on
transport
> > 1: volume callrec-client-0
> > 5: option remote-subvolume /data/brick/callrec
> > 11: volume callrec-client-1
> > 15: option remote-subvolume /data/brick/callrec
> > 21: volume callrec-client-2
> > 25: option remote-subvolume /data/brick/callrec
> > 31: volume callrec-client-3
> > 35: option remote-subvolume /data/brick/callrec
> > 41: volume callrec-replicate-0
> > 50: subvolumes callrec-client-0 callrec-client-1
callrec-client-2 callrec-client-3
> > 159: subvolumes callrec-replicate-0 gv0-replicate-0
voicemail-replicate-0
> > [2016-07-08 08:54:03.458708] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
0-callrec-client-0: changing port to 49153 (from 0)
> > [2016-07-08 08:54:03.465684] I
[client-handshake.c:1413:select_server_supported_programs] 0-callrec-client-0:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2016-07-08 08:54:03.465921] I
[client-handshake.c:1200:client_setvolume_cbk] 0-callrec-client-0: Connected to
callrec-client-0, attached to remote volume '/data/brick/callrec'.
> > [2016-07-08 08:54:03.465927] I
[client-handshake.c:1210:client_setvolume_cbk] 0-callrec-client-0: Server and
Client lk-version numbers are not same, reopening the fds
> > [2016-07-08 08:54:03.465967] I [MSGID: 108005]
[afr-common.c:3669:afr_notify] 0-callrec-replicate-0: Subvolume
'callrec-client-0' came back up; going online.
> > [2016-07-08 08:54:03.466108] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-callrec-client-0: Server lk
version = 1
> > [2016-07-08 08:54:04.266979] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
0-callrec-client-1: changing port to 49153 (from 0)
> > [2016-07-08 08:54:04.732625] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
0-callrec-client-2: changing port to 49153 (from 0)
> > [2016-07-08 08:54:04.738533] I
[client-handshake.c:1413:select_server_supported_programs] 0-callrec-client-2:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2016-07-08 08:54:04.738911] I
[client-handshake.c:1200:client_setvolume_cbk] 0-callrec-client-2: Connected to
callrec-client-2, attached to remote volume '/data/brick/callrec'.
> > [2016-07-08 08:54:04.738921] I
[client-handshake.c:1210:client_setvolume_cbk] 0-callrec-client-2: Server and
Client lk-version numbers are not same, reopening the fds
> > [2016-07-08 08:54:04.739181] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-callrec-client-2: Server lk
version = 1
> > [2016-07-08 08:54:05.271388] I
[client-handshake.c:1413:select_server_supported_programs] 0-callrec-client-1:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2016-07-08 08:54:05.271858] I
[client-handshake.c:1200:client_setvolume_cbk] 0-callrec-client-1: Connected to
callrec-client-1, attached to remote volume '/data/brick/callrec'.
> > [2016-07-08 08:54:05.271879] I
[client-handshake.c:1210:client_setvolume_cbk] 0-callrec-client-1: Server and
Client lk-version numbers are not same, reopening the fds
> > [2016-07-08 08:54:05.272185] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-callrec-client-1: Server lk
version = 1
> > [2016-07-08 08:54:06.302301] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
0-callrec-client-3: changing port to 49153 (from 0)
> > [2016-07-08 08:54:06.305473] I
[client-handshake.c:1413:select_server_supported_programs] 0-callrec-client-3:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2016-07-08 08:54:06.305915] I
[client-handshake.c:1200:client_setvolume_cbk] 0-callrec-client-3: Connected to
callrec-client-3, attached to remote volume '/data/brick/callrec'.
> > [2016-07-08 08:54:06.305925] I
[client-handshake.c:1210:client_setvolume_cbk] 0-callrec-client-3: Server and
Client lk-version numbers are not same, reopening the fds
> > [2016-07-08 08:54:06.306307] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-callrec-client-3: Server lk
version = 1
> >
> >
> > And on the brick that went offline for a few days:
> >
> > [root at gluster2a-1 glusterfs]# grep callrec glustershd.log
> > [2016-07-08 08:54:06.900964] I [graph.c:269:gf_add_cmdline_options]
0-callrec-replicate-0: adding option 'node-uuid' for volume
'callrec-replicate-0' with value
'e96ae8cd-f38f-4c2a-bb3b-baeb78f88f13'
> > [2016-07-08 08:54:06.906449] I [client.c:2280:notify]
0-callrec-client-0: parent translators are ready, attempting connect on
transport
> > [2016-07-08 08:54:06.908851] I [client.c:2280:notify]
0-callrec-client-1: parent translators are ready, attempting connect on
transport
> > [2016-07-08 08:54:06.911045] I [client.c:2280:notify]
0-callrec-client-2: parent translators are ready, attempting connect on
transport
> > [2016-07-08 08:54:06.913528] I [client.c:2280:notify]
0-callrec-client-3: parent translators are ready, attempting connect on
transport
> > 1: volume callrec-client-0
> > 5: option remote-subvolume /data/brick/callrec
> > 11: volume callrec-client-1
> > 15: option remote-subvolume /data/brick/callrec
> > 21: volume callrec-client-2
> > 25: option remote-subvolume /data/brick/callrec
> > 31: volume callrec-client-3
> > 35: option remote-subvolume /data/brick/callrec
> > 41: volume callrec-replicate-0
> > 50: subvolumes callrec-client-0 callrec-client-1
callrec-client-2 callrec-client-3
> > 159: subvolumes callrec-replicate-0 gv0-replicate-0
voicemail-replicate-0
> > [2016-07-08 08:54:06.938769] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
0-callrec-client-2: changing port to 49153 (from 0)
> > [2016-07-08 08:54:06.948204] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
0-callrec-client-1: changing port to 49153 (from 0)
> > [2016-07-08 08:54:06.951625] I
[client-handshake.c:1413:select_server_supported_programs] 0-callrec-client-2:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2016-07-08 08:54:06.951849] I
[client-handshake.c:1200:client_setvolume_cbk] 0-callrec-client-2: Connected to
callrec-client-2, attached to remote volume '/data/brick/callrec'.
> > [2016-07-08 08:54:06.951858] I
[client-handshake.c:1210:client_setvolume_cbk] 0-callrec-client-2: Server and
Client lk-version numbers are not same, reopening the fds
> > [2016-07-08 08:54:06.951906] I [MSGID: 108005]
[afr-common.c:3669:afr_notify] 0-callrec-replicate-0: Subvolume
'callrec-client-2' came back up; going online.
> > [2016-07-08 08:54:06.951938] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-callrec-client-2: Server lk
version = 1
> > [2016-07-08 08:54:07.152217] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
0-callrec-client-3: changing port to 49153 (from 0)
> > [2016-07-08 08:54:07.167137] I
[client-handshake.c:1413:select_server_supported_programs] 0-callrec-client-1:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2016-07-08 08:54:07.167474] I
[client-handshake.c:1200:client_setvolume_cbk] 0-callrec-client-1: Connected to
callrec-client-1, attached to remote volume '/data/brick/callrec'.
> > [2016-07-08 08:54:07.167483] I
[client-handshake.c:1210:client_setvolume_cbk] 0-callrec-client-1: Server and
Client lk-version numbers are not same, reopening the fds
> > [2016-07-08 08:54:07.167664] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-callrec-client-1: Server lk
version = 1
> > [2016-07-08 08:54:07.240249] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
0-callrec-client-0: changing port to 49153 (from 0)
> > [2016-07-08 08:54:07.243156] I
[client-handshake.c:1413:select_server_supported_programs] 0-callrec-client-0:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2016-07-08 08:54:07.243512] I
[client-handshake.c:1200:client_setvolume_cbk] 0-callrec-client-0: Connected to
callrec-client-0, attached to remote volume '/data/brick/callrec'.
> > [2016-07-08 08:54:07.243520] I
[client-handshake.c:1210:client_setvolume_cbk] 0-callrec-client-0: Server and
Client lk-version numbers are not same, reopening the fds
> > [2016-07-08 08:54:07.243804] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-callrec-client-0: Server lk
version = 1
> > [2016-07-08 08:54:07.400188] I
[client-handshake.c:1413:select_server_supported_programs] 0-callrec-client-3:
Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2016-07-08 08:54:07.400574] I
[client-handshake.c:1200:client_setvolume_cbk] 0-callrec-client-3: Connected to
callrec-client-3, attached to remote volume '/data/brick/callrec'.
> > [2016-07-08 08:54:07.400583] I
[client-handshake.c:1210:client_setvolume_cbk] 0-callrec-client-3: Server and
Client lk-version numbers are not same, reopening the fds
> > [2016-07-08 08:54:07.400802] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-callrec-client-3: Server lk
version = 1
> >
> > Cheers,
> > Kingsley.
> >
> > On Fri, 2016-07-08 at 10:08 +0100, Kingsley wrote:
> >> Hi,
> >>
> >> One of our bricks was offline for a few days when it didn't
reboot after
> >> a yum update (the gluster version wasn't changed). The volume
heal info
> >> is showing the same 129 entries, all of the format
> >> <gfid:08713e43-7bcb-43f3-818a-7b062abd6e95> on the 3 bricks
that
> >> remained up, and no entries on the brick that was offline.
> >>
> >> glustershd.log on the brick that was offline has stuff like this
in it:
> >>
> >> [2016-07-08 08:54:07.411486] I
[client-handshake.c:1200:client_setvolume_cbk] 0-gv0-client-1: Connected to
gv0-client-1, attached to remote volume '/data/brick/gv0'.
> >> [2016-07-08 08:54:07.411493] I
[client-handshake.c:1210:client_setvolume_cbk] 0-gv0-client-1: Server and Client
lk-version numbers are not same, reopening the fds
> >> [2016-07-08 08:54:07.411678] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-gv0-client-1: Server lk
version = 1
> >> [2016-07-08 08:54:07.793661] I
[client-handshake.c:1200:client_setvolume_cbk] 0-gv0-client-3: Connected to
gv0-client-3, attached to remote volume '/data/brick/gv0'.
> >> [2016-07-08 08:54:07.793688] I
[client-handshake.c:1210:client_setvolume_cbk] 0-gv0-client-3: Server and Client
lk-version numbers are not same, reopening the fds
> >> [2016-07-08 08:54:07.794091] I
[client-handshake.c:188:client_set_lk_version_cbk] 0-gv0-client-3: Server lk
version = 1
> >>
> >> but glustershd.log on the other 3 bricks has many lines looking
like
> >> this:
> >>
> >> [2016-07-08 09:05:17.203017] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-gv0-client-3: remote operation
failed: No such file or directory. Path:
<gfid:81dc9194-2379-40b5-a949-f7550433b2e0>
(81dc9194-2379-40b5-a949-f7550433b2e0)
> >> [2016-07-08 09:05:17.203405] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-gv0-client-0: remote operation
failed: No such file or directory. Path:
<gfid:b1e273ad-9eb1-4f97-a41c-39eecb149bd6>
(b1e273ad-9eb1-4f97-a41c-39eecb149bd6)
> >> [2016-07-08 09:05:17.204035] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-gv0-client-0: remote operation
failed: No such file or directory. Path:
<gfid:436dcbec-a12a-4df9-b8ef-bae977c98537>
(436dcbec-a12a-4df9-b8ef-bae977c98537)
> >> [2016-07-08 09:05:17.204225] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-gv0-client-1: remote operation
failed: No such file or directory. Path:
<gfid:436dcbec-a12a-4df9-b8ef-bae977c98537>
(436dcbec-a12a-4df9-b8ef-bae977c98537)
> >> [2016-07-08 09:05:17.204651] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-gv0-client-0: remote operation
failed: No such file or directory. Path:
<gfid:08713e43-7bcb-43f3-818a-7b062abd6e95>
(08713e43-7bcb-43f3-818a-7b062abd6e95)
> >> [2016-07-08 09:05:17.204879] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-gv0-client-1: remote operation
failed: No such file or directory. Path:
<gfid:08713e43-7bcb-43f3-818a-7b062abd6e95>
(08713e43-7bcb-43f3-818a-7b062abd6e95)
> >> [2016-07-08 09:05:17.205042] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-gv0-client-3: remote operation
failed: No such file or directory. Path:
<gfid:08713e43-7bcb-43f3-818a-7b062abd6e95>
(08713e43-7bcb-43f3-818a-7b062abd6e95)
> >>
> >> How do I fix this? I need to update the other bricks but am
reluctant to
> >> do so until the volume is in good shape first.
> >>
> >> We're running Gluster 3.6.3 on CentOS 7. Volume info:
> >>
> >> Volume Name: callrec
> >> Type: Replicate
> >> Volume ID: a39830b7-eddb-4061-b381-39411274131a
> >> Status: Started
> >> Number of Bricks: 1 x 4 = 4
> >> Transport-type: tcp
> >> Bricks:
> >> Brick1: gluster1a-1:/data/brick/callrec
> >> Brick2: gluster1b-1:/data/brick/callrec
> >> Brick3: gluster2a-1:/data/brick/callrec
> >> Brick4: gluster2b-1:/data/brick/callrec
> >> Options Reconfigured:
> >> performance.flush-behind: off
> >>
> >>
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>