Gluster users - Jun 2017 - afr-self-heald.c:479:afr_shd_index_sweep

Hi list,

yesterday I noted the following lines into the glustershd.log log file:

[2017-06-28 11:53:05.000890] W [MSGID: 108034]
[afr-self-heald.c:479:afr_shd_index_sweep]
0-iso-images-repo-replicate-0: unable to get index-dir on
iso-images-repo-client-0
[2017-06-28 11:53:05.001146] W [MSGID: 108034]
[afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-0:
unable to get index-dir on vm-images-repo-client-0
[2017-06-28 11:53:06.001141] W [MSGID: 108034]
[afr-self-heald.c:479:afr_shd_index_sweep] 0-hosted-engine-replicate-0:
unable to get index-dir on hosted-engine-client-0
[2017-06-28 11:53:08.001094] W [MSGID: 108034]
[afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-2:
unable to get index-dir on vm-images-repo-client-6
[2017-06-28 11:53:08.001170] W [MSGID: 108034]
[afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-1:
unable to get index-dir on vm-images-repo-client-3

Digging into the mailing list archive I've found another user with a
similar issue (the thread was '[Gluster-users] glustershd: unable to get
index-dir on myvolume-client-0'), the solution suggested was to verify
if the  /<path-to-backend-brick>/.glusterfs/indices directory contains
all these sub directories: 'dirty', 'entry-changes' and
'xattrop' and if
some of them does not exists simply create it with mkdir.

In my case the 'entry-changes' directory is not present on all the
bricks and on all the servers:

/data/glusterfs/brick1a/hosted-engine/.glusterfs/indices/:
total 0
drw------- 2 root root 55 Jun 28 15:02 dirty
drw------- 2 root root 57 Jun 28 15:02 xattrop

/data/glusterfs/brick1b/iso-images-repo/.glusterfs/indices/:
total 0
drw------- 2 root root 55 May 29 14:04 dirty
drw------- 2 root root 57 May 29 14:04 xattrop

/data/glusterfs/brick2/vm-images-repo/.glusterfs/indices/:
total 0
drw------- 2 root root 112 Jun 28 15:02 dirty
drw------- 2 root root  66 Jun 28 15:02 xattrop

/data/glusterfs/brick3/vm-images-repo/.glusterfs/indices/:
total 0
drw------- 2 root root 64 Jun 28 15:02 dirty
drw------- 2 root root 66 Jun 28 15:02 xattrop

/data/glusterfs/brick4/vm-images-repo/.glusterfs/indices/:
total 0
drw------- 2 root root 112 Jun 28 15:02 dirty
drw------- 2 root root  66 Jun 28 15:02 xattrop

I've recently upgraded gluster from 3.7.16 to 3.8.12 with the rolling
upgrade procedure and I haven't noted this issue prior of the update, on
another system upgraded with the same procedure I haven't encountered
this problem.

Currently all VM images appear to be OK but prior to create the
'entry-changes' I would like to ask if this is still the correct
procedure to fix this issue and if this problem could have affected the
heal operations occurred meanwhile.

Thanks.


Greetings,

    Paolo Margara

hi Paolo,
   I just checked code in v3.8.12 and it should have been created when the
brick starts after you upgrade the node. How did you do the upgrade?

On Wed, Jun 28, 2017 at 6:52 PM, Paolo Margara <paolo.margara at
polito.it>
wrote:
> Hi list,
>
> yesterday I noted the following lines into the glustershd.log log file:
>
> [2017-06-28 11:53:05.000890] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep]
> 0-iso-images-repo-replicate-0: unable to get index-dir on
> iso-images-repo-client-0
> [2017-06-28 11:53:05.001146] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-0:
> unable to get index-dir on vm-images-repo-client-0
> [2017-06-28 11:53:06.001141] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep] 0-hosted-engine-replicate-0:
> unable to get index-dir on hosted-engine-client-0
> [2017-06-28 11:53:08.001094] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-2:
> unable to get index-dir on vm-images-repo-client-6
> [2017-06-28 11:53:08.001170] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-1:
> unable to get index-dir on vm-images-repo-client-3
>
> Digging into the mailing list archive I've found another user with a
> similar issue (the thread was '[Gluster-users] glustershd: unable to
get
> index-dir on myvolume-client-0'), the solution suggested was to verify
> if the  /<path-to-backend-brick>/.glusterfs/indices directory
contains
> all these sub directories: 'dirty', 'entry-changes' and
'xattrop' and if
> some of them does not exists simply create it with mkdir.
>
> In my case the 'entry-changes' directory is not present on all the
> bricks and on all the servers:
>
> /data/glusterfs/brick1a/hosted-engine/.glusterfs/indices/:
> total 0
> drw------- 2 root root 55 Jun 28 15:02 dirty
> drw------- 2 root root 57 Jun 28 15:02 xattrop
>
> /data/glusterfs/brick1b/iso-images-repo/.glusterfs/indices/:
> total 0
> drw------- 2 root root 55 May 29 14:04 dirty
> drw------- 2 root root 57 May 29 14:04 xattrop
>
> /data/glusterfs/brick2/vm-images-repo/.glusterfs/indices/:
> total 0
> drw------- 2 root root 112 Jun 28 15:02 dirty
> drw------- 2 root root  66 Jun 28 15:02 xattrop
>
> /data/glusterfs/brick3/vm-images-repo/.glusterfs/indices/:
> total 0
> drw------- 2 root root 64 Jun 28 15:02 dirty
> drw------- 2 root root 66 Jun 28 15:02 xattrop
>
> /data/glusterfs/brick4/vm-images-repo/.glusterfs/indices/:
> total 0
> drw------- 2 root root 112 Jun 28 15:02 dirty
> drw------- 2 root root  66 Jun 28 15:02 xattrop
>
> I've recently upgraded gluster from 3.7.16 to 3.8.12 with the rolling
> upgrade procedure and I haven't noted this issue prior of the update,
on
> another system upgraded with the same procedure I haven't encountered
> this problem.
>
> Currently all VM images appear to be OK but prior to create the
> 'entry-changes' I would like to ask if this is still the correct
> procedure to fix this issue and if this problem could have affected the
> heal operations occurred meanwhile.
>
> Thanks.
>
>
> Greetings,
>
>     Paolo Margara
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170628/3703d726/attachment.html>

On 06/28/2017 06:52 PM, Paolo Margara wrote:
> Hi list,
>
> yesterday I noted the following lines into the glustershd.log log file:
>
> [2017-06-28 11:53:05.000890] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep]
> 0-iso-images-repo-replicate-0: unable to get index-dir on
> iso-images-repo-client-0
> [2017-06-28 11:53:05.001146] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-0:
> unable to get index-dir on vm-images-repo-client-0
> [2017-06-28 11:53:06.001141] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep] 0-hosted-engine-replicate-0:
> unable to get index-dir on hosted-engine-client-0
> [2017-06-28 11:53:08.001094] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-2:
> unable to get index-dir on vm-images-repo-client-6
> [2017-06-28 11:53:08.001170] W [MSGID: 108034]
> [afr-self-heald.c:479:afr_shd_index_sweep] 0-vm-images-repo-replicate-1:
> unable to get index-dir on vm-images-repo-client-3
>
> Digging into the mailing list archive I've found another user with a
> similar issue (the thread was '[Gluster-users] glustershd: unable to
get
> index-dir on myvolume-client-0'), the solution suggested was to verify
> if the  /<path-to-backend-brick>/.glusterfs/indices directory
contains
> all these sub directories: 'dirty', 'entry-changes' and
'xattrop' and if
> some of them does not exists simply create it with mkdir.
>
> In my case the 'entry-changes' directory is not present on all the
> bricks and on all the servers:
>
> /data/glusterfs/brick1a/hosted-engine/.glusterfs/indices/:
> total 0
> drw------- 2 root root 55 Jun 28 15:02 dirty
> drw------- 2 root root 57 Jun 28 15:02 xattrop
>
> /data/glusterfs/brick1b/iso-images-repo/.glusterfs/indices/:
> total 0
> drw------- 2 root root 55 May 29 14:04 dirty
> drw------- 2 root root 57 May 29 14:04 xattrop
>
> /data/glusterfs/brick2/vm-images-repo/.glusterfs/indices/:
> total 0
> drw------- 2 root root 112 Jun 28 15:02 dirty
> drw------- 2 root root  66 Jun 28 15:02 xattrop
>
> /data/glusterfs/brick3/vm-images-repo/.glusterfs/indices/:
> total 0
> drw------- 2 root root 64 Jun 28 15:02 dirty
> drw------- 2 root root 66 Jun 28 15:02 xattrop
>
> /data/glusterfs/brick4/vm-images-repo/.glusterfs/indices/:
> total 0
> drw------- 2 root root 112 Jun 28 15:02 dirty
> drw------- 2 root root  66 Jun 28 15:02 xattrop
>
> I've recently upgraded gluster from 3.7.16 to 3.8.12 with the rolling
> upgrade procedure and I haven't noted this issue prior of the update,
on
> another system upgraded with the same procedure I haven't encountered
> this problem.
>
> Currently all VM images appear to be OK but prior to create the
> 'entry-changes' I would like to ask if this is still the correct
> procedure to fix this issue
Did you restart the bricks after the upgrade? That should have created 
the entry-changes directory. Can you kill the brick and restart it and 
see if the dir is created? Double check from the brick logs that you're 
indeed running 3.12:  "Started running /usr/local/sbin/glusterfsd 
version 3.8.12" should appear when the brick starts.

-Ravi
>   and if this problem could have affected the
> heal operations occurred meanwhile.
>
> Thanks.
>
>
> Greetings,
>
>      Paolo Margara
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

On Wed, Jun 28, 2017 at 9:45 PM, Ravishankar N <ravishankar at redhat.com>
wrote:
> On 06/28/2017 06:52 PM, Paolo Margara wrote:
>
>> Hi list,
>>
>> yesterday I noted the following lines into the glustershd.log log file:
>>
>> [2017-06-28 11:53:05.000890] W [MSGID: 108034]
>> [afr-self-heald.c:479:afr_shd_index_sweep]
>> 0-iso-images-repo-replicate-0: unable to get index-dir on
>> iso-images-repo-client-0
>> [2017-06-28 11:53:05.001146] W [MSGID: 108034]
>> [afr-self-heald.c:479:afr_shd_index_sweep]
0-vm-images-repo-replicate-0:
>> unable to get index-dir on vm-images-repo-client-0
>> [2017-06-28 11:53:06.001141] W [MSGID: 108034]
>> [afr-self-heald.c:479:afr_shd_index_sweep] 0-hosted-engine-replicate-0:
>> unable to get index-dir on hosted-engine-client-0
>> [2017-06-28 11:53:08.001094] W [MSGID: 108034]
>> [afr-self-heald.c:479:afr_shd_index_sweep]
0-vm-images-repo-replicate-2:
>> unable to get index-dir on vm-images-repo-client-6
>> [2017-06-28 11:53:08.001170] W [MSGID: 108034]
>> [afr-self-heald.c:479:afr_shd_index_sweep]
0-vm-images-repo-replicate-1:
>> unable to get index-dir on vm-images-repo-client-3
>>
>> Digging into the mailing list archive I've found another user with
a
>> similar issue (the thread was '[Gluster-users] glustershd: unable
to get
>> index-dir on myvolume-client-0'), the solution suggested was to
verify
>> if the  /<path-to-backend-brick>/.glusterfs/indices directory
contains
>> all these sub directories: 'dirty', 'entry-changes' and
'xattrop' and if
>> some of them does not exists simply create it with mkdir.
>>
>> In my case the 'entry-changes' directory is not present on all
the
>> bricks and on all the servers:
>>
>> /data/glusterfs/brick1a/hosted-engine/.glusterfs/indices/:
>> total 0
>> drw------- 2 root root 55 Jun 28 15:02 dirty
>> drw------- 2 root root 57 Jun 28 15:02 xattrop
>>
>> /data/glusterfs/brick1b/iso-images-repo/.glusterfs/indices/:
>> total 0
>> drw------- 2 root root 55 May 29 14:04 dirty
>> drw------- 2 root root 57 May 29 14:04 xattrop
>>
>> /data/glusterfs/brick2/vm-images-repo/.glusterfs/indices/:
>> total 0
>> drw------- 2 root root 112 Jun 28 15:02 dirty
>> drw------- 2 root root  66 Jun 28 15:02 xattrop
>>
>> /data/glusterfs/brick3/vm-images-repo/.glusterfs/indices/:
>> total 0
>> drw------- 2 root root 64 Jun 28 15:02 dirty
>> drw------- 2 root root 66 Jun 28 15:02 xattrop
>>
>> /data/glusterfs/brick4/vm-images-repo/.glusterfs/indices/:
>> total 0
>> drw------- 2 root root 112 Jun 28 15:02 dirty
>> drw------- 2 root root  66 Jun 28 15:02 xattrop
>>
>> I've recently upgraded gluster from 3.7.16 to 3.8.12 with the
rolling
>> upgrade procedure and I haven't noted this issue prior of the
update, on
>> another system upgraded with the same procedure I haven't
encountered
>> this problem.
>>
>> Currently all VM images appear to be OK but prior to create the
>> 'entry-changes' I would like to ask if this is still the
correct
>> procedure to fix this issue
>>
>
> Did you restart the bricks after the upgrade? That should have created the
> entry-changes directory. Can you kill the brick and restart it and see if
> the dir is created? Double check from the brick logs that you're indeed
> running 3.12:  "Started running /usr/local/sbin/glusterfsd version
3.8.12"
> should appear when the brick starts.
>
Please note that if you are going the route of killing and restarting, you
need to do it in the same way you did rolling upgrade. You need to wait for
heal to complete before you kill the other nodes. But before you do this,
it is better you look at the logs or confirm the steps you used for doing
upgrade.

>
> -Ravi
>
>
>   and if this problem could have affected the
>> heal operations occurred meanwhile.
>>
>> Thanks.
>>
>>
>> Greetings,
>>
>>      Paolo Margara
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170628/c2150da1/attachment.html>