thr3ads.net - Gluster users - [Gluster-users] One error/warning message after upgrade 5.11 -> 6.8 [Jun 2020]

If this information is useful, please help other people find it:
Share via:

Strahil Nikolov

2020-May-08 19:46 UTC

[Gluster-users] One error/warning message after upgrade 5.11 -> 6.8

On April 21, 2020 8:00:32 PM GMT+03:00, Amar Tumballi <amar at kadalu.io>
wrote:>There seems to be a burst of issues when people upgraded to 5.x or 6.x
>from
>3.12 (Thanks to you and Strahil, who have reported most of them).
>
>Latest update from Strahil is that if files are copied fresh on 7.5
>series,
>there are no issues.
>
>We are in process of identifying the patch, and also provide an option
>to
>disable 'acl' for testing. Will update once we identify the issue.
>
>Regards,
>Amar
>
>
>
>On Sat, Apr 11, 2020 at 11:10 AM Hu Bert <revirii at googlemail.com>
>wrote:
>
>> Hi,
>>
>> no one has seen such messages?
>>
>> Regards,
>> Hubert
>>
>> Am Mo., 6. Apr. 2020 um 06:13 Uhr schrieb Hu Bert
><revirii at googlemail.com
>> >:
>> >
>> > Hello,
>> >
>> > i just upgraded my servers and clients from 5.11 to 6.8; besides
>one
>> > connection problem to the gluster download server everything went
>> > fine.
>> >
>> > On the 3 gluster servers i mount the 2 volumes as well, and only
>there
>> > (and not on all the other clients) there are some messages in the
>log
>> > file of both mount logs:
>> >
>> > [2020-04-06 04:10:53.552561] W [MSGID: 114031]
>> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
>> > 0-persistent-client-2: remote operation failed [Permission denied]
>> > [2020-04-06 04:10:53.552635] W [MSGID: 114031]
>> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
>> > 0-persistent-client-1: remote operation failed [Permission denied]
>> > [2020-04-06 04:10:53.552639] W [MSGID: 114031]
>> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
>> > 0-persistent-client-0: remote operation failed [Permission denied]
>> > [2020-04-06 04:10:53.553226] E [MSGID: 148002]
>> > [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-persistent-utime:
>dict
>> > set of key for set-ctime-mdata failed [Permission denied]
>> > The message "W [MSGID: 114031]
>> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
>> > 0-persistent-client-2: remote operation failed [Permission
denied]"
>> > repeated 4 times between [2020-04-06 04:10:53.552561] and
>[2020-04-06
>> > 04:10:53.745542]
>> > The message "W [MSGID: 114031]
>> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
>> > 0-persistent-client-1: remote operation failed [Permission
denied]"
>> > repeated 4 times between [2020-04-06 04:10:53.552635] and
>[2020-04-06
>> > 04:10:53.745610]
>> > The message "W [MSGID: 114031]
>> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
>> > 0-persistent-client-0: remote operation failed [Permission
denied]"
>> > repeated 4 times between [2020-04-06 04:10:53.552639] and
>[2020-04-06
>> > 04:10:53.745632]
>> > The message "E [MSGID: 148002]
>> > [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-persistent-utime:
>dict
>> > set of key for set-ctime-mdata failed [Permission denied]"
repeated
>4
>> > times between [2020-04-06 04:10:53.553226] and [2020-04-06
>> > 04:10:53.746080]
>> >
>> > Anything to worry about?
>> >
>> >
>> > Regards,
>> > Hubert
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
Hi, 

Can you provide the xfs_info for the bricks from the volume ?

I have a theory that I want to confirm or reject.

Best Regards,
Strahil Nikolov

Hu Bert

2020-May-11 04:59 UTC

head link

[Gluster-users] One error/warning message after upgrade 5.11 -> 6.8

Hi,

no problem. Output is identical for all 3 bricks (1 x 3 = 3)

:xfs_info /gluster/md4
meta-data=/dev/md4               isize=512    agcount=32, agsize=152598400 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=4883148800, imaxpct=5
         =                       sunit=128    swidth=256 blks
naming   =version 2              bsize=8192   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Best Regards,
Hubert

Am Fr., 8. Mai 2020 um 21:47 Uhr schrieb Strahil Nikolov
<hunter86_bg at yahoo.com>:>
> On April 21, 2020 8:00:32 PM GMT+03:00, Amar Tumballi <amar at
kadalu.io> wrote:
> >There seems to be a burst of issues when people upgraded to 5.x or 6.x
> >from
> >3.12 (Thanks to you and Strahil, who have reported most of them).
> >
> >Latest update from Strahil is that if files are copied fresh on 7.5
> >series,
> >there are no issues.
> >
> >We are in process of identifying the patch, and also provide an option
> >to
> >disable 'acl' for testing. Will update once we identify the
issue.
> >
> >Regards,
> >Amar
> >
> >
> >
> >On Sat, Apr 11, 2020 at 11:10 AM Hu Bert <revirii at
googlemail.com>
> >wrote:
> >
> >> Hi,
> >>
> >> no one has seen such messages?
> >>
> >> Regards,
> >> Hubert
> >>
> >> Am Mo., 6. Apr. 2020 um 06:13 Uhr schrieb Hu Bert
> ><revirii at googlemail.com
> >> >:
> >> >
> >> > Hello,
> >> >
> >> > i just upgraded my servers and clients from 5.11 to 6.8;
besides
> >one
> >> > connection problem to the gluster download server everything
went
> >> > fine.
> >> >
> >> > On the 3 gluster servers i mount the 2 volumes as well, and
only
> >there
> >> > (and not on all the other clients) there are some messages in
the
> >log
> >> > file of both mount logs:
> >> >
> >> > [2020-04-06 04:10:53.552561] W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-2: remote operation failed [Permission
denied]
> >> > [2020-04-06 04:10:53.552635] W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-1: remote operation failed [Permission
denied]
> >> > [2020-04-06 04:10:53.552639] W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-0: remote operation failed [Permission
denied]
> >> > [2020-04-06 04:10:53.553226] E [MSGID: 148002]
> >> > [utime.c:146:gf_utime_set_mdata_setxattr_cbk]
0-persistent-utime:
> >dict
> >> > set of key for set-ctime-mdata failed [Permission denied]
> >> > The message "W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-2: remote operation failed [Permission
denied]"
> >> > repeated 4 times between [2020-04-06 04:10:53.552561] and
> >[2020-04-06
> >> > 04:10:53.745542]
> >> > The message "W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-1: remote operation failed [Permission
denied]"
> >> > repeated 4 times between [2020-04-06 04:10:53.552635] and
> >[2020-04-06
> >> > 04:10:53.745610]
> >> > The message "W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-0: remote operation failed [Permission
denied]"
> >> > repeated 4 times between [2020-04-06 04:10:53.552639] and
> >[2020-04-06
> >> > 04:10:53.745632]
> >> > The message "E [MSGID: 148002]
> >> > [utime.c:146:gf_utime_set_mdata_setxattr_cbk]
0-persistent-utime:
> >dict
> >> > set of key for set-ctime-mdata failed [Permission
denied]" repeated
> >4
> >> > times between [2020-04-06 04:10:53.553226] and [2020-04-06
> >> > 04:10:53.746080]
> >> >
> >> > Anything to worry about?
> >> >
> >> >
> >> > Regards,
> >> > Hubert
> >> ________
> >>
> >>
> >>
> >> Community Meeting Calendar:
> >>
> >> Schedule -
> >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >> Bridge: https://bluejeans.com/441850968
> >>
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org
> >> https://lists.gluster.org/mailman/listinfo/gluster-users
> >>
>
> Hi,
>
> Can you provide the xfs_info for the bricks from the volume ?
>
> I have a theory that I want to confirm or reject.
>
> Best Regards,
> Strahil Nikolov

Hu Bert

2020-Jun-08 05:30 UTC

head link

[Gluster-users] One error/warning message after upgrade 5.11 -> 6.8

Good morning,

i just wanted to update the version from 6.8 to 6.9 on our replicate 3
system (formerly was version 5.11), and i see tons of these messages:

[2020-06-08 05:25:55.192301] E [MSGID: 113001]
[posix-metadata.c:234:posix_store_mdata_xattr] 0-persistent-posix:
file:
/gluster/md3/persistent/.glusterfs/43/31/43312aba-75c6-42c2-855c-e0db66d7748f:
gfid: 43312aba-75c6-42c2-855c-e0db66d7748f key:trusted.glusterfs.mdata
 [No such file or directory]
[2020-06-08 05:25:55.192375] E [MSGID: 113114]
[posix-metadata.c:433:posix_set_mdata_xattr_legacy_files]
0-persistent-posix: gfid: 43312aba-75c6-42c2-855c-e0db66d7748f
key:trusted.glusterfs.mdata  [No such file or directory]
[2020-06-08 05:25:55.192426] I [MSGID: 115060]
[server-rpc-fops.c:938:_gf_server_log_setxattr_failure]
0-persistent-server: 13382741: SETXATTR
<gfid:43312aba-75c6-42c2-855c-e0db66d7748f>
(43312aba-75c6-42c2-855c-e0db66d7748f) ==> set-ctime-mdata, client:
CTX_ID:e223ca30-6c30-4a40-ae98-a418143ce548-GRAPH_ID:0-PID:1006-HOST:sam-PC_NAME:persistent-client-2-RECON_NO:-1,
error-xlator: persistent-posix

Still the ctime-message. And a lot of these messages:

[2020-06-08 05:25:53.016606] W [MSGID: 101159]
[inode.c:1330:__inode_unlink] 0-inode:
7043eed7-dbd7-4277-976f-d467349c1361/21194684.jpg: dentry not found in
839512f0-75de-414f-993d-1c35892f8560

Well... the problem is: the volume seems to be in a permanent heal status:

Gathering count of entries to be healed on volume persistent has been successful
Brick gluster1:/gluster/md3/persistent
Number of entries: 31
Brick gluster2:/gluster/md3/persistent
Number of entries: 6
Brick gluster3:/gluster/md3/persistent
Number of entries: 5

a bit later:
Gathering count of entries to be healed on volume persistent has been successful
Brick gluster1:/gluster/md3/persistent
Number of entries: 100
Brick gluster2:/gluster/md3/persistent
Number of entries: 74
Brick gluster3:/gluster/md3/persistent
Number of entries: 1

The number of entries never reach 0-0-0; i already updated one of the
systems from 6.8 to 6.9, but updating the other 2 when heal isn't zero
doesn't seem to be a good idea. Well... any idea?


Best regards,
Hubert

Am Fr., 8. Mai 2020 um 21:47 Uhr schrieb Strahil Nikolov
<hunter86_bg at yahoo.com>:>
> On April 21, 2020 8:00:32 PM GMT+03:00, Amar Tumballi <amar at
kadalu.io> wrote:
> >There seems to be a burst of issues when people upgraded to 5.x or 6.x
> >from
> >3.12 (Thanks to you and Strahil, who have reported most of them).
> >
> >Latest update from Strahil is that if files are copied fresh on 7.5
> >series,
> >there are no issues.
> >
> >We are in process of identifying the patch, and also provide an option
> >to
> >disable 'acl' for testing. Will update once we identify the
issue.
> >
> >Regards,
> >Amar
> >
> >
> >
> >On Sat, Apr 11, 2020 at 11:10 AM Hu Bert <revirii at
googlemail.com>
> >wrote:
> >
> >> Hi,
> >>
> >> no one has seen such messages?
> >>
> >> Regards,
> >> Hubert
> >>
> >> Am Mo., 6. Apr. 2020 um 06:13 Uhr schrieb Hu Bert
> ><revirii at googlemail.com
> >> >:
> >> >
> >> > Hello,
> >> >
> >> > i just upgraded my servers and clients from 5.11 to 6.8;
besides
> >one
> >> > connection problem to the gluster download server everything
went
> >> > fine.
> >> >
> >> > On the 3 gluster servers i mount the 2 volumes as well, and
only
> >there
> >> > (and not on all the other clients) there are some messages in
the
> >log
> >> > file of both mount logs:
> >> >
> >> > [2020-04-06 04:10:53.552561] W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-2: remote operation failed [Permission
denied]
> >> > [2020-04-06 04:10:53.552635] W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-1: remote operation failed [Permission
denied]
> >> > [2020-04-06 04:10:53.552639] W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-0: remote operation failed [Permission
denied]
> >> > [2020-04-06 04:10:53.553226] E [MSGID: 148002]
> >> > [utime.c:146:gf_utime_set_mdata_setxattr_cbk]
0-persistent-utime:
> >dict
> >> > set of key for set-ctime-mdata failed [Permission denied]
> >> > The message "W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-2: remote operation failed [Permission
denied]"
> >> > repeated 4 times between [2020-04-06 04:10:53.552561] and
> >[2020-04-06
> >> > 04:10:53.745542]
> >> > The message "W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-1: remote operation failed [Permission
denied]"
> >> > repeated 4 times between [2020-04-06 04:10:53.552635] and
> >[2020-04-06
> >> > 04:10:53.745610]
> >> > The message "W [MSGID: 114031]
> >> > [client-rpc-fops_v2.c:851:client4_0_setxattr_cbk]
> >> > 0-persistent-client-0: remote operation failed [Permission
denied]"
> >> > repeated 4 times between [2020-04-06 04:10:53.552639] and
> >[2020-04-06
> >> > 04:10:53.745632]
> >> > The message "E [MSGID: 148002]
> >> > [utime.c:146:gf_utime_set_mdata_setxattr_cbk]
0-persistent-utime:
> >dict
> >> > set of key for set-ctime-mdata failed [Permission
denied]" repeated
> >4
> >> > times between [2020-04-06 04:10:53.553226] and [2020-04-06
> >> > 04:10:53.746080]
> >> >
> >> > Anything to worry about?
> >> >
> >> >
> >> > Regards,
> >> > Hubert
> >> ________
> >>
> >>
> >>
> >> Community Meeting Calendar:
> >>
> >> Schedule -
> >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >> Bridge: https://bluejeans.com/441850968
> >>
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org
> >> https://lists.gluster.org/mailman/listinfo/gluster-users
> >>
>
> Hi,
>
> Can you provide the xfs_info for the bricks from the volume ?
>
> I have a theory that I want to confirm or reject.
>
> Best Regards,
> Strahil Nikolov

Gluster users - Jun 2020 - One error/warning message after upgrade 5.11 -> 6.8

[Gluster-users] One error/warning message after upgrade 5.11 -> 6.8

[Gluster-users] One error/warning message after upgrade 5.11 -> 6.8

[Gluster-users] One error/warning message after upgrade 5.11 -> 6.8