Gluster users - Feb 2020 - Permission denied at some directories/files after a split brain

Hi Strahil,

On Mon, 10 Feb 2020 at 15:28, Strahil Nikolov <hunter86_bg at yahoo.com>
wrote:
> Hi Alberto,
> Sadly you should verify if the issue is the same.
> Enable the trace logs for the bricks and verify if the errors in the logs
> with those in the bugzilla.
Don't forget to stop the trace log or your logs' dir will get
full.
>
Yes, the log is quite similar:

[2020-02-10 10:38:17.402080] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied]
0-app_data-access-control: client: CTX_ID:7d744c50-43a1-4f81-
9330-001b5dcaddb7-GRAPH_ID:0-PID:2310-HOST:ast10.local.
domain-PC_NAME:app_data-client-1-RECON_NO:-1, gfid:
092f1e28-d6a8-4ca9-95d5-75dc8ad1c835,
req(uid:498,gid:498,perm:4,ngrps:1),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID,
acl:-) [Permission denied]
[2020-02-10 10:38:17.402182] E [MSGID: 115056]
[server-rpc-fops_v2.c:687:server4_opendir_cbk] 0-app_data-server: 6257941:
OPENDIR /mailboxes.old/8692/211411002/Old
(092f1e28-d6a8-4ca9-95d5-75dc8ad1c835), client:
CTX_ID:7d744c50-43a1-4f81-9330-001b5dcaddb7-GRAPH_ID:0-PID:2310-HOST:ast10.local.domain-PC_NAME:app_data-client-1-RECON_NO:-1,
error-xlator: app_data-access-control [Permission denied]

> What version of gluster are you using ?
>
Gluster 6.6.

> In my case only a  downgrade has restored the operation of the cluster, so
> you should consider that as an option (last, but still an option).
>
>
We created a copy of the faulty's directory and put it in place of the
older one to solve our issues for now. We kept the old one for further
investigation.

> You can try to run a find against the fuse and 'find  /path/to/fuse
-exec
> setfacl -m u:root:rw {} \;'
> Maybe that will force gluster to read the ACLs again.
>
Running a setfacl doesn't make any difference. If we do a chmod it
"fixes"
the permission problem.

>
> Good luck!
> If you have the option, join the next gluster meeting and ask for an
> update (if the issue is actually the same).
>
> Best Regards,
> Strahil Nikolov
>
Thank you,
Alberto Bengoa
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200211/5c1915ea/attachment.html>

Hi Alberto,
We face a similar problem with Gluster 6.7 + NFS-Ganesha after a heal. However,
on our side, the problem also affected the root user. We were able to move and
delete the affected files but unable to copy, edit or chmod. The brick logs was
the same as yours.

Since we restarted nfs-ganesha, it working fine.

Renaud

De : gluster-users-bounces at gluster.org <gluster-users-bounces at
gluster.org> De la part de Alberto Bengoa
Envoy? : 11 f?vrier 2020 05:02
? : Strahil Nikolov <hunter86_bg at yahoo.com>
Cc : gluster-users <gluster-users at gluster.org>
Objet : Re: [Gluster-users] Permission denied at some directories/files after a
split brain

[Externe UL*]
Hi Strahil,

On Mon, 10 Feb 2020 at 15:28, Strahil Nikolov <hunter86_bg at
yahoo.com<mailto:hunter86_bg at yahoo.com>> wrote:
Hi Alberto,
Sadly you should verify if the issue is the same.
Enable the trace logs for the bricks and verify if the errors in the logs with
those in the bugzilla.
Don't forget to stop the trace log or your logs' dir will get full.

Yes, the log is quite similar:

[2020-02-10 10:38:17.402080] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-app_data-access-control: client:
CTX_ID:7d744c50-43a1-4f81-9330-001b5dcaddb7-GRAPH_ID:0-PID:2310-HOST:ast10.local.domain-PC_NAME:app_data-client-1-RECON_NO:-1,
gfid: 092f1e28-d6a8-4ca9-95d5-75dc8ad1c835, req(uid:498,gid:498,perm:4,ngrps:1),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) [Permission
denied]
[2020-02-10 10:38:17.402182] E [MSGID: 115056]
[server-rpc-fops_v2.c:687:server4_opendir_cbk] 0-app_data-server: 6257941:
OPENDIR /mailboxes.old/8692/211411002/Old
(092f1e28-d6a8-4ca9-95d5-75dc8ad1c835), client:
CTX_ID:7d744c50-43a1-4f81-9330-001b5dcaddb7-GRAPH_ID:0-PID:2310-HOST:ast10.local.domain-PC_NAME:app_data-client-1-RECON_NO:-1,
error-xlator: app_data-access-control [Permission denied]


What version of gluster are you using ?

Gluster 6.6.

In my case only a  downgrade has restored the operation of the cluster, so you
should consider that as an option (last, but still an option).

We created a copy of the faulty's directory and put it in place of the older
one to solve our issues for now. We kept the old one for further investigation.

You can try to run a find against the fuse and 'find  /path/to/fuse -exec
setfacl -m u:root:rw {} \;'
Maybe that will force gluster to read the ACLs again.

Running a setfacl doesn't make any difference. If we do a chmod it
"fixes" the permission problem.


Good luck!
If you have the option, join the next gluster meeting and ask for an update (if
the issue is actually the same).

Best Regards,
Strahil Nikolov

Thank you,
Alberto Bengoa
*ATTENTION : L??metteur de ce courriel est externe ? l?Universit? Laval.
?vitez de cliquer sur un hyperlien, d?ouvrir une pi?ce jointe ou de transmettre
des informations si vous ne connaissez pas l?exp?diteur du courriel. En cas de
doute, contactez l??quipe de soutien informatique de votre unit? ou hameconnage
at ulaval.ca<mailto:hameconnage at ulaval.ca>.
________________________________

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200211/535381a9/attachment.html>