Gluster users - May 2017 - URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

Hi,

Over the weekend we updated a two server glusterfs 3.6.6 install to
3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
enabled several of the newer caching features from gluster 3.9 for
small file performance and samba, and we now seem to have some issues
with accessing files from glusterfs. When users try to access some
files, they get a Permission denied message. This seems to be only via
samba as I am able to su - username and then do strings on the file.

[root at ysmha02 gluster-backups]# rpm -qa | grep gluster
glusterfs-client-xlators-3.10.2-1.el7.x86_64
glusterfs-server-3.10.2-1.el7.x86_64
glusterfs-api-3.10.2-1.el7.x86_64
glusterfs-3.10.2-1.el7.x86_64
glusterfs-cli-3.10.2-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
glusterfs-fuse-3.10.2-1.el7.x86_64
glusterfs-libs-3.10.2-1.el7.x86_64
glusterfs-rdma-3.10.2-1.el7.x86_64

[root at ysmha02 gluster-backups]# rpm -qa | grep samba
samba-common-libs-4.4.4-14.el7_3.x86_64
samba-common-tools-4.4.4-14.el7_3.x86_64
samba-libs-4.4.4-14.el7_3.x86_64
samba-4.4.4-14.el7_3.x86_64
samba-client-libs-4.4.4-14.el7_3.x86_64
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
samba-common-4.4.4-14.el7_3.noarch

On the samba logs for the machine I notice something weird, samba
seems to be trying to stat the file we are trying as a directory to
see if it contains desktop.ini:

[2017/05/30 10:13:07.297026,  0]
../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
 glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
Not a directory
[2017/05/30 10:13:07.298155,  0]
../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
 glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
Not a directory

This seems to be happening only with files with the .rvt extension.
Though these files are usually larger in size vs other smaller excel,
power point, etc files.

Here are the complete options for the volume:

https://pastebin.com/ZH2vMsMN

I turned off performance.cache-samba-metadata again to see if that
would help, but seems it does not help.

I really appreciate any help with this.

DIego

This is what I see in the logs set from smb.conf via line ->
glusterfs:logfile = /var/log/samba/glusterfs-projects.log

[2017-05-30 14:52:31.051524] E [MSGID: 123001]
[io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
(a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
[2017-05-30 14:52:31.241850] W [MSGID: 114031]
[client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
remote operation failed. Path:
/projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
a directory]
[2017-05-30 14:52:31.242956] W [MSGID: 114031]
[client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
remote operation failed. Path:
/projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
a directory]


On Tue, May 30, 2017 at 10:37 AM, Diego Remolina <dijuremo at gmail.com>
wrote:
> Hi,
>
> Over the weekend we updated a two server glusterfs 3.6.6 install to
> 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
> enabled several of the newer caching features from gluster 3.9 for
> small file performance and samba, and we now seem to have some issues
> with accessing files from glusterfs. When users try to access some
> files, they get a Permission denied message. This seems to be only via
> samba as I am able to su - username and then do strings on the file.
>
> [root at ysmha02 gluster-backups]# rpm -qa | grep gluster
> glusterfs-client-xlators-3.10.2-1.el7.x86_64
> glusterfs-server-3.10.2-1.el7.x86_64
> glusterfs-api-3.10.2-1.el7.x86_64
> glusterfs-3.10.2-1.el7.x86_64
> glusterfs-cli-3.10.2-1.el7.x86_64
> centos-release-gluster310-1.0-1.el7.centos.noarch
> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
> glusterfs-fuse-3.10.2-1.el7.x86_64
> glusterfs-libs-3.10.2-1.el7.x86_64
> glusterfs-rdma-3.10.2-1.el7.x86_64
>
> [root at ysmha02 gluster-backups]# rpm -qa | grep samba
> samba-common-libs-4.4.4-14.el7_3.x86_64
> samba-common-tools-4.4.4-14.el7_3.x86_64
> samba-libs-4.4.4-14.el7_3.x86_64
> samba-4.4.4-14.el7_3.x86_64
> samba-client-libs-4.4.4-14.el7_3.x86_64
> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
> samba-common-4.4.4-14.el7_3.noarch
>
> On the samba logs for the machine I notice something weird, samba
> seems to be trying to stat the file we are trying as a directory to
> see if it contains desktop.ini:
>
> [2017/05/30 10:13:07.297026,  0]
> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
> Not a directory
> [2017/05/30 10:13:07.298155,  0]
> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
> Not a directory
>
> This seems to be happening only with files with the .rvt extension.
> Though these files are usually larger in size vs other smaller excel,
> power point, etc files.
>
> Here are the complete options for the volume:
>
> https://pastebin.com/ZH2vMsMN
>
> I turned off performance.cache-samba-metadata again to see if that
> would help, but seems it does not help.
>
> I really appreciate any help with this.
>
> DIego