thr3ads.net - Gluster users - [Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied [Jun 2017]

If this information is useful, please help other people find it:
Share via:

Diego Remolina

2017-Jun-03 16:16 UTC

[Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

Thanks for taking the time to look into this. Since we needed downtime
due to the gluster update, we also updated the OS, including samba. We
went from 4.2.x to 4.4.4 and many other packages for CentOS were
updated as well. OS and samba updates were installed, then server
rebooted, then gluster was updated.

Created a new test samba share to minimize logs, etc:

[VfsGluster]
   path = /vfsgluster
   browseable = yes
   write list = @Staff,root, at Admin, at Managers
   writeable = yes
   guest ok = no
   create mask = 660
   directory mask = 770
   kernel share modes = No
   vfs objects = glusterfs
   glusterfs:loglevel = 7
   glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
   glusterfs:volume = export
   veto files = desktop.ini


I am attaching the logs for the test of the rename operation. The file
I try to access is called:2016-9-13_Herrin Gear BMW_REV-3
The timeline is as follows (times are UTC):

15:18:00 -> Change smb.conf to glusterfs:loglevel = 9 and reload smbd
via  smbcontrol smbd reload-config
See log: glusterfs-vfsgluster.log
15:19:31 -> Adjust gluster client and brick log levels to TRACE
export.log is the Gluster volume log
bricks-hdds-brick.log is the Brick log on ysmha02 (did you need the
other one?) Samba is running on ysmah02.

15:20:40 -> Get the extended attributes for all files in the bricks on
both servers ysmha01 and ysmha02 before accessing the file:
cd /bricks/hdds/brick/vfsgluster/
 find . -type f -exec getfattr -m .  -d -e hex {} \; >>
/tmp/ysmha01-attributes-before-rename-run2
 find . -type f -exec getfattr -m .  -d -e hex {} \; >>
/tmp/ysmha02-attributes-before-rename-run2

15:21-ish -> Cannot open file so I do a rename and rename back via
Windows explorer
15:22-ish -> Get the extended attirbutes again after renaming the
file. At this point I can open the file.
 find . -type f -exec getfattr -m .  -d -e hex {} \; >>
/tmp/ysmha01-attributes-after-rename-run2
 find . -type f -exec getfattr -m .  -d -e hex {} \; >>
/tmp/ysmha02-attributes-after-rename-run2

I did all the files here, because Revit uses a file and some folders
to keep track of it changes, versions, etc.

15:23:25 -> Change gluster client and brick logs back to info.

Let me know if I need to redo any tests, change any steps or provide
more logs I may have missed.

Diego

On Fri, Jun 2, 2017 at 8:19 PM, Raghavendra Talur <rtalur at redhat.com>
wrote:>
>
> On 03-Jun-2017 3:27 AM, "Diego Remolina" <dijuremo at
gmail.com> wrote:
>
> Hi everyone,
>
> Is there anything else we could do to check on this problem and try to
> fix it? The issue is definitively related to either the samba vfs
> gluster plugin or gluster itself. I am not sure how to pin it down
> futher.
>
>
> I don't think it is vfs plugin because you haven't updated samba
packages
> and vfs plugin is same as before.
>
> It is either gfapi in Gluster or one of the lower xlators.
>
> Changing to fuse based shares might make the problem go away and can be
used
> as a workaround.
>
>
>
> I went ahead and created a new share in the samba server which is on a
> local filesystem where the OS is installed, not part of gluster:
>
> # mount | grep home
> ]# ls -ld /home
> drwxr-xr-x. 14 root root 4096 Jun  2 17:21 /home
>
> This shows this is not mounted from anywhere. so I created a folder
> and shared it:
>
> ]# ls -ld /home/rvtsharetest/
> drwxrwx---. 3 dijuremo Staff 95 Jun  2 17:31 /home/rvtsharetest/
>
> ]# tail -7 /etc/samba/smb.conf
> [rvtsharetest]
>    path=/home/rvtsharetest
>    browseable = yes
>    write list = @Staff,dijuremo
>    guest ok = yes
>    create mask = 664
>    directory mask = 775
>
> When accessing Revit files in this new share the problem is *not* observed.
>
> When accessing Revit files on any of the samba shares that use the vfs
> gluster plugin and are stored in the gluster volume, we see the
> problems.
>
> Further analysis of the issue is even more disconcerting. As you may
> remember, I have found a workaround about renaming the file and back
> to the original, where things work. Here is where it gets more
> interesting. This seems to be workstation dependent, not user
> dependent.
>
> 1. User1 logs into PC1 and tries to access the file and gets the error
> message from Revit "ACCESS DENIED".
>
> 2. User1 uses windows explorer to go to the file location (we tried
> doing this from the server itself on the command line and it did not
> change anything, i.e. su - User1 then mv command). User1 renames the
> file, Revit.rvt -> Revit2.rvt, and clicks away for the rename to take
> place. User1 immediately renames the file to the original, Revit2.rvt
> -> Revit.rvt
>
> 3. User1 opens the file and everything works properly.
>
> 4. User2 logs into PC1 and tries to open the file. The file opens and
> works properly.
>
> 5. User2 logs into PC2 and tries to open the file, the problem comes
> up. User2 uses the rename trick and this fixes the problem on PC2.
> Even if User1 now comes to use PC2, User1 will have no problems with
> the file.
>
> 6. User1 now goes to PC3 where nobody has used the file rename trick,
> and experiences the problem. Only solution is to play the rename trick
> again in PC3.
>
> So it seems you have to play the rename trick at least one per
> workstation and that "fixes" the issue for any user who logs into
that
> workstation.
>
> What other suggestions do you have? What debugging can I do next? I am
> planning once everyone leaves the office today on changing the share
> to bypass vfs gluster plugin and access the file directly from the
> fuse mount, ie.:
>
> # mount | grep export
> 10.0.1.7:/export on /export type fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
>
> Then set samba to share without using the vfs gluster plugin as follows
>
> [Projects]
>    path = /export/projects
>    browseable = yes
>    write list = @Staff,root, at Admin, at Managers
>    writeable = yes
>    guest ok = no
>    create mask = 660
>    directory mask = 770
>
> This test will determine if the issue is the samba vfs gluster plugin
> or if it is the fact that the file is stored in the gluster volume.
>
> Any other thoughts?
>
>
> I suspect this has to do with locking and such.
> Can create a new volume and share through vfs plugin.
> Set logging to number 5 for samba and Gluster volume set client-log-level
> and server-log-level to TRACE.
> Copy one rvt file.
> Performs open, this is like fail.
> Perform rename
> Perform open, this would pass
>
> Send us the logs.
>
> I know it is troublesome but detailed logs are the only things that would
> help us analyze the issue.
>
> Talur
>
>
> Diego
>
> On Wed, May 31, 2017 at 12:59 PM, Diego Remolina <dijuremo at
gmail.com> wrote:
>> The vfs gluster plugin for samba is linked against libglusterfs.so.0
>> which is provided by glusterfs-libs-3.10.2-1.el7.x86_64, please see
>> below.
>>
>> # ldd /usr/lib64/samba/vfs/glusterfs.so | grep glus
>>        libglusterfs.so.0 => /lib64/libglusterfs.so.0
(0x00007f61da858000)
>>
>> ]# yum provides /lib64/libglusterfs.so.0
>> Loaded plugins: fastestmirror
>> Loading mirror speeds from cached hostfile
>> * base: centos.vwtonline.net
>> * extras: mirror.cs.vt.edu
>> * updates: centosv.centos.org
>> glusterfs-libs-3.10.2-1.el7.x86_64 : GlusterFS common libraries
>> Repo        : @centos-gluster310
>> Matched from:
>> Filename    : /lib64/libglusterfs.so.0
>>
>>
>>
>> On Wed, May 31, 2017 at 12:39 PM, Diego Remolina <dijuremo at
gmail.com>
>> wrote:
>>> Samba is running in the same machine as glusterd. The machines were
>>> rebooted after the upgrades and samba has been restarted a few
times.
>>>
>>> # rpm -qa | grep gluster
>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>> glusterfs-server-3.10.2-1.el7.x86_64
>>> glusterfs-api-3.10.2-1.el7.x86_64
>>> glusterfs-3.10.2-1.el7.x86_64
>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>
>>> # rpm -qa | grep samba
>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>> samba-libs-4.4.4-14.el7_3.x86_64
>>> samba-4.4.4-14.el7_3.x86_64
>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>> samba-common-4.4.4-14.el7_3.noarch
>>>
>>> # cat /etc/redhat-release
>>> CentOS Linux release 7.3.1611 (Core)
>>>
>>> I also raised the op version.
>>>
>>> # gluster volume get all cluster.op-version
>>> Option                                  Value
>>> ------                                  -----
>>> cluster.op-version                      31000
>>>
>>> # gluster volume get all cluster.max-op-version
>>> Option                                  Value
>>> ------                                  -----
>>> cluster.max-op-version                  31000
>>>
>>> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur <rtalur at
redhat.com>
>>> wrote:
>>>> Also, please attach your smb.conf. You can directly attach in
the list
>>>> and need not have a google drive link.
>>>>
>>>>
>>>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur <rtalur
at redhat.com>
>>>> wrote:
>>>>> Diego,
>>>>>
>>>>> I see that Samba is still using 3.6.6 Gluster. Is it
possible you did
>>>>> not restart smb after upgrading gluster(if samba is on same
machine as
>>>>> Gluster) or if you forgot to update Gluster client packages
on Samba
>>>>> node?
>>>>>
>>>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina
<dijuremo at gmail.com>
>>>>> wrote:
>>>>>> Please download the log file from this link:
>>>>>>
>>>>>>
https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>>>>>
>>>>>> Let me know if you need any other log files.
>>>>>>
>>>>>> Diego
>>>>>>
>>>>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur
>>>>>> <rtalur at redhat.com> wrote:
>>>>>>> If possible please share the glusterfs-* log files
from
>>>>>>> /var/log/samba.
>>>>>>>
>>>>>>> This might be because of cluster.lookup-optimize.
Adding Poornima and
>>>>>>> Raghavendra Gowdappa to help with this.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina
<dijuremo at gmail.com>
>>>>>>> wrote:
>>>>>>>> This is a bit puzzling, not sure what
difference it would make, but:
>>>>>>>>
>>>>>>>> 1. Try to open file that has a problem, ie.
MyRevitFile.rvt
>>>>>>>> Revit opens and shows a window that says access
denied.
>>>>>>>>
>>>>>>>> 2. Rename file, i.e from windows explorer right
click, rename to
>>>>>>>> MyRevitFile2.rvt
>>>>>>>>
>>>>>>>> 3. Without opening the file, rename file back
to the original, i.e
>>>>>>>> MyRevitFile.rvt
>>>>>>>>
>>>>>>>> 4. Double click on file and now it will open
just fine without the
>>>>>>>> Access Denied error.
>>>>>>>>
>>>>>>>> Any explanation for this? Could the rename
operation be forcing or
>>>>>>>> updating some attributes that then allow the
file to open?
>>>>>>>>
>>>>>>>> Diego
>>>>>>>>
>>>>>>>> On Tue, May 30, 2017 at 10:57 AM, Diego
Remolina
>>>>>>>> <dijuremo at gmail.com> wrote:
>>>>>>>>> This is what I see in the logs set from
smb.conf via line ->
>>>>>>>>> glusterfs:logfile =
/var/log/samba/glusterfs-projects.log
>>>>>>>>>
>>>>>>>>> [2017-05-30 14:52:31.051524] E [MSGID:
123001]
>>>>>>>>> [io-cache.c:564:ioc_open_cbk]
0-export-io-cache: inode context is
>>>>>>>>> NULL
>>>>>>>>> (a97bc9bb-68cf-4a69-aef7-39766b323c14)
[Invalid argument]
>>>>>>>>> [2017-05-30 14:52:31.241850] W [MSGID:
114031]
>>>>>>>>>
[client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>>>>>>>> remote operation failed. Path:
>>>>>>>>>
/projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>> Richmond/02
>>>>>>>>> DRAWINGS/02
ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>> aef7-39766b323c14). Key:
glusterfs.get_real_filename:desktop.ini
>>>>>>>>> [Not
>>>>>>>>> a directory]
>>>>>>>>> [2017-05-30 14:52:31.242956] W [MSGID:
114031]
>>>>>>>>>
[client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>>>>>>>> remote operation failed. Path:
>>>>>>>>>
/projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>> Richmond/02
>>>>>>>>> DRAWINGS/02
ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>> aef7-39766b323c14). Key:
glusterfs.get_real_filename:desktop.ini
>>>>>>>>> [Not
>>>>>>>>> a directory]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 30, 2017 at 10:37 AM, Diego
Remolina
>>>>>>>>> <dijuremo at gmail.com> wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Over the weekend we updated a two
server glusterfs 3.6.6 install
>>>>>>>>>> to
>>>>>>>>>> 3.10.2 We also updated samba and
samba-vfs to the latest in
>>>>>>>>>> CentOS. I
>>>>>>>>>> enabled several of the newer caching
features from gluster 3.9 for
>>>>>>>>>> small file performance and samba, and
we now seem to have some
>>>>>>>>>> issues
>>>>>>>>>> with accessing files from glusterfs.
When users try to access some
>>>>>>>>>> files, they get a Permission denied
message. This seems to be only
>>>>>>>>>> via
>>>>>>>>>> samba as I am able to su - username and
then do strings on the
>>>>>>>>>> file.
>>>>>>>>>>
>>>>>>>>>> [root at ysmha02 gluster-backups]# rpm
-qa | grep gluster
>>>>>>>>>>
glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>>>>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>>>>>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>>>>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>>>>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>>>>>>>>
centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>>>>>>>
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>>>>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>>>>>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>>>>>>
>>>>>>>>>> [root at ysmha02 gluster-backups]# rpm
-qa | grep samba
>>>>>>>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>
samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>>>>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>>>>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>>>>>>>
>>>>>>>>>> On the samba logs for the machine I
notice something weird, samba
>>>>>>>>>> seems to be trying to stat the file we
are trying as a directory
>>>>>>>>>> to
>>>>>>>>>> see if it contains desktop.ini:
>>>>>>>>>>
>>>>>>>>>> [2017/05/30 10:13:07.297026,  0]
>>>>>>>>>>
../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>  glfs_stat(ACTIVE/Automotive/FORD/AN -
Ford East/02
>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD
EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>> failed:
>>>>>>>>>> Not a directory
>>>>>>>>>> [2017/05/30 10:13:07.298155,  0]
>>>>>>>>>>
../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>  glfs_stat(ACTIVE/Automotive/FORD/AN -
Ford East/02
>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD
EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>> failed:
>>>>>>>>>> Not a directory
>>>>>>>>>>
>>>>>>>>>> This seems to be happening only with
files with the .rvt
>>>>>>>>>> extension.
>>>>>>>>>> Though these files are usually larger
in size vs other smaller
>>>>>>>>>> excel,
>>>>>>>>>> power point, etc files.
>>>>>>>>>>
>>>>>>>>>> Here are the complete options for the
volume:
>>>>>>>>>>
>>>>>>>>>> https://pastebin.com/ZH2vMsMN
>>>>>>>>>>
>>>>>>>>>> I turned off
performance.cache-samba-metadata again to see if that
>>>>>>>>>> would help, but seems it does not help.
>>>>>>>>>>
>>>>>>>>>> I really appreciate any help with this.
>>>>>>>>>>
>>>>>>>>>> DIego
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>
>-------------- next part --------------
A non-text attachment was scrubbed...
Name: logs-revit-rename-fix.tar.bz2
Type: application/x-bzip2
Size: 1557241 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170603/13949382/attachment-0001.bz2>

Diego Remolina

2017-Jun-12 12:20 UTC

head link

[Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

Did the logs provide any hints as to what the issue may be?

Diego

On Sat, Jun 3, 2017 at 12:16 PM, Diego Remolina <dijuremo at gmail.com>
wrote:> Thanks for taking the time to look into this. Since we needed downtime
> due to the gluster update, we also updated the OS, including samba. We
> went from 4.2.x to 4.4.4 and many other packages for CentOS were
> updated as well. OS and samba updates were installed, then server
> rebooted, then gluster was updated.
>
> Created a new test samba share to minimize logs, etc:
>
> [VfsGluster]
>    path = /vfsgluster
>    browseable = yes
>    write list = @Staff,root, at Admin, at Managers
>    writeable = yes
>    guest ok = no
>    create mask = 660
>    directory mask = 770
>    kernel share modes = No
>    vfs objects = glusterfs
>    glusterfs:loglevel = 7
>    glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
>    glusterfs:volume = export
>    veto files = desktop.ini
>
>
> I am attaching the logs for the test of the rename operation. The file
> I try to access is called:2016-9-13_Herrin Gear BMW_REV-3
> The timeline is as follows (times are UTC):
>
> 15:18:00 -> Change smb.conf to glusterfs:loglevel = 9 and reload smbd
> via  smbcontrol smbd reload-config
> See log: glusterfs-vfsgluster.log
> 15:19:31 -> Adjust gluster client and brick log levels to TRACE
> export.log is the Gluster volume log
> bricks-hdds-brick.log is the Brick log on ysmha02 (did you need the
> other one?) Samba is running on ysmah02.
>
> 15:20:40 -> Get the extended attributes for all files in the bricks on
> both servers ysmha01 and ysmha02 before accessing the file:
> cd /bricks/hdds/brick/vfsgluster/
>  find . -type f -exec getfattr -m .  -d -e hex {} \; >>
> /tmp/ysmha01-attributes-before-rename-run2
>  find . -type f -exec getfattr -m .  -d -e hex {} \; >>
> /tmp/ysmha02-attributes-before-rename-run2
>
> 15:21-ish -> Cannot open file so I do a rename and rename back via
> Windows explorer
> 15:22-ish -> Get the extended attirbutes again after renaming the
> file. At this point I can open the file.
>  find . -type f -exec getfattr -m .  -d -e hex {} \; >>
> /tmp/ysmha01-attributes-after-rename-run2
>  find . -type f -exec getfattr -m .  -d -e hex {} \; >>
> /tmp/ysmha02-attributes-after-rename-run2
>
> I did all the files here, because Revit uses a file and some folders
> to keep track of it changes, versions, etc.
>
> 15:23:25 -> Change gluster client and brick logs back to info.
>
> Let me know if I need to redo any tests, change any steps or provide
> more logs I may have missed.
>
> Diego
>
> On Fri, Jun 2, 2017 at 8:19 PM, Raghavendra Talur <rtalur at
redhat.com> wrote:
>>
>>
>> On 03-Jun-2017 3:27 AM, "Diego Remolina" <dijuremo at
gmail.com> wrote:
>>
>> Hi everyone,
>>
>> Is there anything else we could do to check on this problem and try to
>> fix it? The issue is definitively related to either the samba vfs
>> gluster plugin or gluster itself. I am not sure how to pin it down
>> futher.
>>
>>
>> I don't think it is vfs plugin because you haven't updated
samba packages
>> and vfs plugin is same as before.
>>
>> It is either gfapi in Gluster or one of the lower xlators.
>>
>> Changing to fuse based shares might make the problem go away and can be
used
>> as a workaround.
>>
>>
>>
>> I went ahead and created a new share in the samba server which is on a
>> local filesystem where the OS is installed, not part of gluster:
>>
>> # mount | grep home
>> ]# ls -ld /home
>> drwxr-xr-x. 14 root root 4096 Jun  2 17:21 /home
>>
>> This shows this is not mounted from anywhere. so I created a folder
>> and shared it:
>>
>> ]# ls -ld /home/rvtsharetest/
>> drwxrwx---. 3 dijuremo Staff 95 Jun  2 17:31 /home/rvtsharetest/
>>
>> ]# tail -7 /etc/samba/smb.conf
>> [rvtsharetest]
>>    path=/home/rvtsharetest
>>    browseable = yes
>>    write list = @Staff,dijuremo
>>    guest ok = yes
>>    create mask = 664
>>    directory mask = 775
>>
>> When accessing Revit files in this new share the problem is *not*
observed.
>>
>> When accessing Revit files on any of the samba shares that use the vfs
>> gluster plugin and are stored in the gluster volume, we see the
>> problems.
>>
>> Further analysis of the issue is even more disconcerting. As you may
>> remember, I have found a workaround about renaming the file and back
>> to the original, where things work. Here is where it gets more
>> interesting. This seems to be workstation dependent, not user
>> dependent.
>>
>> 1. User1 logs into PC1 and tries to access the file and gets the error
>> message from Revit "ACCESS DENIED".
>>
>> 2. User1 uses windows explorer to go to the file location (we tried
>> doing this from the server itself on the command line and it did not
>> change anything, i.e. su - User1 then mv command). User1 renames the
>> file, Revit.rvt -> Revit2.rvt, and clicks away for the rename to
take
>> place. User1 immediately renames the file to the original, Revit2.rvt
>> -> Revit.rvt
>>
>> 3. User1 opens the file and everything works properly.
>>
>> 4. User2 logs into PC1 and tries to open the file. The file opens and
>> works properly.
>>
>> 5. User2 logs into PC2 and tries to open the file, the problem comes
>> up. User2 uses the rename trick and this fixes the problem on PC2.
>> Even if User1 now comes to use PC2, User1 will have no problems with
>> the file.
>>
>> 6. User1 now goes to PC3 where nobody has used the file rename trick,
>> and experiences the problem. Only solution is to play the rename trick
>> again in PC3.
>>
>> So it seems you have to play the rename trick at least one per
>> workstation and that "fixes" the issue for any user who logs
into that
>> workstation.
>>
>> What other suggestions do you have? What debugging can I do next? I am
>> planning once everyone leaves the office today on changing the share
>> to bypass vfs gluster plugin and access the file directly from the
>> fuse mount, ie.:
>>
>> # mount | grep export
>> 10.0.1.7:/export on /export type fuse.glusterfs
>> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
>>
>> Then set samba to share without using the vfs gluster plugin as follows
>>
>> [Projects]
>>    path = /export/projects
>>    browseable = yes
>>    write list = @Staff,root, at Admin, at Managers
>>    writeable = yes
>>    guest ok = no
>>    create mask = 660
>>    directory mask = 770
>>
>> This test will determine if the issue is the samba vfs gluster plugin
>> or if it is the fact that the file is stored in the gluster volume.
>>
>> Any other thoughts?
>>
>>
>> I suspect this has to do with locking and such.
>> Can create a new volume and share through vfs plugin.
>> Set logging to number 5 for samba and Gluster volume set
client-log-level
>> and server-log-level to TRACE.
>> Copy one rvt file.
>> Performs open, this is like fail.
>> Perform rename
>> Perform open, this would pass
>>
>> Send us the logs.
>>
>> I know it is troublesome but detailed logs are the only things that
would
>> help us analyze the issue.
>>
>> Talur
>>
>>
>> Diego
>>
>> On Wed, May 31, 2017 at 12:59 PM, Diego Remolina <dijuremo at
gmail.com> wrote:
>>> The vfs gluster plugin for samba is linked against
libglusterfs.so.0
>>> which is provided by glusterfs-libs-3.10.2-1.el7.x86_64, please see
>>> below.
>>>
>>> # ldd /usr/lib64/samba/vfs/glusterfs.so | grep glus
>>>        libglusterfs.so.0 => /lib64/libglusterfs.so.0
(0x00007f61da858000)
>>>
>>> ]# yum provides /lib64/libglusterfs.so.0
>>> Loaded plugins: fastestmirror
>>> Loading mirror speeds from cached hostfile
>>> * base: centos.vwtonline.net
>>> * extras: mirror.cs.vt.edu
>>> * updates: centosv.centos.org
>>> glusterfs-libs-3.10.2-1.el7.x86_64 : GlusterFS common libraries
>>> Repo        : @centos-gluster310
>>> Matched from:
>>> Filename    : /lib64/libglusterfs.so.0
>>>
>>>
>>>
>>> On Wed, May 31, 2017 at 12:39 PM, Diego Remolina <dijuremo at
gmail.com>
>>> wrote:
>>>> Samba is running in the same machine as glusterd. The machines
were
>>>> rebooted after the upgrades and samba has been restarted a few
times.
>>>>
>>>> # rpm -qa | grep gluster
>>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>> glusterfs-3.10.2-1.el7.x86_64
>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>
>>>> # rpm -qa | grep samba
>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>> samba-4.4.4-14.el7_3.x86_64
>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>
>>>> # cat /etc/redhat-release
>>>> CentOS Linux release 7.3.1611 (Core)
>>>>
>>>> I also raised the op version.
>>>>
>>>> # gluster volume get all cluster.op-version
>>>> Option                                  Value
>>>> ------                                  -----
>>>> cluster.op-version                      31000
>>>>
>>>> # gluster volume get all cluster.max-op-version
>>>> Option                                  Value
>>>> ------                                  -----
>>>> cluster.max-op-version                  31000
>>>>
>>>> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur <rtalur
at redhat.com>
>>>> wrote:
>>>>> Also, please attach your smb.conf. You can directly attach
in the list
>>>>> and need not have a google drive link.
>>>>>
>>>>>
>>>>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur
<rtalur at redhat.com>
>>>>> wrote:
>>>>>> Diego,
>>>>>>
>>>>>> I see that Samba is still using 3.6.6 Gluster. Is it
possible you did
>>>>>> not restart smb after upgrading gluster(if samba is on
same machine as
>>>>>> Gluster) or if you forgot to update Gluster client
packages on Samba
>>>>>> node?
>>>>>>
>>>>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina
<dijuremo at gmail.com>
>>>>>> wrote:
>>>>>>> Please download the log file from this link:
>>>>>>>
>>>>>>>
https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>>>>>>
>>>>>>> Let me know if you need any other log files.
>>>>>>>
>>>>>>> Diego
>>>>>>>
>>>>>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur
>>>>>>> <rtalur at redhat.com> wrote:
>>>>>>>> If possible please share the glusterfs-* log
files from
>>>>>>>> /var/log/samba.
>>>>>>>>
>>>>>>>> This might be because of
cluster.lookup-optimize. Adding Poornima and
>>>>>>>> Raghavendra Gowdappa to help with this.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina
<dijuremo at gmail.com>
>>>>>>>> wrote:
>>>>>>>>> This is a bit puzzling, not sure what
difference it would make, but:
>>>>>>>>>
>>>>>>>>> 1. Try to open file that has a problem, ie.
MyRevitFile.rvt
>>>>>>>>> Revit opens and shows a window that says
access denied.
>>>>>>>>>
>>>>>>>>> 2. Rename file, i.e from windows explorer
right click, rename to
>>>>>>>>> MyRevitFile2.rvt
>>>>>>>>>
>>>>>>>>> 3. Without opening the file, rename file
back to the original, i.e
>>>>>>>>> MyRevitFile.rvt
>>>>>>>>>
>>>>>>>>> 4. Double click on file and now it will
open just fine without the
>>>>>>>>> Access Denied error.
>>>>>>>>>
>>>>>>>>> Any explanation for this? Could the rename
operation be forcing or
>>>>>>>>> updating some attributes that then allow
the file to open?
>>>>>>>>>
>>>>>>>>> Diego
>>>>>>>>>
>>>>>>>>> On Tue, May 30, 2017 at 10:57 AM, Diego
Remolina
>>>>>>>>> <dijuremo at gmail.com> wrote:
>>>>>>>>>> This is what I see in the logs set from
smb.conf via line ->
>>>>>>>>>> glusterfs:logfile =
/var/log/samba/glusterfs-projects.log
>>>>>>>>>>
>>>>>>>>>> [2017-05-30 14:52:31.051524] E [MSGID:
123001]
>>>>>>>>>> [io-cache.c:564:ioc_open_cbk]
0-export-io-cache: inode context is
>>>>>>>>>> NULL
>>>>>>>>>> (a97bc9bb-68cf-4a69-aef7-39766b323c14)
[Invalid argument]
>>>>>>>>>> [2017-05-30 14:52:31.241850] W [MSGID:
114031]
>>>>>>>>>>
[client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>>>>>>>>> remote operation failed. Path:
>>>>>>>>>>
/projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>>> Richmond/02
>>>>>>>>>> DRAWINGS/02
ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>>> aef7-39766b323c14). Key:
glusterfs.get_real_filename:desktop.ini
>>>>>>>>>> [Not
>>>>>>>>>> a directory]
>>>>>>>>>> [2017-05-30 14:52:31.242956] W [MSGID:
114031]
>>>>>>>>>>
[client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>>>>>>>>> remote operation failed. Path:
>>>>>>>>>>
/projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>>> Richmond/02
>>>>>>>>>> DRAWINGS/02
ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>>> aef7-39766b323c14). Key:
glusterfs.get_real_filename:desktop.ini
>>>>>>>>>> [Not
>>>>>>>>>> a directory]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 30, 2017 at 10:37 AM, Diego
Remolina
>>>>>>>>>> <dijuremo at gmail.com> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Over the weekend we updated a two
server glusterfs 3.6.6 install
>>>>>>>>>>> to
>>>>>>>>>>> 3.10.2 We also updated samba and
samba-vfs to the latest in
>>>>>>>>>>> CentOS. I
>>>>>>>>>>> enabled several of the newer
caching features from gluster 3.9 for
>>>>>>>>>>> small file performance and samba,
and we now seem to have some
>>>>>>>>>>> issues
>>>>>>>>>>> with accessing files from
glusterfs. When users try to access some
>>>>>>>>>>> files, they get a Permission denied
message. This seems to be only
>>>>>>>>>>> via
>>>>>>>>>>> samba as I am able to su - username
and then do strings on the
>>>>>>>>>>> file.
>>>>>>>>>>>
>>>>>>>>>>> [root at ysmha02 gluster-backups]#
rpm -qa | grep gluster
>>>>>>>>>>>
glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>>>>>>>>
glusterfs-server-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>>>>>>>>>
centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>>>>>>>>
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>>>>>>>
>>>>>>>>>>> [root at ysmha02 gluster-backups]#
rpm -qa | grep samba
>>>>>>>>>>>
samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>
samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>
samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>>>>>>>>
>>>>>>>>>>> On the samba logs for the machine I
notice something weird, samba
>>>>>>>>>>> seems to be trying to stat the file
we are trying as a directory
>>>>>>>>>>> to
>>>>>>>>>>> see if it contains desktop.ini:
>>>>>>>>>>>
>>>>>>>>>>> [2017/05/30 10:13:07.297026,  0]
>>>>>>>>>>>
../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>> 
glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD
EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>>> failed:
>>>>>>>>>>> Not a directory
>>>>>>>>>>> [2017/05/30 10:13:07.298155,  0]
>>>>>>>>>>>
../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>> 
glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD
EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>>> failed:
>>>>>>>>>>> Not a directory
>>>>>>>>>>>
>>>>>>>>>>> This seems to be happening only
with files with the .rvt
>>>>>>>>>>> extension.
>>>>>>>>>>> Though these files are usually
larger in size vs other smaller
>>>>>>>>>>> excel,
>>>>>>>>>>> power point, etc files.
>>>>>>>>>>>
>>>>>>>>>>> Here are the complete options for
the volume:
>>>>>>>>>>>
>>>>>>>>>>> https://pastebin.com/ZH2vMsMN
>>>>>>>>>>>
>>>>>>>>>>> I turned off
performance.cache-samba-metadata again to see if that
>>>>>>>>>>> would help, but seems it does not
help.
>>>>>>>>>>>
>>>>>>>>>>> I really appreciate any help with
this.
>>>>>>>>>>>
>>>>>>>>>>> DIego
>>>>>>>>>
_______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>

Diego Remolina

2017-Jun-26 14:14 UTC

head link

[Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

Hi,

I was wondering if you have had a chance to review the logs and if you
can tell me know if more information is needed.

Diego

On Mon, Jun 12, 2017 at 8:20 AM, Diego Remolina <dijuremo at gmail.com>
wrote:> Did the logs provide any hints as to what the issue may be?
>
> Diego
>
> On Sat, Jun 3, 2017 at 12:16 PM, Diego Remolina <dijuremo at
gmail.com> wrote:
>> Thanks for taking the time to look into this. Since we needed downtime
>> due to the gluster update, we also updated the OS, including samba. We
>> went from 4.2.x to 4.4.4 and many other packages for CentOS were
>> updated as well. OS and samba updates were installed, then server
>> rebooted, then gluster was updated.
>>
>> Created a new test samba share to minimize logs, etc:
>>
>> [VfsGluster]
>>    path = /vfsgluster
>>    browseable = yes
>>    write list = @Staff,root, at Admin, at Managers
>>    writeable = yes
>>    guest ok = no
>>    create mask = 660
>>    directory mask = 770
>>    kernel share modes = No
>>    vfs objects = glusterfs
>>    glusterfs:loglevel = 7
>>    glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
>>    glusterfs:volume = export
>>    veto files = desktop.ini
>>
>>
>> I am attaching the logs for the test of the rename operation. The file
>> I try to access is called:2016-9-13_Herrin Gear BMW_REV-3
>> The timeline is as follows (times are UTC):
>>
>> 15:18:00 -> Change smb.conf to glusterfs:loglevel = 9 and reload
smbd
>> via  smbcontrol smbd reload-config
>> See log: glusterfs-vfsgluster.log
>> 15:19:31 -> Adjust gluster client and brick log levels to TRACE
>> export.log is the Gluster volume log
>> bricks-hdds-brick.log is the Brick log on ysmha02 (did you need the
>> other one?) Samba is running on ysmah02.
>>
>> 15:20:40 -> Get the extended attributes for all files in the bricks
on
>> both servers ysmha01 and ysmha02 before accessing the file:
>> cd /bricks/hdds/brick/vfsgluster/
>>  find . -type f -exec getfattr -m .  -d -e hex {} \; >>
>> /tmp/ysmha01-attributes-before-rename-run2
>>  find . -type f -exec getfattr -m .  -d -e hex {} \; >>
>> /tmp/ysmha02-attributes-before-rename-run2
>>
>> 15:21-ish -> Cannot open file so I do a rename and rename back via
>> Windows explorer
>> 15:22-ish -> Get the extended attirbutes again after renaming the
>> file. At this point I can open the file.
>>  find . -type f -exec getfattr -m .  -d -e hex {} \; >>
>> /tmp/ysmha01-attributes-after-rename-run2
>>  find . -type f -exec getfattr -m .  -d -e hex {} \; >>
>> /tmp/ysmha02-attributes-after-rename-run2
>>
>> I did all the files here, because Revit uses a file and some folders
>> to keep track of it changes, versions, etc.
>>
>> 15:23:25 -> Change gluster client and brick logs back to info.
>>
>> Let me know if I need to redo any tests, change any steps or provide
>> more logs I may have missed.
>>
>> Diego
>>
>> On Fri, Jun 2, 2017 at 8:19 PM, Raghavendra Talur <rtalur at
redhat.com> wrote:
>>>
>>>
>>> On 03-Jun-2017 3:27 AM, "Diego Remolina" <dijuremo at
gmail.com> wrote:
>>>
>>> Hi everyone,
>>>
>>> Is there anything else we could do to check on this problem and try
to
>>> fix it? The issue is definitively related to either the samba vfs
>>> gluster plugin or gluster itself. I am not sure how to pin it down
>>> futher.
>>>
>>>
>>> I don't think it is vfs plugin because you haven't updated
samba packages
>>> and vfs plugin is same as before.
>>>
>>> It is either gfapi in Gluster or one of the lower xlators.
>>>
>>> Changing to fuse based shares might make the problem go away and
can be used
>>> as a workaround.
>>>
>>>
>>>
>>> I went ahead and created a new share in the samba server which is
on a
>>> local filesystem where the OS is installed, not part of gluster:
>>>
>>> # mount | grep home
>>> ]# ls -ld /home
>>> drwxr-xr-x. 14 root root 4096 Jun  2 17:21 /home
>>>
>>> This shows this is not mounted from anywhere. so I created a folder
>>> and shared it:
>>>
>>> ]# ls -ld /home/rvtsharetest/
>>> drwxrwx---. 3 dijuremo Staff 95 Jun  2 17:31 /home/rvtsharetest/
>>>
>>> ]# tail -7 /etc/samba/smb.conf
>>> [rvtsharetest]
>>>    path=/home/rvtsharetest
>>>    browseable = yes
>>>    write list = @Staff,dijuremo
>>>    guest ok = yes
>>>    create mask = 664
>>>    directory mask = 775
>>>
>>> When accessing Revit files in this new share the problem is *not*
observed.
>>>
>>> When accessing Revit files on any of the samba shares that use the
vfs
>>> gluster plugin and are stored in the gluster volume, we see the
>>> problems.
>>>
>>> Further analysis of the issue is even more disconcerting. As you
may
>>> remember, I have found a workaround about renaming the file and
back
>>> to the original, where things work. Here is where it gets more
>>> interesting. This seems to be workstation dependent, not user
>>> dependent.
>>>
>>> 1. User1 logs into PC1 and tries to access the file and gets the
error
>>> message from Revit "ACCESS DENIED".
>>>
>>> 2. User1 uses windows explorer to go to the file location (we tried
>>> doing this from the server itself on the command line and it did
not
>>> change anything, i.e. su - User1 then mv command). User1 renames
the
>>> file, Revit.rvt -> Revit2.rvt, and clicks away for the rename to
take
>>> place. User1 immediately renames the file to the original,
Revit2.rvt
>>> -> Revit.rvt
>>>
>>> 3. User1 opens the file and everything works properly.
>>>
>>> 4. User2 logs into PC1 and tries to open the file. The file opens
and
>>> works properly.
>>>
>>> 5. User2 logs into PC2 and tries to open the file, the problem
comes
>>> up. User2 uses the rename trick and this fixes the problem on PC2.
>>> Even if User1 now comes to use PC2, User1 will have no problems
with
>>> the file.
>>>
>>> 6. User1 now goes to PC3 where nobody has used the file rename
trick,
>>> and experiences the problem. Only solution is to play the rename
trick
>>> again in PC3.
>>>
>>> So it seems you have to play the rename trick at least one per
>>> workstation and that "fixes" the issue for any user who
logs into that
>>> workstation.
>>>
>>> What other suggestions do you have? What debugging can I do next? I
am
>>> planning once everyone leaves the office today on changing the
share
>>> to bypass vfs gluster plugin and access the file directly from the
>>> fuse mount, ie.:
>>>
>>> # mount | grep export
>>> 10.0.1.7:/export on /export type fuse.glusterfs
>>> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
>>>
>>> Then set samba to share without using the vfs gluster plugin as
follows
>>>
>>> [Projects]
>>>    path = /export/projects
>>>    browseable = yes
>>>    write list = @Staff,root, at Admin, at Managers
>>>    writeable = yes
>>>    guest ok = no
>>>    create mask = 660
>>>    directory mask = 770
>>>
>>> This test will determine if the issue is the samba vfs gluster
plugin
>>> or if it is the fact that the file is stored in the gluster volume.
>>>
>>> Any other thoughts?
>>>
>>>
>>> I suspect this has to do with locking and such.
>>> Can create a new volume and share through vfs plugin.
>>> Set logging to number 5 for samba and Gluster volume set
client-log-level
>>> and server-log-level to TRACE.
>>> Copy one rvt file.
>>> Performs open, this is like fail.
>>> Perform rename
>>> Perform open, this would pass
>>>
>>> Send us the logs.
>>>
>>> I know it is troublesome but detailed logs are the only things that
would
>>> help us analyze the issue.
>>>
>>> Talur
>>>
>>>
>>> Diego
>>>
>>> On Wed, May 31, 2017 at 12:59 PM, Diego Remolina <dijuremo at
gmail.com> wrote:
>>>> The vfs gluster plugin for samba is linked against
libglusterfs.so.0
>>>> which is provided by glusterfs-libs-3.10.2-1.el7.x86_64, please
see
>>>> below.
>>>>
>>>> # ldd /usr/lib64/samba/vfs/glusterfs.so | grep glus
>>>>        libglusterfs.so.0 => /lib64/libglusterfs.so.0
(0x00007f61da858000)
>>>>
>>>> ]# yum provides /lib64/libglusterfs.so.0
>>>> Loaded plugins: fastestmirror
>>>> Loading mirror speeds from cached hostfile
>>>> * base: centos.vwtonline.net
>>>> * extras: mirror.cs.vt.edu
>>>> * updates: centosv.centos.org
>>>> glusterfs-libs-3.10.2-1.el7.x86_64 : GlusterFS common libraries
>>>> Repo        : @centos-gluster310
>>>> Matched from:
>>>> Filename    : /lib64/libglusterfs.so.0
>>>>
>>>>
>>>>
>>>> On Wed, May 31, 2017 at 12:39 PM, Diego Remolina <dijuremo
at gmail.com>
>>>> wrote:
>>>>> Samba is running in the same machine as glusterd. The
machines were
>>>>> rebooted after the upgrades and samba has been restarted a
few times.
>>>>>
>>>>> # rpm -qa | grep gluster
>>>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>
>>>>> # rpm -qa | grep samba
>>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>>
>>>>> # cat /etc/redhat-release
>>>>> CentOS Linux release 7.3.1611 (Core)
>>>>>
>>>>> I also raised the op version.
>>>>>
>>>>> # gluster volume get all cluster.op-version
>>>>> Option                                  Value
>>>>> ------                                  -----
>>>>> cluster.op-version                      31000
>>>>>
>>>>> # gluster volume get all cluster.max-op-version
>>>>> Option                                  Value
>>>>> ------                                  -----
>>>>> cluster.max-op-version                  31000
>>>>>
>>>>> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur
<rtalur at redhat.com>
>>>>> wrote:
>>>>>> Also, please attach your smb.conf. You can directly
attach in the list
>>>>>> and need not have a google drive link.
>>>>>>
>>>>>>
>>>>>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur
<rtalur at redhat.com>
>>>>>> wrote:
>>>>>>> Diego,
>>>>>>>
>>>>>>> I see that Samba is still using 3.6.6 Gluster. Is
it possible you did
>>>>>>> not restart smb after upgrading gluster(if samba is
on same machine as
>>>>>>> Gluster) or if you forgot to update Gluster client
packages on Samba
>>>>>>> node?
>>>>>>>
>>>>>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina
<dijuremo at gmail.com>
>>>>>>> wrote:
>>>>>>>> Please download the log file from this link:
>>>>>>>>
>>>>>>>>
https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>>>>>>>
>>>>>>>> Let me know if you need any other log files.
>>>>>>>>
>>>>>>>> Diego
>>>>>>>>
>>>>>>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra
Talur
>>>>>>>> <rtalur at redhat.com> wrote:
>>>>>>>>> If possible please share the glusterfs-*
log files from
>>>>>>>>> /var/log/samba.
>>>>>>>>>
>>>>>>>>> This might be because of
cluster.lookup-optimize. Adding Poornima and
>>>>>>>>> Raghavendra Gowdappa to help with this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 31, 2017 at 1:03 AM, Diego
Remolina <dijuremo at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>> This is a bit puzzling, not sure what
difference it would make, but:
>>>>>>>>>>
>>>>>>>>>> 1. Try to open file that has a problem,
ie. MyRevitFile.rvt
>>>>>>>>>> Revit opens and shows a window that
says access denied.
>>>>>>>>>>
>>>>>>>>>> 2. Rename file, i.e from windows
explorer right click, rename to
>>>>>>>>>> MyRevitFile2.rvt
>>>>>>>>>>
>>>>>>>>>> 3. Without opening the file, rename
file back to the original, i.e
>>>>>>>>>> MyRevitFile.rvt
>>>>>>>>>>
>>>>>>>>>> 4. Double click on file and now it will
open just fine without the
>>>>>>>>>> Access Denied error.
>>>>>>>>>>
>>>>>>>>>> Any explanation for this? Could the
rename operation be forcing or
>>>>>>>>>> updating some attributes that then
allow the file to open?
>>>>>>>>>>
>>>>>>>>>> Diego
>>>>>>>>>>
>>>>>>>>>> On Tue, May 30, 2017 at 10:57 AM, Diego
Remolina
>>>>>>>>>> <dijuremo at gmail.com> wrote:
>>>>>>>>>>> This is what I see in the logs set
from smb.conf via line ->
>>>>>>>>>>> glusterfs:logfile =
/var/log/samba/glusterfs-projects.log
>>>>>>>>>>>
>>>>>>>>>>> [2017-05-30 14:52:31.051524] E
[MSGID: 123001]
>>>>>>>>>>> [io-cache.c:564:ioc_open_cbk]
0-export-io-cache: inode context is
>>>>>>>>>>> NULL
>>>>>>>>>>>
(a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>>>>>>>>>>> [2017-05-30 14:52:31.241850] W
[MSGID: 114031]
>>>>>>>>>>>
[client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>>>>>>>>>> remote operation failed. Path:
>>>>>>>>>>>
/projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>>>> Richmond/02
>>>>>>>>>>> DRAWINGS/02
ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>>>> aef7-39766b323c14). Key:
glusterfs.get_real_filename:desktop.ini
>>>>>>>>>>> [Not
>>>>>>>>>>> a directory]
>>>>>>>>>>> [2017-05-30 14:52:31.242956] W
[MSGID: 114031]
>>>>>>>>>>>
[client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>>>>>>>>>> remote operation failed. Path:
>>>>>>>>>>>
/projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>>>> Richmond/02
>>>>>>>>>>> DRAWINGS/02
ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>>>> aef7-39766b323c14). Key:
glusterfs.get_real_filename:desktop.ini
>>>>>>>>>>> [Not
>>>>>>>>>>> a directory]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 30, 2017 at 10:37 AM,
Diego Remolina
>>>>>>>>>>> <dijuremo at gmail.com>
wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Over the weekend we updated a
two server glusterfs 3.6.6 install
>>>>>>>>>>>> to
>>>>>>>>>>>> 3.10.2 We also updated samba
and samba-vfs to the latest in
>>>>>>>>>>>> CentOS. I
>>>>>>>>>>>> enabled several of the newer
caching features from gluster 3.9 for
>>>>>>>>>>>> small file performance and
samba, and we now seem to have some
>>>>>>>>>>>> issues
>>>>>>>>>>>> with accessing files from
glusterfs. When users try to access some
>>>>>>>>>>>> files, they get a Permission
denied message. This seems to be only
>>>>>>>>>>>> via
>>>>>>>>>>>> samba as I am able to su -
username and then do strings on the
>>>>>>>>>>>> file.
>>>>>>>>>>>>
>>>>>>>>>>>> [root at ysmha02
gluster-backups]# rpm -qa | grep gluster
>>>>>>>>>>>>
glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>>>>>>>>>
glusterfs-server-3.10.2-1.el7.x86_64
>>>>>>>>>>>>
glusterfs-api-3.10.2-1.el7.x86_64
>>>>>>>>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>>>>>>>>>
glusterfs-cli-3.10.2-1.el7.x86_64
>>>>>>>>>>>>
centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>>>>>>>>>
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>>
glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>>>>>>>>>
glusterfs-libs-3.10.2-1.el7.x86_64
>>>>>>>>>>>>
glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>>>>>>>>
>>>>>>>>>>>> [root at ysmha02
gluster-backups]# rpm -qa | grep samba
>>>>>>>>>>>>
samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>>
samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>>
samba-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>>
samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>>
samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>>
samba-common-4.4.4-14.el7_3.noarch
>>>>>>>>>>>>
>>>>>>>>>>>> On the samba logs for the
machine I notice something weird, samba
>>>>>>>>>>>> seems to be trying to stat the
file we are trying as a directory
>>>>>>>>>>>> to
>>>>>>>>>>>> see if it contains desktop.ini:
>>>>>>>>>>>>
>>>>>>>>>>>> [2017/05/30 10:13:07.297026, 
0]
>>>>>>>>>>>>
../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>>> 
glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD
EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>>>> failed:
>>>>>>>>>>>> Not a directory
>>>>>>>>>>>> [2017/05/30 10:13:07.298155, 
0]
>>>>>>>>>>>>
../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>>> 
glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD
EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>>>> failed:
>>>>>>>>>>>> Not a directory
>>>>>>>>>>>>
>>>>>>>>>>>> This seems to be happening only
with files with the .rvt
>>>>>>>>>>>> extension.
>>>>>>>>>>>> Though these files are usually
larger in size vs other smaller
>>>>>>>>>>>> excel,
>>>>>>>>>>>> power point, etc files.
>>>>>>>>>>>>
>>>>>>>>>>>> Here are the complete options
for the volume:
>>>>>>>>>>>>
>>>>>>>>>>>> https://pastebin.com/ZH2vMsMN
>>>>>>>>>>>>
>>>>>>>>>>>> I turned off
performance.cache-samba-metadata again to see if that
>>>>>>>>>>>> would help, but seems it does
not help.
>>>>>>>>>>>>
>>>>>>>>>>>> I really appreciate any help
with this.
>>>>>>>>>>>>
>>>>>>>>>>>> DIego
>>>>>>>>>>
_______________________________________________
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>

Reasonably Related Threads

Search for more maybe matching threads

Gluster users - Jun 2017 - URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

[Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

Reasonably Related Threads