Gluster users - Sep 2018 - vfs_gluster broken

On Mon, 2018-09-17 at 10:31 -0600, Terry McGuire wrote:
> > On Sep 14, 2018, at 02:32, Anoop C S <anoopcs at autistici.org>
wrote:
> > 
> > On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:
> > > > > My gluster version initially was 3.10.12.  I?ve since
updated
> > > > > to gluster 3.12.13, but the
> > > > > symptoms
> > > > > are the same.
> > > > > 
> > 
> > Did you mean 3.10.12 or 3.12.12?(your initial GlusterFS version)?
> > 
> > If GlusterFS was at v3.10.12, are you sure that you were
> > experiencing issues with samba v4.7.1?
> 
> Definitely 3.10.12.  Our production service is still using it:
> 
> root at mfs-01 ~]#gluster --version
> glusterfs 3.10.12
> 
> 
> Definitely samba 4.7.1:
> 
> root at mfs-01 ~]#smbstatus --version
> Version 4.7.1
Let me confirm one last time:
In your production environment, with GlusterFS v3.10.12 and Samba
v4.7.1, can you confirm whether issues are visible?

If so, do you see any issues for files/directories other than those
present directly under root of the share?

Also is it possible for you to check with `smbclient` utility from a
Linux machine to see whether issues are seen?
> As an experiment, I?ve now upgraded our test machines all the way to
> gluster 4.1.4, but the symptoms are the same.
Yeah.. because the fix for upstream samba bug I mentioned in my last
reply is not yet available with any released version in 4.7.x series.
Thus updating GlusterFS will not get rid of that bug.
> > > Actually, yes.  Large numbers of:
> > > 
> > > [2018/09/12 09:37:17.873711,  0]
> > > ../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat)
> > >  glfs_stat(.) failed: Invalid argument
> > 
> > Might be related to 
> > https://bugzilla.samba.org/show_bug.cgi?id=13585.
> > 
> > Can you try installing a version of GlusterFS < 3.12.13 and see if
> > it works normally?
> 
> This bug does seem to fit the symptoms, however my prod service is
> already < 3.12.3 and is not normal.
I hope you meant < 3.12.13 this time :-)
> Perhaps my best option is to try downgrading samba to the original
> version and see if it restores proper function, until a patched
> version of Gluster is available.
As I explained above, issue was with Samba and fix has not yet made
into a release version. In short, following combinations should work
fine:

* samba version > 4.7.10 + any supported GlusterFS version
* samba version <= 4.7.10 + any supported GlusterFS version < 3.12.13
> Or just live with slow FUSE-backed SMB for the time being.
Let's track down the issue if there is one which is not yet identified.

> On Sep 18, 2018, at 04:44, Anoop C S <anoopcs at autistici.org>
wrote:
> 
> On Mon, 2018-09-17 at 10:31 -0600, Terry McGuire wrote:
>>> On Sep 14, 2018, at 02:32, Anoop C S <anoopcs at
autistici.org> wrote:
>>> 
>>> On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:
>>>>>> My gluster version initially was 3.10.12.  I?ve since
updated
>>>>>> to gluster 3.12.13, but the
>>>>>> symptoms
>>>>>> are the same.
>>>>>> 
>>> 
>>> Did you mean 3.10.12 or 3.12.12?(your initial GlusterFS version)?
>>> 
>>> If GlusterFS was at v3.10.12, are you sure that you were
>>> experiencing issues with samba v4.7.1?
>> 
>> Definitely 3.10.12.  Our production service is still using it:
>> 
>> root at mfs-01 ~]#gluster --version
>> glusterfs 3.10.12
>> 
>> 
>> Definitely samba 4.7.1:
>> 
>> root at mfs-01 ~]#smbstatus --version
>> Version 4.7.1
> 
> Let me confirm one last time:
> In your production environment, with GlusterFS v3.10.12 and Samba
> v4.7.1, can you confirm whether issues are visible?
Can confirm.  Issues visible with GlusterFS v3.10.12 and Samba v4.7.1.

> If so, do you see any issues for files/directories other than those
> present directly under root of the share?
Interesting!  When I use Mac or Windows, I still see issues deeper in the
hierarchy, but when I use Linux, I actually don?t see issues when I?m deeper in
the hierarchy.

> Also is it possible for you to check with `smbclient` utility from a
> Linux machine to see whether issues are seen?
Yup, I see the same error as you mentioned in the bug report:

smb: \> mkdir test
NT_STATUS_INVALID_PARAMETER making remote directory \test

>>> Might be related to 
>>> https://bugzilla.samba.org/show_bug.cgi?id=13585
<https://bugzilla.samba.org/show_bug.cgi?id=13585>.
>>> 
>>> Can you try installing a version of GlusterFS < 3.12.13 and see
if
>>> it works normally?
>> 
>> This bug does seem to fit the symptoms, however my prod service is
>> already < 3.12.3 and is not normal.
> 
> I hope you meant < 3.12.13 this time :-)
I did (but, pedantically-speaking, it is also < 3.12.3).

>> Perhaps my best option is to try downgrading samba to the original
>> version and see if it restores proper function, until a patched
>> version of Gluster is available.
> 
> As I explained above, issue was with Samba and fix has not yet made
> into a release version. In short, following combinations should work
> fine:
> 
> * samba version > 4.7.10 + any supported GlusterFS version
> * samba version <= 4.7.10 + any supported GlusterFS version < 3.12.13
> 
>> Or just live with slow FUSE-backed SMB for the time being.
> 
> Let's track down the issue if there is one which is not yet identified.

Okay.  It does seem, considering the versions I?m running, that there?s
something else (but similar) going on.

Terry


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180918/4515e629/attachment.html>