Gluster users - Sep 2018 - vfs_gluster broken

> On Sep 14, 2018, at 02:32, Anoop C S <anoopcs at autistici.org>
wrote:
> 
> On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:
>> 
>>>> My gluster version initially was 3.10.12.  I?ve since updated
to gluster 3.12.13, but the
>>>> symptoms
>>>> are the same.
>>>> 
> 
> Did you mean 3.10.12 or 3.12.12?(your initial GlusterFS version)?
> 
> If GlusterFS was at v3.10.12, are you sure that you were experiencing
issues with samba v4.7.1?
Definitely 3.10.12.  Our production service is still using it:

root at mfs-01 ~]#gluster --version
glusterfs 3.10.12


Definitely samba 4.7.1:

root at mfs-01 ~]#smbstatus --version
Version 4.7.1


As an experiment, I?ve now upgraded our test machines all the way to gluster
4.1.4, but the symptoms are the same.

>> Actually, yes.  Large numbers of:
>> 
>> [2018/09/12 09:37:17.873711,  0]
../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat)
>>  glfs_stat(.) failed: Invalid argument
> 
> Might be related to https://bugzilla.samba.org/show_bug.cgi?id=13585.
> 
> Can you try installing a version of GlusterFS < 3.12.13 and see if it
works normally?
This bug does seem to fit the symptoms, however my prod service is already <
3.12.3 and is not normal.

Perhaps my best option is to try downgrading samba to the original version and
see if it restores proper function, until a patched version of Gluster is
available.  Or just live with slow FUSE-backed SMB for the time being.

Terry

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180917/0177307d/attachment.html>

On Mon, 2018-09-17 at 10:31 -0600, Terry McGuire wrote:
> > On Sep 14, 2018, at 02:32, Anoop C S <anoopcs at autistici.org>
wrote:
> > 
> > On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:
> > > > > My gluster version initially was 3.10.12.  I?ve since
updated
> > > > > to gluster 3.12.13, but the
> > > > > symptoms
> > > > > are the same.
> > > > > 
> > 
> > Did you mean 3.10.12 or 3.12.12?(your initial GlusterFS version)?
> > 
> > If GlusterFS was at v3.10.12, are you sure that you were
> > experiencing issues with samba v4.7.1?
> 
> Definitely 3.10.12.  Our production service is still using it:
> 
> root at mfs-01 ~]#gluster --version
> glusterfs 3.10.12
> 
> 
> Definitely samba 4.7.1:
> 
> root at mfs-01 ~]#smbstatus --version
> Version 4.7.1
Let me confirm one last time:
In your production environment, with GlusterFS v3.10.12 and Samba
v4.7.1, can you confirm whether issues are visible?

If so, do you see any issues for files/directories other than those
present directly under root of the share?

Also is it possible for you to check with `smbclient` utility from a
Linux machine to see whether issues are seen?
> As an experiment, I?ve now upgraded our test machines all the way to
> gluster 4.1.4, but the symptoms are the same.
Yeah.. because the fix for upstream samba bug I mentioned in my last
reply is not yet available with any released version in 4.7.x series.
Thus updating GlusterFS will not get rid of that bug.
> > > Actually, yes.  Large numbers of:
> > > 
> > > [2018/09/12 09:37:17.873711,  0]
> > > ../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat)
> > >  glfs_stat(.) failed: Invalid argument
> > 
> > Might be related to 
> > https://bugzilla.samba.org/show_bug.cgi?id=13585.
> > 
> > Can you try installing a version of GlusterFS < 3.12.13 and see if
> > it works normally?
> 
> This bug does seem to fit the symptoms, however my prod service is
> already < 3.12.3 and is not normal.
I hope you meant < 3.12.13 this time :-)
> Perhaps my best option is to try downgrading samba to the original
> version and see if it restores proper function, until a patched
> version of Gluster is available.
As I explained above, issue was with Samba and fix has not yet made
into a release version. In short, following combinations should work
fine:

* samba version > 4.7.10 + any supported GlusterFS version
* samba version <= 4.7.10 + any supported GlusterFS version < 3.12.13
> Or just live with slow FUSE-backed SMB for the time being.
Let's track down the issue if there is one which is not yet identified.