Gluster users - Sep 2018 - vfs_gluster broken

On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:
> > Can you please attach the output of `testparm -s` so as to look
through how Samba is setup?
I have a setup where I could browse and work with a GlusterFS volume share made
available to Windows
via vfs_glusterfs module on CentOS 7.5.1804 with glusterfs-3.10.12-1.el7 and
samba-4.7.1-9.el7_5.

What am I missing? Are there any specific operation that leads to abnormal
behaviours?
> From our test server (?nomodule-nofruit? is currently the only well-behaved
share):
> 
> root at mfsuat-01 ~]#testparm -s
> Load smb config files from /etc/samba/smb.conf
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
> Processing section "[share1]"
> Processing section "[share2]"
> Processing section "[nomodule]"
> Processing section "[nomodule-nofruit]"
> Processing section "[module]"
> Processing section "[IPC$]"
> WARNING: No path in service IPC$ - making it unavailable!
> NOTE: Service IPC$ is flagged unavailable.
On an unrelated note:

I don't think your intention to make [IPC$] unavailable using the
'available' parameter would work
at all. 
> Loaded services file OK.
> idmap range not specified for domain '*'
> ERROR: Invalid idmap range for domain *!
On an unrelated note:

Why haven't you specified range for default configuration? I think it is a
must to set range for the
default configuration.
> WARNING: You have some share names that are longer than 12 characters.
> These may not be accessible to some older clients.
> (Eg. Windows9x, WindowsMe, and smbclient prior to Samba 3.0.)
> WARNING: some services use vfs_fruit, others don't. Mounting them in
conjunction on OS X clients
> results in undefined behaviour.
> 
> Server role: ROLE_DOMAIN_MEMBER
> 
> # Global parameters
> [global]
> 	log file = /var/log/samba/log.%m
> 	map to guest = Bad User
> 	max log size = 50
> 	realm = XXXX.AD.UALBERTA.CA
> 	security = ADS
> 	workgroup = STS
> 	glusterfs:volume = mfs1
> 	idmap config * : backend = tdb
> 	access based share enum = Yes
> 	force create mode = 0777
> 	force directory mode = 0777
> 	include = /mfsmount/admin/etc/mfs/smb_shares.conf
> 	kernel share modes = No
> 	read only = No
> 	smb encrypt = desired
> 	vfs objects = glusterfs
> [share1]
> 	path = /share1
> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
> [share2]
> 	path = /share2
> 	valid users = @mfs-test-group at xxxx.ad.ualberta.ca
Oh.. you are sharing sub-directories which is also fine.
> [nomodule]
> 	kernel share modes = Yes
> 	path = /mfsmount/share1
> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
> 	vfs objects = fruit streams_xattr
Interesting..

Even this FUSE mounted GlusterFS share is not behaving normal? What errors do
you see in glusterfs
fuse mount log(/var/log/glusterfs/mfsmount-.log) while accessing this share?
> > 
> [nomodule-nofruit]
> 	kernel share modes = Yes
> 	path = /mfsmount/share1
> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
> 	vfs objects = 
> 
> 
> [module]
> 	path = /share1
> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
> [IPC$]
> 	available = No
> 	vfs objects = 
You may remove the whole [IPC$] section.
> > > My gluster version initially was 3.10.12.  I?ve since updated to
gluster 3.12.13, but the
> > > symptoms
> > > are the same.
> > > 
> > > Does this sound familiar to anyone?
> > 
> > All mentioned symptoms point towards a disconnection. We need to find
out the origin of this
> > disconnection. What do we have in logs under /var/log/samba/? Any
errors?
> 
> Actually, yes.  Large numbers of:
> 
> [2018/09/12 09:37:17.873711,  0]
../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat)
>   glfs_stat(.) failed: Invalid argument
> 
> There appears to be some sort of connection remaining, as I can continue to
cause these errors in
> the server log by attempting I/O with the share.
> 
> This seems like the most promising lead to find the root cause.  Hopefully
you (or someone) can
> interpret what it means, and what I might do about it (besides not using
vfs_gluster anymore).
> 
> Regards,
> Terry

> On Sep 19, 2018, at 06:37, Anoop C S <anoopcs at autistici.org>
wrote:
> 
> On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:
>>> Can you please attach the output of `testparm -s` so as to look
through how Samba is setup?
> 
> I have a setup where I could browse and work with a GlusterFS volume share
made available to Windows
> via vfs_glusterfs module on CentOS 7.5.1804 with glusterfs-3.10.12-1.el7
and samba-4.7.1-9.el7_5.
> 
> What am I missing? Are there any specific operation that leads to abnormal
behaviours?
It seems that, when using smbclient to keep things simple, the first command in
the share?s root works, even if it?s a write.  The following commands, even if
it?s just an ls, fails.

One difference that might be making the difference is that my gluster volume is
distributed-dispersed.  This seems also to break the vfs_fruit module.

>> From our test server (?nomodule-nofruit? is currently the only
well-behaved share):
>> 
>> root at mfsuat-01 ~]#testparm -s
>> Load smb config files from /etc/samba/smb.conf
>> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit
(16384)
>> Processing section "[share1]"
>> Processing section "[share2]"
>> Processing section "[nomodule]"
>> Processing section "[nomodule-nofruit]"
>> Processing section "[module]"
>> Processing section "[IPC$]"
>> WARNING: No path in service IPC$ - making it unavailable!
>> NOTE: Service IPC$ is flagged unavailable.
> 
> On an unrelated note:
> 
> I don't think your intention to make [IPC$] unavailable using the
'available' parameter would work
> at all.
I assume this IPC$ stuff is just an artifact of how I built my smb.conf file.  I
put "vfs objects = glusterfs? into the global stanza, so it didn?t need to
be added to each share stanza, but I discovered that this broke IPC$ in such a
way as to make no share accessible.  To fix this, I added a share stanza for
IPC$ that includes only ?vfs objects =".  Maybe that?s dumb, but it worked,
and, I guess, results in what we see here.

>> Loaded services file OK.
>> idmap range not specified for domain '*'
>> ERROR: Invalid idmap range for domain *!
> 
> On an unrelated note:
> 
> Why haven't you specified range for default configuration? I think it
is a must to set range for the
> default configuration.
I guess because I?m clueless enough not to know that I should.  We?re
authenticating samba users from Windows Active Directory, and I?m really
clueless about that.  We got things to the point that they were working, and
called it a day.

>> WARNING: You have some share names that are longer than 12 characters.
>> These may not be accessible to some older clients.
>> (Eg. Windows9x, WindowsMe, and smbclient prior to Samba 3.0.)
>> WARNING: some services use vfs_fruit, others don't. Mounting them
in conjunction on OS X clients
>> results in undefined behaviour.
>> 
>> Server role: ROLE_DOMAIN_MEMBER
>> 
>> # Global parameters
>> [global]
>> 	log file = /var/log/samba/log.%m
>> 	map to guest = Bad User
>> 	max log size = 50
>> 	realm = XXXX.AD.UALBERTA.CA
>> 	security = ADS
>> 	workgroup = STS
>> 	glusterfs:volume = mfs1
>> 	idmap config * : backend = tdb
>> 	access based share enum = Yes
>> 	force create mode = 0777
>> 	force directory mode = 0777
>> 	include = /mfsmount/admin/etc/mfs/smb_shares.conf
>> 	kernel share modes = No
>> 	read only = No
>> 	smb encrypt = desired
>> 	vfs objects = glusterfs
>> [share1]
>> 	path = /share1
>> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
>> [share2]
>> 	path = /share2
>> 	valid users = @mfs-test-group at xxxx.ad.ualberta.ca
> 
> Oh.. you are sharing sub-directories which is also fine.
We *only* share subdirs of the gluster volume, never the volume itself.

>> [nomodule]
>> 	kernel share modes = Yes
>> 	path = /mfsmount/share1
>> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca <mailto:mfs-sa1 at
xxxx.ad.ualberta.ca>
>> 	vfs objects = fruit streams_xattr
> 
> Interesting..
> 
> Even this FUSE mounted GlusterFS share is not behaving normal? What errors
do you see in glusterfs
> fuse mount log(/var/log/glusterfs/mfsmount-.log) while accessing this
share?
This one isn?t affected by the current samba issue, but it?s affected by another
issue.  (I actually posted about it a while ago - subject ?vfs_fruit and
extended attributes?- and I think you responded, but it didn?t go anywhere.)  We
discovered that the vfs_fruit module seems to interact poorly with our
distributed-dispersed gluster volume.  To learn this, we made a test replicated
volume and it worked fine.

Ironically, we decided we could live without vfs_fruit when we discovered
vfs_glusterfs.  Now we have neither.  :-(

>> [nomodule-nofruit]
>> 	kernel share modes = Yes
>> 	path = /mfsmount/share1
>> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca <mailto:mfs-sa1 at
xxxx.ad.ualberta.ca>
>> 	vfs objects = 
>> 
>> 
>> [module]
>> 	path = /share1
>> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca <mailto:mfs-sa1 at
xxxx.ad.ualberta.ca>
>> [IPC$]
>> 	available = No
>> 	vfs objects = 
> 
> You may remove the whole [IPC$] section.
IPC$ stuff explained above.

Terry

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180920/2c126ed0/attachment.html>