Hey Anoop. Thanks for looking into this. Responses inline:> On Sep 11, 2018, at 23:42, Anoop C S <anoopcs at autistici.org> wrote: > > On Tue, 2018-09-11 at 15:10 -0600, Terry McGuire wrote: >> Hello list. I had happily been sharing a Gluster volume with Samba using vfs_gluster, but it has >> recently stopped working right. I think it might have been after updating Samba from 4.6.2 to >> 4.7.1 (as part of updating CentOS 7.4 to 7.5). The shares suffer a variety of weird issues, >> including: >> >> - sporadic connection refusals (credentials are accepted as valid, but volume is unavailable) > > Does that mean after authentication share is not at all listed?On a Mac, after the auth dialog disappears (suggesting the auth was valid), a dialog appears saying the volume is unavailable. Can?t recall what the behaviour on Windows is, but it would be equivalent. (This error doesn?t happen often, so I can?t quickly reproduce it, and I can?t even quite be sure it?s related to this problem, but I suspect it is.)> >> - on Mac, when attempting to write a file: "The operation can?t be completed because an unexpected >> error occurred (error code -50)." > > How is this write performed? via Finder or via command-line?That error appears when using the Finder. It happens pretty much all the time, and is the clearest symptom of this problem. Using the command-line, anything that attempts I/O with the share gives an ?Invalid argument? error: Mac:~ root# ls /Volumes/ Macintosh HD module Mac:~ root# ls -l /Volumes/ ls: module: Invalid argument total 8 lrwxr-xr-x 1 root wheel 1 24 Aug 15:48 Macintosh HD -> / Mac:~ root# touch /Volumes/module/test touch: /Volumes/module/test: Invalid argument Mac:~ root#> >> - on Windows, sometimes when writing and sometimes when reading: "Z:\ is not accessible. The >> parameter is incorrect" > > How is this write performed? via Explorer or Powershell?Explorer.> >> -on Mac and Windows, the contents of the volume in Finder/Explorer windows sometimes disappears, >> sometimes reappearing later, sometimes not. >> - on Mac (and similar on Windows), volume icon sometimes disappears - maybe the volume unmounts, >> but it's unclear. > > Is this a clustered Samba setup i.e, with CTDB for high availability?Not clustered. Just a plain vanilla Samba.> >> All these issues vanish when I switch to sharing the FUSE-mounted volume, but, of course, I lose >> the advantages of vfs_gluster. > > Can you please attach the output of `testparm -s` so as to look through how Samba is setup?From our test server (?nomodule-nofruit? is currently the only well-behaved share): root at mfsuat-01 ~]#testparm -s Load smb config files from /etc/samba/smb.conf rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) Processing section "[share1]" Processing section "[share2]" Processing section "[nomodule]" Processing section "[nomodule-nofruit]" Processing section "[module]" Processing section "[IPC$]" WARNING: No path in service IPC$ - making it unavailable! NOTE: Service IPC$ is flagged unavailable. Loaded services file OK. idmap range not specified for domain '*' ERROR: Invalid idmap range for domain *! WARNING: You have some share names that are longer than 12 characters. These may not be accessible to some older clients. (Eg. Windows9x, WindowsMe, and smbclient prior to Samba 3.0.) WARNING: some services use vfs_fruit, others don't. Mounting them in conjunction on OS X clients results in undefined behaviour. Server role: ROLE_DOMAIN_MEMBER # Global parameters [global] log file = /var/log/samba/log.%m map to guest = Bad User max log size = 50 realm = XXXX.AD.UALBERTA.CA security = ADS workgroup = STS glusterfs:volume = mfs1 idmap config * : backend = tdb access based share enum = Yes force create mode = 0777 force directory mode = 0777 include = /mfsmount/admin/etc/mfs/smb_shares.conf kernel share modes = No read only = No smb encrypt = desired vfs objects = glusterfs [share1] path = /share1 valid users = @mfs-sa1 at xxxx.ad.ualberta.ca [share2] path = /share2 valid users = @mfs-test-group at xxxx.ad.ualberta.ca [nomodule] kernel share modes = Yes path = /mfsmount/share1 valid users = @mfs-sa1 at xxxx.ad.ualberta.ca vfs objects = fruit streams_xattr [nomodule-nofruit] kernel share modes = Yes path = /mfsmount/share1 valid users = @mfs-sa1 at xxxx.ad.ualberta.ca vfs objects = [module] path = /share1 valid users = @mfs-sa1 at xxxx.ad.ualberta.ca [IPC$] available = No vfs objects => >> My gluster version initially was 3.10.12. I?ve since updated to gluster 3.12.13, but the symptoms >> are the same. >> >> Does this sound familiar to anyone? > > All mentioned symptoms point towards a disconnection. We need to find out the origin of this > disconnection. What do we have in logs under /var/log/samba/? Any errors?Actually, yes. Large numbers of: [2018/09/12 09:37:17.873711, 0] ../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat) glfs_stat(.) failed: Invalid argument There appears to be some sort of connection remaining, as I can continue to cause these errors in the server log by attempting I/O with the share. This seems like the most promising lead to find the root cause. Hopefully you (or someone) can interpret what it means, and what I might do about it (besides not using vfs_gluster anymore). Regards, Terry
On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:> Hey Anoop. Thanks for looking into this. Responses inline: > > > On Sep 11, 2018, at 23:42, Anoop C S <anoopcs at autistici.org> wrote: > > > > On Tue, 2018-09-11 at 15:10 -0600, Terry McGuire wrote: > > > Hello list. I had happily been sharing a Gluster volume with Samba using vfs_gluster, but it > > > has > > > recently stopped working right. I think it might have been after updating Samba from 4.6.2 to > > > 4.7.1 (as part of updating CentOS 7.4 to 7.5). The shares suffer a variety of weird issues, > > > including: > > > > > > - sporadic connection refusals (credentials are accepted as valid, but volume is unavailable) > > > > Does that mean after authentication share is not at all listed? > > On a Mac, after the auth dialog disappears (suggesting the auth was valid), a dialog appears > saying the volume is unavailable. Can?t recall what the behaviour on Windows is, but it would be > equivalent. (This error doesn?t happen often, so I can?t quickly reproduce it, and I can?t even > quite be sure it?s related to this problem, but I suspect it is.) > > > > > > - on Mac, when attempting to write a file: "The operation can?t be completed because an > > > unexpected > > > error occurred (error code -50)." > > > > How is this write performed? via Finder or via command-line? > > That error appears when using the Finder. It happens pretty much all the time, and is the > clearest symptom of this problem. Using the command-line, anything that attempts I/O with the > share gives an ?Invalid argument? error: > > Mac:~ root# ls /Volumes/ > Macintosh HD module > > Mac:~ root# ls -l /Volumes/ > ls: module: Invalid argument > total 8 > lrwxr-xr-x 1 root wheel 1 24 Aug 15:48 Macintosh HD -> / > > Mac:~ root# touch /Volumes/module/test > touch: /Volumes/module/test: Invalid argument > Mac:~ root# > > > > > > - on Windows, sometimes when writing and sometimes when reading: "Z:\ is not accessible. The > > > parameter is incorrect" > > > > How is this write performed? via Explorer or Powershell? > > Explorer. > > > > > -on Mac and Windows, the contents of the volume in Finder/Explorer windows sometimes > > > disappears, > > > sometimes reappearing later, sometimes not. > > > - on Mac (and similar on Windows), volume icon sometimes disappears - maybe the volume > > > unmounts, > > > but it's unclear. > > > > Is this a clustered Samba setup i.e, with CTDB for high availability? > > Not clustered. Just a plain vanilla Samba. > > > > > All these issues vanish when I switch to sharing the FUSE-mounted volume, but, of course, I > > > lose > > > the advantages of vfs_gluster. > > > > Can you please attach the output of `testparm -s` so as to look through how Samba is setup? > > From our test server (?nomodule-nofruit? is currently the only well-behaved share): > > root at mfsuat-01 ~]#testparm -s > Load smb config files from /etc/samba/smb.conf > rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) > Processing section "[share1]" > Processing section "[share2]" > Processing section "[nomodule]" > Processing section "[nomodule-nofruit]" > Processing section "[module]" > Processing section "[IPC$]" > WARNING: No path in service IPC$ - making it unavailable! > NOTE: Service IPC$ is flagged unavailable. > Loaded services file OK. > idmap range not specified for domain '*' > ERROR: Invalid idmap range for domain *! > > WARNING: You have some share names that are longer than 12 characters. > These may not be accessible to some older clients. > (Eg. Windows9x, WindowsMe, and smbclient prior to Samba 3.0.) > WARNING: some services use vfs_fruit, others don't. Mounting them in conjunction on OS X clients > results in undefined behaviour. > > Server role: ROLE_DOMAIN_MEMBER > > # Global parameters > [global] > log file = /var/log/samba/log.%m > map to guest = Bad User > max log size = 50 > realm = XXXX.AD.UALBERTA.CA > security = ADS > workgroup = STS > glusterfs:volume = mfs1 > idmap config * : backend = tdb > access based share enum = Yes > force create mode = 0777 > force directory mode = 0777 > include = /mfsmount/admin/etc/mfs/smb_shares.conf > kernel share modes = No > read only = No > smb encrypt = desired > vfs objects = glusterfs > > > [share1] > path = /share1 > valid users = @mfs-sa1 at xxxx.ad.ualberta.ca > > > [share2] > path = /share2 > valid users = @mfs-test-group at xxxx.ad.ualberta.ca > > > [nomodule] > kernel share modes = Yes > path = /mfsmount/share1 > valid users = @mfs-sa1 at xxxx.ad.ualberta.ca > vfs objects = fruit streams_xattr > > > [nomodule-nofruit] > kernel share modes = Yes > path = /mfsmount/share1 > valid users = @mfs-sa1 at xxxx.ad.ualberta.ca > vfs objects = > > > [module] > path = /share1 > valid users = @mfs-sa1 at xxxx.ad.ualberta.ca > > > [IPC$] > available = No > vfs objects = > > > > > > > My gluster version initially was 3.10.12. I?ve since updated to gluster 3.12.13, but the > > > symptoms > > > are the same. > > >Did you mean 3.10.12 or 3.12.12?(your initial GlusterFS version)? If GlusterFS was at v3.10.12, are you sure that you were experiencing issues with samba v4.7.1?> > > Does this sound familiar to anyone? > > > > All mentioned symptoms point towards a disconnection. We need to find out the origin of this > > disconnection. What do we have in logs under /var/log/samba/? Any errors? > > Actually, yes. Large numbers of: > > [2018/09/12 09:37:17.873711, 0] ../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat) > glfs_stat(.) failed: Invalid argumentMight be related to https://bugzilla.samba.org/show_bug.cgi?id=13585. Can you try installing a version of GlusterFS < 3.12.13 and see if it works normally?> There appears to be some sort of connection remaining, as I can continue to cause these errors in > the server log by attempting I/O with the share. > > This seems like the most promising lead to find the root cause. Hopefully you (or someone) can > interpret what it means, and what I might do about it (besides not using vfs_gluster anymore). > > Regards, > Terry >
On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:> > Can you please attach the output of `testparm -s` so as to look through how Samba is setup?I have a setup where I could browse and work with a GlusterFS volume share made available to Windows via vfs_glusterfs module on CentOS 7.5.1804 with glusterfs-3.10.12-1.el7 and samba-4.7.1-9.el7_5. What am I missing? Are there any specific operation that leads to abnormal behaviours?> From our test server (?nomodule-nofruit? is currently the only well-behaved share): > > root at mfsuat-01 ~]#testparm -s > Load smb config files from /etc/samba/smb.conf > rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) > Processing section "[share1]" > Processing section "[share2]" > Processing section "[nomodule]" > Processing section "[nomodule-nofruit]" > Processing section "[module]" > Processing section "[IPC$]" > WARNING: No path in service IPC$ - making it unavailable! > NOTE: Service IPC$ is flagged unavailable.On an unrelated note: I don't think your intention to make [IPC$] unavailable using the 'available' parameter would work at all.> Loaded services file OK. > idmap range not specified for domain '*' > ERROR: Invalid idmap range for domain *!On an unrelated note: Why haven't you specified range for default configuration? I think it is a must to set range for the default configuration.> WARNING: You have some share names that are longer than 12 characters. > These may not be accessible to some older clients. > (Eg. Windows9x, WindowsMe, and smbclient prior to Samba 3.0.) > WARNING: some services use vfs_fruit, others don't. Mounting them in conjunction on OS X clients > results in undefined behaviour. > > Server role: ROLE_DOMAIN_MEMBER > > # Global parameters > [global] > log file = /var/log/samba/log.%m > map to guest = Bad User > max log size = 50 > realm = XXXX.AD.UALBERTA.CA > security = ADS > workgroup = STS > glusterfs:volume = mfs1 > idmap config * : backend = tdb > access based share enum = Yes > force create mode = 0777 > force directory mode = 0777 > include = /mfsmount/admin/etc/mfs/smb_shares.conf > kernel share modes = No > read only = No > smb encrypt = desired > vfs objects = glusterfs > [share1] > path = /share1 > valid users = @mfs-sa1 at xxxx.ad.ualberta.ca > [share2] > path = /share2 > valid users = @mfs-test-group at xxxx.ad.ualberta.caOh.. you are sharing sub-directories which is also fine.> [nomodule] > kernel share modes = Yes > path = /mfsmount/share1 > valid users = @mfs-sa1 at xxxx.ad.ualberta.ca > vfs objects = fruit streams_xattrInteresting.. Even this FUSE mounted GlusterFS share is not behaving normal? What errors do you see in glusterfs fuse mount log(/var/log/glusterfs/mfsmount-.log) while accessing this share?> > > [nomodule-nofruit] > kernel share modes = Yes > path = /mfsmount/share1 > valid users = @mfs-sa1 at xxxx.ad.ualberta.ca > vfs objects = > > > [module] > path = /share1 > valid users = @mfs-sa1 at xxxx.ad.ualberta.ca > [IPC$] > available = No > vfs objects =You may remove the whole [IPC$] section.> > > My gluster version initially was 3.10.12. I?ve since updated to gluster 3.12.13, but the > > > symptoms > > > are the same. > > > > > > Does this sound familiar to anyone? > > > > All mentioned symptoms point towards a disconnection. We need to find out the origin of this > > disconnection. What do we have in logs under /var/log/samba/? Any errors? > > Actually, yes. Large numbers of: > > [2018/09/12 09:37:17.873711, 0] ../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat) > glfs_stat(.) failed: Invalid argument > > There appears to be some sort of connection remaining, as I can continue to cause these errors in > the server log by attempting I/O with the share. > > This seems like the most promising lead to find the root cause. Hopefully you (or someone) can > interpret what it means, and what I might do about it (besides not using vfs_gluster anymore). > > Regards, > Terry