It seems not many folks have much to say about this.
Here?s an update for anyone in a similar situation:
We narrowed down the issue to being with dispersed volumes. When we created a
replicated volume, vfs_fruit worked without issue.
Ultimately, what we were seeking was performance. Even for our Windows and
Linux users, things could be slow. The good news is that we discovered an SMB
config that helped immensely: vfs_glusterfs. In retrospect, I suppose it?s a
bit obvious, but we somehow failed to discover this module. Using it, instead
of pointing Samba at the Gluster volume mounted with FUSE, we?re seeing 3x
performance improvement with small files.
The bad news is that it looks like these 2 vfs modules can?t be used together.
I can?t make ?em work anyway.
The good news about the bad news is that, with the vfs_glusterfs speedup, things
are fast enough on our dispersed volume, even for Mac users, that we?re ok
without vfs_fruit. That said, it?d still be nice to get both vfs_fruit and
vfs_glusterfs working at the same time. And, of course, to get vfs_fruit to
play nice with dispersed volumes.
Hope that helps anyone else out there dealing with something like this.
Regards,
Terry
> On Sep 22, 2017, at 15:38, Terry McGuire <tmcguire at ualberta.ca>
wrote:
>
> Hello Anoop. Thanks for helping with this!
>
>> On Sep 22, 2017, at 00:11, Anoop C S <anoopcs at autistici.org>
wrote:
>>
>> On Thu, 2017-09-21 at 10:35 -0600, Terry McGuire wrote:
>>> Hello list. I?m attempting to improve how Samba shares directories
on our Gluster volume to Mac
>>> users by using the vfs_fruit module.
>>
>> What versions of GlusterFS and Samba have you installed? And which
platform/distro are you using as
>> Samba server?
>
> glusterfs-3.10.5-1.el7.x86_64
> samba-4.4.4-14.el7_3.x86_64
>
> CentOS Linux release 7.3.1611 (Core)
>
> This is a brand new service. We?re running gluster clients, and samba, on
the servers themselves.
>
>>
>> Please paste the output of `gluster volume info <VOLNAME>`.
>
> (Apologies for the length of this?)
>
> root at mfs-01 ~]#gluster volume info mfs1
>
> Volume Name: mfs1
> Type: Distributed-Disperse
> Volume ID: 2fa02e5d-95b4-4aaa-b16c-5de90e0b11b2
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 6 x (8 + 4) = 72
> Transport-type: tcp
> Bricks:
> Brick1: mfs-b01:/mnt/gfs001/data
> Brick2: mfs-b01:/mnt/gfs002/data
> Brick3: mfs-b01:/mnt/gfs003/data
> Brick4: mfs-b02:/mnt/gfs019/data
> Brick5: mfs-b02:/mnt/gfs020/data
> Brick6: mfs-b02:/mnt/gfs021/data
> Brick7: mfs-b03:/mnt/gfs037/data
> Brick8: mfs-b03:/mnt/gfs038/data
> Brick9: mfs-b03:/mnt/gfs039/data
> Brick10: mfs-b04:/mnt/gfs055/data
> Brick11: mfs-b04:/mnt/gfs056/data
> Brick12: mfs-b04:/mnt/gfs057/data
> Brick13: mfs-b01:/mnt/gfs004/data
> Brick14: mfs-b01:/mnt/gfs005/data
> Brick15: mfs-b01:/mnt/gfs006/data
> Brick16: mfs-b02:/mnt/gfs022/data
> Brick17: mfs-b02:/mnt/gfs023/data
> Brick18: mfs-b02:/mnt/gfs024/data
> Brick19: mfs-b03:/mnt/gfs040/data
> Brick20: mfs-b03:/mnt/gfs041/data
> Brick21: mfs-b03:/mnt/gfs042/data
> Brick22: mfs-b04:/mnt/gfs058/data
> Brick23: mfs-b04:/mnt/gfs059/data
> Brick24: mfs-b04:/mnt/gfs060/data
> Brick25: mfs-b01:/mnt/gfs007/data
> Brick26: mfs-b01:/mnt/gfs008/data
> Brick27: mfs-b01:/mnt/gfs009/data
> Brick28: mfs-b02:/mnt/gfs025/data
> Brick29: mfs-b02:/mnt/gfs026/data
> Brick30: mfs-b02:/mnt/gfs027/data
> Brick31: mfs-b03:/mnt/gfs043/data
> Brick32: mfs-b03:/mnt/gfs044/data
> Brick33: mfs-b03:/mnt/gfs045/data
> Brick34: mfs-b04:/mnt/gfs061/data
> Brick35: mfs-b04:/mnt/gfs062/data
> Brick36: mfs-b04:/mnt/gfs063/data
> Brick37: mfs-b01:/mnt/gfs010/data
> Brick38: mfs-b01:/mnt/gfs011/data
> Brick39: mfs-b01:/mnt/gfs012/data
> Brick40: mfs-b02:/mnt/gfs028/data
> Brick41: mfs-b02:/mnt/gfs029/data
> Brick42: mfs-b02:/mnt/gfs030/data
> Brick43: mfs-b03:/mnt/gfs046/data
> Brick44: mfs-b03:/mnt/gfs047/data
> Brick45: mfs-b03:/mnt/gfs048/data
> Brick46: mfs-b04:/mnt/gfs064/data
> Brick47: mfs-b04:/mnt/gfs065/data
> Brick48: mfs-b04:/mnt/gfs066/data
> Brick49: mfs-b01:/mnt/gfs013/data
> Brick50: mfs-b01:/mnt/gfs014/data
> Brick51: mfs-b01:/mnt/gfs015/data
> Brick52: mfs-b02:/mnt/gfs031/data
> Brick53: mfs-b02:/mnt/gfs032/data
> Brick54: mfs-b02:/mnt/gfs033/data
> Brick55: mfs-b03:/mnt/gfs049/data
> Brick56: mfs-b03:/mnt/gfs050/data
> Brick57: mfs-b03:/mnt/gfs051/data
> Brick58: mfs-b04:/mnt/gfs067/data
> Brick59: mfs-b04:/mnt/gfs068/data
> Brick60: mfs-b04:/mnt/gfs069/data
> Brick61: mfs-b01:/mnt/gfs016/data
> Brick62: mfs-b01:/mnt/gfs017/data
> Brick63: mfs-b01:/mnt/gfs018/data
> Brick64: mfs-b02:/mnt/gfs034/data
> Brick65: mfs-b02:/mnt/gfs035/data
> Brick66: mfs-b02:/mnt/gfs036/data
> Brick67: mfs-b03:/mnt/gfs052/data
> Brick68: mfs-b03:/mnt/gfs053/data
> Brick69: mfs-b03:/mnt/gfs054/data
> Brick70: mfs-b04:/mnt/gfs070/data
> Brick71: mfs-b04:/mnt/gfs071/data
> Brick72: mfs-b04:/mnt/gfs072/data
> Options Reconfigured:
> features.quota-deem-statfs: on
> features.inode-quota: on
> features.quota: on
> nfs.disable: on
> transport.address-family: inet
>
>>
>>> This module does wonders for speeding listings and downloads of
directories with large numbers of
>>> files in the Finder, but it kills uploads dead. Finder gives an
error:
>>>
>>> The Finder can?t complete the operation because some data in
?[filename]? can?t be read or
>>> written.
>>> (Error code -36)
>>
>> Can you please check for any errors in brick logs under
/var/log/glusterfs/bricks/ while you face
>> this Finder error?
>
> There?s stuff in those logs, but nothing is added when the error occurs.
>
>>
>>> The man page for the module indicates that the vfs_streams_xattr
module must also be loaded, like
>>> this:
>>>
>>> vfs objects = fruit streams_xattr
>>
>> Can you please share the output of `testparm -s`?
>
> root at mfs-01 ~]#testparm -s
> Load smb config files from /etc/samba/smb.conf
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
> Processing section "[sa1]"
> Processing section "[nongluster]"
> Processing section "[share1]"
> Processing section "[share2]"
> Loaded services file OK.
> Server role: ROLE_DOMAIN_MEMBER
>
> # Global parameters
> [global]
> realm = XXXX.UALBERTA.CA
> workgroup = STS
> log file = /var/log/samba/log.%m
> max log size = 50
> server min protocol = SMB2
> map to guest = Bad User
> security = ADS
> idmap config * : backend = tdb
> access based share enum = Yes
> force create mode = 0777
> force directory mode = 0777
> smb encrypt = desired
> vfs objects = fruit streams_xattr
>
>
> [sa1]
> path = /mfsmount/sa1
> read only = No
> valid users = @mfs-sa1 at XXXX.ualberta.ca
>
>
> [nongluster]
> path = /nongluster
> read only = No
> valid users = @mfs-sa1 at XXXX.ualberta.ca
>
>
> [share1]
> path = /mfsmount/share1
> read only = No
> valid users = @mfs-ga at XXXX.ualberta.ca
>
>
> [share2]
> path = /mfsmount/share2
> read only = No
> valid users = @mfs-gb at XXXX.ualberta.ca
>
>
>>> I?ve done this, and I?ve futzed with various combinations of
options, but haven?t got writes
>>> working.
>>
>> Is that a simple copy operation? Did you verify whether the file
content itself is written
>> completely? We have experienced similar error while copying large files
into GlusterFS volumes where
>> the real copy operation was completed successfully and the remaining
extra extended attribute
>> setting failed which resulted in a Finder error. Just wanted to check
whether it is the same case or
>> not?
>
> Yup, just a simple drag and drop of a file into the share?s window. No
content is written, though a file of zero size is created.
>
> Woah. I?ve just discovered something. While *files* cause the error,
*folders* sometimes do not. I can drag and drop certain folders of files and it
works fine, but if I drag the files individually, it fails. Other folders fail,
but some of the files in the folder are successful. Perhaps the Finder attaches
problem metadata to files but only under certain circumstances.
>
>>
>>> This issue seems specific to Gluster, as I can share a directory
that?s not in the Gluster volume
>>> and it works fine. I?m guessing the problem has something to do
with extended attributes, but,
>>> near as I can tell, the Gluster volume ought to do xattrs just
fine, as its bricks are XFS.
>>>
>>> Anyone have any experience with this? Anyone have vfs_fruit
working with Gluster?
>>
>> Apart from this write issue, are you facing any other issues while
accessing GlusterFS volumes from
>> Mac via Samba using vfs_fruit module? We would like to really fix any
obvious errors which stops Mac
>> users working with GlusterFS volumes.
>
> No other issues. We really want to use this module, as it dramatically
speeds up certain things, like opening folders with many items, or copying large
numbers of small files.
>
> Regards,
> Terry
>