Dan,
Does something like this help for SAMBA?
http://www.e-rave.nl/disable-creation-of-ds_store-files-on-samba-shares
Negative lookups are a particularly expensive operation on distributed
filesystems, GlusterFS in particular. It has to check through each
brick to see that the file *really* doesn't exist, increasing the
lookup overhead immensely. If samba doesn't pass these queries to
GlusterFS, it might help a lot.
--
Adam Tygart
Beocat Sysadmin
On Wed, Dec 19, 2012 at 8:29 PM, Daniel Mons <daemons at kanuka.com.au>
wrote:> I'm rolling out 4 lots of 6-node GlusterFS setups for my employer.
Each
> node is ~33TB of RAID6 backed storage (16x 3TB SATA disks in RAID6 with a
> hot spare hanging off an LSI controller, with 2x SSDs configured for
> caching), and Gluster is configured in distribute-replicate. Each cluster
> is 200TB of raw space, 100TB usable after replication. When complete,
there
> will be 4 of these clusters.
>
> Nodes are configured as XFS with 512byte inodes, running a fully patched
> CentOS6 and Gluster 3.3.1. Each node has a 6 core Xeon processor (with HT
> for 12 threads) with 32GB of RAM. Each node runs 2x 10Gbps Ethernet over
> fiber in a bonded configuration (single IP address per node) for a full
> 20Gbits per node.
>
> GlusterFS FUSE performance under Linux is great (clients run a mix of
Ubuntu
> 12.04 LTS for workstations and CentOS6 for servers). Samba performance
back
> to Windows 7 clients is great. NFS performance via both Gluster's
userspace
> setup as well as CentOS6's native NFS4 kernel server are great to most
other
> systems where we can't get the Gluster FUSE client loaded (large
> industry-specific Linux boxes that are provided by vendors as a "black
box"
> solution, and only allow limited access via NFS or SMB/CIFS). All testing
> so far under those conditions proves orders of magnitude faster throughput
> than our existing single NAS solutions.
>
> MacOSX Finder performance is a problem, however. There's a huge bug in
> MacOSX itself that prevents using NFS at all (discussions on other mailing
> lists suggest it occurred somewhere around 10.6, and continues through into
> 10.7 and 10.8).
>
> Mounting via SMB under OSX is more stable than NFS, however in folders with
> a large amount of files, Finder goes looking for a corresponding Apple
> Resource Fork file (for every "filename.ext", it looks for a
> "._filename.ext"). Running tcpdump and wireshark on the Gluster
nodes shows
> that the resulting "FILE_NOT_FOUND" error back to the client
takes a very
> long time. Configuring a single node as a pure NAS with the same software
> (but no Gluster implementation) is lightening fast. As soon as GlusterFS
> comes in to play, reporting of each "FILE_NOT_FOUND" slows down
the process
> dramatically, causing a directory with ~1000 images in it to take well over
> 5 minutes to display the contents in MacOSX finder.
>
> This problem is resolved somewhat by switching to AFP (via Netatalk loaded
> on the GlusterFS nodes), but it has it's own problems unique to that
> protocol, and I'd rather stick to GlusterFS-FUSE, NFS or SMB in that
order
> of preference.
>
> It's worth noting that through the terminal, these problems don't
exist.
> Mounting via SMB, browsing to the volume in terminal and running
"ls" or
> "find" style commands retrieve file listings at a similar speed
to Linux and
> Windows. The problem is limited to clients using Finder to browse
> directories, and again particularly ones with a large number of files that
> don't have matching Apple Resource Fork files. (Of note, creating
empty
> files of the matching "._filename.ext" format solves the
performance
> problem, but litters our filestores with millions of empty files, which we
> don't want).
>
> I understand the problem is not strictly Gluster's issue. Finder is
looking
> for a heck of a lot of files that don't exist (which is a pretty silly
> design), and it tends to occur only with Samba re-exporting GlusterFS
> volumes that we can see. And likewise Apple's NFS bug that has now
been in
> existence across three releases of their OS is pretty horrible. But
> hopefully I can at least describe the problem and prompt some testing by
> others.
>
> I haven't had a chance to test a MacOSX FUSE client due to time
constraints,
> but that would at least answer the question if the problem is Gluster's
lag
> in reporting of files not found, or Samba's.
>
> -Dan
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users