On 28 October 2013 23:13, Tim van Elteren <timvanelteren at gmail.com>
wrote:
Background: VFX and Post Production studio that relies on Gluster for
all of it's production data storage. Data is mostly in the shape of
image sequences (raw 2K and 4K footage stored in individual frames,
clocking in at 5-100MB per frame, to be played back at 24 or 48 FPS).
We create content from a render farm consisting of many Linux boxes
spewing out these images as fast as they can.
> 1) transparency to the processing layer / application with respect to data
> locality, e.g. know where data is physically located on a server level,
> mainly for resource allocation and fast processing, high performance, how
> can this be accomplished using GlusterFS?
We mount local GlusterFS clusters (we have them at each site) under
the same mount point. We simply rsync chunks of the tree between
sites as required on a job by job basis (we have "pipeline tools" we
write to make this easier for end users, so they call a wrapper
script, and not rsync directly). There are remote mount points for
users to mount a remote site over our WAN and browse it without
needing to rsync big chunks of data if they prefer. This isn't
GlusterFS's problem, per se, but just a regular remote file system
problem. With that said, GlusterFS's geo-replication may assist you
if you want to automate it more.
> 2) posix compliance, or conformance: hadoop for example isn't posix
> compliant by design, what are the pro's and con's? What is
GlusterFSs
> approach with respect to support for posix operations?
We run a setup of 70% Linux, 20% MacOSX and 10% Windows, all of which
require regular file semantics (not object storage). The 90% of our
client base that requires a POSIX compliant file system has no
problems with it. We re-export all shares via Samba3 for Windows
clients, but we are in the process of totally removing Windows from
our setup for a variety of reasons (mostly because it isn't POSIX
complaint, and we need to support a whole new set of tools to deal
with it, and it sucks for our industry).
Anything we need to do on a regular EXT3/EXT4 file system, we can do
on GlusterFS (symlinks, hard links, tools like "rsnapshot", etc).
Users don't see any difference between it and any regular NFS-mounted
NAS type device.
> 3) mainly with respect to evaluating the production readiness of GlusterFS,
> where is it currently used in production environments and for what specific
> usecases it seems most suitable? Are there any known issues / common
> pitfalls and workarounds available?
We've been running 3.3.1 since March this year, and I'm upgrading to
3.4.1 over the coming weeks. I've had outages, but it's never been
Gluster's fault. I initially put AFP for our Macs on our Gluster
nodes, which was a mistake, and has since been removed (MacOSX Finder
is too slow over SMB due to constant resource fork negative lookups,
and we've finally figured out how to get Macs to NFS mount from
Gluster without locking up Finder anyway so we're migrating away from
AFP). Likewise we've had some hardware and network switch failures
that have taken out multiple nodes. Recovery was quick enough though
(I think 30 minutes was the worst outage, but again not Gluster's
fault).
Because we use a distribute+replicate setup, our single-threaded write
speeds aren't amazing. But Gluster's power is in the cluster itself,
not any one write thread, so over the course of millions of rendered
frames, the performance is better than any single NAS device could
give us under the same client load.
Some applications do silly things - we have a handful of apps where
the "Open File" dialog insists on doing background reads of whatever
tree it's browsing, making file browsing excruciatingly slow. But
again, these are rare and typically only on Windows, so we'll be
leaving that behind soon. The other drama we had was a cpio write
operation from one of our production apps was very slow to GlusterFS
(GlusterFS doesn't seem to like anything that requests a portion of a
file only), so we wrote a wrapper script to save to a local tmpfs, and
then copy that back to GlusterFS. That was only for one operation out
of thousands though, and was easy enough to solve (and it gave us the
ability to extend version control into the wrapper script, which we'd
wanted to do anyway).
> 4) It seems Gluster has the advantage in Geo replication versus for example
> Ceph. What are the main advantages here?
We don't use this, so I can't comment. If I required my remote and
local trees to be in sync, I'd definitely use it. But end users here
are happy to transfer data only when required (because even with 800TB
online, space is still precious).
> 5) Finally what would be the most compelling reason to go for Gluster and
> not for the alternatives?
For us, we needed simple file semantics (we specifically don't need
object storage for OpenStack or Hadoop type operations). That gave us
4 options:
1) Continue with our legacy setup of many NAS units. Pros: cheap.
Cons: inflexible to share storage space between departments, single
point of failure per share
2) Buy a NAS or SAN from a vendor. Pros: simple, easily expandible.
Cons: proprietary, expensive, vendor/product lock-in for future
upgrades
3) Proprietary clustered file system (IBM GPFS, etc). Same pros/cons
as a SAN, quite frankly.
4) Ceph or Lustre. Pros: open source, usual clustered storage
benefits, central name space, etc Cons: in-kernel, required many
nodes for a good high-availability setup to start with, needs a few
smart people around to keep it running
5) GlusterFS: Pros: open source, low node count for basic rollout
(cheaper to start), usual clustered storage benefits, MUCH simpler
than Ceph / Lustre. Cons: usual clustered storage cons (single
threaded write speed, etc), young-ish technology
GlusterFS won the day due to simplicity and the ability to start
small. it keeps up with our business needs, and lets us pick and
choose our hardware (including mixing and matching different
specs/vendors of hardware within the one cluster into the future).
Additionally, RedHat's backing of Gluster finally cemented for us that
it was worthwhile, and it wasn't going to be a project that just
vanished overnight or suffered from production-breaking changes
between releases.
As suggested, you should test for yourself, and not take anyone's word
on anything. But for us, they were the decisions we made, and why.
HTH.
--
Dan Mons
R&D SysAdmin
Unbreaker of broken things
Cutting Edge
http://cuttingedge.com.au