Hi Ben,
Sorry I didn't reply earlier, one week of much needed vacation got in
the way...
Anyway, here's what we're trying to do with glusterfs for now. Not much,
but I figured it would be a test:
Supporting a group that uses genomics data, we have a WHOLE bunch of
data that our users keep piling on. We figured we needed a place to
backup all this data (we're talking 100s of TBs, quickly reaching PBs).
In the past, we have been doing some quick backups for disaster recovery
by using rsync to several different machines, separating the data by
[project/group/user] depending on how much data they had. This becomes a
problem because some machines run out of space while others have plenty
of room. So we figured: "What if we make it all one big glusterfs
filesystem, then we can add storage as we need it for everybody, remove
old systems when they become obsolete, and keep it all under one
namespace". We don't need blazing speed, as this is all just disaster
recovery, backups run about every week. We figured the first backup
would take a while, but after that it would just be incrementals using
rsync, so it would be OK. The data is mixed, with some people keeping
some really large files, some people keeping millions of small files,
and some people somewhere in between.
We started our testing with a 2 node glusterfs system. Then we compared
a copy of several TBs to that system vs. copying to an individual
machine. The glusterfs copy was almost 2 orders of magnitude slower.
Which led me to believe we had done something wrong, or we needed to do
some performance tuning. That's when I found out that there's plenty of
information about the basic setup of glusterfs, but not much when you're
trying to get beyond that.
Thanks for your help, we will look at the pointers you gave us, and will
probably report on our results as we get them.
Jorge
On 01/16/15 12:57, Ben Turner wrote:> ----- Original Message -----
>> From: "Jorge Garcia" <jgarcia at soe.ucsc.edu>
>> To: gluster-users at gluster.org
>> Sent: Thursday, January 15, 2015 1:24:35 PM
>> Subject: [Gluster-users] Getting past the basic setup
>>
>>
>> We're looking at using glusterfs for some of our storage needs. I
have
>> read the "Quick start", was able to create a basic
filesystem, and that
>> all worked. But wow, was it slow! I'm not sure if it was due to our
very
>> simple setup, or if that's just the way it is. So, we're trying
to look
>> deeper into options to improve performance, etc. In particular, I'm
>> interested in the NUFA scheduler. But after hours of googling around
for
>> more information, I haven't gotten anywhere. There's a page
with
>> "translator tutorials", but all the links are dead. Jeff
Darcy seems to
>> be working at Red Hat these days, so maybe all his open-source stuff
got
>> removed. I haven't found anything about how to even start setting
up a
>> NUFA scheduled system. Any pointers of where to even start?
>
> Hi Jorge. Here are some of my favorite gluster tuning DOCs:
>
>
https://rhsummit.files.wordpress.com/2014/04/bengland_h_1100_rhs_performance.pdf
>
http://rhsummit.files.wordpress.com/2013/07/england_th_0450_rhs_perf_practices-4_neependra.pdf
> http://rhsummit.files.wordpress.com/2012/03/england-rhs-performance.pdf
>
> They are all pretty much the same just updated as new features were
implemented. I usually think of gluster perf for sequential workloads like:
>
> writes = 1/2 * NIC bandwidth(when using replica 2) - 20%(with a fast enough
back end to service this)
>
> .5 * 1250 - 250 = ~500 MB / sec
>
> reads = NIC bandwidth - 40%
>
> 1250 * .7 = ~750 MB / sec
>
> If you aren't seeing between 400-500 MB / sec sequential writes and
600-750 MB / sec sequential reads on 10G NICs with fast enough back end to
service this then be sure to tune what is suggested in that DOC. What is your
HW like? What are your performance requirements? Just to note, glusterFS
performance really starts to degrade when writing in under 64KB block sizes.
What size of files and what record / block size are you writing in?
>
> -b
>
>> Any help would be extremely appreciated!
>>
>> Thanks,
>>
>> Jorge
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>