Jeremy Enos
2010-Mar-23 08:37 UTC
[Gluster-users] gluster local vs local = gluster x4 slower
This test is pretty easy to replicate anywhere- only takes 1 disk, one
machine, one tarball. Untarring to local disk directly vs thru gluster
is about 4.5x faster. At first I thought this may be due to a slow host
(Opteron 2.4ghz). But it's not- same configuration, on a much faster
machine (dual 3.33ghz Xeon) yields the performance below.
####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER####
[root at ac33 jenos]# time tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
real 0m41.290s
user 0m14.246s
sys 0m2.957s
####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)####
[root at ac33 jenos]# cd /export/jenos/
[root at ac33 jenos]# time tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
real 0m8.983s
user 0m6.857s
sys 0m1.844s
####THESE ARE TEST FILE DETAILS####
[root at ac33 jenos]# tar tzvf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz |wc -l
109
[root at ac33 jenos]# ls -l
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
-rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
[root at ac33 jenos]#
These are the relevant performance options I'm using in my .vol file:
#------------Performance Options-------------------
volume readahead
type performance/read-ahead
option page-count 4 # 2 is default option
option force-atime-update off # default is off
subvolumes ghome
end-volume
volume writebehind
type performance/write-behind
option cache-size 1MB
subvolumes readahead
end-volume
volume cache
type performance/io-cache
option cache-size 1GB
subvolumes writebehind
end-volume
What can I do to improve gluster's performance?
Jeremy
Tejas N. Bhise
2010-Mar-23 08:59 UTC
[Gluster-users] gluster local vs local = gluster x4 slower
Out of curiosity, if you want to do stuff only on one machine,
why do you want to use a distributed, multi node, clustered,
file system ?
Am I missing something here ?
Regards,
Tejas.
----- Original Message -----
From: "Jeremy Enos" <jenos at ncsa.uiuc.edu>
To: gluster-users at gluster.org
Sent: Tuesday, March 23, 2010 2:07:06 PM GMT +05:30 Chennai, Kolkata, Mumbai,
New Delhi
Subject: [Gluster-users] gluster local vs local = gluster x4 slower
This test is pretty easy to replicate anywhere- only takes 1 disk, one
machine, one tarball. Untarring to local disk directly vs thru gluster
is about 4.5x faster. At first I thought this may be due to a slow host
(Opteron 2.4ghz). But it's not- same configuration, on a much faster
machine (dual 3.33ghz Xeon) yields the performance below.
####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER####
[root at ac33 jenos]# time tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
real 0m41.290s
user 0m14.246s
sys 0m2.957s
####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)####
[root at ac33 jenos]# cd /export/jenos/
[root at ac33 jenos]# time tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
real 0m8.983s
user 0m6.857s
sys 0m1.844s
####THESE ARE TEST FILE DETAILS####
[root at ac33 jenos]# tar tzvf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz |wc -l
109
[root at ac33 jenos]# ls -l
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
-rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
[root at ac33 jenos]#
These are the relevant performance options I'm using in my .vol file:
#------------Performance Options-------------------
volume readahead
type performance/read-ahead
option page-count 4 # 2 is default option
option force-atime-update off # default is off
subvolumes ghome
end-volume
volume writebehind
type performance/write-behind
option cache-size 1MB
subvolumes readahead
end-volume
volume cache
type performance/io-cache
option cache-size 1GB
subvolumes writebehind
end-volume
What can I do to improve gluster's performance?
Jeremy
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Vikas Gorur
2010-Mar-23 18:52 UTC
[Gluster-users] gluster local vs local = gluster x4 slower
Jeremy, Can you post your complete volume file? ------------------------------ Vikas Gorur Engineer - Gluster, Inc. +1 (408) 770 1894 ------------------------------
Tejas N. Bhise
2010-Mar-24 03:27 UTC
[Gluster-users] gluster local vs local = gluster x4 slower
It might also be useful overall to know what you want to achieve. Its better to
do sizing, performance etc if there is clarity on what is to be achieved. Once
that is clear, it would be more useful to say if something is possible or not
with the config you are trying and why or why not and whether even the
expectations are justified or not from what is essentially a distributed
networked FS.
----- Original Message -----
From: "Jeremy Enos" <jenos at ncsa.uiuc.edu>
To: "Stephan von Krawczynski" <skraw at ithnet.com>
Cc: "Tejas N. Bhise" <tejas at gluster.com>, gluster-users at
gluster.org
Sent: Wednesday, March 24, 2010 5:41:28 AM GMT +05:30 Chennai, Kolkata, Mumbai,
New Delhi
Subject: Re: [Gluster-users] gluster local vs local = gluster x4 slower
Stephan is correct- I primarily did this test to show a demonstrable
overhead example that I'm trying to eliminate. It's pronounced enough
that it can be seen on a single disk / single node configuration, which
is good in a way (so anyone can easily repro).
My distributed/clustered solution would be ideal if it were fast enough
for small block i/o as well as large block- I was hoping that single
node systems would achieve that, hence the single node test. Because
the single node test performed poorly, I eventually reduced down to
single disk to see if it could still be seen, and it clearly can be.
Perhaps it's something in my configuration? I've pasted my config files
below.
thx-
Jeremy
######################glusterfsd.vol######################
volume posix
type storage/posix
option directory /export
end-volume
volume locks
type features/locks
subvolumes posix
end-volume
volume disk
type performance/io-threads
option thread-count 4
subvolumes locks
end-volume
volume server-ib
type protocol/server
option transport-type ib-verbs/server
option auth.addr.disk.allow *
subvolumes disk
end-volume
volume server-tcp
type protocol/server
option transport-type tcp/server
option auth.addr.disk.allow *
subvolumes disk
end-volume
######################ghome.vol######################
#-----------IB remotes------------------
volume ghome
type protocol/client
option transport-type ib-verbs/client
# option transport-type tcp/client
option remote-host acfs
option remote-subvolume raid
end-volume
#------------Performance Options-------------------
volume readahead
type performance/read-ahead
option page-count 4 # 2 is default option
option force-atime-update off # default is off
subvolumes ghome
end-volume
volume writebehind
type performance/write-behind
option cache-size 1MB
subvolumes readahead
end-volume
volume cache
type performance/io-cache
option cache-size 1GB
subvolumes writebehind
end-volume
######################END######################
On 3/23/2010 6:02 AM, Stephan von Krawczynski wrote:> On Tue, 23 Mar 2010 02:59:35 -0600 (CST)
> "Tejas N. Bhise"<tejas at gluster.com> wrote:
>
>
>> Out of curiosity, if you want to do stuff only on one machine,
>> why do you want to use a distributed, multi node, clustered,
>> file system ?
>>
> Because what he does is a very good way to show the overhead produced only
by
> glusterfs and nothing else (i.e. no network involved).
> A pretty relevant test scenario I would say.
>
> --
> Regards,
> Stephan
>
>
>
>> Am I missing something here ?
>>
>> Regards,
>> Tejas.
>>
>> ----- Original Message -----
>> From: "Jeremy Enos"<jenos at ncsa.uiuc.edu>
>> To: gluster-users at gluster.org
>> Sent: Tuesday, March 23, 2010 2:07:06 PM GMT +05:30 Chennai, Kolkata,
Mumbai, New Delhi
>> Subject: [Gluster-users] gluster local vs local = gluster x4 slower
>>
>> This test is pretty easy to replicate anywhere- only takes 1 disk, one
>> machine, one tarball. Untarring to local disk directly vs thru gluster
>> is about 4.5x faster. At first I thought this may be due to a slow
host
>> (Opteron 2.4ghz). But it's not- same configuration, on a much
faster
>> machine (dual 3.33ghz Xeon) yields the performance below.
>>
>> ####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER####
>> [root at ac33 jenos]# time tar xzf
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>
>> real 0m41.290s
>> user 0m14.246s
>> sys 0m2.957s
>>
>> ####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)####
>> [root at ac33 jenos]# cd /export/jenos/
>> [root at ac33 jenos]# time tar xzf
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>
>> real 0m8.983s
>> user 0m6.857s
>> sys 0m1.844s
>>
>> ####THESE ARE TEST FILE DETAILS####
>> [root at ac33 jenos]# tar tzvf
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz |wc -l
>> 109
>> [root at ac33 jenos]# ls -l
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>> -rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>> [root at ac33 jenos]#
>>
>> These are the relevant performance options I'm using in my .vol
file:
>>
>> #------------Performance Options-------------------
>>
>> volume readahead
>> type performance/read-ahead
>> option page-count 4 # 2 is default option
>> option force-atime-update off # default is off
>> subvolumes ghome
>> end-volume
>>
>> volume writebehind
>> type performance/write-behind
>> option cache-size 1MB
>> subvolumes readahead
>> end-volume
>>
>> volume cache
>> type performance/io-cache
>> option cache-size 1GB
>> subvolumes writebehind
>> end-volume
>>
>> What can I do to improve gluster's performance?
>>
>> Jeremy
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>
>