Gluster users - Jul 2009 - Gluster 2.0.3 + Apache on CentOS5 performance issue

Hello,

We have been evaluating the choice for the new platform for a webboard
system.
The webboard is PHP scripts that generate/modify HTML page when user
posting/add comment to the page, resulting topic is actually stored as a
HTML file with all related file (file attach to the topic, etc.. )stored in
its own directory for each topic. In general, the web site mostly serve a
lot of small static files using Apache while using PHP to do other dynamic
contents. This system has been working very well in the past, with the
increasing page view rate, it is very likely that we will need some kind of
Cluster file system as backend very soon.

We have set up a test system using Grinder as stress test tool. The test
system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock Apache
(prefork, since the goal is to use this with PHP), linked together with
Gigabit Ethernet. We try to compare the performance of either using single
NFS server in sync mode against using 4 Gluster nodes (distribute of 2
replicated nodes) through Fuse. However, the transaction per second (TPS)
result is not good.

NFS (single server, sync mode)
 - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean response
time = 61.63 ms
 - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 ms
 - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
 - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = 287.33ms

4 nodes Gluster (2 distribute of replicated 2 node)
- 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = 207.33ms
- 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = 672.67ms
- 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
(no 400-600 threads since we run out of client machine, sorry).

gfsd is configured to use 32 thread of iothread as brick. gfs-client is
configured to use
io-cache->write-behind->readahead->distribute->replicate.
io-cache cache-size is 256MB. I used patched Fuse downloaded from Gluster
web-site (build through DKMS).

As the result yield, it seems that Gluster performance worse with increasing
no. of client. One observation is that the glusterfs process on client is
taking about 100% of CPU during all the tests. glusterfsd is utilizing only
70-80% of CPUs during the test time. Note that system is Dual core.

I also tried using modglusterfs and not using fuse at all to serve all the
static files and conduct another test with Grinder. The result is about the
same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in this test that
each Apache prefork process used more about twice more memory and we need to
lower number of httpd processes by about half.

I tried disable EnableMMAP and it didn't help much. Adjusting readahead,
write behind according to GlusterOptimization page didn't help much either.

My question is, there seems to be bottleneck in this setup, but how can I
track this? Note that, I didn't do any other optimization other than what
said above. Are there any best practice configuration for using Apache to
serve a bunch of small static files like this around?

Regards,

Somsak
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090712/b590cae7/attachment.html>

Sorry, I forgot to include informatino about my testbed.

4 machines - Gluster (as described below)
3 machines - WWW server, load balance using LVS
1 machine - LVS direct routing and weighted lease connection
3 machines - clients, 100 threads per client.

2009/7/12 Somsak Sriprayoonsakul <somsaks at gmail.com>
> Hello,
>
> We have been evaluating the choice for the new platform for a webboard
> system.
> The webboard is PHP scripts that generate/modify HTML page when user
> posting/add comment to the page, resulting topic is actually stored as a
> HTML file with all related file (file attach to the topic, etc.. )stored in
> its own directory for each topic. In general, the web site mostly serve a
> lot of small static files using Apache while using PHP to do other dynamic
> contents. This system has been working very well in the past, with the
> increasing page view rate, it is very likely that we will need some kind of
> Cluster file system as backend very soon.
>
> We have set up a test system using Grinder as stress test tool. The test
> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock Apache
> (prefork, since the goal is to use this with PHP), linked together with
> Gigabit Ethernet. We try to compare the performance of either using single
> NFS server in sync mode against using 4 Gluster nodes (distribute of 2
> replicated nodes) through Fuse. However, the transaction per second (TPS)
> result is not good.
>
> NFS (single server, sync mode)
>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
> response time = 61.63 ms
>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 ms
>  - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = 287.33ms
>
> 4 nodes Gluster (2 distribute of replicated 2 node)
> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = 207.33ms
> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = 672.67ms
> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
> (no 400-600 threads since we run out of client machine, sorry).
>
> gfsd is configured to use 32 thread of iothread as brick. gfs-client is
> configured to use
io-cache->write-behind->readahead->distribute->replicate.
> io-cache cache-size is 256MB. I used patched Fuse downloaded from Gluster
> web-site (build through DKMS).
>
> As the result yield, it seems that Gluster performance worse with
> increasing no. of client. One observation is that the glusterfs process on
> client is taking about 100% of CPU during all the tests. glusterfsd is
> utilizing only 70-80% of CPUs during the test time. Note that system is
Dual
> core.
>
> I also tried using modglusterfs and not using fuse at all to serve all the
> static files and conduct another test with Grinder. The result is about the
> same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in this test that
> each Apache prefork process used more about twice more memory and we need
to
> lower number of httpd processes by about half.
>
> I tried disable EnableMMAP and it didn't help much. Adjusting
readahead,
> write behind according to GlusterOptimization page didn't help much
either.
>
> My question is, there seems to be bottleneck in this setup, but how can I
> track this? Note that, I didn't do any other optimization other than
what
> said above. Are there any best practice configuration for using Apache to
> serve a bunch of small static files like this around?
>
> Regards,
>
> Somsak
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090713/9cb33ac8/attachment.html>

We use mod_gluster and Apache
2.2 with good results.  We also ran into the same issue as you that we
ran out of memory past 150 threads even on a 8gig machine.  We got
around this by compiling Apache using mpm-worker
(threads) instead of prefork - it uses 1/4 as much ram with the same number
of connections (150-200) and everything has been running smoothly.  I cannot
see any performance difference except it using way less memory.
liam


On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul
<somsaks at gmail.com>wrote:
> Hello,
>
> We have been evaluating the choice for the new platform for a webboard
> system.
> The webboard is PHP scripts that generate/modify HTML page when user
> posting/add comment to the page, resulting topic is actually stored as a
> HTML file with all related file (file attach to the topic, etc.. )stored in
> its own directory for each topic. In general, the web site mostly serve a
> lot of small static files using Apache while using PHP to do other dynamic
> contents. This system has been working very well in the past, with the
> increasing page view rate, it is very likely that we will need some kind of
> Cluster file system as backend very soon.
>
> We have set up a test system using Grinder as stress test tool. The test
> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock Apache
> (prefork, since the goal is to use this with PHP), linked together with
> Gigabit Ethernet. We try to compare the performance of either using single
> NFS server in sync mode against using 4 Gluster nodes (distribute of 2
> replicated nodes) through Fuse. However, the transaction per second (TPS)
> result is not good.
>
> NFS (single server, sync mode)
>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
> response time = 61.63 ms
>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 ms
>  - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = 287.33ms
>
> 4 nodes Gluster (2 distribute of replicated 2 node)
> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = 207.33ms
> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = 672.67ms
> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
> (no 400-600 threads since we run out of client machine, sorry).
>
> gfsd is configured to use 32 thread of iothread as brick. gfs-client is
> configured to use
io-cache->write-behind->readahead->distribute->replicate.
> io-cache cache-size is 256MB. I used patched Fuse downloaded from Gluster
> web-site (build through DKMS).
>
> As the result yield, it seems that Gluster performance worse with
> increasing no. of client. One observation is that the glusterfs process on
> client is taking about 100% of CPU during all the tests. glusterfsd is
> utilizing only 70-80% of CPUs during the test time. Note that system is
Dual
> core.
>
> I also tried using modglusterfs and not using fuse at all to serve all the
> static files and conduct another test with Grinder. The result is about the
> same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in this test that
> each Apache prefork process used more about twice more memory and we need
to
> lower number of httpd processes by about half.
>
> I tried disable EnableMMAP and it didn't help much. Adjusting
readahead,
> write behind according to GlusterOptimization page didn't help much
either.
>
> My question is, there seems to be bottleneck in this setup, but how can I
> track this? Note that, I didn't do any other optimization other than
what
> said above. Are there any best practice configuration for using Apache to
> serve a bunch of small static files like this around?
>
> Regards,
>
> Somsak
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>

I haven't tried an apples to apples comparison with Apache+mod_gluster vs
Apache+fuse+gluster however i do run both setups.  I load tested both setups
so to verified it could handle 4x our normal daily load and left it at that.
 I didn't actually compare the two (although that might be cool to do
someday).
I really like the idea of Apache+mod_gluster as I don't have to deal with
the whole fuse and mounting the filesystem.  It always scares me having a
public facing webserver with your whole backend fileshare mounted locally.
 Its very slick for serving content such as media files.  We serve audio
content to our CDN with a pair of Apache/mod_gluster servers - pushing
200-300mbit on average daily and everything works very well.

We run an apache+fuse+gluster setup because we need to run some mod_perl
before serving the actual content.  However performance is still very good.
 We do around 50-100 requests (all jpeg images) per second off of a fuse
mount and everything works great.  We also have a java tomcat+fuse+gluster
service which does image manipulation on the fly off of a gluster mount.

We have two backend gluster servers using replication which serve all this
content.

If you would like more information on our setup id be happy to share
offline.  Just email me privately.

thanks,
liam

On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
<somsaks at gmail.com>wrote:
> Oh thank you, thought noone will reply me :)
>
> Have you tried Apache + Fuse over GlusterFS? How is the performance?
>
> Also, anyone in this mailing-list have tried Apache with booster? I tried
> it but Apache refuse to start (just hang and freeze).
>
> 2009/7/23 Liam Slusser <lslusser at gmail.com>
>
>
>> We use mod_gluster and Apache
>> 2.2 with good results.  We also ran into the same issue as you that we
ran out of memory past 150 threads even on a 8gig machine.  We got around this
by compiling Apache using mpm-worker
>> (threads) instead of prefork - it uses 1/4 as much ram with the same
number
>> of connections (150-200) and everything has been running smoothly.  I
cannot
>> see any performance difference except it using way less memory.
>> liam
>>
>>
>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
>> somsaks at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> We have been evaluating the choice for the new platform for a
webboard
>>> system.
>>> The webboard is PHP scripts that generate/modify HTML page when
user
>>> posting/add comment to the page, resulting topic is actually stored
as a
>>> HTML file with all related file (file attach to the topic, etc..
)stored in
>>> its own directory for each topic. In general, the web site mostly
serve a
>>> lot of small static files using Apache while using PHP to do other
dynamic
>>> contents. This system has been working very well in the past, with
the
>>> increasing page view rate, it is very likely that we will need some
kind of
>>> Cluster file system as backend very soon.
>>>
>>> We have set up a test system using Grinder as stress test tool. The
test
>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
Apache
>>> (prefork, since the goal is to use this with PHP), linked together
with
>>> Gigabit Ethernet. We try to compare the performance of either using
single
>>> NFS server in sync mode against using 4 Gluster nodes (distribute
of 2
>>> replicated nodes) through Fuse. However, the transaction per second
(TPS)
>>> result is not good.
>>>
>>> NFS (single server, sync mode)
>>>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
>>> response time = 61.63 ms
>>>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
ms
>>>  - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>>>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt
>>> 287.33ms
>>>
>>> 4 nodes Gluster (2 distribute of replicated 2 node)
>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
207.33ms
>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
672.67ms
>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
>>> (no 400-600 threads since we run out of client machine, sorry).
>>>
>>> gfsd is configured to use 32 thread of iothread as brick.
gfs-client is
>>> configured to use
io-cache->write-behind->readahead->distribute->replicate.
>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from
Gluster
>>> web-site (build through DKMS).
>>>
>>> As the result yield, it seems that Gluster performance worse with
>>> increasing no. of client. One observation is that the glusterfs
process on
>>> client is taking about 100% of CPU during all the tests. glusterfsd
is
>>> utilizing only 70-80% of CPUs during the test time. Note that
system is Dual
>>> core.
>>>
>>> I also tried using modglusterfs and not using fuse at all to serve
all
>>> the static files and conduct another test with Grinder. The result
is about
>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
this test
>>> that each Apache prefork process used more about twice more memory
and we
>>> need to lower number of httpd processes by about half.
>>>
>>> I tried disable EnableMMAP and it didn't help much. Adjusting
readahead,
>>> write behind according to GlusterOptimization page didn't help
much either.
>>>
>>> My question is, there seems to be bottleneck in this setup, but how
can I
>>> track this? Note that, I didn't do any other optimization other
than what
>>> said above. Are there any best practice configuration for using
Apache to
>>> serve a bunch of small static files like this around?
>>>
>>> Regards,
>>>
>>> Somsak
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>
>>>
>>
>