Gluster users - Jul 2010 - Gluster web-farm production environment

Hi list

I would like to share with you the issues I have been suffered with
glusterfs in a web-farm production environment.

I hope it can help anybody and if anybody can help me on this

I created a glusterfs environment like the following:

6 servers, amd64, 4 cores, 4/8 Gb RAM, 1TB disk each one
kernel 2.6.26-2-xen-amd64
In each physical server, there is a xen (or two) virtual machine running
Phy server only serves glusterfs to the clients
Xen servers (4 apache, 2 mysql, 2 load balancers -LVS -- and a plesk server)

----------- phy server -----------
volume posix1
   type storage/posix
   option directory /mnt/export
end-volume

volume locks1
     type features/locks
     subvolumes posix1
end-volume

volume brick1
     type performance/io-threads
     option thread-count 8
     subvolumes locks1
end-volume

volume server-tcp
     type protocol/server
     option transport-type tcp
     option auth.addr.brick1.allow *
     option transport.socket.listen-port 6996
     option transport.socket.nodelay on
     subvolumes brick1
end-volume
--------------

5 xen servers has this configuration

------------- xen server------------
volume nodo1
     type protocol/client
     option transport-type tcp
     option remote-host 10.0.0.11
     option transport.socket.nodelay on
     option remote-port 6996
     option remote-subvolume brick1
end-volume

volume nodo2
     type protocol/client
     option transport-type tcp
     option remote-host 10.0.0.12
     option transport.socket.nodelay on
     option remote-port 6996
     option remote-subvolume brick1
end-volume

volume nodo3
     type protocol/client
     option transport-type tcp
     option remote-host 10.0.0.13
     option transport.socket.nodelay on
     option remote-port 6996
     option remote-subvolume brick1
end-volume

volume nodo4
     type protocol/client
     option transport-type tcp
     option remote-host 10.0.0.14
     option transport.socket.nodelay on
     option remote-port 6996
     option remote-subvolume brick1
end-volume

volume nodo5
     type protocol/client
     option transport-type tcp
     option remote-host 10.0.0.15
     option transport.socket.nodelay on
     option remote-port 6996
     option remote-subvolume brick1
end-volume

volume nodo6
     type protocol/client
     option transport-type tcp
     option remote-host 10.0.0.16
     option transport.socket.nodelay on
     option remote-port 6996
     option remote-subvolume brick1
end-volume

volume mirror-0
     type cluster/replicate
     subvolumes nodo1 nodo2 nodo3
end-volume

volume mirror-1
     type cluster/replicate
     subvolumes nodo4 nodo5 nodo6
end-volume

volume distribute
     type cluster/distribute
     subvolumes mirror-0 mirror-1
end-volume

volume readahead
     type performance/read-ahead
     option page-count 4
     subvolumes distribute
end-volume

volume iocache
     type performance/io-cache
     option cache-size `echo $(( $(grep 'MemTotal' /proc/meminfo | sed 
's/[^0-9]//g') / 5120 ))`MB
     option cache-timeout 1
     subvolumes readahead
end-volume

volume quickread
     type performance/quick-read
     option cache-timeout 1
     option max-file-size 512kb
     subvolumes iocache
end-volume

volume writebehind
     type performance/write-behind
     option cache-size 4MB
     subvolumes quickread
end-volume

volume statprefetch
     type performance/stat-prefetch
     subvolumes writebehind
end-volume
-----------------------

There is a dedicated switch to serve glusterfs, and the 2nd nic of the
servers is connected to this.

The problems:

In phy servers and xen servers, glusterfs process is consumming a lot of
RAM... so the overall perfommance in the clients is poor, but not
always... only in a punctual moments. When it occurs, I stop glusterfs in
the nodes and I start them again and everything turns to go ok.

If the web domain served by the apaches is not too much complicated (few
php files, jpg/gif files) the perfommance is good. But if there is a
joomla, magento or some type of product with too much files alocated in
too much directories, the perfommance is not so good (from 2-3 seconds in
optimal conditions to 7-8 seconds, more or less)

The thing is that for the moment I have had to deactivate glusterfs
because the dealers are complaining with the perfommance.. and I'm looking
for other solution (NFS, DRBD), but I don't want to discard glusterfs
because I believe this is a good solution for the future.

I hope glusterfs version 3.1 will solve many problems I have today, but,
looking into my configuration files, can anybody tell me if I could do
better than it is?

Many thanks in advance

Kyle


--------------------------------------------------------------------------------------------
Busca, compara... pero busca en Qu?!
Desc?brelo en http://buscar.que.es/
--------------------------------------------------------------------------------------------
Correo enviado desde http://www.ozu.es

hi kyle,

i would recommend useing glusterfs 3.0.5.
read the "Important Enhancements and Bugs Fixed in GlusterFS 3.0.5" of
the release notes...
(http://ftp.gluster.com/pub/gluster/glusterfs/3.0/3.0.5/GlusterFS_3.0.5_
Release_Notes.pdf)

i've used glusterfs version 3.0.4 and had similar problems with it.
(problems with locks, timeouts, small files ....)
which version do you use?

i think it's good when glusterfs uses much memory => caching :-)

when serving with apache i recommend following apache core config: (it's
also recommended serveing nfs published files)
http://httpd.apache.org/docs/2.2/mod/core.html#enablemmap

EnableMMAP off
EnableSendfile off

my system config:
* glusterfs 3.0.5 (rpm setup) on centos 5.5 kernel:
2.6.18-194.8.1.el5xen
* all systems in "virtual environment" (apache & gluster server)
* i don't use a "dedicaded" nic for gluster environment -> no
bottle
necks ...
* 2 raid1 gluster servers
* 5 clients ...

my speed comparison: (DELL PE R701, 10k SAS disks) citrix xenserver
virtualhost with physical disks: up to 160 mb/sek (one big file)
virtualhost with gluster disks: up to 80 mb/sek (one big file dd
if=/dev/zero of=/mnt/store ...)
for small files i'm getting much lower values...

I'm still expanding and useing glusterfs more and more ...
please update me about your experience or special settings!

regards,
josy
gugler* News der Woche: Erfolgreiches ?ko-Audit! Das g* Umweltmanagementsystem
erf?llt die h?chsten europ?ischen EMAS-Standards. Lesen Sie mehr dar?ber im
neuen gugler* Nachhaltigkeitsbericht:
http://www.gugler.at/unternehmen/publikationen/g-nachhaltigkeitsbericht.html