Hi list I would like to share with you the issues I have been suffered with glusterfs in a web-farm production environment. I hope it can help anybody and if anybody can help me on this I created a glusterfs environment like the following: 6 servers, amd64, 4 cores, 4/8 Gb RAM, 1TB disk each one kernel 2.6.26-2-xen-amd64 In each physical server, there is a xen (or two) virtual machine running Phy server only serves glusterfs to the clients Xen servers (4 apache, 2 mysql, 2 load balancers -LVS -- and a plesk server) ----------- phy server ----------- volume posix1 type storage/posix option directory /mnt/export end-volume volume locks1 type features/locks subvolumes posix1 end-volume volume brick1 type performance/io-threads option thread-count 8 subvolumes locks1 end-volume volume server-tcp type protocol/server option transport-type tcp option auth.addr.brick1.allow * option transport.socket.listen-port 6996 option transport.socket.nodelay on subvolumes brick1 end-volume -------------- 5 xen servers has this configuration ------------- xen server------------ volume nodo1 type protocol/client option transport-type tcp option remote-host 10.0.0.11 option transport.socket.nodelay on option remote-port 6996 option remote-subvolume brick1 end-volume volume nodo2 type protocol/client option transport-type tcp option remote-host 10.0.0.12 option transport.socket.nodelay on option remote-port 6996 option remote-subvolume brick1 end-volume volume nodo3 type protocol/client option transport-type tcp option remote-host 10.0.0.13 option transport.socket.nodelay on option remote-port 6996 option remote-subvolume brick1 end-volume volume nodo4 type protocol/client option transport-type tcp option remote-host 10.0.0.14 option transport.socket.nodelay on option remote-port 6996 option remote-subvolume brick1 end-volume volume nodo5 type protocol/client option transport-type tcp option remote-host 10.0.0.15 option transport.socket.nodelay on option remote-port 6996 option remote-subvolume brick1 end-volume volume nodo6 type protocol/client option transport-type tcp option remote-host 10.0.0.16 option transport.socket.nodelay on option remote-port 6996 option remote-subvolume brick1 end-volume volume mirror-0 type cluster/replicate subvolumes nodo1 nodo2 nodo3 end-volume volume mirror-1 type cluster/replicate subvolumes nodo4 nodo5 nodo6 end-volume volume distribute type cluster/distribute subvolumes mirror-0 mirror-1 end-volume volume readahead type performance/read-ahead option page-count 4 subvolumes distribute end-volume volume iocache type performance/io-cache option cache-size `echo $(( $(grep 'MemTotal' /proc/meminfo | sed 's/[^0-9]//g') / 5120 ))`MB option cache-timeout 1 subvolumes readahead end-volume volume quickread type performance/quick-read option cache-timeout 1 option max-file-size 512kb subvolumes iocache end-volume volume writebehind type performance/write-behind option cache-size 4MB subvolumes quickread end-volume volume statprefetch type performance/stat-prefetch subvolumes writebehind end-volume ----------------------- There is a dedicated switch to serve glusterfs, and the 2nd nic of the servers is connected to this. The problems: In phy servers and xen servers, glusterfs process is consumming a lot of RAM... so the overall perfommance in the clients is poor, but not always... only in a punctual moments. When it occurs, I stop glusterfs in the nodes and I start them again and everything turns to go ok. If the web domain served by the apaches is not too much complicated (few php files, jpg/gif files) the perfommance is good. But if there is a joomla, magento or some type of product with too much files alocated in too much directories, the perfommance is not so good (from 2-3 seconds in optimal conditions to 7-8 seconds, more or less) The thing is that for the moment I have had to deactivate glusterfs because the dealers are complaining with the perfommance.. and I'm looking for other solution (NFS, DRBD), but I don't want to discard glusterfs because I believe this is a good solution for the future. I hope glusterfs version 3.1 will solve many problems I have today, but, looking into my configuration files, can anybody tell me if I could do better than it is? Many thanks in advance Kyle -------------------------------------------------------------------------------------------- Busca, compara... pero busca en Qu?! Desc?brelo en buscar.que.es -------------------------------------------------------------------------------------------- Correo enviado desde ozu.es
it-news (Josef Lahmer)
2010-Jul-15 11:01 UTC
[Gluster-users] Gluster web-farm production environment
hi kyle, i would recommend useing glusterfs 3.0.5. read the "Important Enhancements and Bugs Fixed in GlusterFS 3.0.5" of the release notes... (ftp.gluster.com/pub/gluster/glusterfs/3.0/3.0.5/GlusterFS_3.0.5_ Release_Notes.pdf) i've used glusterfs version 3.0.4 and had similar problems with it. (problems with locks, timeouts, small files ....) which version do you use? i think it's good when glusterfs uses much memory => caching :-) when serving with apache i recommend following apache core config: (it's also recommended serveing nfs published files) httpd.apache.org/docs/2.2/mod/core.html#enablemmap EnableMMAP off EnableSendfile off my system config: * glusterfs 3.0.5 (rpm setup) on centos 5.5 kernel: 2.6.18-194.8.1.el5xen * all systems in "virtual environment" (apache & gluster server) * i don't use a "dedicaded" nic for gluster environment -> no bottle necks ... * 2 raid1 gluster servers * 5 clients ... my speed comparison: (DELL PE R701, 10k SAS disks) citrix xenserver virtualhost with physical disks: up to 160 mb/sek (one big file) virtualhost with gluster disks: up to 80 mb/sek (one big file dd if=/dev/zero of=/mnt/store ...) for small files i'm getting much lower values... I'm still expanding and useing glusterfs more and more ... please update me about your experience or special settings! regards, josy gugler* News der Woche: Erfolgreiches ?ko-Audit! Das g* Umweltmanagementsystem erf?llt die h?chsten europ?ischen EMAS-Standards. Lesen Sie mehr dar?ber im neuen gugler* Nachhaltigkeitsbericht: gugler.at/unternehmen/publikationen/g-nachhaltigkeitsbericht.html