Gluster users - Jun 2011 - Gluster 3.2 to create mirrored volumes between two servers

Hi!

First, let me explain the scenario:

Until now, I am one admin of an OpenSuse 11.3 root server which hosts
several domains and is mainly used for web and mail.

To enhance reliability, availability and performance, we decided to
extend this setup to up to three servers on different locations.

Now we had to find solutions to sync parts of the filebase (especially
web root and mail root). For mysql, solutions exist (either clustering
or master-master-replication).

So, the task is to maintain 2-3 partitions (or a part of them) in sync
between two or three servers, no further clients will access them and
the volumes are solely used locally on the two servers. Gluster should
simply ensure the sync between the two servers.

I found first ocfs2 and later on gluster.

Gluster, not requiring own partitions with a special file system (like
ocfs2/drbd), seemed to be a good solution.

After having worked through the documentation on gluster.org and some
further tutorials (there are not real hints on mounting the volumes), I
got the things working.

Creating and installing the mail root volume (15 GB) worked quite good
and after enabling readahead, iocache and writeback volume settings in
/usr/local/etc/gluster/glusterfs.vol, I achieved acceptable perfomance
and system load parameters.

The setup is as follows. The partitions (xfs) are mounted under /data
and the volumes reference to these directories. The volumes are then
mounted on the two machines to their original location.

The volumes are created as replica 2 transport tcp and the necessary
ports are open for both servers in both iptables.

This morning, I tried to extend this to the web root partition (12.5 GB)
and expected similiar results.

But it did not run stable.

After the first installation, I made an error with the path settings on
the second host so I stopped Apache/... and tried to recreate the
volume. But Apache conflicted in some way with the ongoing first sync,
the httpd-processes freezed and could not be killed. I had to reboot the
machine. After coming up, Gluster treated the part which was synced to
the backup server as valid and deleted the rest from the mail root
partition on the master server. Sometimes, extensive backups are really
useful.

I recreated the volume and copied the backup data from the second server
onto the volume (which synced the data to the main server, too).

After this was nearly complete, I restarted Apache on the main server.

This combination (restoring the last files and Apache accessing the
volume) resulted in an increasing system load (95-97 % wait, load going
from 20 up to 300) until I had a hard reset on the machine.

After the reboot, everything seemed fine so far.

But working with the file system and deleting some duplicate folder
trees with mc, single folders were repeatedly reported as non-existent
or unaccessible, their content vanished and reappeared.

Finally, I dismounted these new gluster volumes and deleted them and
remounted the local partitions to their original place and restored
again missing files.

Additionaly, I observed that in case of a reboot of the backup server,
the mounted volumes become inaccessible on the main server although all
information is local and the mount.glusterfs command always references
the local machine when mounting the volume.

Does anybody have a good hint for me to get this working as intended?

Best regards.