I have posted this to the lists purely to help others - please do not
consider any of the following suitable for a production environment
and follow these rough instructions at your own risk.
Feel free to add your own additions to this mail posting as this may
or may not work for everybody!
I will not be held responsible for data loss, excessive CPU or mem
usage etc etc etc...
INSTALL NEEDED COMPONENTS:
# make sure you have completely removed any other Ubuntu glusterfs packages
apt-get remove --purge glusterfs-*
# install the build environment
apt-get install sshfs build-essential flex bison byacc vim wget libreadline-dev
BUILD AND INSTALL GLUSTERFS 3.1
cd /usr/src
wget
http://download.gluster.com/pub/gluster/glusterfs/3.1/LATEST/glusterfs-3.1.0.tar.gz
tar -zxvf glusterfs-3.1.0.tar.gz
cd glusterfs-3.1.0
./configure
# If all went well, you should see the following:
GlusterFS configure summary
==========================FUSE client : yes
Infiniband verbs : no
epoll IO multiplex : yes
argp-standalone : no
fusermount : no
readline : yes
# Now continue to do the usual:
make
make install
MY GREY AREA - Getting it up and running
There should now be a file /etc/glusterfs/glusterd.vol and it should
contain the following:
volume management
type mgmt/glusterd
option working-directory /etc/glusterd
option transport-type socket,rdma
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
end-volume
I am unsure if I copied the file from elsewhere or if it was
installed, so make sure it is there and contains the above info as it
points to the directory that is now used to store all your
configurations.
The next problem I ran into was with the initscript itself, I have a
strong feeling this should be done elsewhere (i.e. within
/etc/defaults/) but this is possibly a dirty and nasty thing todo:
edit /etc/init.d/glusterd with the editor of your choice and add the
following within the variables at the top:
CONFIGFILE='/etc/glusterfs/glusterd.vol'
You should now be able to start the service.
SETTING UP YOUR OTHER NODES
Depending on how exactly you install on other nodes can cause a few
problems, if you simply copy the src directory and install, you will
find yourself with machines all using the same UID.
cat /etc/glusterd/glusterd.info
UUID=5714e9b0-d8db-11df-937b-0800200c9a66
Make sure this is unique on every node, if it is not - then create a new UID
(I simply googled and generated some UID's online - but whatever
method you use - even rolling a dice or playing cards - this NEEDS to
be unique)
You should now be able to start glusterfs on every node.
/etc/init.d/glusterd start
/etc/init.d/glusterd status
NOTES ON THE NEW CLI USAGE
I love the new CLI - it is great, it will create the directories on
all nodes for you - so from here on in, you can use the documentation
to create the storage you require by starting with assigning your
storage pool:
http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Creating_Trusted_Storage_Pools
To simplify, do this from 1 node with glusterd running on all nodes...
do not try to add the node you are on, it is automatically there - and
as long as the UIDs are unique as mentioned above, then you should be
good to go.
As I said above, the CLI is great, so there is no need to manually
mkdir - simply choose your location and do (to create a simple
distributed volume across 3 nodes):
gluster volume create test-volume transport tcp
node1:/my/storage/test1 tcp node2:/my/storage/test2 tcp
node3:/my/storage/test3
*replacing node with IP addresses or hostnames that all the systems
know along with the locations of the underlying bricks will be stored.
NOTES ON MOUNTING
Well, it is all there and ready for use - now to mount it!
If you have already read the documentation, then you know there are 3
ways to access it.
I used 2 methods:
1: glusterfs native (fuse)
http://www.gluster.com/community/documentation/index.php/Gluster_3.1_Native_Client_Guide
This method has better locking capabilities but can cause big
overheads with CPU and mem usage - make sure you have an ample amount
of mem to use or load will cause your node to suffer - if you notice
swap thrashing your disks, then your system is struggling.
I found that my webserver/mailserver + glusterfs native mount on 512MB
was not enough but did run a lot nicer on 1024MB
2: glusterfs NFS
Obviously make sure you have nfs-common and portmap installed and then
mount in the usual way.
I found this method had less mem and CPU overheads but locking seemed
really bad with some of my services (Dovecot, SVN) and the locks
ultimately caused load to spiral out of control.
It may have been a misconfiguration on my behalf!
Simply using NFS mounting as a read filesystem without the need for
locking worked well... but writing large files seemed to lock up the
system also (i did not test this with 1024MB of mem and again, it is
possibly a configuration on my behalf).
CONCLUSION
Well, I am biast - I am not be affiliated in anyway with gluster - but
- 3.1 is awesome!
AGAIN - do not use my above setup on 32bit within a production
environment - but simply have a play around.
I love the new CLI and the work these people are doing - feel free to
post to this thread with any corrections and findings within your own
experiments.
AGAIN - 32bit is unsupported so do not expect help from the devs, they
are busy enough as it is - I have provided this post for the curious
and brave!