Hello guys,
I was wondering if someone could share their glusterfs volume and system
settings if you are running glusterfs with infiniband networking. In particular
I am interested in using the glusterfs + infiniband + kvm for virtualisation.
However, any other implementation would also be useful for me.
I've tried various versions of glusterfs (versions 3.2, 3.3 and 3.4beta)
over the past year. My experience with gluster + infiniband is below. The
underlying fs is a zfs pool with compression=lz4 made of 8 disks in raid 10
equivalent setup. Filesystem itself can do over 900MB/s with single thread and
around 1050MB/s with multiple threads based on dd with bs=128K with 100GB data
file generated from /dev/urandom.
1. volume transport set to RDMA is very very unstable. disconnecting all the
time even with minimal load. I've found that when it does work the speeds
are very fast - was getting around 1.4 - 1.6GB/s with 16 threads on 40gbit/s QDR
link.
2. volume transport set to tcp works over IPoIB interface, but i've not
managed to get a good level of performance with real data. I was maxing about
220MB/s with a single thread and around 550MB/s with 12 or more threads.
Benchmarking with a huge file made from /dev/zero gives me around 800-900MB/s
with multiple threads and around 750MB/s single thread. Not sure why would
glusterfs has a good level of performance when reading from RAM but is only
working at 1/2 of the disk speed when reading real data from the disk?
3. Glusterfs with NFS also gives me crappy results. Running NFS directly from
zfs pool gives me a far better performance.
4. Tried various performance related options for glusterfs, but with only small
performance increase ((
5. Clients perform horribly when additing new bricks to the cluser. By that I
mean over 2 hours to run "time ls -lhR /glustervolume" which contains
just 10 files. Basically, mounted fs is completely unusable during this time!
6. Virtual machines with volumes stored on the glusterfs mounted filesystem have
an extremely slow performance. I've not managed to get speeds over 50MB/s
using cache=none option.
If any of you guys managed to get a decent level of performance I would be
grateful if you share your experience and any tips.
Many thanks
Andrei
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130512/b187b34d/attachment.html>
