Thanks a lot for sharing your results!
I'm to looking into a similar test scenario.
According to
http://www.gluster.org/community/documentation/index.php/GlusterOnZFS
one should apply this patch
https://github.com/zfsonlinux/zfs/issues/1648
Did you do that? Or don't you use symlinks on your glustermounts?
From your blogs I understand you didn't, did you experienced ny issues?
I like to use this scenario as a backend fro VM images.
The glusterserver mounts the glustervolumes
and are also the hosts for the VM, sort of self describing/self providing
approach.
I created an individual filesystems and along with it a separate glustervolume
for each VM. This way I can snapshot the volumes/VMs individually.
I doubt it would make any performance difference
if I would put all images in one Volume/zfs-Filesystem. ?
I ran several VMs in parallel and let them all perform bonnie++ tests
The whole setup seems quite stable so far
including life migrating of the VMs and reboot of the hosts during the tests.
TIA
Bernhard
Bernhard Glomm
IT Administration
Phone: +49 (30) 86880 134
Fax: +49 (30) 86880 100
Skype: bernhard.glomm.ecologic
Ecologic Institut gemeinn?tzige GmbH | Pfalzburger Str. 43/44 | 10717 Berlin |
Germany
GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 | USt/VAT-IdNr.:
DE811963464
Ecologic? is a Trade Mark (TM) of Ecologic Institut gemeinn?tzige GmbH
On Dec 27, 2013, at 5:52 PM, Tony Maro <tonym at evrichart.com> wrote:
> Off and on for the past 8 months I've been working on setting up a
rather sizable Gluster configuration on top of ZFS. I wanted to report back my
experiences.
>
> First, I'm running Ubuntu 12.04 with zfsonlinux.org packages. Gluster
is installed from the semiosis PPA.
>
> My test config is as follows:
>
> Two brick Gluster in mirroring (replica 2)
> Each brick has 8 TB of hard drives configured with ZFS using RAID-Z.
> Each brick has a 256 GB SSD drive to use as cache for ZFS.
> A third server for geo-replication slave also with RAID-Z storage.
> Private Gigabit network segment for the gluster servers (and the
application servers that talk to them)
> Geo-rep will be placed three states away in a colo center, but is currently
on the same private segment.
>
> Data:
> About 6 TB of test data, the final deploy will be much larger. Most files
are around 40MB in size. Files are stored in directories going 4 deep, no more
than 255 files in a single directory.
>
> All data is already compressed and encrypted with AES-256 prior to storing
on the filesystem, so the comrpession feature of ZFS isn't useful to me.
>
> The default configs for ZFS caused this to be a miserable failure.
Configuring Gluster geo-replication over SSH was also a difficult task, because
the documentation is wrong on many counts when compared to the version in the
PPA.
>
> With a default config, directory listings (remember, less than 255 files
and folders in each directory) would take about 15 seconds to complete over the
Gluster share.
>
> By simply tweaking the following ZFS configs:
> zfs set atime=off [volname]
> zfs set xattr=sa [volname]
> zfs set exec=off [volname]
> zfs set sync=disabled [volname]
>
> I was able to get near instantaneous directory listings and dramatically
improved read performance, although I haven't bothered getting hard numbers
on that.
>
> I just wanted to pass this along. Here's my blog posts about ZFS and
Gluster, which pretty much says the same thing I just did but offers more about
installing ZFS:
>
> http://www.ossramblings.com/zfs-filesystem-on-linux
> http://www.ossramblings.com/gluster-geo-replication-with-zfs
>
> -Tony
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131230/a02c792e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131230/a02c792e/attachment.sig>