Arman Khalatyan
2017-Mar-06 09:51 UTC
[Gluster-users] [ovirt-users] Replicated Glusterfs on top of ZFS
On Fri, Mar 3, 2017 at 7:00 PM, Darrell Budic <budic at onholyground.com> wrote:> Why are you using an arbitrator if all your HW configs are identical? I?d > use a true replica 3 in this case. > >This was just GIU suggestion when I was creating the cluster it was asking for the 3 Hosts , I did not knew even that an Arbiter does not keep the data. I am not so sure if I can change the type of the glusterfs to triplicated one in the running system, probably I need to destroy whole cluster.> Also in my experience with gluster and vm hosting, the ZIL/slog degrades > write performance unless it?s a truly dedicated disk. But I have 8 spinners > backing my ZFS volumes, so trying to share a sata disk wasn?t a good zil. > If yours is dedicated SAS, keep it, if it?s SATA, try testing without it. > >We have also several huge systems running with zfs quite successful over the years. This was an idea to use zfs + glusterfs for the HA solutions.> You don?t have compression enabled on your zfs volume, and I?d recommend > enabling relatime on it. Depending on the amount of RAM in these boxes, you > probably want to limit your zfs arc size to 8G or so (1/4 total ram or > less). Gluster just works volumes hard during a rebuild, what?s the problem > you?re seeing? If it?s affecting your VMs, using shading and tuning client > & server threads can help avoid interruptions to your VMs while repairs are > running. If you really need to limit it, you can use cgroups to keep it > from hogging all the CPU, but it takes longer to heal, of course. There are > a couple older posts and blogs about it, if you go back a while. >Yes I saw that glusterfs is CPU/RAM hugry!!! 99% of all 16 cores used just for healing 500GB vm disks. It was taking almost infinity compare with nfs storage (single disk+zfs ssd cache, for sure one get an penalty for the HA:) ) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170306/11a954e8/attachment.html>
Sahina Bose
2017-Mar-07 07:08 UTC
[Gluster-users] [ovirt-users] Replicated Glusterfs on top of ZFS
On Mon, Mar 6, 2017 at 3:21 PM, Arman Khalatyan <arm2arm at gmail.com> wrote:> > > On Fri, Mar 3, 2017 at 7:00 PM, Darrell Budic <budic at onholyground.com> > wrote: > >> Why are you using an arbitrator if all your HW configs are identical? I?d >> use a true replica 3 in this case. >> >> > This was just GIU suggestion when I was creating the cluster it was asking > for the 3 Hosts , I did not knew even that an Arbiter does not keep the > data. > I am not so sure if I can change the type of the glusterfs to triplicated > one in the running system, probably I need to destroy whole cluster. > > > >> Also in my experience with gluster and vm hosting, the ZIL/slog degrades >> write performance unless it?s a truly dedicated disk. But I have 8 spinners >> backing my ZFS volumes, so trying to share a sata disk wasn?t a good zil. >> If yours is dedicated SAS, keep it, if it?s SATA, try testing without it. >> >> > We have also several huge systems running with zfs quite successful over > the years. This was an idea to use zfs + glusterfs for the HA solutions. > > >> You don?t have compression enabled on your zfs volume, and I?d recommend >> enabling relatime on it. Depending on the amount of RAM in these boxes, you >> probably want to limit your zfs arc size to 8G or so (1/4 total ram or >> less). Gluster just works volumes hard during a rebuild, what?s the problem >> you?re seeing? If it?s affecting your VMs, using shading and tuning client >> & server threads can help avoid interruptions to your VMs while repairs are >> running. If you really need to limit it, you can use cgroups to keep it >> from hogging all the CPU, but it takes longer to heal, of course. There are >> a couple older posts and blogs about it, if you go back a while. >> > > Yes I saw that glusterfs is CPU/RAM hugry!!! 99% of all 16 cores used just > for healing 500GB vm disks. It was taking almost infinity compare with nfs > storage (single disk+zfs ssd cache, for sure one get an penalty for the > HA:) ) >Is your gluster volume configured to use sharding feature? Could you provide output of gluster vol info?> > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170307/b36ad0d3/attachment.html>