Strahil
2019-Oct-21 14:02 UTC
[Gluster-users] Adding arbiter on a large existing replica 2 set
Then, You should expect the arbiter to be setup in less time , but this depends to the amount of files - if you have few large files , sync is faster than 10 mil small files. Best Regards, Strahil NikolovOn Oct 21, 2019 16:23, Thorgeir Marthinussen <thorgeir.marthinussen at basefarm.com> wrote:> > Hi, > > The new cluster is set up with two physical servers with HDDs and a VM backed by an all-flash stretched vSAN. > The old cluster will be set up the same way. > > The main volume that I'm concerned about usually takes about 20-30 minutes to finish the self-heal, the network is 10Gbps. > > > > Best regards > --? > THORGEIR MARTHINUSSEN > Senior Systems Consultant > BASEFARM > > -----Original Message----- > From: Strahil <hunter86_bg at yahoo.com> > To: Thorgeir <thorgeir.marthinussen at basefarm.com>, gluster-users <gluster-users at gluster.org> > Subject: Re: [Gluster-users] Adding arbiter on a large existing replica 2 set > Date: Wed, 16 Oct 2019 21:04:50 +0300 > > Hi Thorgeir, > > Did you try adding an arbiter with SSD brick/bricks ? > > SSD/NVMe is the best type of storage for an arbiter - yes , it's more expensive but you will need less disks than a data brick . > > Of course , arbiter is only one side of the equasion and the time to heal might depend on your data bricks' IOPS. > > How much time does a node in the cluster need to heal after being reboot ? > > Best Regards, > Strahil Nikolov > > On Oct 16, 2019 16:37, Thorgeir Marthinussen <thorgeir.marthinussen at basefarm.com> wrote: >> >> Hi, >> >> We have an old Gluster cluster setup, running a replica 2 across two datacenters, and currently on version 4.1.5 >> >> I need to add an arbiter to this setup, but I'm concerned about the performance impact of this on the volumes. >> >> I recently set up a new cluster, for a different purpose, and decided to test adding an arbiter to the volume after adding in some data. >> Had a volume with ~435,000 files totaling about 12TB. >> Adding the arbiter initiated a heal-operation that took almost 3 hours. >> >> The older cluster, one of the volumes is about 14TB, but ~45,5 million files. >> >> Since arbiter is only concerned about metadata and checksums, I'm concerned about the fact that we have 100 times the amount of files, i.e. 100 times the amount of I/O operations to execute during healing, and possibly 100 times the time which would mean about 12,5 days. >> >> Another "issue" is that the 'gluster volume heal <vol-name> info summary' command seems to "count" all the files, so the command can take a very long time to complete. >> The metrics-scraping script I created for us, with a timeout of 110seconds, fails to complete when a volume has over ~800-900 files unsynced (which happens regularily when taking one cluster-node down for patching). >> >> >> Does anyone have any experience with adding arbiter afterwards, perf-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191021/6e07d9eb/attachment.html>