Good afternoon,
I am looking for additional information about glusterfs, arbiters and
thin-arbiters. The current stable release is gluster 11, so I am
considering that version for deployment. My planned setup is: 4 storage
servers in distributed + replicated mode.
Server1, server2 : replica 2, arbiter 1
Server3, server4 : replica 2, arbiter 1
Since having replica 2 is not recommended due to split-brains I will
have an arbiter.
Generic questions:
* Is arbiter or thin-arbiter recommended in a production, large volume
storage environment?
* Is thin arbiter code stable and deployment ready?
* Arbiter is file based and stores metadata for each file. In this
scenario I would at least double the default inode count on the
arbiter server. Thin-arbiter on the other hand is brick based but I
have not found enough information if its inode usage is as inode
hungry as the arbiter configuration. I am thinking, it should not be
as it is brick based. So do I need to increase the inode count when
using thin-arbiters? If yes, what is recommended?
* I've read old bug reports reporting that thin arbiter was not ready
to serve multiple trusted pools. Is this still the case?? I may
configure multiple trusted pools in the future.
* I have many linux boxes running different linux distributions and
releases. Ideally the assortment of boxes would mount the same
gluster pool/volume. I looked for information about older versions
of gluster clients running on a range of older distributions
mounting the most recent gluster 11 pool/volume? Does that work??
Can gluster client (version 10, 9, 8, 7, etc.) mount gluster 11
volume and run without significant issues?? I understand that older
versions of client will not have the most recent features. Most
recent features aside, is such configuration supported/stable?
*Thin-arbiter approach:*? If I go with the thin-arbiter configuration I
will use a 5^th server as this server can be outside of the trusted pool
and can be shared among multiple trusted pools
Server1, server2: replica 2, thin-arbiter server5
Server3, server4: replica 2, thin-arbiter server5
*Old arbiter approach:*? If I go with the older arbiter configuration, I
am considering using 2 of the storage servers to act as both replica and
an arbiter. Is that configuration possible/supported and reasonable?
/Server1/, server2: replica 2, arbiter *server3*
*Server3*, server4: replica 2, arbiter /server1/
In this configuration, I am considering using server3 to be arbiter for
server{1,2} replica 2,? and using server1 to be arbiter for server{3,4}
replica 2.
Questions:
* Is this a reasonable/recommended configuration?
* Should the arbiter metadata folder be inside or outside of the volume?
o In detail. Say server{1,2} replica has 1 brick each
*/gluster/brick1 *with*/gfs1vol1 *as the volume
o Should the arbiter metadata folder location be:
??/gluster/arbiter/gfs1vol1?? (outside of the? volume path) or
*/gfs1vol1/*arbiter1/ ?(inside the volume path)
Thank you for your thoughts,
Peter
--
This email has been checked for viruses by AVG antivirus software.
www.avg.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20230421/0a9a4796/attachment.html>
Strahil Nikolov
2023-May-14 19:48 UTC
[Gluster-users] gluster, arbiter, thin-arbiter questions
As nobody chimed in, let me reply inline.
Best Regards,Strahil Nikolov
On Sunday, April 23, 2023, 2:35 AM, Peter P <peter.a.pfeiffer at
gmail.com> wrote:
Good afternoon,
I am looking for additional information about glusterfs, arbiters and
thin-arbiters. The current stable release is gluster 11, so I am considering
that version for deployment. My planned setup is:? 4 storage servers in
distributed + replicated mode.
Server1, server2 : replica 2, arbiter 1
Server3, server4 : replica 2, arbiter 1
Since having replica 2 is not recommended due to split-brains I will have an
arbiter.
Generic questions:
- Is arbiter or thin-arbiter recommended in a production, large volume
storage environment?
- Both were introduced a long time ago. Most users prefer full arbiter, so
healing is far more optimal (only changed files will be healed).
- Is thin arbiter code stable and deployment ready?
- I know that it?s in use but full arbiter has been introduced earlier and has a
wider adoption.
- Arbiter is file based and stores metadata for each file. In this scenario
I would at least double the default inode count on the arbiter server.
Thin-arbiter on the other hand is brick based but I have not found enough
information if its inode usage is as inode hungry as the arbiter configuration.
I am thinking, it should not be as it is brick based. So do I need to increase
the inode count when using thin-arbiters? If yes, what is recommended?
- Full arbiter is sensitive to network?latency?and disk speed (a lot of small
I/O for those inodes). Increase macpct (XFS)?on arbiter bricks and prefer using
a SSD/NVME. As full arbiter doesn?t store any data , you can set the maxpct
to?around 75%- Thin arbiter doesn?t have a brick and when you create it, you
just specify the replica id file (
see?https://docs.gluster.org/en/main/Administrator-Guide/Thin-Arbiter-Volumes/?
)
- I've read old bug reports reporting that thin arbiter was not ready to
serve multiple trusted pools. Is this still the case?? I may configure multiple
trusted pools in the future.
- I saw Kadalu uses their own thin arbiter and I never saw issues. I doubt I was
the only one using it, so it should be fine.
- I have many linux boxes running different linux distributions and releases.
Ideally the assortment of boxes would mount the same gluster pool/volume. I
looked for information about older versions of gluster clients running on a
range of older distributions mounting the most recent gluster 11 pool/volume?
Does that work?? Can gluster client (version 10, 9, 8, 7, etc.) mount gluster 11
volume and run without significant issues?? I understand that older versions of
client will not have the most recent features. Most recent features aside, is
such configuration supported/stable?
- For that purpose gluster has 2 settings:cluster.max-op-version -> the max
compatibility?version you can set your cluster based?of the oldest client?s
versioncluster.op-version -> the cluster?s compatibility versionAs long you
keep the cluster.op-version compatible with your client - it should work.
Thin-arbiter approach:? If I go with the thin-arbiter configuration I will use
a 5th server as this server can be outside of the trusted pool and can be shared
among multiple trusted pools
Server1, server2: replica 2, thin-arbiter server5
Server3, server4: replica 2, thin-arbiter server5
Old arbiter approach:? If I go with the older arbiter configuration, I am
considering using 2 of the storage servers to act as both replica and an
arbiter. Is that configuration possible/supported and reasonable?
Server1, server2: replica 2, arbiter server3?
Server3, server4: replica 2, arbiter server1
- Yes, as long as you have a dedicated brick (in this example server3 should
have a data brick and arbiter brick)
In this configuration, I am considering using server3 to be arbiter for
server{1,2} replica 2,? and using server1 to be arbiter for server{3,4} replica
2.?
Questions:
- Is this a reasonable/recommended configuration?
- It?s used quite often
- ?Should the arbiter metadata folder be inside or outside of the volume?
- In detail. Say server{1,2} replica has 1 brick each /gluster/brick1 with
/gfs1vol1 as the volume
- Should the arbiter metadata folder location be:
??/gluster/arbiter/gfs1vol1?? (outside of the? volume path)? or??
/gfs1vol1/arbiter1/ ?(inside the volume path)
- Always keep bricks as separate mount points. For example:/dev/vg/lv mounted
on?/bricks/databricks?with directory vol1/brick1/dev/vg/lv2 mounted
on?/bricks/arbiterbricks with directory vol1/arbiterbrick1
The idea is that if?the device?is not mounted, the brick directory will be
missing and the mess will be far less.
Thank you for your thoughts,
Peter
| | Virus-free.www.avg.com |
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20230514/dd5142d2/attachment.html>