Gluster users - Jan 2019 - usage of harddisks: each hdd a brick? raid?

We ara also using 10TB disks, heal takes 7-8 days.
You can play with "cluster.shd-max-threads" setting. It is default 1 I
think. I am using it with 4.
Below you can find more info:
https://access.redhat.com/solutions/882233

On Thu, Jan 10, 2019 at 9:53 AM Hu Bert <revirii at googlemail.com>
wrote:
>
> Hi Mike,
>
> > We have similar setup, and I do not test restoring...
> > How many volumes do you have - one volume on one (*3) disk 10 TB in
size
> >   - then 4 volumes?
>
> Testing could be quite easy: reset-brick start, then delete&re-create
> partition/fs/etc., reset-brick commit force - and then watch.
>
> We only have 1 big volume over all bricks. Details:
>
> Volume Name: shared
> Type: Distributed-Replicate
> Number of Bricks: 4 x 3 = 12
> Brick1: gluster11:/gluster/bricksda1/shared
> Brick2: gluster12:/gluster/bricksda1/shared
> Brick3: gluster13:/gluster/bricksda1/shared
> Brick4: gluster11:/gluster/bricksdb1/shared
> Brick5: gluster12:/gluster/bricksdb1/shared
> Brick6: gluster13:/gluster/bricksdb1/shared
> Brick7: gluster11:/gluster/bricksdc1/shared
> Brick8: gluster12:/gluster/bricksdc1/shared
> Brick9: gluster13:/gluster/bricksdc1/shared
> Brick10: gluster11:/gluster/bricksdd1/shared
> Brick11: gluster12:/gluster/bricksdd1_new/shared
> Brick12: gluster13:/gluster/bricksdd1_new/shared
>
> Didn't think about creating more volumes (in order to split data),
> e.g. 4 volumes with 3*10TB each, or 2 volumes with 6*10TB each.
>
> Just curious: after splitting into 2 or more volumes - would that make
> the volume with the healthy/non-restoring disks better accessable? And
> only the volume with the once faulty and now restoring disk would be
> in a "bad mood"?
>
> > > Any opinions on that? Maybe it would be better to use more
servers and
> > > smaller disks, but this isn't possible at the moment.
> > Also interested. We can swap SSDs to HDDs for RAID10, but is it
worthless?
>
> Yeah, would be interested in how the glusterfs professionsals deal
> with faulty disks, especially when these are as big as our ones.
>
>
> Thx
> Hubert
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

10.01.2019 11:26, Serkan ?oban ?????:
> We ara also using 10TB disks, heal takes 7-8 days.
> You can play with "cluster.shd-max-threads" setting. It is
default 1 I
> think. I am using it with 4.
> Below you can find more info:
> https://access.redhat.com/solutions/882233
I'm using ovirt, setup script set this values by default:

cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
>> Testing could be quite easy: reset-brick start, then
delete&re-create
>> partition/fs/etc., reset-brick commit force - and then watch.
>>
>> We only have 1 big volume over all bricks. Details:
>>
>> Volume Name: shared
>> Type: Distributed-Replicate
A, you have distributed-replicated volume, but I choose only replicated 
(for beginning simplicity :)
>> Brick12: gluster13:/gluster/bricksdd1_new/shared
>>
>> Didn't think about creating more volumes (in order to split data),
>> e.g. 4 volumes with 3*10TB each, or 2 volumes with 6*10TB each.
May be replicated volume are healing faster?
>> Yeah, would be interested in how the glusterfs professionsals deal
>> with faulty disks, especially when these are as big as our ones.
>>

Hi,
> > We ara also using 10TB disks, heal takes 7-8 days.
> > You can play with "cluster.shd-max-threads" setting. It is
default 1 I
> > think. I am using it with 4.
> > Below you can find more info:
> > https://access.redhat.com/solutions/882233
> cluster.shd-max-threads: 8
> cluster.shd-wait-qlength: 10000
Our setup:
cluster.shd-max-threads: 2
cluster.shd-wait-qlength: 10000
> >> Volume Name: shared
> >> Type: Distributed-Replicate
> A, you have distributed-replicated volume, but I choose only replicated
> (for beginning simplicity :)
> May be replicated volume are healing faster?
Well, maybe our setup with 3 servers and 4 disks=bricks == 12 bricks,
resulting in a distributed-replicate volume (all /dev/sd{a,b,c,d}
identical) , isn't optimal? And it would be better to create a
replicate 3 volume with only 1 (big) brick per server (with 4 disks:
either a logical volume or sw/hw raid)?

But it would be interesting to know if a replicate volume is healing
faster than a distributed-replicate volume - even if there was only 1
faulty brick.


Thx
Hubert