09.01.2019 17:38, Hu Bert ?????:> Hi @all, > > we have 3 servers, 4 disks (10TB) each, in a replicate 3 setup. We're > having some problems after a disk failed; the restore via reset-brick > takes way too long (way over a month)terrible. We have similar setup, and I do not test restoring... How many volumes do you have - one volume on one (*3) disk 10 TB in size - then 4 volumes?> We were thinking about migrating to 3 servers with a RAID10 (HW or > SW), again in a replicate 3 setup. We would waste a lot of space, but > the idea is that, if a hdd fails: > > - the data are still available on the hdd copy > - performance is better than with a failed/restoring hdd > - the restore via SW/HW RAID is faster than the restore via glusterfsOur setup is worse in wasted space - we have a 3 10Tb disks in each server + 1 SSD "for raid controller cache". there is no ability to create raid 10, raid 5 is a no-no on HDDS, and only viable variant is a 1ADM (3*RAID1 + 1 SSD cache) (for using all disks). Or 2*RAID1 + 1 HDD for gluster ...> Any opinions on that? Maybe it would be better to use more servers and > smaller disks, but this isn't possible at the moment.Also interested. We can swap SSDs to HDDs for RAID10, but is it worthless? And what about right way to restore for configs like replica 3 *10Tb volumes?> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >
Hi Mike,> We have similar setup, and I do not test restoring... > How many volumes do you have - one volume on one (*3) disk 10 TB in size > - then 4 volumes?Testing could be quite easy: reset-brick start, then delete&re-create partition/fs/etc., reset-brick commit force - and then watch. We only have 1 big volume over all bricks. Details: Volume Name: shared Type: Distributed-Replicate Number of Bricks: 4 x 3 = 12 Brick1: gluster11:/gluster/bricksda1/shared Brick2: gluster12:/gluster/bricksda1/shared Brick3: gluster13:/gluster/bricksda1/shared Brick4: gluster11:/gluster/bricksdb1/shared Brick5: gluster12:/gluster/bricksdb1/shared Brick6: gluster13:/gluster/bricksdb1/shared Brick7: gluster11:/gluster/bricksdc1/shared Brick8: gluster12:/gluster/bricksdc1/shared Brick9: gluster13:/gluster/bricksdc1/shared Brick10: gluster11:/gluster/bricksdd1/shared Brick11: gluster12:/gluster/bricksdd1_new/shared Brick12: gluster13:/gluster/bricksdd1_new/shared Didn't think about creating more volumes (in order to split data), e.g. 4 volumes with 3*10TB each, or 2 volumes with 6*10TB each. Just curious: after splitting into 2 or more volumes - would that make the volume with the healthy/non-restoring disks better accessable? And only the volume with the once faulty and now restoring disk would be in a "bad mood"?> > Any opinions on that? Maybe it would be better to use more servers and > > smaller disks, but this isn't possible at the moment. > Also interested. We can swap SSDs to HDDs for RAID10, but is it worthless?Yeah, would be interested in how the glusterfs professionsals deal with faulty disks, especially when these are as big as our ones. Thx Hubert