Diego Zuccato
2020-Oct-26 14:39 UTC
[Gluster-users] Upgrade from 6.9 to 7.7 stuck (peer is rejected)
Il 26/10/20 15:09, mabi ha scritto:> Right, seen liked that this sounds reasonable. Do you actually remember the exact command you ran in order to remove the brick? I was thinking this should be it: > gluster volume remove-brick <VOLNAME> <BRICK> force > but should I use "force" or "start"?Memory does not serve me well (there are 28 disks, not 26!), but bash history does :) # gluster volume remove-brick BigVol replica 2 str957-biostq:/srv/arbiters/{00..27}/BigVol force # gluster peer detach str957-biostq # gluster peer probe str957-biostq # gluster volume add-brick BigVol replica 3 arbiter 1 str957-biostq:/srv/arbiters/{00..27}/BigVol You obviously have to wait for remove-brick to complete before detaching arbiter.>> IIRC it took about 3 days, but the arbiters are on a VM (8CPU, 8GB RAM) >> that uses an iSCSI disk. More than 80% continuous load on both CPUs and RAM. > That's quite long I must say and I am in the same case as you, my arbiter is a VM.Give all the CPU and RAM you can. Less than 8GB RAM is asking for troubles (in my case). -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Universit? di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
mabi
2020-Oct-26 14:53 UTC
[Gluster-users] Upgrade from 6.9 to 7.7 stuck (peer is rejected)
??????? Original Message ??????? On Monday, October 26, 2020 3:39 PM, Diego Zuccato <diego.zuccato at unibo.it> wrote:> Memory does not serve me well (there are 28 disks, not 26!), but bash > history does :)Yes, I also too often rely on history ;)> gluster volume remove-brick BigVol replica 2 str957-biostq:/srv/arbiters/{00..27}/BigVol forceThanks for the info, I was missing the "replica 2" inside the command it looks like> gluster peer detach str957-biostq > gluster peer probe str957-biostqDo I really need to do a detach and re-probe the aribter node? I would like to avoid that because I have two other volumes with even more files... so that would mean that I have to remove the arbiter brick of the two other volumes too...> Give all the CPU and RAM you can. Less than 8GB RAM is asking for > troubles (in my case).I have added an extra 4 GB of RAM just in case.
Strahil Nikolov
2020-Oct-26 17:06 UTC
[Gluster-users] Upgrade from 6.9 to 7.7 stuck (peer is rejected)
Detaching the arbiter is pointless... Quota is an extended file attribute, and thus disabling and reenabling quota on a volume with millions of files will take a lot of time and lots of IOPS. I would leave it as a last resort.? Also, it was mentioned in the list about the following script that might help you: https://github.com/gluster/glusterfs/blob/devel/extras/quota/quota_fsck.py You can take a look in the mailing list for usage and more details. Best Regards, Strahil Nikolov ? ??????????, 26 ???????? 2020 ?., 16:40:06 ???????+2, Diego Zuccato <diego.zuccato at unibo.it> ??????: Il 26/10/20 15:09, mabi ha scritto:> Right, seen liked that this sounds reasonable. Do you actually remember the exact command you ran in order to remove the brick? I was thinking this should be it: > gluster volume remove-brick <VOLNAME> <BRICK> force > but should I use "force" or "start"?Memory does not serve me well (there are 28 disks, not 26!), but bash history does :) # gluster volume remove-brick BigVol replica 2 str957-biostq:/srv/arbiters/{00..27}/BigVol force # gluster peer detach str957-biostq # gluster peer probe str957-biostq # gluster volume add-brick BigVol replica 3 arbiter 1 str957-biostq:/srv/arbiters/{00..27}/BigVol You obviously have to wait for remove-brick to complete before detaching arbiter.>> IIRC it took about 3 days, but the arbiters are on a VM (8CPU, 8GB RAM) >> that uses an iSCSI disk. More than 80% continuous load on both CPUs and RAM. > That's quite long I must say and I am in the same case as you, my arbiter is a VM.Give all the CPU and RAM you can. Less than 8GB RAM is asking for troubles (in my case). -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Universit? di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786 ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users