Strahil Nikolov
2021-Nov-30 12:17 UTC
[Gluster-users] force realignement after downing a node?
Than, take a beer/tee/coffee/<whatever is legal> and enjoy the rest of the day ;) Best Regards,Strahil Nikolov On Mon, Nov 29, 2021 at 13:09, Diego Zuccato<diego.zuccato at unibo.it> wrote: Here it is. Seems gluster thinks there's nothing to be done... -8<-- root at str957-clustor00:~# gluster v heal cluster_data info summary Brick clustor00:/srv/bricks/00/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick clustor01:/srv/bricks/00/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick clustor02:/srv/quorum/00/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick clustor02:/srv/bricks/00/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick clustor00:/srv/bricks/01/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick clustor01:/srv/quorum/00/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [... snip: everything reports 0...] Brick clustor01:/srv/bricks/29/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick clustor02:/srv/bricks/29/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick clustor00:/srv/quorum/14/d Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 -8<-- Il 29/11/2021 12:02, Strahil Nikolov ha scritto:> What is the output of 'gluster volume heal VOLUME info summary' ? > > Best Regards, > Strahil Nikolov > >? ? On Mon, Nov 29, 2021 at 10:33, Diego Zuccato >? ? <diego.zuccato at unibo.it> wrote: >? ? Hello all. > >? ? I just brought offline a node (in a replica 3 arbiter 1 volume) to >? ? install more RAM. >? ? The other two nodes kept being used, so I expected to see some >? ? resync at >? ? power on. But I saw nothing unusual: seems it's just serving files >? ? as usual. >? ? Is it normal or should I force a resync? If so, how? > >? ? Regards. > >? ? -- >? ? Diego Zuccato >? ? DIFA - Dip. di Fisica e Astronomia >? ? Servizi Informatici >? ? Alma Mater Studiorum - Universit? di Bologna >? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy >? ? tel.: +39 051 20 95786 >? ? ________ > > > >? ? Community Meeting Calendar: > >? ? Schedule - >? ? Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >? ? Bridge: https://meet.google.com/cpu-eiue-hvk >? ? <https://meet.google.com/cpu-eiue-hvk> >? ? Gluster-users mailing list >? ? Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >? ? https://lists.gluster.org/mailman/listinfo/gluster-users >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users> >-- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Universit? di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20211130/3d0e8a65/attachment.html>
Diego Zuccato
2021-Nov-30 13:10 UTC
[Gluster-users] force realignement after downing a node?
Done just that :) Today I upgraded the second node, with a cleaner shutdown. Strangely, at reboot it worked for about half an hour with all the cores at 100% (but low mem use) and "gluster v heal cluster_data info" apparently hanging. Had lunch and now all cores are back to normal (20-60%), memory use is higher ang gluster is responding again. Still no files in heal pending. I'll skip tomorrow, then upgrade the last server. Hope it all goes smoothly again. Tks. Il 30/11/2021 13:17, Strahil Nikolov ha scritto:> Than, > > take a beer/tee/coffee/<whatever is legal> and enjoy the rest of the day ;) > > Best Regards, > Strahil Nikolov > > On Mon, Nov 29, 2021 at 13:09, Diego Zuccato > <diego.zuccato at unibo.it> wrote: > Here it is. Seems gluster thinks there's nothing to be done... > > -8<-- > root at str957-clustor00 <mailto:root at str957-clustor00>:~# gluster v > heal cluster_data info summary > Brick clustor00:/srv/bricks/00/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick clustor01:/srv/bricks/00/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick clustor02:/srv/quorum/00/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick clustor02:/srv/bricks/00/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick clustor00:/srv/bricks/01/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick clustor01:/srv/quorum/00/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > [... snip: everything reports 0...] > > Brick clustor01:/srv/bricks/29/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick clustor02:/srv/bricks/29/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick clustor00:/srv/quorum/14/d > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > -8<-- > > Il 29/11/2021 12:02, Strahil Nikolov ha scritto: > > What is the output of 'gluster volume heal VOLUME info summary' ? > > > > Best Regards, > > Strahil Nikolov > > > >? ? On Mon, Nov 29, 2021 at 10:33, Diego Zuccato > >? ? <diego.zuccato at unibo.it <mailto:diego.zuccato at unibo.it>> wrote: > >? ? Hello all. > > > >? ? I just brought offline a node (in a replica 3 arbiter 1 volume) to > >? ? install more RAM. > >? ? The other two nodes kept being used, so I expected to see some > >? ? resync at > >? ? power on. But I saw nothing unusual: seems it's just serving files > >? ? as usual. > >? ? Is it normal or should I force a resync? If so, how? > > > >? ? Regards. > > > >? ? -- > >? ? Diego Zuccato > >? ? DIFA - Dip. di Fisica e Astronomia > >? ? Servizi Informatici > >? ? Alma Mater Studiorum - Universit? di Bologna > >? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy > >? ? tel.: +39 051 20 95786 > >? ? ________ > > > > > > > >? ? Community Meeting Calendar: > > > >? ? Schedule - > >? ? Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > >? ? Bridge: https://meet.google.com/cpu-eiue-hvk > <https://meet.google.com/cpu-eiue-hvk> > >? ? <https://meet.google.com/cpu-eiue-hvk > <https://meet.google.com/cpu-eiue-hvk>> > >? ? Gluster-users mailing list > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > <mailto:Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>> > > https://lists.gluster.org/mailman/listinfo/gluster-users > <https://lists.gluster.org/mailman/listinfo/gluster-users> > >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users > <https://lists.gluster.org/mailman/listinfo/gluster-users>> > > > > > -- > Diego Zuccato > DIFA - Dip. di Fisica e Astronomia > Servizi Informatici > Alma Mater Studiorum - Universit? di Bologna > V.le Berti-Pichat 6/2 - 40127 Bologna - Italy > tel.: +39 051 20 95786 >-- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Universit? di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Strahil Nikolov
2021-Dec-01 06:37 UTC
[Gluster-users] force realignement after downing a node?
Usually when healing, gluster uses more cpu, but you can reduce that if needed. Best Regards,Strahil Nikolov On Tue, Nov 30, 2021 at 15:10, Diego Zuccato<diego.zuccato at unibo.it> wrote: Done just that :) Today I upgraded the second node, with a cleaner shutdown. Strangely, at reboot it worked for about half an hour with all the cores at 100% (but low mem use) and "gluster v heal cluster_data info" apparently hanging. Had lunch and now all cores are back to normal (20-60%), memory use is higher ang gluster is responding again. Still no files in heal pending. I'll skip tomorrow, then upgrade the last server. Hope it all goes smoothly again. Tks. Il 30/11/2021 13:17, Strahil Nikolov ha scritto:> Than, > > take a beer/tee/coffee/<whatever is legal> and enjoy the rest of the day ;) > > Best Regards, > Strahil Nikolov > >? ? On Mon, Nov 29, 2021 at 13:09, Diego Zuccato >? ? <diego.zuccato at unibo.it> wrote: >? ? Here it is. Seems gluster thinks there's nothing to be done... > >? ? -8<-- >? ? root at str957-clustor00 <mailto:root at str957-clustor00>:~# gluster v >? ? heal cluster_data info summary >? ? Brick clustor00:/srv/bricks/00/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 > >? ? Brick clustor01:/srv/bricks/00/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 > >? ? Brick clustor02:/srv/quorum/00/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 > >? ? Brick clustor02:/srv/bricks/00/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 > >? ? Brick clustor00:/srv/bricks/01/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 > >? ? Brick clustor01:/srv/quorum/00/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 > >? ? [... snip: everything reports 0...] > >? ? Brick clustor01:/srv/bricks/29/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 > >? ? Brick clustor02:/srv/bricks/29/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 > >? ? Brick clustor00:/srv/quorum/14/d >? ? Status: Connected >? ? Total Number of entries: 0 >? ? Number of entries in heal pending: 0 >? ? Number of entries in split-brain: 0 >? ? Number of entries possibly healing: 0 >? ? -8<-- > >? ? Il 29/11/2021 12:02, Strahil Nikolov ha scritto: >? ? ? > What is the output of 'gluster volume heal VOLUME info summary' ? >? ? ? > >? ? ? > Best Regards, >? ? ? > Strahil Nikolov >? ? ? > >? ? ? >? ? On Mon, Nov 29, 2021 at 10:33, Diego Zuccato >? ? ? >? ? <diego.zuccato at unibo.it <mailto:diego.zuccato at unibo.it>> wrote: >? ? ? >? ? Hello all. >? ? ? > >? ? ? >? ? I just brought offline a node (in a replica 3 arbiter 1 volume) to >? ? ? >? ? install more RAM. >? ? ? >? ? The other two nodes kept being used, so I expected to see some >? ? ? >? ? resync at >? ? ? >? ? power on. But I saw nothing unusual: seems it's just serving files >? ? ? >? ? as usual. >? ? ? >? ? Is it normal or should I force a resync? If so, how? >? ? ? > >? ? ? >? ? Regards. >? ? ? > >? ? ? >? ? -- >? ? ? >? ? Diego Zuccato >? ? ? >? ? DIFA - Dip. di Fisica e Astronomia >? ? ? >? ? Servizi Informatici >? ? ? >? ? Alma Mater Studiorum - Universit? di Bologna >? ? ? >? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy >? ? ? >? ? tel.: +39 051 20 95786 >? ? ? >? ? ________ >? ? ? > >? ? ? > >? ? ? > >? ? ? >? ? Community Meeting Calendar: >? ? ? > >? ? ? >? ? Schedule - >? ? ? >? ? Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >? ? ? >? ? Bridge: https://meet.google.com/cpu-eiue-hvk >? ? <https://meet.google.com/cpu-eiue-hvk> >? ? ? >? ? <https://meet.google.com/cpu-eiue-hvk >? ? <https://meet.google.com/cpu-eiue-hvk>> >? ? ? >? ? Gluster-users mailing list >? ? ? > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >? ? <mailto:Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>> >? ? ? > https://lists.gluster.org/mailman/listinfo/gluster-users >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users> >? ? ? >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users >? ? <https://lists.gluster.org/mailman/listinfo/gluster-users>> > >? ? ? > > >? ? -- >? ? Diego Zuccato >? ? DIFA - Dip. di Fisica e Astronomia >? ? Servizi Informatici >? ? Alma Mater Studiorum - Universit? di Bologna >? ? V.le Berti-Pichat 6/2 - 40127 Bologna - Italy >? ? tel.: +39 051 20 95786 >-- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Universit? di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20211201/15ab4724/attachment.html>