Gluster users - Feb 2024 - Graceful shutdown doesn't stop all Gluster processes

If I might chip in here, this can cause an issue when rebooting nodes unless you
make sure to stop the Gluster processes first. If you don't stop the
processes, then the Gluster volumes can pause for the default 42 seconds(?)
until the other nodes time out the rebooting node. This is of course long enough
to cause any VMs running their volumes on gluster to show I/O errors and
re-mount them as read-only potentially causing all sorts of mischief.

I *think* there's a systemd solution to this somewhere, perhaps the RedHat
packages?, that does stop the gluster processes prior to a reboot or halt.

Certainly the Debian packages *don't* have this solution in place. I wish
they did but I've never mastered enough systemd foo to sort if out myself.
:-).

Ronny


Aravinda wrote on 05/02/2024 19:09:
> Hi Anant,
>
> It was intentional design decision to not stop any gluster processes if
Glusterd need to be upgraded or if Glusterd crashes. Because of this Volume
availability will not be affected if any issues with Glusterd or Glusterd is
upgraded. All the mounts will reconnect once the Glusterd comes back up. CLI
operations from that node may not be available but IO will not be affected if
the Glusterd is down (New mounts can't be created, but existing mounts
should work without Glusterd).
>
> stop-all-gluster-processes.sh is available as part of installation, it can
be used to stop all the processes (Check in /usr/share/glusterfs/scripts)
>
> --
> Thanks and Regards
> Aravinda
> Kadalu Technologies
>
>
>
> ---- On Mon, 05 Feb 2024 22:40:30 +0530 *Anant Saraswat <anant.saraswat
at techblue.co.uk>* wrote ---
>
>     Hello Everyone,
>
>
>     I am using GlusterFS 9.4, and whenever we use the systemctl command to
stop the Gluster server, it leaves many Gluster processes running. So, I just
want to check how to shut down the Gluster server in a graceful manner.
>
>
>     Is there any specific sequence or trick I need to follow? Currently, I
am using the following command:
>
>
>     [root at master2 ~]# systemctl stop glusterd.service
>
>     [root at master2 ~]# ps aux | grep gluster
>     root ? ? 2710138 14.1 ?0.0 2968372 216852 ? ? ? ?Ssl ?Jan27 170:27
/usr/sbin/glusterfsd -s master2 --volfile-id
tier1data.master2.opt-tier1data2019-brick -p
/var/run/gluster/vols/tier1data/master2-opt-tier1data2019-brick.pid -S
/var/run/gluster/97da28e3d5c23317.socket --brick-name /opt/tier1data2019/brick
-l /var/log/glusterfs/bricks/opt-tier1data2019-brick.log --xlator-option
*-posix.glusterd-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --process-name brick
--brick-port 49152 --xlator-option tier1data-server.listen-port=49152
>     root ? ? 2710196 ?0.0 ?0.0 1298116 11544 ? ? ? ? Ssl ?Jan27 ? 0:01
/usr/sbin/glusterfs -s localhost --volfile-id shd/tier1data -p
/var/run/gluster/shd/tier1data/tier1data-shd.pid -l
/var/log/glusterfs/glustershd.log -S /var/run/gluster/1ac2284f75671ffa.socket
--xlator-option *replicate*.node-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d
--process-name glustershd --client-pid=-6
>     root ? ? 3730742 ?0.0 ?0.0 288264 14388 ? ? ? ? ?Ssl ?18:44 ? 0:00
/usr/bin/python3 /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py
--path=/opt/tier1data2019/brick ?--monitor -c
/var/lib/glusterd/geo-replication/tier1data_drtier1data_drtier1data/gsyncd.conf
--iprefix=/var :tier1data --glusterd-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d
drtier1data::drtier1data
>     root ? ? 3730763 ?2.4 ?0.0 2097216 35904 ? ? ? ? Sl ? 18:44 ? 0:09
python3 /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py worker tier1data
drtier1data::drtier1data --feedback-fd 9 --local-path /opt/tier1data2019/brick
--local-node master2 --local-node-id c1591bde-df1c-41b4-8cc3-5eaa02c5b89d
--slave-id eca32e08-c3f8-4883-bef5-84bfb89f4d56 --subvol-num 1 --resource-remote
drtier1data --resource-remote-id 28f3e75b-56aa-43a1-a0ea-a0e5d44d59ea
>     root ? ? 3730768 ?0.7 ?0.0 ?50796 ?9668 ? ? ? ? ?S ? ?18:44 ? 0:02 ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock drtier1data
/nonexistent/gsyncd slave tier1data drtier1data::drtier1data --master-node
master2 --master-node-id c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --master-brick
/opt/tier1data2019/brick --local-node drtier1data --local-node-id
28f3e75b-56aa-43a1-a0ea-a0e5d44d59ea --slave-timeout 120 --slave-log-level INFO
--slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin
--master-dist-count 1
>     root ? ? 3730795 ?1.1 ?0.0 1108268 55596 ? ? ? ? Ssl ?18:44 ? 0:04
/usr/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO
--log-file=/var/log/glusterfs/geo-replication/tier1data_drtier1data_drtier1data/mnt-opt-tier1data2019-brick.log
--volfile-server=localhost --volfile-id=tier1data --client-pid=-1
/tmp/gsyncd-aux-mount-9210kh43
>     root ? ? 3772665 ?0.0 ?0.0 ?12208 ?2400 ? ? ? ? ?S ? ?18:51 ? 0:00
rsync -aR0 --inplace --files-from=- --super --stats --numeric-ids
--no-implied-dirs --existing --xattrs --acls --ignore-missing-args . -e ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock
drtier1data:/proc/897118/cwd
>     root ? ? 3772667 ?0.0 ?0.0 ?44156 ?5640 ? ? ? ? ?S ? ?18:51 ? 0:00 ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock drtier1data
rsync --server -logDtpAXRe.LsfxC --super --stats --numeric-ids --existing
--inplace --no-implied-dirs . /proc/897118/cwd
>
>     For now, We are using
https://github.com/gluster/glusterfs/blob/master/extras/stop-all-gluster-processes.sh?to
kill all the remaining processes.
>
>     Thanks,
>     Anant
>
>
>     DISCLAIMER: This email and any files transmitted with it are
confidential and intended solely for the use of the individual or entity to whom
they are addressed. If you have received this email in error, please notify the
sender. This message contains confidential information and is intended only for
the individual named. If you are not the named addressee, you should not
disseminate, distribute or copy this email. Please notify the sender immediately
by email if you have received this email by mistake and delete this email from
your system.
>
>     If you are not the intended recipient, you are notified that
disclosing, copying, distributing or taking any action in reliance on the
contents of this information is strictly prohibited. Thanks for your
cooperation.
>
>     ________
>
>
>
>     Community Meeting Calendar:
>
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://meet.google.com/cpu-eiue-hvk
>     Gluster-users mailing list
>     Gluster-users at gluster.org
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-- 

Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8977 8943
w: www.amazinginternet.com

Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ
Registered in England. Company No. 4042957

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20240207/267a7685/attachment.html>

I think the service that shutdowns the bricks on EL systems is something like
this - right now I don't have access to my systems to check but you can
extract the rpms and see it:
https://bugzilla.redhat.com/show_bug.cgi?id=1022542#c4
Best Regards,Strahil Nikolov
 
 
  On Wed, Feb 7, 2024 at 19:51, Ronny Adsetts<ronny.adsetts at
amazinginternet.com> wrote:   ________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20240209/075465cf/attachment.html>