Gluster users - Feb 2024 - Graceful shutdown doesn't stop all Gluster processes

Hello Everyone,


I am using GlusterFS 9.4, and whenever we use the systemctl command to stop the
Gluster server, it leaves many Gluster processes running. So, I just want to
check how to shut down the Gluster server in a graceful manner.


Is there any specific sequence or trick I need to follow? Currently, I am using
the following command:


[root at master2 ~]# systemctl stop glusterd.service

[root at master2 ~]# ps aux | grep gluster
root     2710138 14.1  0.0 2968372 216852 ?      Ssl  Jan27 170:27
/usr/sbin/glusterfsd -s master2 --volfile-id
tier1data.master2.opt-tier1data2019-brick -p
/var/run/gluster/vols/tier1data/master2-opt-tier1data2019-brick.pid -S
/var/run/gluster/97da28e3d5c23317.socket --brick-name /opt/tier1data2019/brick
-l /var/log/glusterfs/bricks/opt-tier1data2019-brick.log --xlator-option
*-posix.glusterd-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --process-name brick
--brick-port 49152 --xlator-option tier1data-server.listen-port=49152
root     2710196  0.0  0.0 1298116 11544 ?       Ssl  Jan27   0:01
/usr/sbin/glusterfs -s localhost --volfile-id shd/tier1data -p
/var/run/gluster/shd/tier1data/tier1data-shd.pid -l
/var/log/glusterfs/glustershd.log -S /var/run/gluster/1ac2284f75671ffa.socket
--xlator-option *replicate*.node-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d
--process-name glustershd --client-pid=-6
root     3730742  0.0  0.0 288264 14388 ?        Ssl  18:44   0:00
/usr/bin/python3 /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py
--path=/opt/tier1data2019/brick  --monitor -c
/var/lib/glusterd/geo-replication/tier1data_drtier1data_drtier1data/gsyncd.conf
--iprefix=/var :tier1data --glusterd-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d
drtier1data::drtier1data
root     3730763  2.4  0.0 2097216 35904 ?       Sl   18:44   0:09 python3
/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py worker tier1data
drtier1data::drtier1data --feedback-fd 9 --local-path /opt/tier1data2019/brick
--local-node master2 --local-node-id c1591bde-df1c-41b4-8cc3-5eaa02c5b89d
--slave-id eca32e08-c3f8-4883-bef5-84bfb89f4d56 --subvol-num 1 --resource-remote
drtier1data --resource-remote-id 28f3e75b-56aa-43a1-a0ea-a0e5d44d59ea
root     3730768  0.7  0.0  50796  9668 ?        S    18:44   0:02 ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock drtier1data
/nonexistent/gsyncd slave tier1data drtier1data::drtier1data --master-node
master2 --master-node-id c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --master-brick
/opt/tier1data2019/brick --local-node drtier1data --local-node-id
28f3e75b-56aa-43a1-a0ea-a0e5d44d59ea --slave-timeout 120 --slave-log-level INFO
--slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin
--master-dist-count 1
root     3730795  1.1  0.0 1108268 55596 ?       Ssl  18:44   0:04
/usr/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO
--log-file=/var/log/glusterfs/geo-replication/tier1data_drtier1data_drtier1data/mnt-opt-tier1data2019-brick.log
--volfile-server=localhost --volfile-id=tier1data --client-pid=-1
/tmp/gsyncd-aux-mount-9210kh43
root     3772665  0.0  0.0  12208  2400 ?        S    18:51   0:00 rsync -aR0
--inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs
--existing --xattrs --acls --ignore-missing-args . -e ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock
drtier1data:/proc/897118/cwd
root     3772667  0.0  0.0  44156  5640 ?        S    18:51   0:00 ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock drtier1data
rsync --server -logDtpAXRe.LsfxC --super --stats --numeric-ids --existing
--inplace --no-implied-dirs . /proc/897118/cwd

For now, We are using
https://github.com/gluster/glusterfs/blob/master/extras/stop-all-gluster-processes.sh
to kill all the remaining processes.

Thanks,
Anant

DISCLAIMER: This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error, please notify the sender.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee, you should not
disseminate, distribute or copy this email. Please notify the sender immediately
by email if you have received this email by mistake and delete this email from
your system.

If you are not the intended recipient, you are notified that disclosing,
copying, distributing or taking any action in reliance on the contents of this
information is strictly prohibited. Thanks for your cooperation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20240205/f6bcc316/attachment.html>

Hi Anant,



It was intentional design decision to not stop any gluster processes if Glusterd
need to be upgraded or if Glusterd crashes. Because of this Volume availability
will not be affected if any issues with Glusterd or Glusterd is upgraded. All
the mounts will reconnect once the Glusterd comes back up. CLI operations from
that node may not be available but IO will not be affected if the Glusterd is
down (New mounts can't be created, but existing mounts should work without
Glusterd).



stop-all-gluster-processes.sh is available as part of installation, it can be
used to stop all the processes (Check in /usr/share/glusterfs/scripts)


--

Thanks and Regards
Aravinda

Kadalu Technologies







---- On Mon, 05 Feb 2024 22:40:30 +0530 Anant Saraswat <anant.saraswat at
techblue.co.uk> wrote ---



Hello Everyone,



I am using GlusterFS 9.4, and whenever we
 use the systemctl command to stop the Gluster server, it leaves many Gluster
processes running. So, I just want to check how to shut down the Gluster server
in a graceful manner.



Is there any specific sequence or trick I need to follow? Currently, I am using
the following command:



[root at master2 ~]# systemctl stop glusterd.service

[root at master2 ~]# ps aux | grep gluster

root ? ? 2710138 14.1 ?0.0 2968372 216852 ? ? ? ?Ssl ?Jan27 170:27
/usr/sbin/glusterfsd -s master2 --volfile-id
 tier1data.master2.opt-tier1data2019-brick -p
/var/run/gluster/vols/tier1data/master2-opt-tier1data2019-brick.pid -S
/var/run/gluster/97da28e3d5c23317.socket --brick-name /opt/tier1data2019/brick
-l /var/log/glusterfs/bricks/opt-tier1data2019-brick.log --xlator-option
 *-posix.glusterd-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --process-name brick
--brick-port 49152 --xlator-option tier1data-server.listen-port=49152

root ? ? 2710196 ?0.0 ?0.0 1298116 11544 ? ? ? ? Ssl ?Jan27 ? 0:01
/usr/sbin/glusterfs -s localhost --volfile-id
 shd/tier1data -p /var/run/gluster/shd/tier1data/tier1data-shd.pid -l
/var/log/glusterfs/glustershd.log -S /var/run/gluster/1ac2284f75671ffa.socket
--xlator-option *replicate*.node-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d
--process-name glustershd --client-pid=-6

root ? ? 3730742 ?0.0 ?0.0 288264 14388 ? ? ? ? ?Ssl ?18:44 ? 0:00
/usr/bin/python3 /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py
 --path=/opt/tier1data2019/brick ?--monitor -c
/var/lib/glusterd/geo-replication/tier1data_drtier1data_drtier1data/gsyncd.conf
--iprefix=/var :tier1data --glusterd-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d
drtier1data::drtier1data

root ? ? 3730763 ?2.4 ?0.0 2097216 35904 ? ? ? ? Sl ? 18:44 ? 0:09 python3
/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py
 worker tier1data drtier1data::drtier1data --feedback-fd 9 --local-path
/opt/tier1data2019/brick --local-node master2 --local-node-id
c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --slave-id
eca32e08-c3f8-4883-bef5-84bfb89f4d56 --subvol-num 1 --resource-remote
drtier1data
 --resource-remote-id 28f3e75b-56aa-43a1-a0ea-a0e5d44d59ea

root ? ? 3730768 ?0.7 ?0.0 ?50796 ?9668 ? ? ? ? ?S ? ?18:44 ? 0:02 ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no
 -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock drtier1data
/nonexistent/gsyncd slave tier1data drtier1data::drtier1data --master-node
master2 --master-node-id
 c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --master-brick /opt/tier1data2019/brick
--local-node drtier1data --local-node-id 28f3e75b-56aa-43a1-a0ea-a0e5d44d59ea
--slave-timeout 120 --slave-log-level INFO --slave-gluster-log-level INFO
--slave-gluster-command-dir
 /usr/sbin --master-dist-count 1

root ? ? 3730795 ?1.1 ?0.0 1108268 55596 ? ? ? ? Ssl ?18:44 ? 0:04
/usr/sbin/glusterfs --aux-gfid-mount --acl
 --log-level=INFO
--log-file=/var/log/glusterfs/geo-replication/tier1data_drtier1data_drtier1data/mnt-opt-tier1data2019-brick.log
--volfile-server=localhost --volfile-id=tier1data --client-pid=-1
/tmp/gsyncd-aux-mount-9210kh43

root ? ? 3772665 ?0.0 ?0.0 ?12208 ?2400 ? ? ? ? ?S ? ?18:51 ? 0:00 rsync -aR0
--inplace --files-from=- --super
 --stats --numeric-ids --no-implied-dirs --existing --xattrs --acls
--ignore-missing-args . -e ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock
 drtier1data:/proc/897118/cwd

root ? ? 3772667 ?0.0 ?0.0 ?44156 ?5640 ? ? ? ? ?S ? ?18:51 ? 0:00 ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no
 -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock drtier1data
rsync --server -logDtpAXRe.LsfxC --super --stats --numeric-ids --existing
--inplace --no-implied-dirs
 . /proc/897118/cwd



For now, We are using
https://github.com/gluster/glusterfs/blob/master/extras/stop-all-gluster-processes.sh?to
kill all the remaining processes.



Thanks,

Anant





DISCLAIMER: This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error, please
 notify the sender. This message contains confidential information and is
intended only for the individual named. If you are not the named addressee, you
should not disseminate, distribute or copy this email. Please notify the sender
immediately by email if
 you have received this email by mistake and delete this email from your system.
 
 If you are not the intended recipient, you are notified that disclosing,
copying, distributing or taking any action in reliance on the contents of this
information is strictly prohibited. Thanks for your cooperation.

________
 
 
 
Community Meeting Calendar: 
 
Schedule - 
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC 
Bridge: https://meet.google.com/cpu-eiue-hvk 
Gluster-users mailing list 
Gluster-users at gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20240206/6b0e6397/attachment.html>