Gluster users - Nov 2018 - How to check running transactions in gluster?

Hi,

I am getting output Another transaction is in progress with few gluster
volume commands including stop command. And with gluster volume status
command, it's just hung and fails with timeout error.

So, I want to find out which transaction is hung and how to know this? I
ran volume statedump command, but didn't wait till it's completed to
check
if it's hung or giving any resut, as it is also taking time.

Please help me with this.. I'm struggling with these gluster timeout errors
:(

Besides, I have also tuned
transport.listen-backlog gluster to 200 and following kernel parameters to
avoid syn overflow rejects:
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 20480

Regards,
Jeevan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20181125/9601e9df/attachment.html>

On Sun, Nov 25, 2018 at 8:40 PM Jeevan Patnaik <g1patnaik at gmail.com>
wrote:
> Hi,
>
> I am getting output Another transaction is in progress with few gluster
> volume commands including stop command. And with gluster volume status
> command, it's just hung and fails with timeout error.
>
This is primarily because of not allowing glusterd to complete it's
handshake with others when concurrent restart of glusterd services are done
(as I could understand from your previous email in the list). With GlusterD
(read as GD1) this is a current challenge w.r.t it's design where due to
its N X N handshaking mechanism at the restart sequence to bring all the
configuration data into inconsistent what we've seen is the overall
recovery time of the cluster can take very long if N is on the higher side
(in your case N = 72 which is certainly high) and hence the recommendation
is not to restart the glusterd services concurrently and wait for the
handshaking to complete.

> So, I want to find out which transaction is hung and how to know this? I
> ran volume statedump command, but didn't wait till it's completed
to check
> if it's hung or giving any resut, as it is also taking time.
>
kill -SIGUSR1 $(pidof glusterd) should get you a glusterd statedump file in
/var/run/gluster which can point to a backtrace dump at the bottom to
understand which transaction is currently in progress. In case this
transaction is queued up for more than 180 seconds (which is not usual) the
unlock timer kicks out such locks.

> Please help me with this.. I'm struggling with these gluster timeout
> errors :(
>
> Besides, I have also tuned
> transport.listen-backlog gluster to 200 and following kernel parameters to
> avoid syn overflow rejects:
> net.core.somaxconn = 1024
> net.ipv4.tcp_max_syn_backlog = 20480
>
> Regards,
> Jeevan.
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20181126/e7695e20/attachment.html>