Gluster users - Mar 2019 - Transport endpoint is not connected failures in

On Wed, Mar 27, 2019 at 9:46 PM <brandon at thinkhuge.net> wrote:
> Hello Amar and list,
>
>
>
> I wanted to follow-up to confirm that upgrading to 5.5 seem to fix the
> ?Transport endpoint is not connected failures? for us.
>
What was the version you saw failures in? Were there any logs matching with
the pattern "ping_timer_expired" earlier?

>
> We did not have any of these failures in this past weekend backups cycle.
>
>
>
> Thank you very much for fixing whatever was the problem.
>
>
>
> I also removed some volume config options.  One or more of the settings
> was contributing to the slow directory listing.
>
>
>
> Here is our current volume info.
>
>
>
> [root at lonbaknode3 ~]# gluster volume info
>
>
>
> Volume Name: volbackups
>
> Type: Distribute
>
> Volume ID: 32bf4fe9-5450-49f8-b6aa-05471d3bdffa
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 8
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: lonbaknode3.domain.net:/lvbackups/brick
>
> Brick2: lonbaknode4.domain.net:/lvbackups/brick
>
> Brick3: lonbaknode5.domain.net:/lvbackups/brick
>
> Brick4: lonbaknode6.domain.net:/lvbackups/brick
>
> Brick5: lonbaknode7.domain.net:/lvbackups/brick
>
> Brick6: lonbaknode8.domain.net:/lvbackups/brick
>
> Brick7: lonbaknode9.domain.net:/lvbackups/brick
>
> Brick8: lonbaknode10.domain.net:/lvbackups/brick
>
> Options Reconfigured:
>
> performance.io-thread-count: 32
>
> performance.client-io-threads: on
>
> client.event-threads: 8
>
> diagnostics.brick-sys-log-level: WARNING
>
> diagnostics.brick-log-level: WARNING
>
> performance.cache-max-file-size: 2MB
>
> performance.cache-size: 256MB
>
> cluster.min-free-disk: 1%
>
> nfs.disable: on
>
> transport.address-family: inet
>
> server.event-threads: 8
>
> [root at lonbaknode3 ~]#
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190328/52cf817c/attachment.html>

Hello,

 

Yes I did find some hits on this in the following logs. We started seeing
failures after upgrading to 5.3 from 4.6.  If you want me to check for something
else let me know.   Thank you all on the gluster team for finding and fixing
that problem whatever it was!

 

[root at lonbaknode3 glusterfs]# zgrep ping_timer
/var/log/glusterfs/home-volbackups*

....

/var/log/glusterfs/home-volbackups.log-20190317.gz:[2019-03-16 10:34:44.419605]
C [rpc-clnt-ping.c:162:rpc_clnt_ping_timer_expired] 0-volbackups-client-3:
server 1.2.3.4:49153 has not responded in the last 42 seconds, disconnecting.

/var/log/glusterfs/home-volbackups.log-20190317.gz:[2019-03-16 10:34:44.419672]
C [rpc-clnt-ping.c:162:rpc_clnt_ping_timer_expired] 0-volbackups-client-6:
server 1.2.3.4:49153 has not responded in the last 42 seconds, disconnecting.

/var/log/glusterfs/home-volbackups.log-20190317.gz:[2019-03-16 10:34:57.425211]
C [rpc-clnt-ping.c:162:rpc_clnt_ping_timer_expired] 0-volbackups-client-9:
server 1.2.3.4:49153 has not responded in the last 42 seconds, disconnecting.

/var/log/glusterfs/home-volbackups.log-20190317.gz:[2019-03-16 11:46:25.768650]
C [rpc-clnt-ping.c:162:rpc_clnt_ping_timer_expired] 0-volbackups-client-6:
server 1.2.3.4:49153 has not responded in the last 42 seconds, disconnecting.

/var/log/glusterfs/home-volbackups.log-20190317.gz:[2019-03-16 16:02:29.921450]
C [rpc-clnt-ping.c:162:rpc_clnt_ping_timer_expired] 0-volbackups-client-3:
server 1.2.3.4:49153 has not responded in the last 42 seconds, disconnecting.

....

 

-----

What was the version you saw failures in? Were there any logs matching with the
pattern "ping_timer_expired" earlier?

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190329/a36d6e51/attachment.html>