Okay, I did some digging. On the client there was many errors such as: [2015-04-29 15:47:08.700174] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-img-client-0: remote operation failed: Transport endpoint is not connected. Path: /www/img/gallery/9722926_4130.jpg (00000000-0000-0000-0000-000000000000) [2015-04-29 15:47:08.700268] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-img-replicate-0: looking up /www/img/gallery/9722926_4130.jpg under img-client-0 failed (Transport endpoint is not connected) And at the same time on the cluster: [2015-04-29 15:47:59.989897] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-img-client-0: remote operation failed: Transport endpoint is not connected. Path: /www/pdf/23096091-1722.pdf (00000000-0000-0000-0000-000000000000) [2015-04-29 15:47:59.989923] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-img-replicate-0: looking up /www/pdf/23096091-1722.pdf under img-client-0 failed (Transport endpoint is not connected) What could it mean? Is there some kind of network error? BTW there was nothing that indicated any network connectivity problems between nodes and clients.
----- Original Message -----> From: "Alex" <alex.m at icecat.biz> > To: gluster-users at gluster.org > Sent: Thursday, April 30, 2015 6:52:58 AM > Subject: Re: [Gluster-users] Write operations failing on clients > > Okay, I did some digging. On the client there was many errors such as: > > [2015-04-29 15:47:08.700174] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] > 0-img-client-0: remote operation failed: Transport endpoint is not > connected. Path: /www/img/gallery/9722926_4130.jpg > (00000000-0000-0000-0000-000000000000) > [2015-04-29 15:47:08.700268] I > [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] > 0-img-replicate-0: looking up /www/img/gallery/9722926_4130.jpg under > img-client-0 failed (Transport endpoint is not connected) > > And at the same time on the cluster: > [2015-04-29 15:47:59.989897] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] > 0-img-client-0: remote operation failed: Transport endpoint is not > connected. Path: /www/pdf/23096091-1722.pdf > (00000000-0000-0000-0000-000000000000) > [2015-04-29 15:47:59.989923] I > [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] > 0-img-replicate-0: looking up /www/pdf/23096091-1722.pdf under img-client-0 > failed (Transport endpoint is not connected) > > > What could it mean? Is there some kind of network error? BTW there was > nothing that indicated any network connectivity problems between nodes and > clients.> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >
----- Original Message -----> From: "Alex" <alex.m at icecat.biz> > To: gluster-users at gluster.org > Sent: Thursday, April 30, 2015 6:52:58 AM > Subject: Re: [Gluster-users] Write operations failing on clients > > Okay, I did some digging. On the client there was many errors such as: > > [2015-04-29 15:47:08.700174] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] > 0-img-client-0: remote operation failed: Transport endpoint is not > connected. Path: /www/img/gallery/9722926_4130.jpg > (00000000-0000-0000-0000-000000000000) > [2015-04-29 15:47:08.700268] I > [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] > 0-img-replicate-0: looking up /www/img/gallery/9722926_4130.jpg under > img-client-0 failed (Transport endpoint is not connected) > > And at the same time on the cluster: > [2015-04-29 15:47:59.989897] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] > 0-img-client-0: remote operation failed: Transport endpoint is not > connected. Path: /www/pdf/23096091-1722.pdf > (00000000-0000-0000-0000-000000000000) > [2015-04-29 15:47:59.989923] I > [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] > 0-img-replicate-0: looking up /www/pdf/23096091-1722.pdf under img-client-0 > failed (Transport endpoint is not connected) > > > What could it mean? Is there some kind of network error? BTW there was > nothing that indicated any network connectivity problems between nodes and > clients.Hi Alex. You are correct when you see the "Transport endpoint is not connected." it usually means that the client is unable to access the server. Check gluster v status and make sure all your bricks are online. Try to unmount / remount the client if you see everything is up. If you still cant access and everythin is online I would do some basic NW troubleshooting, make sure selinux is off on the servers, and check /var/log/glusterfs/bricks for errors on each of the servers. If you see any error messages of the severity { M | A | C | E } pastebin them to me and I'll have a look. -b> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >