Carl Boberg
2012-Feb-23 21:49 UTC
[Gluster-users] Help with some socket related logwarnings
Hello I have just started to prepare a smallish production setup (nothing critical running on it yet). I have 2 gluster servers with 8 volumes and Im getting a lot of theese warnings in the cli.log [2012-02-23 22:32:15.808271] W [rpc-transport.c:606:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" Could be related to the warinings in the etc-glusterfs-glusterd.vol.log [2012-02-21 17:28:15.279406] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.131.139.26:1019) [2012-02-21 17:28:15.279521] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.131.139.26:1016) [2012-02-21 17:28:19.236986] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.131.139.25:1013) [2012-02-21 17:28:42.660021] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (127.0.0.1:1012) [2012-02-21 17:34:58.405575] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (10.131.139.9:1021) where 10.131.139.26 and 25 are the 2 servers and 10.131.139.9 is a client (other clients ip:s are also showing up) 'gluster peer status' on either server says everything is fine and connected and im not experiencing any problems. Im just a little nervous about the warnings since they get logged quite often. Would be greatful for som insight in the reason for theese warnings. gluster 3.2.5 rpm install all clients and servers are RedHat 6.1 using native client Best regards --- Carl Boberg Operations Memnon Networks AB Tegn?rgatan 34, SE-113 59 Stockholm Mobile: +46(0)70 467 27 12 www.memnonnetworks.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120223/1985bea8/attachment.html>
David Coulson
2012-Feb-23 21:51 UTC
[Gluster-users] Help with some socket related logwarnings
what's in the client logs and the brick logs in /var/log/glusterfs/*log and /var/log/glusterfs/bricks/*log On 2/23/12 4:49 PM, Carl Boberg wrote:> Hello > > I have just started to prepare a smallish production setup (nothing > critical running on it yet). > > I have 2 gluster servers with 8 volumes and Im getting a lot of theese > warnings in the cli.log > [2012-02-23 22:32:15.808271] W > [rpc-transport.c:606:rpc_transport_load] 0-rpc-transport: missing > 'option transport-type'. defaulting to "socket" > > Could be related to the warinings in the etc-glusterfs-glusterd.vol.log > [2012-02-21 17:28:15.279406] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.131.139.26:1019 <http://10.131.139.26:1019>) > [2012-02-21 17:28:15.279521] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.131.139.26:1016 <http://10.131.139.26:1016>) > [2012-02-21 17:28:19.236986] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.131.139.25:1013 <http://10.131.139.25:1013>) > [2012-02-21 17:28:42.660021] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (127.0.0.1:1012 <http://127.0.0.1:1012>) > [2012-02-21 17:34:58.405575] W > [socket.c:1494:__socket_proto_state_machine] 0-socket.management: > reading from socket failed. Error (Transport endpoint is not > connected), peer (10.131.139.9:1021 <http://10.131.139.9:1021>) > > > where 10.131.139.26 and 25 are the 2 servers and 10.131.139.9 is a > client (other clients ip:s are also showing up) > > 'gluster peer status' on either server says everything is fine and > connected and im not experiencing any problems. Im just a little > nervous about the warnings since they get logged quite often. Would be > greatful for som insight in the reason for theese warnings. > > gluster 3.2.5 rpm install > all clients and servers are RedHat 6.1 using native client > > Best regards > --- > Carl Boberg > Operations > > Memnon Networks AB > Tegn?rgatan 34, SE-113 59 Stockholm > > Mobile: +46(0)70 467 27 12 > www.memnonnetworks.com <http://www.memnonnetworks.com> > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120223/140abd69/attachment.html>
Dan Bretherton
2012-Feb-25 15:30 UTC
[Gluster-users] Help with some socket related logwarnings
(Re-sending with correct subject - sorry) Hello Carl and list members, My etc-glusterfs-glusterd.vol.log files are also flooded with warning messages like the following. bdan4: [2012-01-05 18:54:30.385095] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (192.171.166.72:993) Some of the "peers" mentioned are actually clients but others are GlusterFS servers. The warnings seem to occur repeatedly for a period of time on all the servers and then stop for a while before starting again. I have a feeling it has something to do with NFS, because it seems to occur more frequently on servers I use to export NFS (although all are running the glusterfs NFS process). The warnings seem to coincide with the NFS processes restarting, as they did on all the servers in my cluster at 16:48 today, un-noticed by me, for no apparent reason. The warnings stopped on most servers soon after this, but they have carried on since then on the servers that are exporting NFS. I have been ignoring these warnings for some time because there didn't seem to be any adverse effects, but I am trying to solve a serious performance problem affecting some applications and I want to look at all possible causes for concern. I have seen these warnings in log files posted to the list in the context of various problems which I don't have, but I have never seen them mentioned specifically. Are these spurious warnings that can be ignored or do they point to a problem with my cluster? Any comments or suggestions would be much appreciated. Regards Dan. On 02/23/2012 10:01 PM, gluster-users-request at gluster.org wrote:> Date: Thu, 23 Feb 2012 22:49:52 +0100 > From: Carl Boberg<carl.boberg at memnonnetworks.com> > Subject: [Gluster-users] Help with some socket related logwarnings > To:gluster-users at gluster.org > Message-ID: > <CANnFLQwZHpjrVkHCrME0wWFBUFs_ZJQv1dWVHBQRi6HSkTDQag at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hello > > I have just started to prepare a smallish production setup (nothing > critical running on it yet). > > I have 2 gluster servers with 8 volumes and Im getting a lot of theese > warnings in the cli.log > [2012-02-23 22:32:15.808271] W [rpc-transport.c:606:rpc_transport_load] > 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" > > Could be related to the warinings in the etc-glusterfs-glusterd.vol.log > [2012-02-21 17:28:15.279406] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (10.131.139.26:1019) > [2012-02-21 17:28:15.279521] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (10.131.139.26:1016) > [2012-02-21 17:28:19.236986] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (10.131.139.25:1013) > [2012-02-21 17:28:42.660021] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (127.0.0.1:1012) > [2012-02-21 17:34:58.405575] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (10.131.139.9:1021) > > > where 10.131.139.26 and 25 are the 2 servers and 10.131.139.9 is a client > (other clients ip:s are also showing up) > > 'gluster peer status' on either server says everything is fine and > connected and im not experiencing any problems. Im just a little nervous > about the warnings since they get logged quite often. Would be greatful for > som insight in the reason for theese warnings. > > gluster 3.2.5 rpm install > all clients and servers are RedHat 6.1 using native client > > Best regards > --- > Carl Boberg > Operations > > Memnon Networks AB > Tegn?rgatan 34, SE-113 59 Stockholm > > Mobile: +46(0)70 467 27 12 > www.memnonnetworks.com
Carl Boberg
2012-Feb-27 09:58 UTC
[Gluster-users] Help with some socket related logwarnings
Ok, The line "option transport.socket.read-fail-log off" was not in my /etc/glusterfs/glusterd.vol file on neither of the servers. What does this line do exactly? FYI. My install is just a basic rpm install made by following the docs. I have not added anything to the config files neither by glusterd command or by direct editing. I have added the read-fail-log off option and restarted glusterd on both servers. I still get this in the cli.log about once every minute on one of the servers: [2012-02-27 10:48:55.332085] W [rpc-transport.c:606:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" [2012-02-27 10:48:55.338023] I [cli-cmd-volume.c:1077:cli_check_gsync_present] 0-: geo-replication not installed [2012-02-27 10:48:55.338371] I [cli-rpc-ops.c:251:gf_cli3_1_list_friends_cbk] 0-cli: Received resp to list: 0 [2012-02-27 10:48:55.338450] I [input.c:46:cli_batch] 0-: Exiting with: 0 And this in the etc log at the same interval: [2012-02-27 10:48:55.338244] I [glusterd-handler.c:750:glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2012-02-27 10:48:55.339195] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (127.0.0.1:1023) Cheers --- Carl Boberg Operations Memnon Networks AB Tegn?rgatan 34, SE-113 59 Stockholm Mobile: +46(0)70 467 27 12 www.memnonnetworks.com On Mon, Feb 27, 2012 at 09:56, Amar Tumballi <amarts at redhat.com> wrote:> On 02/26/2012 01:30 AM, Brian Candler wrote: > >> On Sat, Feb 25, 2012 at 08:27:06PM +0100, Carl Boberg wrote: >> >>> [socket.c:1494:__socket_proto_**state_machine] 0-socket.management: >>> reading from socket failed. Error (Transport endpoint is not >>> connected), peer ([2]127.0.0.1:1020) >>> And I cant figure out why I get the socket error with 127.0.0.1 >>> I have checked all network configurations/dns on the server. >>> >> >> Including /etc/hosts? That is, you don't have an entry like >> >> 127.0.0.1 myhostname >> >> (Some distros have a habit of adding an entry like that for you) >> ______________________________**______ >> > > Can you please check if your 'glusterd.vol' file (located in > /etc/glusterfs/glusterd.vol) has below lines? > > " option transport.socket.read-fail-log off" > > If not, you can go ahead and add it. > > Dan, to answer your question, this log in 'glusterd.log' file is not > serious. But it gives a hint of possible network issues in Filesystem logs > (like brick logs and nfs logs and native client logs), as in Filesystem > setup we need to be connected all the time. > > > Regards, > Amar >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120227/8e54ef2d/attachment.html>