On 12/01/2015 04:15 AM, Andrus, Brian Contractor wrote:> All, > > > > I am seeing it VERY consistently that when I do a ?gluster peer status? > or ?gluster pool list?, the system ?hangs? for up to 1 minute before > spitting back results.That's interesting, 'gluster peer status' or 'pool list' is a local node operation and doesn't incur RPC calls. Until and unless it takes time to communicate back to CLI because of a poor N/W, I don't see any reason why would it take such a long time. Can you run glusterd with debug log enabled and share the *glusterd.log file with us? ~Atin> > > > I have 10 nodes all on the same network and currently ZERO volumes or > bricks configured. Just trying to get good performance for the cluster > to be talking to itself?. > > > > > > What is gluster doing that takes so long to respond? Seems there may be > a more efficient way of doing it, whatever it is? > > > > > > > > Brian Andrus > > ITACS/Research Computing > > Naval Postgraduate School > > Monterey, California > > voice: 831-656-6238 > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >
Andrus, Brian Contractor
2015-Dec-01 20:38 UTC
[Gluster-users] Why oh why does gluster delay?
Atin, In debug, the glusterd log file is pretty sizeable. I do see regular: [2015-12-01 20:29:14.407105] E [MSGID: 106167] [glusterd-handshake.c:2073:__glusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later And ones like: [2015-12-01 20:29:45.057998] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-management: server 172.18.56.55:24007 has not responded in the last 30 seconds, disconnecting. Hopefully there is something in there that can give a clue as to what is going on. Incidentally, to enable debug using the RPMs, the code in /etc/sysconfig/glusterd is inaccurate. The init file looks for LOG_FILE and LOG_LEVEL rather than the GLUSTERD_LOGLEVEL and GLUSTERD_LOGFILE that are in there as examples. Brian Andrus ITACS/Research Computing Naval Postgraduate School Monterey, California -----Original Message----- From: Atin Mukherjee [mailto:amukherj at redhat.com] Sent: Monday, November 30, 2015 8:09 PM To: Andrus, Brian Contractor <bdandrus at nps.edu>; gluster-users at gluster.org Subject: Re: [Gluster-users] Why oh why does gluster delay? On 12/01/2015 04:15 AM, Andrus, Brian Contractor wrote:> All, > > > > I am seeing it VERY consistently that when I do a 'gluster peer status' > or 'gluster pool list', the system 'hangs' for up to 1 minute before > spitting back results.That's interesting, 'gluster peer status' or 'pool list' is a local node operation and doesn't incur RPC calls. Until and unless it takes time to communicate back to CLI because of a poor N/W, I don't see any reason why would it take such a long time. Can you run glusterd with debug log enabled and share the *glusterd.log file with us? ~Atin> > > > I have 10 nodes all on the same network and currently ZERO volumes or > bricks configured. Just trying to get good performance for the cluster > to be talking to itself.... > > > > > > What is gluster doing that takes so long to respond? Seems there may > be a more efficient way of doing it, whatever it is... > > > > > > > > Brian Andrus > > ITACS/Research Computing > > Naval Postgraduate School > > Monterey, California > > voice: 831-656-6238 > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- A non-text attachment was scrubbed... Name: gluster.log.tgz Type: application/x-compressed Size: 127536 bytes Desc: gluster.log.tgz URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151201/00c9b39d/attachment.bin>