Atin Mukherjee
2015-May-26 04:01 UTC
[Gluster-users] [Gluster-devel] Gluster 3.7.0 released
On 05/26/2015 03:12 AM, Ted Miller wrote:> > From: Niels de Vos <ndevos at redhat.com> > Sent: Monday, May 25, 2015 4:44 PM > > On Mon, May 25, 2015 at 06:49:26PM +0000, Ted Miller wrote: >> >> ________________________________ >> From: Humble Devassy Chirammal <humble.devassy at gmail.com> >> Sent: Monday, May 18, 2015 9:37 AM >> Hi All, >> >> GlusterFS 3.7.0 RPMs for RHEL, CentOS, Fedora and packages for Debian are available at download.gluster.org<http://download.gluster.org> [1]. >> >> [1] http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/ >> >> --Humble >> >> >> On Thu, May 14, 2015 at 2:49 PM, Vijay Bellur <vbellur at redhat.com<mailto:vbellur at redhat.com>> wrote: >> >> Hi All, >> >> I am happy to announce that Gluster 3.7.0 is now generally available. 3.7.0 contains several >> >> [snip] >> >> Cheers, >> Vijay >> >> [snip] >> >> What happened to packages for RHEL/Centos 5? I have the (probably >> unusual--added gluster to existing servers) setup of running a replica >> 3 cluster where two nodes run on Centos 6 and one is still on Centos >> 5. This is a personal setup, and I have been using >> http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-5/x86_64/repodata/repomod.xml >> as my repo. It has worked fine for a while, but this time the two >> Centos 6 nodes updated to 3.7, but the Centos 5 node got left behind >> at 3.6.3. > > Packages for RHEL/CentOS-5 are not available yet. These will follow > later. Thare are some changes needed to be able to build the packages on > EL5. Because we are currently stabilizing our CI/regression tests, we do > not merge any other changes. Until we provide packages in our > repository, you could apply patch http://review.gluster.org/10803 > yourself and build the EL5 version. I expect that we will do a release > in 2-3 weeks which will have EL5 RPMs too. > > I have no idea about the problem below, it sounds like something the > GlusterD developers could help with. > > Niels > >> Command 'gluster volume status' on the C5 machine makes everything >> look fine: >> >> Status of volume: ISO2 >> Gluster process Port Online Pid >> ------------------------------------------------------------------------------ >> Brick 10.x.x.2:/bricks/01/iso2 49162 Y 4679 >> Brick 10.x.x.4:/bricks/01/iso2 49183 Y 6447 >> Brick 10.x.x.9:/bricks/01/iso2 49169 Y 1985 >> >> But the same command on either of the C6 machines shows the C5 machine >> (10.x.x.2) missing in action (though it does recognize that there are >> NFS and heal daemons there): >> >> Status of volume: ISO2 >> Gluster process TCP Port RDMA Port Online Pid >> ------------------------------------------------------------------------------ >> Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447 >> Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985 >> NFS Server on localhost 2049 0 Y 2279 >> Self-heal Daemon on localhost N/A N/A Y 2754 >> NFS Server on 10.41.65.2 2049 0 Y 4757 >> Self-heal Daemon on 10.41.65.2 N/A N/A Y 4764 >> NFS Server on 10.41.65.4 2049 0 Y 6543 >> Self-heal Daemon on 10.41.65.4 N/A N/A Y 6551 >> >> So, is this just an oversight (I hope), or has support for C5 been dropped? >> If support for C5 is gone, how do I downgrade my Centos6 machines back >> to 3.6.x? (I know how to change the repo, but the actual sequence of >> yum commands and gluster commands is unknown to me).Could you attach the glusterd log file of 10.x.x.2 machine and the node from where you triggered volume status. Could you also share gluster volume info output of all the nodes?>> >> Ted Miller >> Elkhart, IN, USA > > > Thanks for the information. As long as I know it is coming, I can improvise and hang on. > > I am assuming that the problem with the .2 machine not being seen is a result of running a cluster with a version split. > > Ted Miller > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-- ~Atin
responses below Ted Miller On 5/26/2015 12:01 AM, Atin Mukherjee wrote:> > On 05/26/2015 03:12 AM, Ted Miller wrote: >> From: Niels de Vos <ndevos at redhat.com> >> Sent: Monday, May 25, 2015 4:44 PM >> >> On Mon, May 25, 2015 at 06:49:26PM +0000, Ted Miller wrote: >>> ________________________________ >>> From: Humble Devassy Chirammal <humble.devassy at gmail.com> >>> Sent: Monday, May 18, 2015 9:37 AM >>> Hi All, >>> >>> GlusterFS 3.7.0 RPMs for RHEL, CentOS, Fedora and packages for Debian are available at download.gluster.org<http://download.gluster.org> [1]. >>> >>> [1] http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/ >>> >>> --Humble >>> >>> >>> On Thu, May 14, 2015 at 2:49 PM, Vijay Bellur <vbellur at redhat.com<mailto:vbellur at redhat.com>> wrote: >>> >>> Hi All, >>> >>> I am happy to announce that Gluster 3.7.0 is now generally available. 3.7.0 contains several >>> >>> [snip] >>> >>> Cheers, >>> Vijay >>> >>> [snip] >>> >>> What happened to packages for RHEL/Centos 5? I have the (probably >>> unusual--added gluster to existing servers) setup of running a replica >>> 3 cluster where two nodes run on Centos 6 and one is still on Centos >>> 5. This is a personal setup, and I have been using >>> http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-5/x86_64/repodata/repomod.xml >>> as my repo. It has worked fine for a while, but this time the two >>> Centos 6 nodes updated to 3.7, but the Centos 5 node got left behind >>> at 3.6.3. >> Packages for RHEL/CentOS-5 are not available yet. These will follow >> later. Thare are some changes needed to be able to build the packages on >> EL5. Because we are currently stabilizing our CI/regression tests, we do >> not merge any other changes. Until we provide packages in our >> repository, you could apply patch http://review.gluster.org/10803 >> yourself and build the EL5 version. I expect that we will do a release >> in 2-3 weeks which will have EL5 RPMs too. >> >> I have no idea about the problem below, it sounds like something the >> GlusterD developers could help with. >> >> Niels >> >>> Command 'gluster volume status' on the C5 machine makes everything >>> look fine: >>> >>> Status of volume: ISO2 >>> Gluster process Port Online Pid >>> ------------------------------------------------------------------------------ >>> Brick 10.x.x.2:/bricks/01/iso2 49162 Y 4679 >>> Brick 10.x.x.4:/bricks/01/iso2 49183 Y 6447 >>> Brick 10.x.x.9:/bricks/01/iso2 49169 Y 1985 >>> >>> But the same command on either of the C6 machines shows the C5 machine >>> (10.x.x.2) missing in action (though it does recognize that there are >>> NFS and heal daemons there): >>> >>> Status of volume: ISO2 >>> Gluster process TCP Port RDMA Port Online Pid >>> ------------------------------------------------------------------------------ >>> Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447 >>> Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985 >>> NFS Server on localhost 2049 0 Y 2279 >>> Self-heal Daemon on localhost N/A N/A Y 2754 >>> NFS Server on 10.41.65.2 2049 0 Y 4757 >>> Self-heal Daemon on 10.41.65.2 N/A N/A Y 4764 >>> NFS Server on 10.41.65.4 2049 0 Y 6543 >>> Self-heal Daemon on 10.41.65.4 N/A N/A Y 6551 >>> >>> So, is this just an oversight (I hope), or has support for C5 been dropped? >>> If support for C5 is gone, how do I downgrade my Centos6 machines back >>> to 3.6.x? (I know how to change the repo, but the actual sequence of >>> yum commands and gluster commands is unknown to me). > Could you attach the glusterd log file of 10.x.x.2 machineattached as etc-glusterfs-glusterd.vol.log.newer.2, starting from last machine reboot> and the node from where you triggered volume status.attached as etc-glusterfs-glusterd.vol.log.newer4 starting same time as .2 log> Could you also share gluster volume info output of all the nodes?I have several volumes, so I chose the one that shows up first on the listings: *from 10.41.65.2:* [root at office2 /var/log/glusterfs]$ gluster volume info Volume Name: ISO2 Type: Replicate Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.41.65.2:/bricks/01/iso2 Brick2: 10.41.65.4:/bricks/01/iso2 Brick3: 10.41.65.9:/bricks/01/iso2 [root at office2 /var/log/glusterfs]$ gluster volume status ISO2 Status of volume: ISO2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.41.65.2:/bricks/01/iso2 49162 Y 4463 Brick 10.41.65.4:/bricks/01/iso2 49183 Y 6447 Brick 10.41.65.9:/bricks/01/iso2 49169 Y 1985 NFS Server on localhost 2049 Y 4536 Self-heal Daemon on localhost N/A Y 4543 NFS Server on 10.41.65.9 2049 Y 2279 Self-heal Daemon on 10.41.65.9 N/A Y 2754 NFS Server on 10.41.65.4 2049 Y 6543 Self-heal Daemon on 10.41.65.4 N/A Y 6551 Task Status of Volume ISO2 ------------------------------------------------------------------------------ There are no active volume tasks [root at office2 ~]$ gluster peer status Number of Peers: 2 Hostname: 10.41.65.9 Uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 State: Peer in Cluster (Connected) Hostname: 10.41.65.4 Uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c State: Peer in Cluster (Connected) *from 10.41.65.4:* [root at office4b ~]# gluster volume info ISO2 Volume Name: ISO2 Type: Replicate Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.41.65.2:/bricks/01/iso2 Brick2: 10.41.65.4:/bricks/01/iso2 Brick3: 10.41.65.9:/bricks/01/iso2 [root at office4b ~]# gluster volume status ISO2 Status of volume: ISO2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447 Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985 NFS Server on localhost 2049 0 Y 6543 Self-heal Daemon on localhost N/A N/A Y 6551 NFS Server on 10.41.65.2 2049 0 Y 4536 Self-heal Daemon on 10.41.65.2 N/A N/A Y 4543 NFS Server on 10.41.65.9 2049 0 Y 2279 Self-heal Daemon on 10.41.65.9 N/A N/A Y 2754 Task Status of Volume ISO2 ------------------------------------------------------------------------------ There are no active volume tasks [root at office4b ~]# gluster peer status Number of Peers: 2 Hostname: 10.41.65.2 Uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 State: Peer in Cluster (Connected) Hostname: 10.41.65.9 Uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 State: Peer in Cluster (Connected) *from 10.41.65.9:* [root at office9 ~]$ gluster volume info ISO2 Volume Name: ISO2 Type: Replicate Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.41.65.2:/bricks/01/iso2 Brick2: 10.41.65.4:/bricks/01/iso2 Brick3: 10.41.65.9:/bricks/01/iso2 [root at office9 ~]$ gluster volume status ISO2 Status of volume: ISO2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.41.65.4:/bricks/01/iso2 49183 0 Y 6447 Brick 10.41.65.9:/bricks/01/iso2 49169 0 Y 1985 NFS Server on localhost 2049 0 Y 2279 Self-heal Daemon on localhost N/A N/A Y 2754 NFS Server on 10.41.65.2 2049 0 Y 4536 Self-heal Daemon on 10.41.65.2 N/A N/A Y 4543 NFS Server on 10.41.65.4 2049 0 Y 6543 Self-heal Daemon on 10.41.65.4 N/A N/A Y 6551 Task Status of Volume ISO2 ------------------------------------------------------------------------------ There are no active volume tasks [root at office9 ~]$ gluster peer status Number of Peers: 2 Hostname: 10.41.65.2 Uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 State: Peer in Cluster (Connected) Hostname: 10.41.65.4 Uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c State: Peer in Cluster (Connected)>>> Ted Miller >>> Elkhart, IN, USA >> >> Thanks for the information. As long as I know it is coming, I can improvise and hang on. >> >> I am assuming that the problem with the .2 machine not being seen is a result of running a cluster with a version split. >> >> Ted Miller >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >>-- *Ted Miller*, Design Engineer *SonSet Solutions* (formerly HCJB Global Technology Center) my desk +1 574.970.4272 receptionist +1 574.972.4252 http://sonsetsolutions.org /Technology for abundant life!/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150527/040d40f8/attachment.html> -------------- next part -------------- [2015-05-25 14:51:53.898279] W [socket.c:620:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (No data available) [2015-05-25 14:51:53.898403] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer cf2ae9c7-833e-4a73-a996-e72158011c69, in Peer in Cluster state, has disconnected from glusterd. [2015-05-25 14:51:53.898800] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISO2 not held [2015-05-25 14:51:53.899083] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISOold not held [2015-05-25 14:51:53.899356] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM-eng01 not held [2015-05-25 14:51:53.899647] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM1 not held [2015-05-25 14:51:53.899958] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM2 not held [2015-05-25 14:51:53.900230] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM3 not held [2015-05-25 14:51:53.900498] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VMold not held [2015-05-25 14:51:53.900767] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol export1 not held [2015-05-25 14:51:53.901034] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol home1 not held [2015-05-25 14:51:53.901304] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol host_share1 not held [2015-05-25 14:51:53.901591] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol pics not held [2015-05-25 14:51:53.901861] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol video01 not held [2015-05-25 14:52:03.883612] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600 [2015-05-25 14:52:04.100889] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:04.153955] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0 [2015-05-25 14:52:04.214125] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c [2015-05-25 14:52:04.214210] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:04.253289] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c [2015-05-25 14:52:05.876100] W [socket.c:620:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (Connection reset by peer) [2015-05-25 14:52:05.876432] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1d7)[0x315680e887] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x315680e98e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x315680ea31] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd2)[0x315680f152] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2015-05-25 14:52:04.163362 (xid=0xe) [2015-05-25 14:52:05.876479] E [glusterd-rpc-ops.c:619:__glusterd_friend_update_cbk] 0-management: RPC Error [2015-05-25 14:52:05.876515] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000 [2015-05-25 14:52:05.876706] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1d7)[0x315680e887] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x315680e98e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x315680ea31] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xd2)[0x315680f152] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: forced unwinding frame type(Gluster MGMT Handshake) op(MGMT-VERS(1)) called at 2015-05-25 14:52:04.166562 (xid=0xf) [2015-05-25 14:52:05.876751] E [glusterd-handshake.c:1680:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later [2015-05-25 14:52:05.877044] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISO2 not held [2015-05-25 14:52:05.877336] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol ISOold not held [2015-05-25 14:52:05.877610] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM-eng01 not held [2015-05-25 14:52:05.877879] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM1 not held [2015-05-25 14:52:05.878149] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM2 not held [2015-05-25 14:52:05.878418] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VM3 not held [2015-05-25 14:52:05.878686] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol VMold not held [2015-05-25 14:52:05.878955] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol export1 not held [2015-05-25 14:52:05.879235] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol home1 not held [2015-05-25 14:52:05.879538] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol host_share1 not held [2015-05-25 14:52:05.880094] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol pics not held [2015-05-25 14:52:05.880366] W [glusterd-locks.c:647:glusterd_mgmt_v3_unlock] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x3f9)[0x2ab1969e6389] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x273)[0x2ab196945283] (--> /usr/lib64/glusterfs/3.6.3/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x2ab196945e60] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x11c)[0x315680f19c] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)[0x315680b1e7] ))))) 0-management: Lock for vol video01 not held [2015-05-25 14:52:08.846394] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600 [2015-05-25 14:52:10.144090] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.156233] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0 [2015-05-25 14:52:10.161604] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c [2015-05-25 14:52:10.161697] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.161767] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.474292] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69, host: 10.41.65.9, port: 0 [2015-05-25 14:52:10.528558] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.528639] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.813342] I [glusterd-handler.c:2373:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bd3ca8b7-f2da-44ce-8739-c0db5e40158c [2015-05-25 14:52:10.813441] I [glusterd-handler.c:2416:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.997945] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:05.876801] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer cf2ae9c7-833e-4a73-a996-e72158011c69, in Peer in Cluster state, has disconnected from glusterd. -------------- next part -------------- [2015-05-25 14:51:53.902180] W [socket.c:642:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (No data available) [2015-05-25 14:51:53.960843] I [MSGID: 106004] [glusterd-handler.c:4809:__glusterd_peer_rpc_notify] 0-management: Peer <10.41.65.9> (<cf2ae9c7-833e-4a73-a996-e72158011c69>), in state <Peer in Cluster>, has disconnected from glusterd. [2015-05-25 14:51:53.963643] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISO2 not held [2015-05-25 14:51:53.963793] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISOold not held [2015-05-25 14:51:53.963927] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM-eng01 not held [2015-05-25 14:51:53.964057] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM1 not held [2015-05-25 14:51:53.964187] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM2 not held [2015-05-25 14:51:53.964316] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM3 not held [2015-05-25 14:51:53.964452] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VMold not held [2015-05-25 14:51:53.964587] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol export1 not held [2015-05-25 14:51:53.964716] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol home1 not held [2015-05-25 14:51:53.964845] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol host_share1 not held [2015-05-25 14:51:53.964974] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol pics not held [2015-05-25 14:51:53.965103] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol video01 not held [2015-05-25 14:52:03.879790] I [glusterd-handshake.c:1151:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600 [2015-05-25 14:52:04.097083] I [glusterd-handler.c:2422:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:04.161852] I [glusterd-handler.c:3580:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0 [2015-05-25 14:52:04.249180] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 [2015-05-25 14:52:04.249236] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:04.249279] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 [2015-05-25 14:52:05.871153] W [socket.c:642:__socket_rwv] 0-management: readv on 10.41.65.9:24007 failed (Connection reset by peer) [2015-05-25 14:52:05.871411] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f066ce7fa87] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f066ce7fb9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f066ce7fc6b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f066ce8022f] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2015-05-25 14:52:04.210183 (xid=0xa) [2015-05-25 14:52:05.871431] E [glusterd-rpc-ops.c:653:__glusterd_friend_update_cbk] 0-management: RPC Error [2015-05-25 14:52:05.871441] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000 [2015-05-25 14:52:05.871525] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f066ce7fa87] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f066ce7fb9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f066ce7fc6b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f066ce8022f] ))))) 0-management: forced unwinding frame type(Gluster MGMT Handshake) op(MGMT-VERS(1)) called at 2015-05-25 14:52:04.249148 (xid=0xb) [2015-05-25 14:52:05.871545] E [glusterd-handshake.c:1740:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later [2015-05-25 14:52:05.871564] I [MSGID: 106004] [glusterd-handler.c:4809:__glusterd_peer_rpc_notify] 0-management: Peer <10.41.65.9> (<cf2ae9c7-833e-4a73-a996-e72158011c69>), in state <Peer in Cluster>, has disconnected from glusterd. [2015-05-25 14:52:05.871701] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISO2 not held [2015-05-25 14:52:05.871832] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol ISOold not held [2015-05-25 14:52:05.871962] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM-eng01 not held [2015-05-25 14:52:05.872091] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM1 not held [2015-05-25 14:52:05.872248] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM2 not held [2015-05-25 14:52:05.872385] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VM3 not held [2015-05-25 14:52:05.872517] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol VMold not held [2015-05-25 14:52:05.872646] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol export1 not held [2015-05-25 14:52:05.872774] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol home1 not held [2015-05-25 14:52:05.872903] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol host_share1 not held [2015-05-25 14:52:05.873032] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol pics not held [2015-05-25 14:52:05.873167] W [glusterd-locks.c:648:glusterd_mgmt_v3_unlock] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f066d0aefb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x481)[0x7f06625e0881] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify+0x290)[0x7f066255d760] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_big_locked_notify+0x60)[0x7f0662542e70] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1a3)[0x7f066ce80273] ))))) 0-management: Lock for vol video01 not held [2015-05-25 14:52:08.842559] I [glusterd-handshake.c:1151:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600 [2015-05-25 14:52:09.710802] I [glusterd-rpc-ops.c:463:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69, host: 10.41.65.9, port: 0 [2015-05-25 14:52:10.155216] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 [2015-05-25 14:52:10.155267] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.737330] I [glusterd-handler.c:2422:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.766113] I [glusterd-handler.c:3580:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.41.65.9 (0), ret: 0 [2015-05-25 14:52:10.852943] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4a53ed8b-2b41-4a3c-acf7-2dabec431f57 [2015-05-25 14:52:10.852988] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.853013] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:10.853061] I [glusterd-handler.c:2585:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69 [2015-05-25 14:52:10.853079] I [glusterd-handler.c:2628:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2015-05-25 14:52:11.047970] I [glusterd-rpc-ops.c:668:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cf2ae9c7-833e-4a73-a996-e72158011c69