Kaamesh Kamalaaharan
2015-Mar-19 06:40 UTC
[Gluster-users] Gluster volume brick keeps going offline
Hi Atin, Thanks for the reply. Im not sure which logs are relevant so ill just attach them all in a gz file. I ran a sudo gluster volume start gfsvolume force at 2015-03-19 05:49 i hope this helps. Thank You Kindly, Kaamesh On Sun, Mar 15, 2015 at 11:41 PM, Atin Mukherjee <amukherj at redhat.com> wrote:> Could you attach the logs for the analysis? > > ~Atin > > On 03/13/2015 03:29 PM, Kaamesh Kamalaaharan wrote: > > Hi guys. Ive been using gluster for a while now and despite a few > hiccups, > > i find its a great system to use. One of my more persistent hiccups is an > > issue with one brick going offline. > > > > My setup is a 2 brick 2 node setup. my main brick is gfs1 which has not > > given me any problem. gfs2 however keeps going offline. Following > > http://www.gluster.org/pipermail/gluster-users/2014-June/017583.html > > temporarily fixed the error but the brick goes offline within the hour. > > > > This is what i get from my volume status command : > > > > sudo gluster volume status > >> > >> Status of volume: gfsvolume > >> Gluster process Port Online Pid > >> > >> > ------------------------------------------------------------------------------ > >> Brick gfs1:/export/sda/brick 49153 Y 9760 > >> Brick gfs2:/export/sda/brick N/A N 13461 > >> NFS Server on localhost 2049 Y 13473 > >> Self-heal Daemon on localhost N/A Y 13480 > >> NFS Server on gfs1 2049 Y 16166 > >> Self-heal Daemon on gfs1 N/A Y 16173 > >> > >> Task Status of Volume gfsvolume > >> > >> > ------------------------------------------------------------------------------ > >> There are no active volume tasks > >> > >> > > doing sudo gluster volume start gfsvolume force gives me this: > > > > sudo gluster volume status > >> > >> Status of volume: gfsvolume > >> Gluster process Port Online Pid > >> > >> > ------------------------------------------------------------------------------ > >> Brick gfs1:/export/sda/brick 49153 Y 9760 > >> Brick gfs2:/export/sda/brick 49153 Y 13461 > >> NFS Server on localhost 2049 Y 13473 > >> Self-heal Daemon on localhost N/A Y 13480 > >> NFS Server on gfs1 2049 Y 16166 > >> Self-heal Daemon on gfs1 N/A Y 16173 > >> > >> Task Status of Volume gfsvolume > >> > >> > ------------------------------------------------------------------------------ > >> There are no active volume tasks > >> > >> half an hour later and my brick goes down again. > > > >> > >> > >> This is my glustershd.log. I snipped it because the rest of the log is a > > repeat of the same error > > > > > >> > >> [2015-03-13 02:09:41.951556] I [glusterfsd.c:1959:main] > >> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.0 > >> (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p > >> /var/lib/glus > >> terd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log > -S > >> /var/run/deac2f873d0ac5b6c3e84b23c4790172.socket --xlator-option > >> *replicate*.node-uuid=adbb7505-3342-4c6d-be3d-75938633612c) > >> [2015-03-13 02:09:41.954173] I [socket.c:3561:socket_init] > >> 0-socket.glusterfsd: SSL support is NOT enabled > >> [2015-03-13 02:09:41.954236] I [socket.c:3576:socket_init] > >> 0-socket.glusterfsd: using system polling thread > >> [2015-03-13 02:09:41.954421] I [socket.c:3561:socket_init] 0-glusterfs: > >> SSL support is NOT enabled > >> [2015-03-13 02:09:41.954443] I [socket.c:3576:socket_init] 0-glusterfs: > >> using system polling thread > >> [2015-03-13 02:09:41.956731] I [graph.c:254:gf_add_cmdline_options] > >> 0-gfsvolume-replicate-0: adding option 'node-uuid' for volume > >> 'gfsvolume-replicate-0' with value > 'adbb7505-3342-4c6d-be3d-75938633612c' > >> [2015-03-13 02:09:41.960210] I [rpc-clnt.c:972:rpc_clnt_connection_init] > >> 0-gfsvolume-client-1: setting frame-timeout to 90 > >> [2015-03-13 02:09:41.960288] I [socket.c:3561:socket_init] > >> 0-gfsvolume-client-1: SSL support is NOT enabled > >> [2015-03-13 02:09:41.960301] I [socket.c:3576:socket_init] > >> 0-gfsvolume-client-1: using system polling thread > >> [2015-03-13 02:09:41.961095] I [rpc-clnt.c:972:rpc_clnt_connection_init] > >> 0-gfsvolume-client-0: setting frame-timeout to 90 > >> [2015-03-13 02:09:41.961134] I [socket.c:3561:socket_init] > >> 0-gfsvolume-client-0: SSL support is NOT enabled > >> [2015-03-13 02:09:41.961145] I [socket.c:3576:socket_init] > >> 0-gfsvolume-client-0: using system polling thread > >> [2015-03-13 02:09:41.961173] I [client.c:2273:notify] > >> 0-gfsvolume-client-0: parent translators are ready, attempting connect > on > >> transport > >> [2015-03-13 02:09:41.961412] I [client.c:2273:notify] > >> 0-gfsvolume-client-1: parent translators are ready, attempting connect > on > >> transport > >> Final graph: > >> > >> > +------------------------------------------------------------------------------+ > >> 1: volume gfsvolume-client-0 > >> 2: type protocol/client > >> 3: option remote-host gfs1 > >> 4: option remote-subvolume /export/sda/brick > >> 5: option transport-type socket > >> 6: option frame-timeout 90 > >> 7: option ping-timeout 30 > >> 8: end-volume > >> 9: > >> 10: volume gfsvolume-client-1 > >> 11: type protocol/client > >> 12: option remote-host gfs2 > >> 13: option remote-subvolume /export/sda/brick > >> 14: option transport-type socket > >> 15: option frame-timeout 90 > >> 16: option ping-timeout 30 > >> 17: end-volume > >> 18: > >> 19: volume gfsvolume-replicate-0 > >> 20: type cluster/replicate > >> 21: option node-uuid adbb7505-3342-4c6d-be3d-75938633612c > >> 22: option background-self-heal-count 0 > >> 23: option metadata-self-heal on > >> 24: option data-self-heal on > >> 25: option entry-self-heal on > >> 26: option self-heal-daemon on > >> 27: option data-self-heal-algorithm diff > >> 28: option quorum-type fixed > >> 29: option quorum-count 1 > >> 30: option iam-self-heal-daemon yes > >> 31: subvolumes gfsvolume-client-0 gfsvolume-client-1 > >> 32: end-volume > >> 33: > >> 34: volume glustershd > >> 35: type debug/io-stats > >> 36: subvolumes gfsvolume-replicate-0 > >> 37: end-volume > >> > >> > +------------------------------------------------------------------------------+ > >> [2015-03-13 02:09:41.961871] I [rpc-clnt.c:1685:rpc_clnt_reconfig] > >> 0-gfsvolume-client-1: changing port to 49153 (from 0) > >> [2015-03-13 02:09:41.962129] I > >> [client-handshake.c:1659:select_server_supported_programs] > >> 0-gfsvolume-client-1: Using Program GlusterFS 3.3, Num (1298437), > Version > >> (330) > >> [2015-03-13 02:09:41.962344] I > >> [client-handshake.c:1456:client_setvolume_cbk] 0-gfsvolume-client-1: > >> Connected to 172.20.20.22:49153, attached to remote volume > >> '/export/sda/brick'. > >> [2015-03-13 02:09:41.962363] I > >> [client-handshake.c:1468:client_setvolume_cbk] 0-gfsvolume-client-1: > Server > >> and Client lk-version numbers are not same, reopening the fds > >> [2015-03-13 02:09:41.962416] I [afr-common.c:3922:afr_notify] > >> 0-gfsvolume-replicate-0: Subvolume 'gfsvolume-client-1' came back up; > going > >> online. > >> [2015-03-13 02:09:41.962487] I > >> [client-handshake.c:450:client_set_lk_version_cbk] 0-gfsvolume-client-1: > >> Server lk version = 1 > >> [2015-03-13 02:09:41.963109] E > >> [afr-self-heald.c:1479:afr_find_child_position] 0-gfsvolume-replicate-0: > >> getxattr failed on gfsvolume-client-0 - (Transport endpoint is not > >> connected) > >> [2015-03-13 02:09:41.963502] I > >> [afr-self-heald.c:1687:afr_dir_exclusive_crawl] 0-gfsvolume-replicate-0: > >> Another crawl is in progress for gfsvolume-client-1 > >> [2015-03-13 02:09:41.967478] E > >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] > >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for > >> <gfid:66af7dc1-a2e6-4919-9ea1-ad75fe2d40b9>. > >> [2015-03-13 02:09:41.968550] E > >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] > >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for > >> <gfid:8a7cfa39-9a12-43cd-a9f3-9142b7403d0e>. > >> [2015-03-13 02:09:41.969663] E > >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] > >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for > >> <gfid:3762920e-9631-4a52-9a9f-4f04d09e8d84>. > >> [2015-03-13 02:09:41.974345] E > >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] > >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for > >> <gfid:66af7dc1-a2e6-4919-9ea1-ad75fe2d40b9>. > >> [2015-03-13 02:09:41.975657] E > >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] > >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for > >> <gfid:8a7cfa39-9a12-43cd-a9f3-9142b7403d0e>. > >> [2015-03-13 02:09:41.977020] E > >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] > >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for > >> <gfid:3762920e-9631-4a52-9a9f-4f04d09e8d84>. > >> [2015-03-13 02:09:44.307219] I [rpc-clnt.c:1685:rpc_clnt_reconfig] > >> 0-gfsvolume-client-0: changing port to 49153 (from 0) > >> [2015-03-13 02:09:44.307748] I > >> [client-handshake.c:1659:select_server_supported_programs] > >> 0-gfsvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), > Version > >> (330) > >> [2015-03-13 02:09:44.448377] I > >> [client-handshake.c:1456:client_setvolume_cbk] 0-gfsvolume-client-0: > >> Connected to 172.20.20.21:49153, attached to remote volume > >> '/export/sda/brick'. > >> [2015-03-13 02:09:44.448418] I > >> [client-handshake.c:1468:client_setvolume_cbk] 0-gfsvolume-client-0: > Server > >> and Client lk-version numbers are not same, reopening the fds > >> [2015-03-13 02:09:44.448713] I > >> [client-handshake.c:450:client_set_lk_version_cbk] 0-gfsvolume-client-0: > >> Server lk version = 1 > >> [2015-03-13 02:09:44.515112] I > >> [afr-self-heal-common.c:2859:afr_log_self_heal_completion_status] > >> 0-gfsvolume-replicate-0: foreground data self heal is successfully > >> completed, data self heal from gfsvolume-client-0 to sinks > >> gfsvolume-client-1, with 892928 bytes on gfsvolume-client-0, 892928 > bytes > >> on gfsvolume-client-1, data - Pending matrix: [ [ 0 155762 ] [ 0 0 ] ] > >> on <gfid:123536cc-c34b-43d7-b0c6-cf80eefa8322> > >> [2015-03-13 02:09:44.809988] I > >> [afr-self-heal-common.c:2859:afr_log_self_heal_completion_status] > >> 0-gfsvolume-replicate-0: foreground data self heal is successfully > >> completed, data self heal from gfsvolume-client-0 to sinks > >> gfsvolume-client-1, with 15998976 bytes on gfsvolume-client-0, 15998976 > >> bytes on gfsvolume-client-1, data - Pending matrix: [ [ 0 36506 ] [ 0 > 0 ] > >> ] on <gfid:b6dc0e74-31bf-469a-b629-ee51ab4cf729> > >> [2015-03-13 02:09:44.946050] W > >> [client-rpc-fops.c:574:client3_3_readlink_cbk] 0-gfsvolume-client-0: > remote > >> operation failed: Stale NFS file handle > >> [2015-03-13 02:09:44.946097] I > >> [afr-self-heal-entry.c:1538:afr_sh_entry_impunge_readlink_sink_cbk] > >> 0-gfsvolume-replicate-0: readlink of > >> <gfid:66af7dc1-a2e6-4919-9ea1-ad75fe2d40b9>/PB2_corrected.fastq on > >> gfsvolume-client-1 failed (Stale NFS file handle) > >> [2015-03-13 02:09:44.951370] I > >> [afr-self-heal-entry.c:2321:afr_sh_entry_fix] 0-gfsvolume-replicate-0: > >> <gfid:8a7cfa39-9a12-43cd-a9f3-9142b7403d0e>: Performing conservative > merge > >> [2015-03-13 02:09:45.149995] W > >> [client-rpc-fops.c:574:client3_3_readlink_cbk] 0-gfsvolume-client-0: > remote > >> operation failed: Stale NFS file handle > >> [2015-03-13 02:09:45.150036] I > >> [afr-self-heal-entry.c:1538:afr_sh_entry_impunge_readlink_sink_cbk] > >> 0-gfsvolume-replicate-0: readlink of > >> <gfid:8a7cfa39-9a12-43cd-a9f3-9142b7403d0e>/Rscript on > gfsvolume-client-1 > >> failed (Stale NFS file handle) > >> [2015-03-13 02:09:45.214253] W > >> [client-rpc-fops.c:574:client3_3_readlink_cbk] 0-gfsvolume-client-0: > remote > >> operation failed: Stale NFS file handle > >> [2015-03-13 02:09:45.214295] I > >> [afr-self-heal-entry.c:1538:afr_sh_entry_impunge_readlink_sink_cbk] > >> 0-gfsvolume-replicate-0: readlink of > >> <gfid:3762920e-9631-4a52-9a9f-4f04d09e8d84>/ananas_d_tmp on > >> gfsvolume-client-1 failed (Stale NFS file handle) > >> [2015-03-13 02:13:27.324856] W [socket.c:522:__socket_rwv] > >> 0-gfsvolume-client-1: readv on 172.20.20.22:49153 failed (No data > >> available) > >> [2015-03-13 02:13:27.324961] I [client.c:2208:client_rpc_notify] > >> 0-gfsvolume-client-1: disconnected from 172.20.20.22:49153. Client > >> process will keep trying to connect to glusterd until brick's port is > >> available > >> [2015-03-13 02:13:37.981531] I [rpc-clnt.c:1685:rpc_clnt_reconfig] > >> 0-gfsvolume-client-1: changing port to 49153 (from 0) > >> [2015-03-13 02:13:37.981781] E [socket.c:2161:socket_connect_finish] > >> 0-gfsvolume-client-1: connection to 172.20.20.22:49153 failed > (Connection > >> refused) > >> [2015-03-13 02:13:41.982125] I [rpc-clnt.c:1685:rpc_clnt_reconfig] > >> 0-gfsvolume-client-1: changing port to 49153 (from 0) > >> [2015-03-13 02:13:41.982353] E [socket.c:2161:socket_connect_finish] > >> 0-gfsvolume-client-1: connection to 172.20.20.22:49153 failed > (Connection > >> refused) > >> [2015-03-13 02:13:45.982693] I [rpc-clnt.c:1685:rpc_clnt_reconfig] > >> 0-gfsvolume-client-1: changing port to 49153 (from 0) > >> [2015-03-13 02:13:45.982926] E [socket.c:2161:socket_connect_finish] > >> 0-gfsvolume-client-1: connection to 172.20.20.22:49153 failed > (Connection > >> refused) > >> [2015-03-13 02:13:49.983309] I [rpc-clnt.c:1685:rpc_clnt_reconfig] > >> 0-gfsvolume-client-1: changing port to 49153 (from 0) > >> > >> > > > > Any help would be greatly appreciated. > > Thank You Kindly, > > Kaamesh > > > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150319/fcf13473/attachment.html>
Kaamesh Kamalaaharan
2015-Mar-19 07:28 UTC
[Gluster-users] Gluster volume brick keeps going offline
Sorry, forgot to include the attachment Thank You Kindly, Kaamesh Bioinformatician Novocraft Technologies Sdn Bhd C-23A-05, 3 Two Square, Section 19, 46300 Petaling Jaya Selangor Darul Ehsan Malaysia Mobile: +60176562635 Ph: +60379600541 Fax: +60379600540 On Thu, Mar 19, 2015 at 2:40 PM, Kaamesh Kamalaaharan <kaamesh at novocraft.com> wrote:> Hi Atin, Thanks for the reply. Im not sure which logs are relevant so ill > just attach them all in a gz file. > > I ran a sudo gluster volume start gfsvolume force at 2015-03-19 05:49 > i hope this helps. > > Thank You Kindly, > Kaamesh > > On Sun, Mar 15, 2015 at 11:41 PM, Atin Mukherjee <amukherj at redhat.com> > wrote: > >> Could you attach the logs for the analysis? >> >> ~Atin >> >> On 03/13/2015 03:29 PM, Kaamesh Kamalaaharan wrote: >> > Hi guys. Ive been using gluster for a while now and despite a few >> hiccups, >> > i find its a great system to use. One of my more persistent hiccups is >> an >> > issue with one brick going offline. >> > >> > My setup is a 2 brick 2 node setup. my main brick is gfs1 which has not >> > given me any problem. gfs2 however keeps going offline. Following >> > http://www.gluster.org/pipermail/gluster-users/2014-June/017583.html >> > temporarily fixed the error but the brick goes offline within the hour. >> > >> > This is what i get from my volume status command : >> > >> > sudo gluster volume status >> >> >> >> Status of volume: gfsvolume >> >> Gluster process Port Online Pid >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Brick gfs1:/export/sda/brick 49153 Y 9760 >> >> Brick gfs2:/export/sda/brick N/A N 13461 >> >> NFS Server on localhost 2049 Y 13473 >> >> Self-heal Daemon on localhost N/A Y 13480 >> >> NFS Server on gfs1 2049 Y 16166 >> >> Self-heal Daemon on gfs1 N/A Y 16173 >> >> >> >> Task Status of Volume gfsvolume >> >> >> >> >> ------------------------------------------------------------------------------ >> >> There are no active volume tasks >> >> >> >> >> > doing sudo gluster volume start gfsvolume force gives me this: >> > >> > sudo gluster volume status >> >> >> >> Status of volume: gfsvolume >> >> Gluster process Port Online Pid >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Brick gfs1:/export/sda/brick 49153 Y 9760 >> >> Brick gfs2:/export/sda/brick 49153 Y 13461 >> >> NFS Server on localhost 2049 Y 13473 >> >> Self-heal Daemon on localhost N/A Y 13480 >> >> NFS Server on gfs1 2049 Y 16166 >> >> Self-heal Daemon on gfs1 N/A Y 16173 >> >> >> >> Task Status of Volume gfsvolume >> >> >> >> >> ------------------------------------------------------------------------------ >> >> There are no active volume tasks >> >> >> >> half an hour later and my brick goes down again. >> > >> >> >> >> >> >> This is my glustershd.log. I snipped it because the rest of the log is >> a >> > repeat of the same error >> > >> > >> >> >> >> [2015-03-13 02:09:41.951556] I [glusterfsd.c:1959:main] >> >> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version >> 3.5.0 >> >> (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p >> >> /var/lib/glus >> >> terd/glustershd/run/glustershd.pid -l >> /var/log/glusterfs/glustershd.log -S >> >> /var/run/deac2f873d0ac5b6c3e84b23c4790172.socket --xlator-option >> >> *replicate*.node-uuid=adbb7505-3342-4c6d-be3d-75938633612c) >> >> [2015-03-13 02:09:41.954173] I [socket.c:3561:socket_init] >> >> 0-socket.glusterfsd: SSL support is NOT enabled >> >> [2015-03-13 02:09:41.954236] I [socket.c:3576:socket_init] >> >> 0-socket.glusterfsd: using system polling thread >> >> [2015-03-13 02:09:41.954421] I [socket.c:3561:socket_init] 0-glusterfs: >> >> SSL support is NOT enabled >> >> [2015-03-13 02:09:41.954443] I [socket.c:3576:socket_init] 0-glusterfs: >> >> using system polling thread >> >> [2015-03-13 02:09:41.956731] I [graph.c:254:gf_add_cmdline_options] >> >> 0-gfsvolume-replicate-0: adding option 'node-uuid' for volume >> >> 'gfsvolume-replicate-0' with value >> 'adbb7505-3342-4c6d-be3d-75938633612c' >> >> [2015-03-13 02:09:41.960210] I >> [rpc-clnt.c:972:rpc_clnt_connection_init] >> >> 0-gfsvolume-client-1: setting frame-timeout to 90 >> >> [2015-03-13 02:09:41.960288] I [socket.c:3561:socket_init] >> >> 0-gfsvolume-client-1: SSL support is NOT enabled >> >> [2015-03-13 02:09:41.960301] I [socket.c:3576:socket_init] >> >> 0-gfsvolume-client-1: using system polling thread >> >> [2015-03-13 02:09:41.961095] I >> [rpc-clnt.c:972:rpc_clnt_connection_init] >> >> 0-gfsvolume-client-0: setting frame-timeout to 90 >> >> [2015-03-13 02:09:41.961134] I [socket.c:3561:socket_init] >> >> 0-gfsvolume-client-0: SSL support is NOT enabled >> >> [2015-03-13 02:09:41.961145] I [socket.c:3576:socket_init] >> >> 0-gfsvolume-client-0: using system polling thread >> >> [2015-03-13 02:09:41.961173] I [client.c:2273:notify] >> >> 0-gfsvolume-client-0: parent translators are ready, attempting connect >> on >> >> transport >> >> [2015-03-13 02:09:41.961412] I [client.c:2273:notify] >> >> 0-gfsvolume-client-1: parent translators are ready, attempting connect >> on >> >> transport >> >> Final graph: >> >> >> >> >> +------------------------------------------------------------------------------+ >> >> 1: volume gfsvolume-client-0 >> >> 2: type protocol/client >> >> 3: option remote-host gfs1 >> >> 4: option remote-subvolume /export/sda/brick >> >> 5: option transport-type socket >> >> 6: option frame-timeout 90 >> >> 7: option ping-timeout 30 >> >> 8: end-volume >> >> 9: >> >> 10: volume gfsvolume-client-1 >> >> 11: type protocol/client >> >> 12: option remote-host gfs2 >> >> 13: option remote-subvolume /export/sda/brick >> >> 14: option transport-type socket >> >> 15: option frame-timeout 90 >> >> 16: option ping-timeout 30 >> >> 17: end-volume >> >> 18: >> >> 19: volume gfsvolume-replicate-0 >> >> 20: type cluster/replicate >> >> 21: option node-uuid adbb7505-3342-4c6d-be3d-75938633612c >> >> 22: option background-self-heal-count 0 >> >> 23: option metadata-self-heal on >> >> 24: option data-self-heal on >> >> 25: option entry-self-heal on >> >> 26: option self-heal-daemon on >> >> 27: option data-self-heal-algorithm diff >> >> 28: option quorum-type fixed >> >> 29: option quorum-count 1 >> >> 30: option iam-self-heal-daemon yes >> >> 31: subvolumes gfsvolume-client-0 gfsvolume-client-1 >> >> 32: end-volume >> >> 33: >> >> 34: volume glustershd >> >> 35: type debug/io-stats >> >> 36: subvolumes gfsvolume-replicate-0 >> >> 37: end-volume >> >> >> >> >> +------------------------------------------------------------------------------+ >> >> [2015-03-13 02:09:41.961871] I [rpc-clnt.c:1685:rpc_clnt_reconfig] >> >> 0-gfsvolume-client-1: changing port to 49153 (from 0) >> >> [2015-03-13 02:09:41.962129] I >> >> [client-handshake.c:1659:select_server_supported_programs] >> >> 0-gfsvolume-client-1: Using Program GlusterFS 3.3, Num (1298437), >> Version >> >> (330) >> >> [2015-03-13 02:09:41.962344] I >> >> [client-handshake.c:1456:client_setvolume_cbk] 0-gfsvolume-client-1: >> >> Connected to 172.20.20.22:49153, attached to remote volume >> >> '/export/sda/brick'. >> >> [2015-03-13 02:09:41.962363] I >> >> [client-handshake.c:1468:client_setvolume_cbk] 0-gfsvolume-client-1: >> Server >> >> and Client lk-version numbers are not same, reopening the fds >> >> [2015-03-13 02:09:41.962416] I [afr-common.c:3922:afr_notify] >> >> 0-gfsvolume-replicate-0: Subvolume 'gfsvolume-client-1' came back up; >> going >> >> online. >> >> [2015-03-13 02:09:41.962487] I >> >> [client-handshake.c:450:client_set_lk_version_cbk] >> 0-gfsvolume-client-1: >> >> Server lk version = 1 >> >> [2015-03-13 02:09:41.963109] E >> >> [afr-self-heald.c:1479:afr_find_child_position] >> 0-gfsvolume-replicate-0: >> >> getxattr failed on gfsvolume-client-0 - (Transport endpoint is not >> >> connected) >> >> [2015-03-13 02:09:41.963502] I >> >> [afr-self-heald.c:1687:afr_dir_exclusive_crawl] >> 0-gfsvolume-replicate-0: >> >> Another crawl is in progress for gfsvolume-client-1 >> >> [2015-03-13 02:09:41.967478] E >> >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] >> >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for >> >> <gfid:66af7dc1-a2e6-4919-9ea1-ad75fe2d40b9>. >> >> [2015-03-13 02:09:41.968550] E >> >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] >> >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for >> >> <gfid:8a7cfa39-9a12-43cd-a9f3-9142b7403d0e>. >> >> [2015-03-13 02:09:41.969663] E >> >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] >> >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for >> >> <gfid:3762920e-9631-4a52-9a9f-4f04d09e8d84>. >> >> [2015-03-13 02:09:41.974345] E >> >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] >> >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for >> >> <gfid:66af7dc1-a2e6-4919-9ea1-ad75fe2d40b9>. >> >> [2015-03-13 02:09:41.975657] E >> >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] >> >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for >> >> <gfid:8a7cfa39-9a12-43cd-a9f3-9142b7403d0e>. >> >> [2015-03-13 02:09:41.977020] E >> >> [afr-self-heal-entry.c:2364:afr_sh_post_nonblocking_entry_cbk] >> >> 0-gfsvolume-replicate-0: Non Blocking entrylks failed for >> >> <gfid:3762920e-9631-4a52-9a9f-4f04d09e8d84>. >> >> [2015-03-13 02:09:44.307219] I [rpc-clnt.c:1685:rpc_clnt_reconfig] >> >> 0-gfsvolume-client-0: changing port to 49153 (from 0) >> >> [2015-03-13 02:09:44.307748] I >> >> [client-handshake.c:1659:select_server_supported_programs] >> >> 0-gfsvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), >> Version >> >> (330) >> >> [2015-03-13 02:09:44.448377] I >> >> [client-handshake.c:1456:client_setvolume_cbk] 0-gfsvolume-client-0: >> >> Connected to 172.20.20.21:49153, attached to remote volume >> >> '/export/sda/brick'. >> >> [2015-03-13 02:09:44.448418] I >> >> [client-handshake.c:1468:client_setvolume_cbk] 0-gfsvolume-client-0: >> Server >> >> and Client lk-version numbers are not same, reopening the fds >> >> [2015-03-13 02:09:44.448713] I >> >> [client-handshake.c:450:client_set_lk_version_cbk] >> 0-gfsvolume-client-0: >> >> Server lk version = 1 >> >> [2015-03-13 02:09:44.515112] I >> >> [afr-self-heal-common.c:2859:afr_log_self_heal_completion_status] >> >> 0-gfsvolume-replicate-0: foreground data self heal is successfully >> >> completed, data self heal from gfsvolume-client-0 to sinks >> >> gfsvolume-client-1, with 892928 bytes on gfsvolume-client-0, 892928 >> bytes >> >> on gfsvolume-client-1, data - Pending matrix: [ [ 0 155762 ] [ 0 0 ] >> ] >> >> on <gfid:123536cc-c34b-43d7-b0c6-cf80eefa8322> >> >> [2015-03-13 02:09:44.809988] I >> >> [afr-self-heal-common.c:2859:afr_log_self_heal_completion_status] >> >> 0-gfsvolume-replicate-0: foreground data self heal is successfully >> >> completed, data self heal from gfsvolume-client-0 to sinks >> >> gfsvolume-client-1, with 15998976 bytes on gfsvolume-client-0, >> 15998976 >> >> bytes on gfsvolume-client-1, data - Pending matrix: [ [ 0 36506 ] [ >> 0 0 ] >> >> ] on <gfid:b6dc0e74-31bf-469a-b629-ee51ab4cf729> >> >> [2015-03-13 02:09:44.946050] W >> >> [client-rpc-fops.c:574:client3_3_readlink_cbk] 0-gfsvolume-client-0: >> remote >> >> operation failed: Stale NFS file handle >> >> [2015-03-13 02:09:44.946097] I >> >> [afr-self-heal-entry.c:1538:afr_sh_entry_impunge_readlink_sink_cbk] >> >> 0-gfsvolume-replicate-0: readlink of >> >> <gfid:66af7dc1-a2e6-4919-9ea1-ad75fe2d40b9>/PB2_corrected.fastq on >> >> gfsvolume-client-1 failed (Stale NFS file handle) >> >> [2015-03-13 02:09:44.951370] I >> >> [afr-self-heal-entry.c:2321:afr_sh_entry_fix] 0-gfsvolume-replicate-0: >> >> <gfid:8a7cfa39-9a12-43cd-a9f3-9142b7403d0e>: Performing conservative >> merge >> >> [2015-03-13 02:09:45.149995] W >> >> [client-rpc-fops.c:574:client3_3_readlink_cbk] 0-gfsvolume-client-0: >> remote >> >> operation failed: Stale NFS file handle >> >> [2015-03-13 02:09:45.150036] I >> >> [afr-self-heal-entry.c:1538:afr_sh_entry_impunge_readlink_sink_cbk] >> >> 0-gfsvolume-replicate-0: readlink of >> >> <gfid:8a7cfa39-9a12-43cd-a9f3-9142b7403d0e>/Rscript on >> gfsvolume-client-1 >> >> failed (Stale NFS file handle) >> >> [2015-03-13 02:09:45.214253] W >> >> [client-rpc-fops.c:574:client3_3_readlink_cbk] 0-gfsvolume-client-0: >> remote >> >> operation failed: Stale NFS file handle >> >> [2015-03-13 02:09:45.214295] I >> >> [afr-self-heal-entry.c:1538:afr_sh_entry_impunge_readlink_sink_cbk] >> >> 0-gfsvolume-replicate-0: readlink of >> >> <gfid:3762920e-9631-4a52-9a9f-4f04d09e8d84>/ananas_d_tmp on >> >> gfsvolume-client-1 failed (Stale NFS file handle) >> >> [2015-03-13 02:13:27.324856] W [socket.c:522:__socket_rwv] >> >> 0-gfsvolume-client-1: readv on 172.20.20.22:49153 failed (No data >> >> available) >> >> [2015-03-13 02:13:27.324961] I [client.c:2208:client_rpc_notify] >> >> 0-gfsvolume-client-1: disconnected from 172.20.20.22:49153. Client >> >> process will keep trying to connect to glusterd until brick's port is >> >> available >> >> [2015-03-13 02:13:37.981531] I [rpc-clnt.c:1685:rpc_clnt_reconfig] >> >> 0-gfsvolume-client-1: changing port to 49153 (from 0) >> >> [2015-03-13 02:13:37.981781] E [socket.c:2161:socket_connect_finish] >> >> 0-gfsvolume-client-1: connection to 172.20.20.22:49153 failed >> (Connection >> >> refused) >> >> [2015-03-13 02:13:41.982125] I [rpc-clnt.c:1685:rpc_clnt_reconfig] >> >> 0-gfsvolume-client-1: changing port to 49153 (from 0) >> >> [2015-03-13 02:13:41.982353] E [socket.c:2161:socket_connect_finish] >> >> 0-gfsvolume-client-1: connection to 172.20.20.22:49153 failed >> (Connection >> >> refused) >> >> [2015-03-13 02:13:45.982693] I [rpc-clnt.c:1685:rpc_clnt_reconfig] >> >> 0-gfsvolume-client-1: changing port to 49153 (from 0) >> >> [2015-03-13 02:13:45.982926] E [socket.c:2161:socket_connect_finish] >> >> 0-gfsvolume-client-1: connection to 172.20.20.22:49153 failed >> (Connection >> >> refused) >> >> [2015-03-13 02:13:49.983309] I [rpc-clnt.c:1685:rpc_clnt_reconfig] >> >> 0-gfsvolume-client-1: changing port to 49153 (from 0) >> >> >> >> >> > >> > Any help would be greatly appreciated. >> > Thank You Kindly, >> > Kaamesh >> > >> > >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> > >> >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150319/5aa8e709/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: gluster_logs.tgz Type: application/x-gzip Size: 4273876 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150319/5aa8e709/attachment-0001.tgz>