Qaiser Shah (Banking Division)
2014-Jun-19 10:05 UTC
[Gluster-users] glusterd 3.2.7 crash on start
Hi I have a problem with glusterd 3.2.7. I have two nodes each configured with one brick in a replicate configuration. Both have been running for 9 months. One node recently started to crash with kernel core dump. The issue was a corrupt ext4 filesystem on the glusterfs brick. I unmounted the volume and fixed the filesystem. Now I have problems starting glusterd: [2014-06-19 10:20:33.526714] I [glusterfsd.c:1493:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3. 2.7 [2014-06-19 10:20:33.538144] I [glusterd.c:550:init] 0-management: Using /var/lib/glusterd as working directory [2014-06-19 10:20:33.539148] E [rpc-transport.c:677:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.2.7/rpc-tra nsport/rdma.so: cannot open shared object file: No such file or directory [2014-06-19 10:20:33.539175] E [rpc-transport.c:681:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transpor t-type 'rdma' is not valid or not found on this machine [2014-06-19 10:20:33.539183] W [rpcsvc.c:1288:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the t ransport failed [2014-06-19 10:20:33.539263] I [glusterd.c:88:glusterd_uuid_init] 0-glusterd: retrieved UUID: 8b6e691c-e4c1-4107-944f-fee9c fb55269 [2014-06-19 10:20:34.854729] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-06-19 10:20:34.854778] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-06-19 10:20:35.104082] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0 [2014-06-19 10:20:35.107779] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not know n) [2014-06-19 10:20:35.107843] E [name.c:253:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host all pending frames: patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-06-19 10:20:35 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.2.7 /lib64/libc.so.6(+0x32920)[0x7f10fe1ff920] /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_friend_sm+0x25)[0x7f10fc761055] /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_peer_rpc_notify+0x1d6)[0x7f10fc749026] /usr/lib64/libgfrpc.so.0(rpc_clnt_start+0x17)[0x7f10feba38b7] /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_rpc_create+0x108)[0x7f10fc749f18] /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_friend_add+0x33c)[0x7f10fc74a30c] /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_store_retrieve_peers+0x4d6)[0x7f10fc7932f6] /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_restore+0x78)[0x7f10fc794cb8] /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(init+0xbec)[0x7f10fc74884c] /usr/lib64/libglusterfs.so.0(xlator_init+0x52)[0x7f10fedc4af2] /usr/lib64/libglusterfs.so.0(glusterfs_graph_init+0x31)[0x7f10fedf57a1] /usr/lib64/libglusterfs.so.0(glusterfs_graph_activate+0x88)[0x7f10fedf5938] /usr/sbin/glusterd(glusterfs_process_volfp+0xd5)[0x404b35] /usr/sbin/glusterd(glusterfs_volumes_init+0x145)[0x404d45] /usr/sbin/glusterd(main+0x295)[0x405e85] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f10fe1ebcdd] /usr/sbin/glusterd[0x4036c9] Any pointers on solving this would be helpful. -------------------------------------------------- PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS E-MAIL Close Brothers Limited (Company No. 195626) and Close Brothers Group plc (Company No. 520241) are registered in England and Wales, and their registered office is 10 Crown Place, London EC2A 4FT. Close Brothers Limited is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Firm reference number 124750. The information in this e-mail and any files transmitted with it is confidential and may be legally privileged. It is intended solely for the addressee and others authorised to receive it. If you are not the intended recipient, any disclosure, copying, distribution or action taken in reliance on its contents isprohibited and may be unlawful. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140619/3217ad74/attachment.html>
Hi Qaiser, This is fixed in 3.3. Tracked as: https://bugzilla.redhat.com/show_bug.cgi?id=820673 Thanks, Santosh On 06/19/2014 03:35 PM, Qaiser Shah (Banking Division) wrote:> > Hi > > I have a problem with glusterd 3.2.7. I have two nodes each > configured with one brick in a replicate configuration. Both have been > running for 9 months. One node recently started to crash with kernel > core dump. The issue was a corrupt ext4 filesystem on the glusterfs > brick. I unmounted the volume and fixed the filesystem. Now I have > problems starting glusterd: > > [2014-06-19 10:20:33.526714] I [glusterfsd.c:1493:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3. > > 2.7 > > [2014-06-19 10:20:33.538144] I [glusterd.c:550:init] 0-management: > Using /var/lib/glusterd as working directory > > [2014-06-19 10:20:33.539148] E > [rpc-transport.c:677:rpc_transport_load] 0-rpc-transport: > /usr/lib64/glusterfs/3.2.7/rpc-tra > > nsport/rdma.so: cannot open shared object file: No such file or directory > > [2014-06-19 10:20:33.539175] E > [rpc-transport.c:681:rpc_transport_load] 0-rpc-transport: volume > 'rdma.management': transpor > > t-type 'rdma' is not valid or not found on this machine > > [2014-06-19 10:20:33.539183] W [rpcsvc.c:1288:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the t > > ransport failed > > [2014-06-19 10:20:33.539263] I [glusterd.c:88:glusterd_uuid_init] > 0-glusterd: retrieved UUID: 8b6e691c-e4c1-4107-944f-fee9c > > fb55269 > > [2014-06-19 10:20:34.854729] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-0 > > [2014-06-19 10:20:34.854778] E > [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown > key: brick-1 > > [2014-06-19 10:20:35.104082] I > [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect > returned 0 > > [2014-06-19 10:20:35.107779] E [common-utils.c:125:gf_resolve_ip6] > 0-resolver: getaddrinfo failed (Name or service not know > > n) > > [2014-06-19 10:20:35.107843] E > [name.c:253:af_inet_client_get_remote_sockaddr] 0-management: DNS > resolution failed on host > > all > > pending frames: > > patchset: git://git.gluster.com/glusterfs.git > > signal received: 11 > > time of crash: 2014-06-19 10:20:35 > > configuration details: > > argp 1 > > backtrace 1 > > dlfcn 1 > > fdatasync 1 > > libpthread 1 > > llistxattr 1 > > setfsid 1 > > spinlock 1 > > epoll.h 1 > > xattr.h 1 > > st_atim.tv_nsec 1 > > package-string: glusterfs 3.2.7 > > /lib64/libc.so.6(+0x32920)[0x7f10fe1ff920] > > /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_friend_sm+0x25)[0x7f10fc761055] > > /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_peer_rpc_notify+0x1d6)[0x7f10fc749026] > > /usr/lib64/libgfrpc.so.0(rpc_clnt_start+0x17)[0x7f10feba38b7] > > /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_rpc_create+0x108)[0x7f10fc749f18] > > /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_friend_add+0x33c)[0x7f10fc74a30c] > > /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_store_retrieve_peers+0x4d6)[0x7f10fc7932f6] > > /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_restore+0x78)[0x7f10fc794cb8] > > /usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(init+0xbec)[0x7f10fc74884c] > > /usr/lib64/libglusterfs.so.0(xlator_init+0x52)[0x7f10fedc4af2] > > /usr/lib64/libglusterfs.so.0(glusterfs_graph_init+0x31)[0x7f10fedf57a1] > > /usr/lib64/libglusterfs.so.0(glusterfs_graph_activate+0x88)[0x7f10fedf5938] > > /usr/sbin/glusterd(glusterfs_process_volfp+0xd5)[0x404b35] > > /usr/sbin/glusterd(glusterfs_volumes_init+0x145)[0x404d45] > > /usr/sbin/glusterd(main+0x295)[0x405e85] > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f10fe1ebcdd] > > /usr/sbin/glusterd[0x4036c9] > > Any pointers on solving this would be helpful. > > > > ------------------------------------------------------------------------ > PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS E-MAIL. > Close Brothers Limited (Company No. 195626) and Close Brothers Group > plc (Company No. 520241) are registered in England and Wales, and > their registered office is 10 Crown Place, London EC2A 4FT. > Close Brothers Limited is authorised by the Prudential Regulation > Authority and regulated by the Financial Conduct Authority and > Prudential Regulation Authority. Firm reference number 124750. > ------------------------------------------------------------------------ > The information in this e-mail and any files transmitted with it is > confidential and may be legally privileged. It is intended solely for > the addressee and others authorised to receive it. If you are not the > intended recipient, any disclosure, copying, distribution or action > taken in reliance on its contents is prohibited and may be unlawful. > ------------------------------------------------------------------------ > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140620/9e184e76/attachment.html>