Frank Yu
2019-Aug-12 12:37 UTC
[Gluster-users] 【replace-brick failed but make there’re two same client-id of the gluster cluster, which lead can’t mount the gluster anymore】
Hi guys, I met a terrible situations need all your helps. I have a production cluster running well at first. the version of gluster is 3.12.15-1.el7.x86_64, the cluster has 12 nodes, 12 brick(disk) per nodes, there is one distributed-replicate volume, with 144 bricks(48 * 3). then there is a node crushed(the node named nodeA), and all it?s disk can?t be used anymore, but since the os of nodes run on kvm machine, so it came back with 12 new disks. I try to replace the first brick of nodeA with cmd ?gluster volume replace-brick VOLUMENAME nodeA:/mnt/data-1/data nodeA:/mnt/data-1/data01 commit force?, after some times, it failed with error ?Error : Request timed out?. here came the problem, both ?nodeA:/mnt/data-1/data? and ?nodeA:/mnt/data-1/data01? show in the output of cmd ?gluster volume info? When I try to mount gluster to client with fuse, it report error like below: [2019-08-12 12:27:42.395440] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.15 (args: /usr/sbin/glusterfs --volfile-server=xxxxx --volfile-id=/training-data-ali /mnt/glusterfs) [2019-08-12 12:27:42.400015] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2019-08-12 12:27:42.404994] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 *[2019-08-12 12:27:42.415971] E [MSGID: 101179] [graph.y:153:new_volume] 0-parser: Line 1381: volume ?VOLUME-NAME-client-74' defined again* [2019-08-12 12:27:42.416124] E [MSGID: 100026] [glusterfsd.c:2358:glusterfs_process_volfp] 0-: failed to construct the graph [2019-08-12 12:27:42.416376] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x55898e35e092] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x150) [0x55898e357da0] -->/lib64/libglusterfs.so.0(glusterfs_graph_destroy+0x84) [0x7f95f7318754] ) 0-graph: invalid argument: graph [Invalid argument] [2019-08-12 12:27:42.416425] W [glusterfsd.c:1375:cleanup_and_exit] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x55898e35e092] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x163) [0x55898e357db3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55898e35732b] ) 0-: received signum (-1), shutting down [2019-08-12 12:27:42.416455] I [fuse-bridge.c:5852:fini] 0-fuse: Unmounting '/mnt/glusterfs'. [2019-08-12 12:27:42.429655] I [fuse-bridge.c:5857:fini] 0-fuse: Closing fuse connection to '/mnt/glusterfs-aliyun'. [2019-08-12 12:27:42.429759] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f95f6140e25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55898e3574b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55898e35732b] ) 0-: received signum (15), shutting down So, how can I solve error *?Line 1381: volume ?VOLUME-NAME-client-74' defined again? * -- Regards Frank Yu -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190812/bb176b28/attachment.html>