Here is the tail of the failed rebalance log, any clue? [2015-12-13 21:30:31.527493] I [dht-rebalance.c:2340:gf_defrag_process_dir] 0-FastVol-dht: Migration operation on dir /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/5F/1MsH5--BcoGRAJPI took 20.95 secs [2015-12-13 21:30:31.528704] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-13 21:30:31.543901] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-13 21:31:37.210496] I [MSGID: 109081] [dht-common.c:3780:dht_setxattr] 0-FastVol-dht: fixing the layout of /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q [2015-12-13 21:31:37.722825] I [MSGID: 109045] [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 0 (FastVol-client-0): 1032124 chunks [2015-12-13 21:31:37.722837] I [MSGID: 109045] [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 1 (FastVol-client-1): 1032124 chunks [2015-12-13 21:33:03.955539] I [MSGID: 109064] [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol: FastVol-client-0; inode layout - 0 - 2146817919 - 1; disk layout - 2146817920 - 4294967295 - 1 [2015-12-13 21:33:04.069859] I [MSGID: 109018] [dht-common.c:806:dht_revalidate_cbk] 0-FastVol-dht: Mismatching layouts for /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q, gfid = f38c4ed2-a26a-4d83-adfd-6b0331831738 [2015-12-13 21:33:04.118800] I [MSGID: 109064] [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol: FastVol-client-1; inode layout - 2146817920 - 4294967295 - 1; disk layout - 0 - 2146817919 - 1 [2015-12-13 21:33:19.979507] I [MSGID: 109022] [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration of /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished from subvolume FastVol-client-0 to FastVol-client-1 [2015-12-13 21:33:19.979459] I [MSGID: 109022] [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration of /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint from subvolume FastVol-client-0 to FastVol-client-1 [2015-12-13 21:33:25.543941] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_origin.jpg: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-13 21:33:25.962547] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_small.jpg: attempting to move from FastVol-client-0 to FastVol-client-1 Cloudor From: Sakshi Bansal Date: 2015-12-12 13:02 To: ?? CC: gluster-users Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? In the rebalance log file you can check the file/directory for which the rebalance has failed. It can mention what was the fop for whihc the failure happened. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151214/cf2c6b09/attachment.html>
Nithya Balachandran
2015-Dec-14 10:57 UTC
[Gluster-users] How to diagnose volume rebalance failure?
Hi, Can you send us the rebalance log? Regards, Nithya ----- Original Message -----> From: "PuYun" <cloudor at 126.com> > To: "gluster-users" <gluster-users at gluster.org> > Sent: Monday, December 14, 2015 11:33:40 AM > Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? > > Here is the tail of the failed rebalance log, any clue? > > [2015-12-13 21:30:31.527493] I [dht-rebalance.c:2340:gf_defrag_process_dir] > 0-FastVol-dht: Migration operation on dir > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/5F/1MsH5--BcoGRAJPI took 20.95 secs > [2015-12-13 21:30:31.528704] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:30:31.543901] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:31:37.210496] I [MSGID: 109081] > [dht-common.c:3780:dht_setxattr] 0-FastVol-dht: fixing the layout of > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q > [2015-12-13 21:31:37.722825] I [MSGID: 109045] > [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 0 > (FastVol-client-0): 1032124 chunks > [2015-12-13 21:31:37.722837] I [MSGID: 109045] > [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 1 > (FastVol-client-1): 1032124 chunks > [2015-12-13 21:33:03.955539] I [MSGID: 109064] > [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol: > FastVol-client-0; inode layout - 0 - 2146817919 - 1; disk layout - > 2146817920 - 4294967295 - 1 > [2015-12-13 21:33:04.069859] I [MSGID: 109018] > [dht-common.c:806:dht_revalidate_cbk] 0-FastVol-dht: Mismatching layouts for > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q, gfid > f38c4ed2-a26a-4d83-adfd-6b0331831738 > [2015-12-13 21:33:04.118800] I [MSGID: 109064] > [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol: > FastVol-client-1; inode layout - 2146817920 - 4294967295 - 1; disk layout - > 0 - 2146817919 - 1 > [2015-12-13 21:33:19.979507] I [MSGID: 109022] > [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration > of > /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished > from subvolume FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:19.979459] I [MSGID: 109022] > [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration > of /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint > from subvolume FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:25.543941] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_origin.jpg: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:25.962547] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_small.jpg: > attempting to move from FastVol-client-0 to FastVol-client-1 > > > Cloudor > > > > From: Sakshi Bansal > Date: 2015-12-12 13:02 > To: ?? > CC: gluster-users > Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? > In the rebalance log file you can check the file/directory for which the > rebalance has failed. It can mention what was the fop for whihc the failure > happened. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users