Gluster users - Jun 2017 - Rebalance failing on fix-layout

Hello,

The past couple of weeks I had some issues with firmware on the OS hard drives
in my gluster cluster. I have recently fixed the issue, and am bringing my
bricks back into the volume. I am running gluster 3.7.6 and am running into the
following issue:

When I add the brick and rebalance, the operation fails after a couple minutes.
The errors I find in the rebalance log is this:

[2017-06-05 13:38:40.441671] E [MSGID: 109010]
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: /LV_Fitting/code/C
gfid not present
[2017-06-05 13:38:40.450341] E [MSGID: 109010]
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht:
/LV_Fitting/code/C/NoCov_NoImm gfid not present
[2017-06-05 13:38:40.450380] E [MSGID: 109010]
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht:
/LV_Fitting/code/C/simulate gfid not present
[2017-06-05 13:38:40.459365] E [MSGID: 109010]
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht:
/LV_Fitting/code/C/NoCov_NoImm/fits_generate gfid not present
[2017-06-05 13:38:40.468756] E [MSGID: 109010]
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht:
/LV_Fitting/code/C/NoCov_NoImm/fits_generate/N_0_vector.dat gfid not present
[2017-06-05 13:38:40.495645] E [MSGID: 109010]
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht:
LV_Fitting/code/C/simulate/RK45_Integrate.c gfid not present
[2017-06-05 13:38:40.512336] E [MSGID: 109010]
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht:
/LV_Fitting/output/line_search gfid not present
[2017-06-05 13:38:40.512373] E [MSGID: 109010]
[dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: 
/LV_Fitting/output/mcmc gfid not present
[2017-06-05 13:38:40.517808] E [dht-rebalance.c:2992:gf_defrag_fix_layout]
0-hpcscratch-dht: Setxattr failed for /LV_Fitting/output/line_search
[2017-06-05 13:38:40.518025] E [MSGID: 109016]
[dht-rebalance.c:3006:gf_defrag_fix_layout] 0-hpcscratch-dht: Fix layout failed
for /LV_Fitting/output
[2017-06-05 13:38:40.518136] E [MSGID: 109016]
[dht-rebalance.c:3006:gf_defrag_fix_layout] 0-hpcscratch-dht: Fix layout failed
for /LV_Fitting


There are about 102,000 of the gfid error, but only a few errors for the fix
layout failed. Is there any way to recover from this issue?

--
Mike Jarsulic
Sr. HPC Administrator
Center for Research Informatics | University of Chicago
773.702.2066