Hi Nithya, the files were sent to priv to avoid spamming the list with large attachments. Could someone explain what is index in Gluster? Unfortunately index is popular word, so googling is not very helpful. Best regards, Szymon Miotk On Sun, Jul 9, 2017 at 6:37 PM, Nithya Balachandran <nbalacha at redhat.com> wrote:> > On 7 July 2017 at 15:42, Szymon Miotk <szymon.miotk at gmail.com> wrote: >> >> Hello everyone, >> >> >> I have problem rebalancing Gluster volume. >> Gluster version is 3.7.3. >> My 1x3 replicated volume become full, so I've added three more bricks >> to make it 2x3 and wanted to rebalance. >> But every time I start rebalancing, it fails immediately. >> Rebooting Gluster nodes doesn't help. >> >> # gluster volume rebalance gsae_artifactory_cluster_storage start >> volume rebalance: gsae_artifactory_cluster_storage: success: Rebalance >> on gsae_artifactory_cluster_storage has been started successfully. Use >> rebalance status command to check status of the rebalance process. >> ID: b22572ff-7575-4557-8317-765f7e52d445 >> >> # gluster volume rebalance gsae_artifactory_cluster_storage status >> Node Rebalanced-files >> size scanned failures skipped status >> run time in secs >> --------- ----------- >> ----------- ----------- ----------- ----------- >> ------------ -------------- >> localhost 0 >> 0Bytes 0 0 0 failed >> 0.00 >> 10.239.40.9 0 >> 0Bytes 0 0 0 failed >> 0.00 >> 10.239.40.8 0 >> 0Bytes 0 0 0 failed >> 0.00 >> volume rebalance: gsae_artifactory_cluster_storage: success: >> >> The messages in logfiles mention 'failed to get index': >> [2017-07-07 10:07:18.230202] E [MSGID: 106062] >> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict] >> 0-glusterd: failed to get index >> >> and then the rebalance process crashes: >> pending frames: >> frame : type(0) op(0) >> frame : type(0) op(0) >> frame : type(0) op(0) >> patchset: git://git.gluster.com/glusterfs.git >> signal received: 4 >> time of crash: >> 2017-07-07 10:07:23 >> configuration details: >> argp 1 >> backtrace 1 >> dlfcn 1 >> libpthread 1 >> llistxattr 1 >> setfsid 1 >> spinlock 1 >> epoll.h 1 >> xattr.h 1 >> st_atim.tv_nsec 1 >> package-string: glusterfs 3.7.3 >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f24de214502] >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f24de23059d] >> /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f24dd612d40] >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x11f6b)[0x7f24dd9b2f6b] >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_ref+0x19)[0x7f24de234e69] >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_copy+0x4a)[0x7f24de21291a] >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_local_init+0x4b)[0x7f24d851f51b] >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_lookup+0x91)[0x7f24d8550521] >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(syncop_lookup+0x1a2)[0x7f24de258fc2] >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_fix_layout+0x87)[0x7f24d85289e7] >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_start_crawl+0x6d3)[0x7f24d8529ce3] >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_wrap+0x12)[0x7f24de255822] >> /lib/x86_64-linux-gnu/libc.so.6(+0x498b0)[0x7f24dd6258b0] >> >> >> Anybody has a clue how to fix 'failed to get index' error? > > > Do you have the coredump? If yes, please send that and the rebalance logs > across. > > Regards, > Nithya >> >> >> Thank you in advance! >> Szymon Miotk >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users > >
Hi Szymon, I have received the files and will take a look and get back to you. In what context are you seeing index? Thanks, Nithya On 11 July 2017 at 01:15, Szymon Miotk <szymon.miotk at gmail.com> wrote:> Hi Nithya, > > the files were sent to priv to avoid spamming the list with large > attachments. > Could someone explain what is index in Gluster? > Unfortunately index is popular word, so googling is not very helpful. > > Best regards, > Szymon Miotk > > On Sun, Jul 9, 2017 at 6:37 PM, Nithya Balachandran <nbalacha at redhat.com> > wrote: > > > > On 7 July 2017 at 15:42, Szymon Miotk <szymon.miotk at gmail.com> wrote: > >> > >> Hello everyone, > >> > >> > >> I have problem rebalancing Gluster volume. > >> Gluster version is 3.7.3. > >> My 1x3 replicated volume become full, so I've added three more bricks > >> to make it 2x3 and wanted to rebalance. > >> But every time I start rebalancing, it fails immediately. > >> Rebooting Gluster nodes doesn't help. > >> > >> # gluster volume rebalance gsae_artifactory_cluster_storage start > >> volume rebalance: gsae_artifactory_cluster_storage: success: Rebalance > >> on gsae_artifactory_cluster_storage has been started successfully. Use > >> rebalance status command to check status of the rebalance process. > >> ID: b22572ff-7575-4557-8317-765f7e52d445 > >> > >> # gluster volume rebalance gsae_artifactory_cluster_storage status > >> Node Rebalanced-files > >> size scanned failures skipped status > >> run time in secs > >> --------- ----------- > >> ----------- ----------- ----------- ----------- > >> ------------ -------------- > >> localhost 0 > >> 0Bytes 0 0 0 failed > >> 0.00 > >> 10.239.40.9 0 > >> 0Bytes 0 0 0 failed > >> 0.00 > >> 10.239.40.8 0 > >> 0Bytes 0 0 0 failed > >> 0.00 > >> volume rebalance: gsae_artifactory_cluster_storage: success: > >> > >> The messages in logfiles mention 'failed to get index': > >> [2017-07-07 10:07:18.230202] E [MSGID: 106062] > >> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict] > >> 0-glusterd: failed to get index > >> > >> and then the rebalance process crashes: > >> pending frames: > >> frame : type(0) op(0) > >> frame : type(0) op(0) > >> frame : type(0) op(0) > >> patchset: git://git.gluster.com/glusterfs.git > >> signal received: 4 > >> time of crash: > >> 2017-07-07 10:07:23 > >> configuration details: > >> argp 1 > >> backtrace 1 > >> dlfcn 1 > >> libpthread 1 > >> llistxattr 1 > >> setfsid 1 > >> spinlock 1 > >> epoll.h 1 > >> xattr.h 1 > >> st_atim.tv_nsec 1 > >> package-string: glusterfs 3.7.3 > >> > >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_ > backtrace_nomem+0x92)[0x7f24de214502] > >> > >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_ > trace+0x31d)[0x7f24de23059d] > >> /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f24dd612d40] > >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x11f6b)[0x7f24dd9b2f6b] > >> > >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_ref+ > 0x19)[0x7f24de234e69] > >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_copy+ > 0x4a)[0x7f24de21291a] > >> > >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/ > cluster/distribute.so(dht_local_init+0x4b)[0x7f24d851f51b] > >> > >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/ > cluster/distribute.so(dht_lookup+0x91)[0x7f24d8550521] > >> > >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(syncop_ > lookup+0x1a2)[0x7f24de258fc2] > >> > >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/ > cluster/distribute.so(gf_defrag_fix_layout+0x87)[0x7f24d85289e7] > >> > >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/ > cluster/distribute.so(gf_defrag_start_crawl+0x6d3)[0x7f24d8529ce3] > >> > >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_ > wrap+0x12)[0x7f24de255822] > >> /lib/x86_64-linux-gnu/libc.so.6(+0x498b0)[0x7f24dd6258b0] > >> > >> > >> Anybody has a clue how to fix 'failed to get index' error? > > > > > > Do you have the coredump? If yes, please send that and the rebalance logs > > across. > > > > Regards, > > Nithya > >> > >> > >> Thank you in advance! > >> Szymon Miotk > >> _______________________________________________ > >> Gluster-users mailing list > >> Gluster-users at gluster.org > >> http://lists.gluster.org/mailman/listinfo/gluster-users > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170713/6413c6f7/attachment.html>
Hi Nithya, I see index in context: [2017-07-07 10:07:18.230202] E [MSGID: 106062] [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index I wonder if there is anything I can do to fix it. I was trying to strace gluster process but still have no clue what exactly is gluster index. Best regards, Szymon Miotk On Thu, Jul 13, 2017 at 10:12 AM, Nithya Balachandran <nbalacha at redhat.com> wrote:> Hi Szymon, > > I have received the files and will take a look and get back to you. > > In what context are you seeing index? > > Thanks, > Nithya > > On 11 July 2017 at 01:15, Szymon Miotk <szymon.miotk at gmail.com> wrote: >> >> Hi Nithya, >> >> the files were sent to priv to avoid spamming the list with large >> attachments. >> Could someone explain what is index in Gluster? >> Unfortunately index is popular word, so googling is not very helpful. >> >> Best regards, >> Szymon Miotk >> >> On Sun, Jul 9, 2017 at 6:37 PM, Nithya Balachandran <nbalacha at redhat.com> >> wrote: >> > >> > On 7 July 2017 at 15:42, Szymon Miotk <szymon.miotk at gmail.com> wrote: >> >> >> >> Hello everyone, >> >> >> >> >> >> I have problem rebalancing Gluster volume. >> >> Gluster version is 3.7.3. >> >> My 1x3 replicated volume become full, so I've added three more bricks >> >> to make it 2x3 and wanted to rebalance. >> >> But every time I start rebalancing, it fails immediately. >> >> Rebooting Gluster nodes doesn't help. >> >> >> >> # gluster volume rebalance gsae_artifactory_cluster_storage start >> >> volume rebalance: gsae_artifactory_cluster_storage: success: Rebalance >> >> on gsae_artifactory_cluster_storage has been started successfully. Use >> >> rebalance status command to check status of the rebalance process. >> >> ID: b22572ff-7575-4557-8317-765f7e52d445 >> >> >> >> # gluster volume rebalance gsae_artifactory_cluster_storage status >> >> Node Rebalanced-files >> >> size scanned failures skipped status >> >> run time in secs >> >> --------- ----------- >> >> ----------- ----------- ----------- ----------- >> >> ------------ -------------- >> >> localhost 0 >> >> 0Bytes 0 0 0 failed >> >> 0.00 >> >> 10.239.40.9 0 >> >> 0Bytes 0 0 0 failed >> >> 0.00 >> >> 10.239.40.8 0 >> >> 0Bytes 0 0 0 failed >> >> 0.00 >> >> volume rebalance: gsae_artifactory_cluster_storage: success: >> >> >> >> The messages in logfiles mention 'failed to get index': >> >> [2017-07-07 10:07:18.230202] E [MSGID: 106062] >> >> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict] >> >> 0-glusterd: failed to get index >> >> >> >> and then the rebalance process crashes: >> >> pending frames: >> >> frame : type(0) op(0) >> >> frame : type(0) op(0) >> >> frame : type(0) op(0) >> >> patchset: git://git.gluster.com/glusterfs.git >> >> signal received: 4 >> >> time of crash: >> >> 2017-07-07 10:07:23 >> >> configuration details: >> >> argp 1 >> >> backtrace 1 >> >> dlfcn 1 >> >> libpthread 1 >> >> llistxattr 1 >> >> setfsid 1 >> >> spinlock 1 >> >> epoll.h 1 >> >> xattr.h 1 >> >> st_atim.tv_nsec 1 >> >> package-string: glusterfs 3.7.3 >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f24de214502] >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f24de23059d] >> >> /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f24dd612d40] >> >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x11f6b)[0x7f24dd9b2f6b] >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_ref+0x19)[0x7f24de234e69] >> >> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_copy+0x4a)[0x7f24de21291a] >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_local_init+0x4b)[0x7f24d851f51b] >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_lookup+0x91)[0x7f24d8550521] >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(syncop_lookup+0x1a2)[0x7f24de258fc2] >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_fix_layout+0x87)[0x7f24d85289e7] >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_start_crawl+0x6d3)[0x7f24d8529ce3] >> >> >> >> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_wrap+0x12)[0x7f24de255822] >> >> /lib/x86_64-linux-gnu/libc.so.6(+0x498b0)[0x7f24dd6258b0] >> >> >> >> >> >> Anybody has a clue how to fix 'failed to get index' error? >> > >> > >> > Do you have the coredump? If yes, please send that and the rebalance >> > logs >> > across. >> > >> > Regards, >> > Nithya >> >> >> >> >> >> Thank you in advance! >> >> Szymon Miotk >> >> _______________________________________________ >> >> Gluster-users mailing list >> >> Gluster-users at gluster.org >> >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > >> > > >