Pranith Kumar Karampuri
2017-May-05 12:10 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> wrote:> So should i start using gluster-block with elasticsearch in kubernetes > environment? > > My expectation from gluster-block is, it should not CorruptIndex > of elasticsearch...and issue facing in previous mails. > > Please let me know whether should i processed with above mentioned > combination. >We are still in the process of fixing the failure scenarios of tcmu-runner dying and failingover in the multipath scenarios.> > On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > >> Abhijit we just started making the efforts to get all of this stable. >> >> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >> wrote: >> >>> I yet to try gluster-block with elasticsearch...but carious to know does >>> this combination plays well in kubernetes environment? >>> >>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>> wrote: >>> >>>> thanks Krutika for the alternative. >>>> >>>> @*Prasanna @**Pranith* >>>> I was going thorough the mentioned blog post and saw that used tested >>>> environment was Fedora , >>>> but i am using RHEL based Oracle linux so does gluster-block compatible >>>> with RHEL as well? >>>> >>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay <kdhananj at redhat.com >>>> > wrote: >>>> >>>>> Yeah, there are a couple of cache consistency issues with performance >>>>> translators that are causing these exceptions. >>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>> >>>>> Alternatively you can give gluster-block + elasticsearch a try, which >>>>> doesn't require solving all these caching issues. >>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>> >>>>> Adding Prasanna and Pranith who worked on this, in case you need more >>>>> info on this. >>>>> >>>>> -Krutika >>>>> >>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>> er.abhijitpaul at gmail.com> wrote: >>>>> >>>>>> Thanks for the reply, i will try it out but i am also facing one more >>>>>> issue "i.e. replicated volumes returning different timestamps" >>>>>> so is this because of Bug 1426548 - Openshift Logging ElasticSearch >>>>>> FSLocks when using GlusterFS storage backend >>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>> >>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>> >>>>>> Regards, >>>>>> Abhijit >>>>>> >>>>>> >>>>>> >>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi <atumball at redhat.com> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>> >>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>> "perf xlator options"? >>>>>>>> >>>>>>>> >>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Amar >>>>>>> >>>>>>>> >>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>> >>>>>>>>>> I think there is still some pending stuffs in some of the gluster >>>>>>>>>> perf xlators to make that work complete. Cced the relevant folks for more >>>>>>>>>> information. Can you please turn off all the perf xlator options as a work >>>>>>>>>> around to move forward? >>>>>>>>>> >>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Dear folks, >>>>>>>>>>> >>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>> *with Elasticseach logs and due to that index health trued RED >>>>>>>>>>> in Elasticsearch. >>>>>>>>>>> >>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but even >>>>>>>>>>> after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>> >>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>> issue.* >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Abhijit >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Amar Tumballi (amarts) >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>> >>>>> >>>> >>> >> >> >> -- >> Pranith >> > >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170505/60ffde57/attachment.html>
Pranith Kumar Karampuri
2017-May-05 12:12 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:> > > On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> > wrote: > >> So should i start using gluster-block with elasticsearch in kubernetes >> environment? >> >> My expectation from gluster-block is, it should not CorruptIndex >> of elasticsearch...and issue facing in previous mails. >> >> Please let me know whether should i processed with above mentioned >> combination. >> > > We are still in the process of fixing the failure scenarios of tcmu-runner > dying and failingover in the multipath scenarios. >Prasanna did test that elasticsearch itself worked fine in gluster-block environment when all the machines are up etc i.e. success path. We are doing failure path testing and fixing things at the moment.> > >> >> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> Abhijit we just started making the efforts to get all of this stable. >>> >>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>> wrote: >>> >>>> I yet to try gluster-block with elasticsearch...but carious to know >>>> does this combination plays well in kubernetes environment? >>>> >>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul <er.abhijitpaul at gmail.com >>>> > wrote: >>>> >>>>> thanks Krutika for the alternative. >>>>> >>>>> @*Prasanna @**Pranith* >>>>> I was going thorough the mentioned blog post and saw that used tested >>>>> environment was Fedora , >>>>> but i am using RHEL based Oracle linux so does gluster-block >>>>> compatible with RHEL as well? >>>>> >>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay < >>>>> kdhananj at redhat.com> wrote: >>>>> >>>>>> Yeah, there are a couple of cache consistency issues with performance >>>>>> translators that are causing these exceptions. >>>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>>> >>>>>> Alternatively you can give gluster-block + elasticsearch a try, which >>>>>> doesn't require solving all these caching issues. >>>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>>> >>>>>> Adding Prasanna and Pranith who worked on this, in case you need more >>>>>> info on this. >>>>>> >>>>>> -Krutika >>>>>> >>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>> >>>>>>> Thanks for the reply, i will try it out but i am also facing one >>>>>>> more issue "i.e. replicated volumes returning different timestamps" >>>>>>> so is this because of Bug 1426548 - Openshift Logging ElasticSearch >>>>>>> FSLocks when using GlusterFS storage backend >>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>>> >>>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>>> >>>>>>> Regards, >>>>>>> Abhijit >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi <atumball at redhat.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>> >>>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>>> "perf xlator options"? >>>>>>>>> >>>>>>>>> >>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Amar >>>>>>>> >>>>>>>>> >>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>>> >>>>>>>>>>> I think there is still some pending stuffs in some of the >>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks >>>>>>>>>>> for more information. Can you please turn off all the perf xlator options >>>>>>>>>>> as a work around to move forward? >>>>>>>>>>> >>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Dear folks, >>>>>>>>>>>> >>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>>> *with Elasticseach logs and due to that index health trued RED >>>>>>>>>>>> in Elasticsearch. >>>>>>>>>>>> >>>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but even >>>>>>>>>>>> after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>>> >>>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>>> issue.* >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Abhijit >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Amar Tumballi (amarts) >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Pranith >>> >> >> > > > -- > Pranith >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170505/3926858a/attachment.html>