Pranith Kumar Karampuri
2017-May-05 12:12 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:> > > On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> > wrote: > >> So should i start using gluster-block with elasticsearch in kubernetes >> environment? >> >> My expectation from gluster-block is, it should not CorruptIndex >> of elasticsearch...and issue facing in previous mails. >> >> Please let me know whether should i processed with above mentioned >> combination. >> > > We are still in the process of fixing the failure scenarios of tcmu-runner > dying and failingover in the multipath scenarios. >Prasanna did test that elasticsearch itself worked fine in gluster-block environment when all the machines are up etc i.e. success path. We are doing failure path testing and fixing things at the moment.> > >> >> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> Abhijit we just started making the efforts to get all of this stable. >>> >>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>> wrote: >>> >>>> I yet to try gluster-block with elasticsearch...but carious to know >>>> does this combination plays well in kubernetes environment? >>>> >>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul <er.abhijitpaul at gmail.com >>>> > wrote: >>>> >>>>> thanks Krutika for the alternative. >>>>> >>>>> @*Prasanna @**Pranith* >>>>> I was going thorough the mentioned blog post and saw that used tested >>>>> environment was Fedora , >>>>> but i am using RHEL based Oracle linux so does gluster-block >>>>> compatible with RHEL as well? >>>>> >>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay < >>>>> kdhananj at redhat.com> wrote: >>>>> >>>>>> Yeah, there are a couple of cache consistency issues with performance >>>>>> translators that are causing these exceptions. >>>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>>> >>>>>> Alternatively you can give gluster-block + elasticsearch a try, which >>>>>> doesn't require solving all these caching issues. >>>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>>> >>>>>> Adding Prasanna and Pranith who worked on this, in case you need more >>>>>> info on this. >>>>>> >>>>>> -Krutika >>>>>> >>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>> >>>>>>> Thanks for the reply, i will try it out but i am also facing one >>>>>>> more issue "i.e. replicated volumes returning different timestamps" >>>>>>> so is this because of Bug 1426548 - Openshift Logging ElasticSearch >>>>>>> FSLocks when using GlusterFS storage backend >>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>>> >>>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>>> >>>>>>> Regards, >>>>>>> Abhijit >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi <atumball at redhat.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>> >>>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>>> "perf xlator options"? >>>>>>>>> >>>>>>>>> >>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Amar >>>>>>>> >>>>>>>>> >>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>>> >>>>>>>>>>> I think there is still some pending stuffs in some of the >>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks >>>>>>>>>>> for more information. Can you please turn off all the perf xlator options >>>>>>>>>>> as a work around to move forward? >>>>>>>>>>> >>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Dear folks, >>>>>>>>>>>> >>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>>> *with Elasticseach logs and due to that index health trued RED >>>>>>>>>>>> in Elasticsearch. >>>>>>>>>>>> >>>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but even >>>>>>>>>>>> after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>>> >>>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>>> issue.* >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Abhijit >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Amar Tumballi (amarts) >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Pranith >>> >> >> > > > -- > Pranith >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170505/3926858a/attachment.html>
Abhijit Paul
2017-May-06 19:36 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
https://pkalever.wordpress.com/2017/03/14/elasticsearch-with-gluster-block/ here used tested environment is Fedora , but i am using RHEL based Oracle linux so does gluster-block compatible with RHEL as well? What i needs to change & make it work? On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:> > > On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > >> >> >> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >> wrote: >> >>> So should i start using gluster-block with elasticsearch in kubernetes >>> environment? >>> >>> My expectation from gluster-block is, it should not CorruptIndex >>> of elasticsearch...and issue facing in previous mails. >>> >>> Please let me know whether should i processed with above mentioned >>> combination. >>> >> >> We are still in the process of fixing the failure scenarios of >> tcmu-runner dying and failingover in the multipath scenarios. >> > > Prasanna did test that elasticsearch itself worked fine in gluster-block > environment when all the machines are up etc i.e. success path. We are > doing failure path testing and fixing things at the moment. > > >> >> >>> >>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < >>> pkarampu at redhat.com> wrote: >>> >>>> Abhijit we just started making the efforts to get all of this stable. >>>> >>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>>> wrote: >>>> >>>>> I yet to try gluster-block with elasticsearch...but carious to know >>>>> does this combination plays well in kubernetes environment? >>>>> >>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul < >>>>> er.abhijitpaul at gmail.com> wrote: >>>>> >>>>>> thanks Krutika for the alternative. >>>>>> >>>>>> @*Prasanna @**Pranith* >>>>>> I was going thorough the mentioned blog post and saw that used >>>>>> tested environment was Fedora , >>>>>> but i am using RHEL based Oracle linux so does gluster-block >>>>>> compatible with RHEL as well? >>>>>> >>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay < >>>>>> kdhananj at redhat.com> wrote: >>>>>> >>>>>>> Yeah, there are a couple of cache consistency issues with >>>>>>> performance translators that are causing these exceptions. >>>>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>>>> >>>>>>> Alternatively you can give gluster-block + elasticsearch a try, >>>>>>> which doesn't require solving all these caching issues. >>>>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>>>> >>>>>>> Adding Prasanna and Pranith who worked on this, in case you need >>>>>>> more info on this. >>>>>>> >>>>>>> -Krutika >>>>>>> >>>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>> >>>>>>>> Thanks for the reply, i will try it out but i am also facing one >>>>>>>> more issue "i.e. replicated volumes returning different timestamps" >>>>>>>> so is this because of Bug 1426548 - Openshift Logging >>>>>>>> ElasticSearch FSLocks when using GlusterFS storage backend >>>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>>>> >>>>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>>>> >>>>>>>> Regards, >>>>>>>> Abhijit >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi <atumball at redhat.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>>>> "perf xlator options"? >>>>>>>>>> >>>>>>>>>> >>>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Amar >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I think there is still some pending stuffs in some of the >>>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks >>>>>>>>>>>> for more information. Can you please turn off all the perf xlator options >>>>>>>>>>>> as a work around to move forward? >>>>>>>>>>>> >>>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Dear folks, >>>>>>>>>>>>> >>>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>>>> *with Elasticseach logs and due to that index health trued >>>>>>>>>>>>> RED in Elasticsearch. >>>>>>>>>>>>> >>>>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but even >>>>>>>>>>>>> after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>>>> >>>>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>>>> issue.* >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Abhijit >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Gluster-users mailing list >>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Amar Tumballi (amarts) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Pranith >>>> >>> >>> >> >> >> -- >> Pranith >> > > > > -- > Pranith >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170507/de3b7b69/attachment.html>