Abhijit Paul
2017-May-06 19:36 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
https://pkalever.wordpress.com/2017/03/14/elasticsearch-with-gluster-block/ here used tested environment is Fedora , but i am using RHEL based Oracle linux so does gluster-block compatible with RHEL as well? What i needs to change & make it work? On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:> > > On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > >> >> >> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >> wrote: >> >>> So should i start using gluster-block with elasticsearch in kubernetes >>> environment? >>> >>> My expectation from gluster-block is, it should not CorruptIndex >>> of elasticsearch...and issue facing in previous mails. >>> >>> Please let me know whether should i processed with above mentioned >>> combination. >>> >> >> We are still in the process of fixing the failure scenarios of >> tcmu-runner dying and failingover in the multipath scenarios. >> > > Prasanna did test that elasticsearch itself worked fine in gluster-block > environment when all the machines are up etc i.e. success path. We are > doing failure path testing and fixing things at the moment. > > >> >> >>> >>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < >>> pkarampu at redhat.com> wrote: >>> >>>> Abhijit we just started making the efforts to get all of this stable. >>>> >>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>>> wrote: >>>> >>>>> I yet to try gluster-block with elasticsearch...but carious to know >>>>> does this combination plays well in kubernetes environment? >>>>> >>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul < >>>>> er.abhijitpaul at gmail.com> wrote: >>>>> >>>>>> thanks Krutika for the alternative. >>>>>> >>>>>> @*Prasanna @**Pranith* >>>>>> I was going thorough the mentioned blog post and saw that used >>>>>> tested environment was Fedora , >>>>>> but i am using RHEL based Oracle linux so does gluster-block >>>>>> compatible with RHEL as well? >>>>>> >>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay < >>>>>> kdhananj at redhat.com> wrote: >>>>>> >>>>>>> Yeah, there are a couple of cache consistency issues with >>>>>>> performance translators that are causing these exceptions. >>>>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>>>> >>>>>>> Alternatively you can give gluster-block + elasticsearch a try, >>>>>>> which doesn't require solving all these caching issues. >>>>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>>>> >>>>>>> Adding Prasanna and Pranith who worked on this, in case you need >>>>>>> more info on this. >>>>>>> >>>>>>> -Krutika >>>>>>> >>>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>> >>>>>>>> Thanks for the reply, i will try it out but i am also facing one >>>>>>>> more issue "i.e. replicated volumes returning different timestamps" >>>>>>>> so is this because of Bug 1426548 - Openshift Logging >>>>>>>> ElasticSearch FSLocks when using GlusterFS storage backend >>>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>>>> >>>>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>>>> >>>>>>>> Regards, >>>>>>>> Abhijit >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi <atumball at redhat.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>>>> "perf xlator options"? >>>>>>>>>> >>>>>>>>>> >>>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Amar >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I think there is still some pending stuffs in some of the >>>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks >>>>>>>>>>>> for more information. Can you please turn off all the perf xlator options >>>>>>>>>>>> as a work around to move forward? >>>>>>>>>>>> >>>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Dear folks, >>>>>>>>>>>>> >>>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>>>> *with Elasticseach logs and due to that index health trued >>>>>>>>>>>>> RED in Elasticsearch. >>>>>>>>>>>>> >>>>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but even >>>>>>>>>>>>> after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>>>> >>>>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>>>> issue.* >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Abhijit >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Gluster-users mailing list >>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Amar Tumballi (amarts) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Pranith >>>> >>> >>> >> >> >> -- >> Pranith >> > > > > -- > Pranith >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170507/de3b7b69/attachment.html>
Abhijit Paul
2017-May-08 08:49 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
poking for previous mail reply On Sun, May 7, 2017 at 1:06 AM, Abhijit Paul <er.abhijitpaul at gmail.com> wrote:> https://pkalever.wordpress.com/2017/03/14/elasticsearch-wit > h-gluster-block/ > here used tested environment is Fedora , > but i am using RHEL based Oracle linux so does gluster-block compatible > with RHEL as well? What i needs to change & make it work? > > On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > >> >> >> On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> >>> >>> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>> wrote: >>> >>>> So should i start using gluster-block with elasticsearch in kubernetes >>>> environment? >>>> >>>> My expectation from gluster-block is, it should not CorruptIndex >>>> of elasticsearch...and issue facing in previous mails. >>>> >>>> Please let me know whether should i processed with above mentioned >>>> combination. >>>> >>> >>> We are still in the process of fixing the failure scenarios of >>> tcmu-runner dying and failingover in the multipath scenarios. >>> >> >> Prasanna did test that elasticsearch itself worked fine in gluster-block >> environment when all the machines are up etc i.e. success path. We are >> doing failure path testing and fixing things at the moment. >> >> >>> >>> >>>> >>>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < >>>> pkarampu at redhat.com> wrote: >>>> >>>>> Abhijit we just started making the efforts to get all of this stable. >>>>> >>>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul <er.abhijitpaul at gmail.com >>>>> > wrote: >>>>> >>>>>> I yet to try gluster-block with elasticsearch...but carious to know >>>>>> does this combination plays well in kubernetes environment? >>>>>> >>>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul < >>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>> >>>>>>> thanks Krutika for the alternative. >>>>>>> >>>>>>> @*Prasanna @**Pranith* >>>>>>> I was going thorough the mentioned blog post and saw that used >>>>>>> tested environment was Fedora , >>>>>>> but i am using RHEL based Oracle linux so does gluster-block >>>>>>> compatible with RHEL as well? >>>>>>> >>>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay < >>>>>>> kdhananj at redhat.com> wrote: >>>>>>> >>>>>>>> Yeah, there are a couple of cache consistency issues with >>>>>>>> performance translators that are causing these exceptions. >>>>>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>>>>> >>>>>>>> Alternatively you can give gluster-block + elasticsearch a try, >>>>>>>> which doesn't require solving all these caching issues. >>>>>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>>>>> >>>>>>>> Adding Prasanna and Pranith who worked on this, in case you need >>>>>>>> more info on this. >>>>>>>> >>>>>>>> -Krutika >>>>>>>> >>>>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks for the reply, i will try it out but i am also facing one >>>>>>>>> more issue "i.e. replicated volumes returning different >>>>>>>>> timestamps" >>>>>>>>> so is this because of Bug 1426548 - Openshift Logging >>>>>>>>> ElasticSearch FSLocks when using GlusterFS storage backend >>>>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>>>>> >>>>>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Abhijit >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi < >>>>>>>>> atumball at redhat.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>>>>> "perf xlator options"? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Amar >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I think there is still some pending stuffs in some of the >>>>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks >>>>>>>>>>>>> for more information. Can you please turn off all the perf xlator options >>>>>>>>>>>>> as a work around to move forward? >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Dear folks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>>>>> *with Elasticseach logs and due to that index health trued >>>>>>>>>>>>>> RED in Elasticsearch. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but >>>>>>>>>>>>>> even after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>>>>> >>>>>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>>>>> issue.* >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Abhijit >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Amar Tumballi (amarts) >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pranith >>>>> >>>> >>>> >>> >>> >>> -- >>> Pranith >>> >> >> >> >> -- >> Pranith >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170508/adae79ab/attachment.html>