Abhijit Paul
2017-May-08 08:49 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
poking for previous mail reply On Sun, May 7, 2017 at 1:06 AM, Abhijit Paul <er.abhijitpaul at gmail.com> wrote:> https://pkalever.wordpress.com/2017/03/14/elasticsearch-wit > h-gluster-block/ > here used tested environment is Fedora , > but i am using RHEL based Oracle linux so does gluster-block compatible > with RHEL as well? What i needs to change & make it work? > > On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > >> >> >> On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> >>> >>> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>> wrote: >>> >>>> So should i start using gluster-block with elasticsearch in kubernetes >>>> environment? >>>> >>>> My expectation from gluster-block is, it should not CorruptIndex >>>> of elasticsearch...and issue facing in previous mails. >>>> >>>> Please let me know whether should i processed with above mentioned >>>> combination. >>>> >>> >>> We are still in the process of fixing the failure scenarios of >>> tcmu-runner dying and failingover in the multipath scenarios. >>> >> >> Prasanna did test that elasticsearch itself worked fine in gluster-block >> environment when all the machines are up etc i.e. success path. We are >> doing failure path testing and fixing things at the moment. >> >> >>> >>> >>>> >>>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < >>>> pkarampu at redhat.com> wrote: >>>> >>>>> Abhijit we just started making the efforts to get all of this stable. >>>>> >>>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul <er.abhijitpaul at gmail.com >>>>> > wrote: >>>>> >>>>>> I yet to try gluster-block with elasticsearch...but carious to know >>>>>> does this combination plays well in kubernetes environment? >>>>>> >>>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul < >>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>> >>>>>>> thanks Krutika for the alternative. >>>>>>> >>>>>>> @*Prasanna @**Pranith* >>>>>>> I was going thorough the mentioned blog post and saw that used >>>>>>> tested environment was Fedora , >>>>>>> but i am using RHEL based Oracle linux so does gluster-block >>>>>>> compatible with RHEL as well? >>>>>>> >>>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay < >>>>>>> kdhananj at redhat.com> wrote: >>>>>>> >>>>>>>> Yeah, there are a couple of cache consistency issues with >>>>>>>> performance translators that are causing these exceptions. >>>>>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>>>>> >>>>>>>> Alternatively you can give gluster-block + elasticsearch a try, >>>>>>>> which doesn't require solving all these caching issues. >>>>>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>>>>> >>>>>>>> Adding Prasanna and Pranith who worked on this, in case you need >>>>>>>> more info on this. >>>>>>>> >>>>>>>> -Krutika >>>>>>>> >>>>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks for the reply, i will try it out but i am also facing one >>>>>>>>> more issue "i.e. replicated volumes returning different >>>>>>>>> timestamps" >>>>>>>>> so is this because of Bug 1426548 - Openshift Logging >>>>>>>>> ElasticSearch FSLocks when using GlusterFS storage backend >>>>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>>>>> >>>>>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Abhijit >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi < >>>>>>>>> atumball at redhat.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>>>>> "perf xlator options"? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Amar >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I think there is still some pending stuffs in some of the >>>>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks >>>>>>>>>>>>> for more information. Can you please turn off all the perf xlator options >>>>>>>>>>>>> as a work around to move forward? >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Dear folks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>>>>> *with Elasticseach logs and due to that index health trued >>>>>>>>>>>>>> RED in Elasticsearch. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but >>>>>>>>>>>>>> even after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>>>>> >>>>>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>>>>> issue.* >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Abhijit >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Amar Tumballi (amarts) >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pranith >>>>> >>>> >>>> >>> >>> >>> -- >>> Pranith >>> >> >> >> >> -- >> Pranith >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170508/adae79ab/attachment.html>
Pranith Kumar Karampuri
2017-May-08 09:19 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
Wait, wait we are discussing this issue only. Expect a reply in some time :-) On Mon, May 8, 2017 at 2:19 PM, Abhijit Paul <er.abhijitpaul at gmail.com> wrote:> poking for previous mail reply > > On Sun, May 7, 2017 at 1:06 AM, Abhijit Paul <er.abhijitpaul at gmail.com> > wrote: > >> https://pkalever.wordpress.com/2017/03/14/elasticsearch-wit >> h-gluster-block/ >> here used tested environment is Fedora , >> but i am using RHEL based Oracle linux so does gluster-block compatible >> with RHEL as well? What i needs to change & make it work? >> >> On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> >>> >>> On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri < >>> pkarampu at redhat.com> wrote: >>> >>>> >>>> >>>> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>>> wrote: >>>> >>>>> So should i start using gluster-block with elasticsearch in kubernetes >>>>> environment? >>>>> >>>>> My expectation from gluster-block is, it should not CorruptIndex >>>>> of elasticsearch...and issue facing in previous mails. >>>>> >>>>> Please let me know whether should i processed with above mentioned >>>>> combination. >>>>> >>>> >>>> We are still in the process of fixing the failure scenarios of >>>> tcmu-runner dying and failingover in the multipath scenarios. >>>> >>> >>> Prasanna did test that elasticsearch itself worked fine in gluster-block >>> environment when all the machines are up etc i.e. success path. We are >>> doing failure path testing and fixing things at the moment. >>> >>> >>>> >>>> >>>>> >>>>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < >>>>> pkarampu at redhat.com> wrote: >>>>> >>>>>> Abhijit we just started making the efforts to get all of this stable. >>>>>> >>>>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul < >>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>> >>>>>>> I yet to try gluster-block with elasticsearch...but carious to know >>>>>>> does this combination plays well in kubernetes environment? >>>>>>> >>>>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul < >>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>> >>>>>>>> thanks Krutika for the alternative. >>>>>>>> >>>>>>>> @*Prasanna @**Pranith* >>>>>>>> I was going thorough the mentioned blog post and saw that used >>>>>>>> tested environment was Fedora , >>>>>>>> but i am using RHEL based Oracle linux so does gluster-block >>>>>>>> compatible with RHEL as well? >>>>>>>> >>>>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay < >>>>>>>> kdhananj at redhat.com> wrote: >>>>>>>> >>>>>>>>> Yeah, there are a couple of cache consistency issues with >>>>>>>>> performance translators that are causing these exceptions. >>>>>>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>>>>>> >>>>>>>>> Alternatively you can give gluster-block + elasticsearch a try, >>>>>>>>> which doesn't require solving all these caching issues. >>>>>>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>>>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>>>>>> >>>>>>>>> Adding Prasanna and Pranith who worked on this, in case you need >>>>>>>>> more info on this. >>>>>>>>> >>>>>>>>> -Krutika >>>>>>>>> >>>>>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the reply, i will try it out but i am also facing one >>>>>>>>>> more issue "i.e. replicated volumes returning different >>>>>>>>>> timestamps" >>>>>>>>>> so is this because of Bug 1426548 - Openshift Logging >>>>>>>>>> ElasticSearch FSLocks when using GlusterFS storage backend >>>>>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>>>>>> >>>>>>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Abhijit >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi < >>>>>>>>>> atumball at redhat.com> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>>>>>> "perf xlator options"? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Amar >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I think there is still some pending stuffs in some of the >>>>>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks >>>>>>>>>>>>>> for more information. Can you please turn off all the perf xlator options >>>>>>>>>>>>>> as a work around to move forward? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dear folks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>>>>>> *with Elasticseach logs and due to that index health trued >>>>>>>>>>>>>>> RED in Elasticsearch. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but >>>>>>>>>>>>>>> even after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>>>>>> issue.* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Abhijit >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Amar Tumballi (amarts) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Gluster-users mailing list >>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pranith >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Pranith >>>> >>> >>> >>> >>> -- >>> Pranith >>> >> >> >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170508/8c6a130c/attachment.html>