Pranith Kumar Karampuri
2017-May-08 09:19 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
Wait, wait we are discussing this issue only. Expect a reply in some time :-) On Mon, May 8, 2017 at 2:19 PM, Abhijit Paul <er.abhijitpaul at gmail.com> wrote:> poking for previous mail reply > > On Sun, May 7, 2017 at 1:06 AM, Abhijit Paul <er.abhijitpaul at gmail.com> > wrote: > >> https://pkalever.wordpress.com/2017/03/14/elasticsearch-wit >> h-gluster-block/ >> here used tested environment is Fedora , >> but i am using RHEL based Oracle linux so does gluster-block compatible >> with RHEL as well? What i needs to change & make it work? >> >> On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> >>> >>> On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri < >>> pkarampu at redhat.com> wrote: >>> >>>> >>>> >>>> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul <er.abhijitpaul at gmail.com> >>>> wrote: >>>> >>>>> So should i start using gluster-block with elasticsearch in kubernetes >>>>> environment? >>>>> >>>>> My expectation from gluster-block is, it should not CorruptIndex >>>>> of elasticsearch...and issue facing in previous mails. >>>>> >>>>> Please let me know whether should i processed with above mentioned >>>>> combination. >>>>> >>>> >>>> We are still in the process of fixing the failure scenarios of >>>> tcmu-runner dying and failingover in the multipath scenarios. >>>> >>> >>> Prasanna did test that elasticsearch itself worked fine in gluster-block >>> environment when all the machines are up etc i.e. success path. We are >>> doing failure path testing and fixing things at the moment. >>> >>> >>>> >>>> >>>>> >>>>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri < >>>>> pkarampu at redhat.com> wrote: >>>>> >>>>>> Abhijit we just started making the efforts to get all of this stable. >>>>>> >>>>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul < >>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>> >>>>>>> I yet to try gluster-block with elasticsearch...but carious to know >>>>>>> does this combination plays well in kubernetes environment? >>>>>>> >>>>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul < >>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>> >>>>>>>> thanks Krutika for the alternative. >>>>>>>> >>>>>>>> @*Prasanna @**Pranith* >>>>>>>> I was going thorough the mentioned blog post and saw that used >>>>>>>> tested environment was Fedora , >>>>>>>> but i am using RHEL based Oracle linux so does gluster-block >>>>>>>> compatible with RHEL as well? >>>>>>>> >>>>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay < >>>>>>>> kdhananj at redhat.com> wrote: >>>>>>>> >>>>>>>>> Yeah, there are a couple of cache consistency issues with >>>>>>>>> performance translators that are causing these exceptions. >>>>>>>>> Some of them were fixed by 3.10.1. Some still remain. >>>>>>>>> >>>>>>>>> Alternatively you can give gluster-block + elasticsearch a try, >>>>>>>>> which doesn't require solving all these caching issues. >>>>>>>>> Here's a blog post on the same - https://pkalever.wordpress.com >>>>>>>>> /2017/03/14/elasticsearch-with-gluster-block/ >>>>>>>>> >>>>>>>>> Adding Prasanna and Pranith who worked on this, in case you need >>>>>>>>> more info on this. >>>>>>>>> >>>>>>>>> -Krutika >>>>>>>>> >>>>>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul < >>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the reply, i will try it out but i am also facing one >>>>>>>>>> more issue "i.e. replicated volumes returning different >>>>>>>>>> timestamps" >>>>>>>>>> so is this because of Bug 1426548 - Openshift Logging >>>>>>>>>> ElasticSearch FSLocks when using GlusterFS storage backend >>>>>>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ? >>>>>>>>>> >>>>>>>>>> *FYI i am using glusterfs 3.10.1 tar.gz* >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Abhijit >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi < >>>>>>>>>> atumball at redhat.com> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul < >>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Since i am new to gluster, can please provide how to turn off/disable >>>>>>>>>>>> "perf xlator options"? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off >>>>>>>>>>> $ gluster volume set <volname> performance.read-ahead off >>>>>>>>>>> $ gluster volume set <volname> performance.write-behind off >>>>>>>>>>> $ gluster volume set <volname> performance.io-cache off >>>>>>>>>>> $ gluster volume set <volname> performance.quick-read off >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Amar >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee < >>>>>>>>>>>>> amukherj at redhat.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I think there is still some pending stuffs in some of the >>>>>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks >>>>>>>>>>>>>> for more information. Can you please turn off all the perf xlator options >>>>>>>>>>>>>> as a work around to move forward? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul < >>>>>>>>>>>>>> er.abhijitpaul at gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dear folks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume for >>>>>>>>>>>>>>> Elasticsearch(5.1.2) but currently facing issue with *"CorruptIndexException" >>>>>>>>>>>>>>> *with Elasticseach logs and due to that index health trued >>>>>>>>>>>>>>> RED in Elasticsearch. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Later found that there was an issue with gluster < 3.10 ( >>>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but >>>>>>>>>>>>>>> even after *upgrading to 3.10.1 issue is still there.* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *So curios to know what would be the root cause to fix this >>>>>>>>>>>>>>> issue.* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Abhijit >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Gluster-users mailing list >>>>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Amar Tumballi (amarts) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Gluster-users mailing list >>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pranith >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Pranith >>>> >>> >>> >>> >>> -- >>> Pranith >>> >> >> >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170508/8c6a130c/attachment.html>
Pranith Kumar Karampuri
2017-May-08 09:32 UTC
[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1
Abhijit,
We released gluster-block v0.2 just this Friday for which RHEL
packages are yet to be built.
+Kaleb,
Could you help with this please?
Prasanna,
Could you let Abhijit know the rpm versions for tcmu-runner and other
packages so that this feature can be used?
On Mon, May 8, 2017 at 2:49 PM, Pranith Kumar Karampuri <pkarampu at
redhat.com> wrote:
> Wait, wait we are discussing this issue only. Expect a reply in some time
> :-)
>
> On Mon, May 8, 2017 at 2:19 PM, Abhijit Paul <er.abhijitpaul at
gmail.com>
> wrote:
>
>> poking for previous mail reply
>>
>> On Sun, May 7, 2017 at 1:06 AM, Abhijit Paul <er.abhijitpaul at
gmail.com>
>> wrote:
>>
>>> https://pkalever.wordpress.com/2017/03/14/elasticsearch-wit
>>> h-gluster-block/
>>> here used tested environment is Fedora ,
>>> but i am using RHEL based Oracle linux so does gluster-block
compatible
>>> with RHEL as well? What i needs to change & make it work?
>>>
>>> On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri <
>>> pkarampu at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri <
>>>> pkarampu at redhat.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul
<er.abhijitpaul at gmail.com
>>>>> > wrote:
>>>>>
>>>>>> So should i start using gluster-block with
elasticsearch in kubernetes
>>>>>> environment?
>>>>>>
>>>>>> My expectation from gluster-block is, it should not
CorruptIndex
>>>>>> of elasticsearch...and issue facing in previous mails.
>>>>>>
>>>>>> Please let me know whether should i processed with
above mentioned
>>>>>> combination.
>>>>>>
>>>>>
>>>>> We are still in the process of fixing the failure scenarios
of
>>>>> tcmu-runner dying and failingover in the multipath
scenarios.
>>>>>
>>>>
>>>> Prasanna did test that elasticsearch itself worked fine in
>>>> gluster-block environment when all the machines are up etc i.e.
success
>>>> path. We are doing failure path testing and fixing things at
the moment.
>>>>
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri
<
>>>>>> pkarampu at redhat.com> wrote:
>>>>>>
>>>>>>> Abhijit we just started making the efforts to get
all of this stable.
>>>>>>>
>>>>>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul <
>>>>>>> er.abhijitpaul at gmail.com> wrote:
>>>>>>>
>>>>>>>> I yet to try gluster-block with
elasticsearch...but carious to know
>>>>>>>> does this combination plays well in kubernetes
environment?
>>>>>>>>
>>>>>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul
<
>>>>>>>> er.abhijitpaul at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> thanks Krutika for the alternative.
>>>>>>>>>
>>>>>>>>> @*Prasanna @**Pranith*
>>>>>>>>> I was going thorough the mentioned blog
post and saw that used
>>>>>>>>> tested environment was Fedora ,
>>>>>>>>> but i am using RHEL based Oracle linux so
does gluster-block
>>>>>>>>> compatible with RHEL as well?
>>>>>>>>>
>>>>>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika
Dhananjay <
>>>>>>>>> kdhananj at redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yeah, there are a couple of cache
consistency issues with
>>>>>>>>>> performance translators that are
causing these exceptions.
>>>>>>>>>> Some of them were fixed by 3.10.1. Some
still remain.
>>>>>>>>>>
>>>>>>>>>> Alternatively you can give
gluster-block + elasticsearch a try,
>>>>>>>>>> which doesn't require solving all
these caching issues.
>>>>>>>>>> Here's a blog post on the same -
https://pkalever.wordpress.com
>>>>>>>>>>
/2017/03/14/elasticsearch-with-gluster-block/
>>>>>>>>>>
>>>>>>>>>> Adding Prasanna and Pranith who worked
on this, in case you need
>>>>>>>>>> more info on this.
>>>>>>>>>>
>>>>>>>>>> -Krutika
>>>>>>>>>>
>>>>>>>>>> On Fri, May 5, 2017 at 12:15 AM,
Abhijit Paul <
>>>>>>>>>> er.abhijitpaul at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the reply, i will try it
out but i am also facing one
>>>>>>>>>>> more issue "i.e. replicated
volumes returning different
>>>>>>>>>>> timestamps"
>>>>>>>>>>> so is this because of Bug 1426548 -
Openshift Logging
>>>>>>>>>>> ElasticSearch FSLocks when using
GlusterFS storage backend
>>>>>>>>>>>
<https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ?
>>>>>>>>>>>
>>>>>>>>>>> *FYI i am using glusterfs 3.10.1
tar.gz*
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Abhijit
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, May 4, 2017 at 10:58 PM,
Amar Tumballi <
>>>>>>>>>>> atumball at redhat.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, May 4, 2017 at 10:41
PM, Abhijit Paul <
>>>>>>>>>>>> er.abhijitpaul at gmail.com>
wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Since i am new to gluster,
can please provide how to turn off/disable
>>>>>>>>>>>>> "perf xlator
options"?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> $ gluster volume set
<volname> performance.stat-prefetch off
>>>>>>>>>>>> $ gluster volume set
<volname> performance.read-ahead off
>>>>>>>>>>>> $ gluster volume set
<volname> performance.write-behind off
>>>>>>>>>>>> $ gluster volume set
<volname> performance.io-cache off
>>>>>>>>>>>> $ gluster volume set
<volname> performance.quick-read off
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Amar
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, May 3, 2017 at
8:51 PM, Atin Mukherjee <
>>>>>>>>>>>>>> amukherj at
redhat.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think there is
still some pending stuffs in some of the
>>>>>>>>>>>>>>> gluster perf
xlators to make that work complete. Cced the relevant folks
>>>>>>>>>>>>>>> for more
information. Can you please turn off all the perf xlator options
>>>>>>>>>>>>>>> as a work around to
move forward?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, May 3, 2017
at 8:04 PM, Abhijit Paul <
>>>>>>>>>>>>>>> er.abhijitpaul at
gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Dear folks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I setup
Glusterfs(3.10.1) NFS type as persistence volume
>>>>>>>>>>>>>>>> for
Elasticsearch(5.1.2) but currently facing issue with
*"CorruptIndexException"
>>>>>>>>>>>>>>>> *with
Elasticseach logs and due to that index health trued
>>>>>>>>>>>>>>>> RED in
Elasticsearch.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Later found
that there was an issue with gluster < 3.10 (
>>>>>>>>>>>>>>>>
https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but
>>>>>>>>>>>>>>>> even after
*upgrading to 3.10.1 issue is still there.*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *So curios to
know what would be the root cause to fix this
>>>>>>>>>>>>>>>> issue.*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Abhijit
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>>>>> Gluster-users
mailing list
>>>>>>>>>>>>>>>> Gluster-users
at gluster.org
>>>>>>>>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>> Gluster-users at
gluster.org
>>>>>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Amar Tumballi (amarts)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Pranith
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Pranith
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Pranith
>>>>
>>>
>>>
>>
>
>
> --
> Pranith
>
--
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170508/3efedac1/attachment.html>