Gluster users - Aug 2016 - memory leak in glusterd 3.7.x

hi Zdenek,
        I recently found this issue and there has been a discussion on
gluster-devel about how to fix this. It is a bit involved, so taking more
time than I would like to fix, this bug has been there for almost 3-4 years
I guess.

You can find the discussion here:
http://www.gluster.org/pipermail/gluster-devel/2016-July/050085.html

On Fri, Aug 19, 2016 at 7:10 PM, Zdenek Styblik <zdenek.styblik at
showmax.com>
wrote:
> Hello,
>
> we've found a memory leak in glusterd v3.7.x(currently at v3.7.14, but
> we are users of v3.7.x from the beginning).
> It seems, we've empirically verified, that continuous execution of %
> gluster volume set <volname> <key> <value>; leads to
memory leaks in
> glusterd and OOM, although not necessarily OOM of glusterd itself.
> Settings which were being set over and over again are
> `nfs.addr-namelookup false` and `nfs.disable true`. There might have
> been other settings, but I was able to find these in recent logs.
> Unfortunately, we don't have capacity to debug this issue
> further(statedumps are quite overwhelming :] ).
> Repeated execution has been caused due to bug in Puppet module we're
> using(and we were able to address this issue). Therefore, it's safe to
> say that the number of affected users or like hood of somebody else
> having these problem is probably low. It's still a memory leak and,
> well, rather serious one if you happen to stumble upon it. Also, it
> must be noted that this gets amplified if you have more than just
> volume.
>
> If there is anything I can help with, let me know.
>
> Please, keep me on CC as I'm not subscribed to the mailing list.
>
> Best regards,
> Zdenek Styblik
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160820/1c6c7b65/attachment.html>

On Sat, Aug 20, 2016 at 11:48 AM, Pranith Kumar Karampuri
<pkarampu at redhat.com> wrote:
> hi Zdenek,
>         I recently found this issue and there has been a discussion on
> gluster-devel about how to fix this. It is a bit involved, so taking more
> time than I would like to fix, this bug has been there for almost 3-4 years
> I guess.
>
> You can find the discussion here:
> http://www.gluster.org/pipermail/gluster-devel/2016-July/050085.html
>
Hello Pranith,

thank you for reply and heads up. I've read only through mails with
subject "memory leak" and didn't know you, or somebody else,
already
found this issue.

Best regards,
Z.
> On Fri, Aug 19, 2016 at 7:10 PM, Zdenek Styblik <zdenek.styblik at
showmax.com>
> wrote:
>>
>> Hello,
>>
>> we've found a memory leak in glusterd v3.7.x(currently at v3.7.14,
but
>> we are users of v3.7.x from the beginning).
>> It seems, we've empirically verified, that continuous execution of
%
>> gluster volume set <volname> <key> <value>; leads to
memory leaks in
>> glusterd and OOM, although not necessarily OOM of glusterd itself.
>> Settings which were being set over and over again are
>> `nfs.addr-namelookup false` and `nfs.disable true`. There might have
>> been other settings, but I was able to find these in recent logs.
>> Unfortunately, we don't have capacity to debug this issue
>> further(statedumps are quite overwhelming :] ).
>> Repeated execution has been caused due to bug in Puppet module
we're
>> using(and we were able to address this issue). Therefore, it's safe
to
>> say that the number of affected users or like hood of somebody else
>> having these problem is probably low. It's still a memory leak and,
>> well, rather serious one if you happen to stumble upon it. Also, it
>> must be noted that this gets amplified if you have more than just
>> volume.
>>
>> If there is anything I can help with, let me know.
>>
>> Please, keep me on CC as I'm not subscribed to the mailing list.
>>
>> Best regards,
>> Zdenek Styblik
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> Pranith

On Mon, Aug 22, 2016 at 9:03 PM, Zdenek Styblik <zdenek.styblik at
showmax.com>
wrote:
> On Sat, Aug 20, 2016 at 11:48 AM, Pranith Kumar Karampuri
> <pkarampu at redhat.com> wrote:
> > hi Zdenek,
> >         I recently found this issue and there has been a discussion on
> > gluster-devel about how to fix this. It is a bit involved, so taking
more
> > time than I would like to fix, this bug has been there for almost 3-4
> years
> > I guess.
> >
> > You can find the discussion here:
> > http://www.gluster.org/pipermail/gluster-devel/2016-July/050085.html
> >
>
> Hello Pranith,
>
> thank you for reply and heads up. I've read only through mails with
> subject "memory leak" and didn't know you, or somebody else,
already
> found this issue.
>
Hey,
       Please err on posting these kinds of mails in future as well :-). In
the worst case it will be something we already know. But in the best case
it wll be something new :-).

>
> Best regards,
> Z.
>
> > On Fri, Aug 19, 2016 at 7:10 PM, Zdenek Styblik <
> zdenek.styblik at showmax.com>
> > wrote:
> >>
> >> Hello,
> >>
> >> we've found a memory leak in glusterd v3.7.x(currently at
v3.7.14, but
> >> we are users of v3.7.x from the beginning).
> >> It seems, we've empirically verified, that continuous
execution of %
> >> gluster volume set <volname> <key> <value>;
leads to memory leaks in
> >> glusterd and OOM, although not necessarily OOM of glusterd itself.
> >> Settings which were being set over and over again are
> >> `nfs.addr-namelookup false` and `nfs.disable true`. There might
have
> >> been other settings, but I was able to find these in recent logs.
> >> Unfortunately, we don't have capacity to debug this issue
> >> further(statedumps are quite overwhelming :] ).
> >> Repeated execution has been caused due to bug in Puppet module
we're
> >> using(and we were able to address this issue). Therefore, it's
safe to
> >> say that the number of affected users or like hood of somebody
else
> >> having these problem is probably low. It's still a memory leak
and,
> >> well, rather serious one if you happen to stumble upon it. Also,
it
> >> must be noted that this gets amplified if you have more than just
> >> volume.
> >>
> >> If there is anything I can help with, let me know.
> >>
> >> Please, keep me on CC as I'm not subscribed to the mailing
list.
> >>
> >> Best regards,
> >> Zdenek Styblik
> >> _______________________________________________
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org
> >> http://www.gluster.org/mailman/listinfo/gluster-users
> >
> >
> >
> >
> > --
> > Pranith
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160822/a1733a44/attachment.html>