On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote:> Brief test shows that Ganesha stopped leaking and crashing, so it seems > to be good for me. >Thanks for checking.> Nevertheless, back to my original question: what about FUSE client? It > is still leaking despite all the fixes applied. Should it be considered > another issue?For fuse client, I tried vfs drop_caches as suggested by Vijay in an earlier mail. Though all the inodes get purged, I still doesn't see much difference in the memory footprint drop. Need to investigate what else is consuming so much memory here. Thanks, Soumya> > 11.01.2016 12:26, Soumya Koduri ???????: >> I have made changes to fix the lookup leak in a different way (as >> discussed with Pranith) and uploaded them in the latest patch set #4 >> - http://review.gluster.org/#/c/13096/ >> >> Please check if it resolves the mem leak and hopefully doesn't result >> in any assertion :) >> >> Thanks, >> Soumya >> >> On 01/08/2016 05:04 PM, Soumya Koduri wrote: >>> I could reproduce while testing deep directories with in the mount >>> point. I root caus'ed the issue & had discussion with Pranith to >>> understand the purpose and recommended way of taking nlookup on inodes. >>> >>> I shall make changes to my existing fix and post the patch soon. >>> Thanks for your patience! >>> >>> -Soumya >>> >>> On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote: >>>> OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most >>>> recent >>>> revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision >>>> too). >>>> >>>> On traversing GlusterFS volume with many files in one folder via NFS >>>> mount I >>>> get an assertion: >>>> >>>> ==>>>> ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >>>>> nlookup' failed. >>>> ==>>>> >>>> I used GDB on NFS-Ganesha process to get appropriate stacktraces: >>>> >>>> 1. short stacktrace of failed thread: >>>> >>>> https://gist.github.com/7f63bb99c530d26ded18 >>>> >>>> 2. full stacktrace of failed thread: >>>> >>>> https://gist.github.com/d9bc7bc8f6a0bbff9e86 >>>> >>>> 3. short stacktrace of all threads: >>>> >>>> https://gist.github.com/f31da7725306854c719f >>>> >>>> 4. full stacktrace of all threads: >>>> >>>> https://gist.github.com/65cbc562b01211ea5612 >>>> >>>> GlusterFS volume configuration: >>>> >>>> https://gist.github.com/30f0129d16e25d4a5a52 >>>> >>>> ganesha.conf: >>>> >>>> https://gist.github.com/9b5e59b8d6d8cb84c85d >>>> >>>> How I mount NFS share: >>>> >>>> ==>>>> mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o >>>> defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100 >>>> ==>>>> >>>> On ??????, 7 ????? 2016 ?. 12:06:42 EET Soumya Koduri wrote: >>>>> Entries_HWMark = 500; >>>> >>>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users
Mathieu Chateau
2016-Jan-12 06:34 UTC
[Gluster-users] Memory leak in GlusterFS FUSE client
Hello, I also experience high memory usage on my gluster clients. Sample : [image: Images int?gr?es 1] Can I help in testing/debugging ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-12 7:24 GMT+01:00 Soumya Koduri <skoduri at redhat.com>:> > > On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote: > >> Brief test shows that Ganesha stopped leaking and crashing, so it seems >> to be good for me. >> >> Thanks for checking. > > Nevertheless, back to my original question: what about FUSE client? It >> is still leaking despite all the fixes applied. Should it be considered >> another issue? >> > > For fuse client, I tried vfs drop_caches as suggested by Vijay in an > earlier mail. Though all the inodes get purged, I still doesn't see much > difference in the memory footprint drop. Need to investigate what else is > consuming so much memory here. > > Thanks, > Soumya > > > >> 11.01.2016 12:26, Soumya Koduri ???????: >> >>> I have made changes to fix the lookup leak in a different way (as >>> discussed with Pranith) and uploaded them in the latest patch set #4 >>> - http://review.gluster.org/#/c/13096/ >>> >>> Please check if it resolves the mem leak and hopefully doesn't result >>> in any assertion :) >>> >>> Thanks, >>> Soumya >>> >>> On 01/08/2016 05:04 PM, Soumya Koduri wrote: >>> >>>> I could reproduce while testing deep directories with in the mount >>>> point. I root caus'ed the issue & had discussion with Pranith to >>>> understand the purpose and recommended way of taking nlookup on inodes. >>>> >>>> I shall make changes to my existing fix and post the patch soon. >>>> Thanks for your patience! >>>> >>>> -Soumya >>>> >>>> On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote: >>>> >>>>> OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most >>>>> recent >>>>> revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision >>>>> too). >>>>> >>>>> On traversing GlusterFS volume with many files in one folder via NFS >>>>> mount I >>>>> get an assertion: >>>>> >>>>> ==>>>>> ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >>>>>> nlookup' failed. >>>>> ==>>>>> >>>>> I used GDB on NFS-Ganesha process to get appropriate stacktraces: >>>>> >>>>> 1. short stacktrace of failed thread: >>>>> >>>>> https://gist.github.com/7f63bb99c530d26ded18 >>>>> >>>>> 2. full stacktrace of failed thread: >>>>> >>>>> https://gist.github.com/d9bc7bc8f6a0bbff9e86 >>>>> >>>>> 3. short stacktrace of all threads: >>>>> >>>>> https://gist.github.com/f31da7725306854c719f >>>>> >>>>> 4. full stacktrace of all threads: >>>>> >>>>> https://gist.github.com/65cbc562b01211ea5612 >>>>> >>>>> GlusterFS volume configuration: >>>>> >>>>> https://gist.github.com/30f0129d16e25d4a5a52 >>>>> >>>>> ganesha.conf: >>>>> >>>>> https://gist.github.com/9b5e59b8d6d8cb84c85d >>>>> >>>>> How I mount NFS share: >>>>> >>>>> ==>>>>> mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o >>>>> defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100 >>>>> ==>>>>> >>>>> On ??????, 7 ????? 2016 ?. 12:06:42 EET Soumya Koduri wrote: >>>>> >>>>>> Entries_HWMark = 500; >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>> >>> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160112/35d2d99f/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 10189 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160112/35d2d99f/attachment.png>
Oleksandr Natalenko
2016-Jan-12 07:16 UTC
[Gluster-users] Memory leak in GlusterFS FUSE client
Just in case, here is Valgrind output on FUSE client with 3.7.6 + API-related patches we discussed before: https://gist.github.com/cd6605ca19734c1496a4 12.01.2016 08:24, Soumya Koduri ???????:> For fuse client, I tried vfs drop_caches as suggested by Vijay in an > earlier mail. Though all the inodes get purged, I still doesn't see > much difference in the memory footprint drop. Need to investigate what > else is consuming so much memory here.