Oleksandr Natalenko
2015-Sep-24 08:14 UTC
[Gluster-users] Memory leak in GlusterFS FUSE client
I've checked statedump of volume in question and haven't found lots of iobuf as mentioned in that bugreport. However, I've noticed that there are lots of LRU records like this: ==[conn.1.bound_xl./bricks/r6sdLV07_vd0_mail/mail.lru.1] gfid=c4b29310-a19d-451b-8dd1-b3ac2d86b595 nlookup=1 fd-count=0 ref=0 ia_type=1 == In fact, there are 16383 of them. I've checked "gluster volume set help" in order to find something LRU-related and have found this: ==Option: network.inode-lru-limit Default Value: 16384 Description: Specifies the maximum megabytes of memory to be used in the inode cache. == Is there error in description stating "maximum megabytes of memory"? Shouldn't it mean "maximum amount of LRU records"? If no, is that true, that inode cache could grow up to 16 GiB for client, and one must lower network.inode-lru-limit value? Another thought: we've enabled write-behind, and the default write-behind-window-size value is 1 MiB. So, one may conclude that with lots of small files written, write-behind buffer could grow up to inode-lru-limit?write-behind-window-size=16 GiB? Who could explain that to me? 24.09.2015 10:42, Gabi C write:> oh, my bad... > coulb be this one? > > https://bugzilla.redhat.com/show_bug.cgi?id=1126831 [2] > Anyway, on ovirt+gluster w I experienced similar behavior...
Raghavendra Bhat
2015-Sep-28 11:31 UTC
[Gluster-users] [Gluster-devel] Memory leak in GlusterFS FUSE client
Hi Oleksandr, You are right. The description should have said it as the limit on the number of inodes in the lru list of the inode cache. I have sent a patch for that. http://review.gluster.org/#/c/12242/ Regards, Raghavendra Bhat On Thu, Sep 24, 2015 at 1:44 PM, Oleksandr Natalenko < oleksandr at natalenko.name> wrote:> I've checked statedump of volume in question and haven't found lots of > iobuf as mentioned in that bugreport. > > However, I've noticed that there are lots of LRU records like this: > > ==> [conn.1.bound_xl./bricks/r6sdLV07_vd0_mail/mail.lru.1] > gfid=c4b29310-a19d-451b-8dd1-b3ac2d86b595 > nlookup=1 > fd-count=0 > ref=0 > ia_type=1 > ==> > In fact, there are 16383 of them. I've checked "gluster volume set help" > in order to find something LRU-related and have found this: > > ==> Option: network.inode-lru-limit > Default Value: 16384 > Description: Specifies the maximum megabytes of memory to be used in the > inode cache. > ==> > Is there error in description stating "maximum megabytes of memory"? > Shouldn't it mean "maximum amount of LRU records"? If no, is that true, > that inode cache could grow up to 16 GiB for client, and one must lower > network.inode-lru-limit value? > > Another thought: we've enabled write-behind, and the default > write-behind-window-size value is 1 MiB. So, one may conclude that with > lots of small files written, write-behind buffer could grow up to > inode-lru-limit?write-behind-window-size=16 GiB? Who could explain that to > me? > > 24.09.2015 10:42, Gabi C write: > >> oh, my bad... >> coulb be this one? >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1126831 [2] >> Anyway, on ovirt+gluster w I experienced similar behavior... >> > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150928/2ad35ce5/attachment.html>
Oleksandr Natalenko
2015-Oct-13 08:57 UTC
[Gluster-users] [Gluster-devel] Memory leak in GlusterFS FUSE client
Here are two consecutive statedumps of brick in question memory usage [1] [2]. glusterfs client process went from ~630 MB to ~1350 MB of memory usage in less than one hour. Volume options: ==cluster.lookup-optimize: on cluster.readdir-optimize: on client.event-threads: 4 network.inode-lru-limit: 4096 server.event-threads: 8 performance.client-io-threads: on storage.linux-aio: on performance.write-behind-window-size: 4194304 performance.stat-prefetch: on performance.quick-read: on performance.read-ahead: on performance.flush-behind: on performance.write-behind: on performance.io-thread-count: 4 performance.cache-max-file-size: 1048576 performance.cache-size: 33554432 performance.readdir-ahead: on == I observe such a behavior on similar volumes where millions of files are stored. The volume in question holds ~11M of small files (mail storage). So, memleak persists. Had to switch to NFS temporarily :(. Any idea? [1] https://gist.github.com/46697b70ffe193fa797e [2] https://gist.github.com/3a968ca909bfdeb31cca 28.09.2015 14:31, Raghavendra Bhat ???????:> Hi Oleksandr, > > You are right. The description should have said it as the limit on the > number of inodes in the lru list of the inode cache. I have sent a > patch for that. > http://review.gluster.org/#/c/12242/ [3] > > Regards, > Raghavendra Bhat > > On Thu, Sep 24, 2015 at 1:44 PM, Oleksandr Natalenko > <oleksandr at natalenko.name> wrote: > >> I've checked statedump of volume in question and haven't found lots >> of iobuf as mentioned in that bugreport. >> >> However, I've noticed that there are lots of LRU records like this: >> >> ==>> [conn.1.bound_xl./bricks/r6sdLV07_vd0_mail/mail.lru.1] >> gfid=c4b29310-a19d-451b-8dd1-b3ac2d86b595 >> nlookup=1 >> fd-count=0 >> ref=0 >> ia_type=1 >> ==>> >> In fact, there are 16383 of them. I've checked "gluster volume set >> help" in order to find something LRU-related and have found this: >> >> ==>> Option: network.inode-lru-limit >> Default Value: 16384 >> Description: Specifies the maximum megabytes of memory to be used in >> the inode cache. >> ==>> >> Is there error in description stating "maximum megabytes of memory"? >> Shouldn't it mean "maximum amount of LRU records"? If no, is that >> true, that inode cache could grow up to 16 GiB for client, and one >> must lower network.inode-lru-limit value? >> >> Another thought: we've enabled write-behind, and the default >> write-behind-window-size value is 1 MiB. So, one may conclude that >> with lots of small files written, write-behind buffer could grow up >> to inode-lru-limit?write-behind-window-size=16 GiB? Who could >> explain that to me? >> >> 24.09.2015 10:42, Gabi C write: >> >>> oh, my bad... >>> coulb be this one? >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1126831 [1] [2] >>> Anyway, on ovirt+gluster w I experienced similar behavior... >> >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-devel [2] > > > > Links: > ------ > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1126831 > [2] http://www.gluster.org/mailman/listinfo/gluster-devel > [3] http://review.gluster.org/#/c/12242/