thr3ads.net - Gluster users - [Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary

If this information is useful, please help other people find it:
Share via:

Soumya Koduri

2016-Feb-01 08:54 UTC

[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I

On 02/01/2016 01:39 PM, Oleksandr Natalenko wrote:> Wait. It seems to be my bad.
>
> Before unmounting I do drop_caches (2), and glusterfs process CPU usage
> goes to 100% for a while. I haven't waited for it to drop to 0%, and
> instead perform unmount. It seems glusterfs is purging inodes and
that's
> why it uses 100% of CPU. I've re-tested it, waiting for CPU usage to
> become normal, and got no leaks.
>
> Will verify this once again and report more.
>
> BTW, if that works, how could I limit inode cache for FUSE client? I do
> not want it to go beyond 1G, for example, even if I have 48G of RAM on
> my server.
Its hard-coded for now. For fuse the lru limit (of the inodes which are 
not active) is (32*1024).
One of the ways to address this (which we were discussing earlier) is to 
have an option to configure inode cache limit. If that sounds good, we 
can then check on if it has to be global/volume-level, client/server/both.

Thanks,
Soumya
>
> 01.02.2016 09:54, Soumya Koduri ???????:
>> On 01/31/2016 03:05 PM, Oleksandr Natalenko wrote:
>>> Unfortunately, this patch doesn't help.
>>>
>>> RAM usage on "find" finish is ~9G.
>>>
>>> Here is statedump before drop_caches: https://gist.github.com/
>>> fc1647de0982ab447e20
>>
>> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
>> size=706766688
>> num_allocs=2454051
>>
>>>
>>> And after drop_caches: https://gist.github.com/5eab63bc13f78787ed19
>>
>> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
>> size=550996416
>> num_allocs=1913182
>>
>> There isn't much significant drop in inode contexts. One of the
>> reasons could be because of dentrys holding a refcount on the inodes
>> which shall result in inodes not getting purged even after
>> fuse_forget.
>>
>>
>> pool-name=fuse:dentry_t
>> hot-count=32761
>>
>> if  '32761' is the current active dentry count, it still
doesn't seem
>> to match up to inode count.
>>
>> Thanks,
>> Soumya
>>>
>>> And here is Valgrind output:
>>> https://gist.github.com/2490aeac448320d98596
>>>
>>> On ??????, 30 ????? 2016 ?. 22:56:37 EET Xavier Hernandez wrote:
>>>> There's another inode leak caused by an incorrect counting
of
>>>> lookups on directory reads.
>>>>
>>>> Here's a patch that solves the problem for
>>>> 3.7:
>>>>
>>>> http://review.gluster.org/13324
>>>>
>>>> Hopefully with this patch the
>>>> memory leaks should disapear.
>>>>
>>>> Xavi
>>>>
>>>> On 29.01.2016 19:09, Oleksandr
>>>>
>>>> Natalenko wrote:
>>>>> Here is intermediate summary of current memory
>>>>
>>>> leaks in FUSE client
>>>>
>>>>> investigation.
>>>>>
>>>>> I use GlusterFS v3.7.6
>>>>
>>>> release with the following patches:
>>>>> ==>>>>
>>>>> Kaleb S KEITHLEY (1):
>>>> fuse: use-after-free fix in fuse-bridge, revisited
>>>>
>>>>> Pranith Kumar K
>>>>
>>>> (1):
>>>>> mount/fuse: Fix use-after-free crash
>>>>
>>>>> Soumya Koduri (3):
>>>> gfapi: Fix inode nlookup counts
>>>>
>>>>> inode: Retire the inodes from the lru
>>>>
>>>> list in inode_table_destroy
>>>>
>>>>> upcall: free the xdr* allocations
>>>>> ==>>>>>
>>>>>
>>>>> With those patches we got API leaks fixed (I hope, brief
tests show
>>>>
>>>> that) and
>>>>
>>>>> got rid of "kernel notifier loop terminated"
message.
>>>>
>>>> Nevertheless, FUSE
>>>>
>>>>> client still leaks.
>>>>>
>>>>> I have several test
>>>>
>>>> volumes with several million of small files (100K?2M in
>>>>
>>>>> average). I
>>>>
>>>> do 2 types of FUSE client testing:
>>>>> 1) find /mnt/volume -type d
>>>>> 2)
>>>>
>>>> rsync -av -H /mnt/source_volume/* /mnt/target_volume/
>>>>
>>>>> And most
>>>>
>>>> up-to-date results are shown below:
>>>>> === find /mnt/volume -type d
>>>>
>>>> ==>>>>
>>>>> Memory consumption: ~4G
>>>>
>>>>> Statedump:
>>>> https://gist.github.com/10cde83c63f1b4f1dd7a
>>>>
>>>>> Valgrind:
>>>> https://gist.github.com/097afb01ebb2c5e9e78d
>>>>
>>>>> I guess,
>>>>
>>>> fuse-bridge/fuse-resolve. related.
>>>>
>>>>> === rsync -av -H
>>>>
>>>> /mnt/source_volume/* /mnt/target_volume/ ==>>>>
>>>>> Memory consumption:
>>>> ~3.3...4G
>>>>
>>>>> Statedump (target volume):
>>>> https://gist.github.com/31e43110eaa4da663435
>>>>
>>>>> Valgrind (target volume):
>>>> https://gist.github.com/f8e0151a6878cacc9b1a
>>>>
>>>>> I guess,
>>>>
>>>> DHT-related.
>>>>
>>>>> Give me more patches to test :).
>>>>
>>>> _______________________________________________
>>>>
>>>>> Gluster-devel mailing
>>>>
>>>> list
>>>>
>>>>> Gluster-devel at gluster.org
>>>>
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>

Xavier Hernandez

2016-Feb-01 09:18 UTC

head link

[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I

Hi,

On 01/02/16 09:54, Soumya Koduri wrote:>
>
> On 02/01/2016 01:39 PM, Oleksandr Natalenko wrote:
>> Wait. It seems to be my bad.
>>
>> Before unmounting I do drop_caches (2), and glusterfs process CPU usage
>> goes to 100% for a while. I haven't waited for it to drop to 0%,
and
>> instead perform unmount. It seems glusterfs is purging inodes and
that's
>> why it uses 100% of CPU. I've re-tested it, waiting for CPU usage
to
>> become normal, and got no leaks.
>>
>> Will verify this once again and report more.
>>
>> BTW, if that works, how could I limit inode cache for FUSE client? I do
>> not want it to go beyond 1G, for example, even if I have 48G of RAM on
>> my server.
>
> Its hard-coded for now. For fuse the lru limit (of the inodes which are
> not active) is (32*1024).
This is not exact for current implementation. The inode memory pool is 
configured with 32*1024 entries, but the lru limit is set to infinite: 
currently inode_table_prune() takes lru_limit == 0 as infinite, and the 
inode table created by fuse is initialized with 0.

Anyway this should not be a big problem in normal conditions. After 
having fixed the incorrect nlookup count for "." and ".."
directory
entries, when the kernel detects memory pressure and sends inode 
forgets, the memory will be released.
> One of the ways to address this (which we were discussing earlier) is to
> have an option to configure inode cache limit.
I think this will need more thinking. I've made a fast test forcing 
lru_limit to a small value and weird errors have appeared (probably from 
inodes being expected to exist when kernel sends new requests). Anyway I 
haven't spent time on this. I haven't tested in on master either.

Xavi
> If that sounds good, we
> can then check on if it has to be global/volume-level, client/server/both.
>
> Thanks,
> Soumya
>
>>
>> 01.02.2016 09:54, Soumya Koduri ???????:
>>> On 01/31/2016 03:05 PM, Oleksandr Natalenko wrote:
>>>> Unfortunately, this patch doesn't help.
>>>>
>>>> RAM usage on "find" finish is ~9G.
>>>>
>>>> Here is statedump before drop_caches: https://gist.github.com/
>>>> fc1647de0982ab447e20
>>>
>>> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
>>> size=706766688
>>> num_allocs=2454051
>>>
>>>>
>>>> And after drop_caches:
https://gist.github.com/5eab63bc13f78787ed19
>>>
>>> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
>>> size=550996416
>>> num_allocs=1913182
>>>
>>> There isn't much significant drop in inode contexts. One of the
>>> reasons could be because of dentrys holding a refcount on the
inodes
>>> which shall result in inodes not getting purged even after
>>> fuse_forget.
>>>
>>>
>>> pool-name=fuse:dentry_t
>>> hot-count=32761
>>>
>>> if  '32761' is the current active dentry count, it still
doesn't seem
>>> to match up to inode count.
>>>
>>> Thanks,
>>> Soumya
>>>>
>>>> And here is Valgrind output:
>>>> https://gist.github.com/2490aeac448320d98596
>>>>
>>>> On ??????, 30 ????? 2016 ?. 22:56:37 EET Xavier Hernandez
wrote:
>>>>> There's another inode leak caused by an incorrect
counting of
>>>>> lookups on directory reads.
>>>>>
>>>>> Here's a patch that solves the problem for
>>>>> 3.7:
>>>>>
>>>>> http://review.gluster.org/13324
>>>>>
>>>>> Hopefully with this patch the
>>>>> memory leaks should disapear.
>>>>>
>>>>> Xavi
>>>>>
>>>>> On 29.01.2016 19:09, Oleksandr
>>>>>
>>>>> Natalenko wrote:
>>>>>> Here is intermediate summary of current memory
>>>>>
>>>>> leaks in FUSE client
>>>>>
>>>>>> investigation.
>>>>>>
>>>>>> I use GlusterFS v3.7.6
>>>>>
>>>>> release with the following patches:
>>>>>> ==>>>>>
>>>>>> Kaleb S KEITHLEY (1):
>>>>> fuse: use-after-free fix in fuse-bridge, revisited
>>>>>
>>>>>> Pranith Kumar K
>>>>>
>>>>> (1):
>>>>>> mount/fuse: Fix use-after-free crash
>>>>>
>>>>>> Soumya Koduri (3):
>>>>> gfapi: Fix inode nlookup counts
>>>>>
>>>>>> inode: Retire the inodes from the lru
>>>>>
>>>>> list in inode_table_destroy
>>>>>
>>>>>> upcall: free the xdr* allocations
>>>>>> ==>>>>>>
>>>>>>
>>>>>> With those patches we got API leaks fixed (I hope,
brief tests show
>>>>>
>>>>> that) and
>>>>>
>>>>>> got rid of "kernel notifier loop terminated"
message.
>>>>>
>>>>> Nevertheless, FUSE
>>>>>
>>>>>> client still leaks.
>>>>>>
>>>>>> I have several test
>>>>>
>>>>> volumes with several million of small files (100K?2M in
>>>>>
>>>>>> average). I
>>>>>
>>>>> do 2 types of FUSE client testing:
>>>>>> 1) find /mnt/volume -type d
>>>>>> 2)
>>>>>
>>>>> rsync -av -H /mnt/source_volume/* /mnt/target_volume/
>>>>>
>>>>>> And most
>>>>>
>>>>> up-to-date results are shown below:
>>>>>> === find /mnt/volume -type d
>>>>>
>>>>> ==>>>>>
>>>>>> Memory consumption: ~4G
>>>>>
>>>>>> Statedump:
>>>>> https://gist.github.com/10cde83c63f1b4f1dd7a
>>>>>
>>>>>> Valgrind:
>>>>> https://gist.github.com/097afb01ebb2c5e9e78d
>>>>>
>>>>>> I guess,
>>>>>
>>>>> fuse-bridge/fuse-resolve. related.
>>>>>
>>>>>> === rsync -av -H
>>>>>
>>>>> /mnt/source_volume/* /mnt/target_volume/
==>>>>>
>>>>>> Memory consumption:
>>>>> ~3.3...4G
>>>>>
>>>>>> Statedump (target volume):
>>>>> https://gist.github.com/31e43110eaa4da663435
>>>>>
>>>>>> Valgrind (target volume):
>>>>> https://gist.github.com/f8e0151a6878cacc9b1a
>>>>>
>>>>>> I guess,
>>>>>
>>>>> DHT-related.
>>>>>
>>>>>> Give me more patches to test :).
>>>>>
>>>>> _______________________________________________
>>>>>
>>>>>> Gluster-devel mailing
>>>>>
>>>>> list
>>>>>
>>>>>> Gluster-devel at gluster.org
>>>>>
>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel

Oleksandr Natalenko

2016-Feb-01 18:28 UTC

head link

[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I

Please take a look at updated test results.

Test: find /mnt/volume -type d

RAM usage after "find" finishes: ~ 10.8G (see "ps" output
[1]).

Statedump after "find" finishes: [2].

Then I did drop_caches, and RAM usage dropped to ~4.7G [3].

Statedump after drop_caches: [4].

Here is diff between statedumps: [5].

And, finally, Valgrind output: [6].

Definitely, no major leaks on exit, but why glusterfs process uses almost 5G 
of RAM after drop_caches? Examining statedump shows only the following snippet 
with high "size" value:

==[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
size=4234592647
num_allocs=1
max_size=4294935223
max_num_allocs=3
total_allocs=4186991
==
Another leak?

Grepping "gf_fuse_mt_iov_base" on GlusterFS source tree shows the
following:

==$ grep -Rn gf_fuse_mt_iov_base
xlators/mount/fuse/src/fuse-mem-types.h:20:        gf_fuse_mt_iov_base,
xlators/mount/fuse/src/fuse-bridge.c:4887:
gf_fuse_mt_iov_base);
==
fuse-bridge.c snippet:

==                /* Add extra 128 byte to the first iov so that it can
                 * accommodate "ordinary" non-write requests. It's
not
                 * guaranteed to be big enough, as SETXATTR and namespace
                 * operations with very long names may grow behind it,
                 * but it's good enough in most cases (and we can handle
                 * rest via realloc).
                 */
                iov_in[0].iov_base = GF_CALLOC (1, msg0_size,
                                                gf_fuse_mt_iov_base);
==
Probably, some freeing missing for iov_base?

[1] https://gist.github.com/f0cf98e8bff0c13ea38f
[2] https://gist.github.com/87baa0a778ba54f0f7f7
[3] https://gist.github.com/7013b493d19c8c5fffae
[4] https://gist.github.com/cc38155b57e68d7e86d5
[5] https://gist.github.com/6a24000c77760a97976a
[6] https://gist.github.com/74bd7a9f734c2fd21c33

On ?????????, 1 ?????? 2016 ?. 14:24:22 EET Soumya Koduri
wrote:> On 02/01/2016 01:39 PM, Oleksandr Natalenko wrote:
> > Wait. It seems to be my bad.
> > 
> > Before unmounting I do drop_caches (2), and glusterfs process CPU
usage
> > goes to 100% for a while. I haven't waited for it to drop to 0%,
and
> > instead perform unmount. It seems glusterfs is purging inodes and
that's
> > why it uses 100% of CPU. I've re-tested it, waiting for CPU usage
to
> > become normal, and got no leaks.
> > 
> > Will verify this once again and report more.
> > 
> > BTW, if that works, how could I limit inode cache for FUSE client? I
do
> > not want it to go beyond 1G, for example, even if I have 48G of RAM on
> > my server.
> 
> Its hard-coded for now. For fuse the lru limit (of the inodes which are
> not active) is (32*1024).
> One of the ways to address this (which we were discussing earlier) is to
> have an option to configure inode cache limit. If that sounds good, we
> can then check on if it has to be global/volume-level, client/server/both.
> 
> Thanks,
> Soumya
> 
> > 01.02.2016 09:54, Soumya Koduri ???????:
> >> On 01/31/2016 03:05 PM, Oleksandr Natalenko wrote:
> >>> Unfortunately, this patch doesn't help.
> >>> 
> >>> RAM usage on "find" finish is ~9G.
> >>> 
> >>> Here is statedump before drop_caches: https://gist.github.com/
> >>> fc1647de0982ab447e20
> >> 
> >> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
> >> size=706766688
> >> num_allocs=2454051
> >> 
> >>> And after drop_caches:
https://gist.github.com/5eab63bc13f78787ed19
> >> 
> >> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
> >> size=550996416
> >> num_allocs=1913182
> >> 
> >> There isn't much significant drop in inode contexts. One of
the
> >> reasons could be because of dentrys holding a refcount on the
inodes
> >> which shall result in inodes not getting purged even after
> >> fuse_forget.
> >> 
> >> 
> >> pool-name=fuse:dentry_t
> >> hot-count=32761
> >> 
> >> if  '32761' is the current active dentry count, it still
doesn't seem
> >> to match up to inode count.
> >> 
> >> Thanks,
> >> Soumya
> >> 
> >>> And here is Valgrind output:
> >>> https://gist.github.com/2490aeac448320d98596
> >>> 
> >>> On ??????, 30 ????? 2016 ?. 22:56:37 EET Xavier Hernandez
wrote:
> >>>> There's another inode leak caused by an incorrect
counting of
> >>>> lookups on directory reads.
> >>>> 
> >>>> Here's a patch that solves the problem for
> >>>> 3.7:
> >>>> 
> >>>> http://review.gluster.org/13324
> >>>> 
> >>>> Hopefully with this patch the
> >>>> memory leaks should disapear.
> >>>> 
> >>>> Xavi
> >>>> 
> >>>> On 29.01.2016 19:09, Oleksandr
> >>>> 
> >>>> Natalenko wrote:
> >>>>> Here is intermediate summary of current memory
> >>>> 
> >>>> leaks in FUSE client
> >>>> 
> >>>>> investigation.
> >>>>> 
> >>>>> I use GlusterFS v3.7.6
> >>>> 
> >>>> release with the following patches:
> >>>>> ==> >>>> 
> >>>>> Kaleb S KEITHLEY (1):
> >>>> fuse: use-after-free fix in fuse-bridge, revisited
> >>>> 
> >>>>> Pranith Kumar K
> >>>> 
> >>>> (1):
> >>>>> mount/fuse: Fix use-after-free crash
> >>>> 
> >>>>> Soumya Koduri (3):
> >>>> gfapi: Fix inode nlookup counts
> >>>> 
> >>>>> inode: Retire the inodes from the lru
> >>>> 
> >>>> list in inode_table_destroy
> >>>> 
> >>>>> upcall: free the xdr* allocations
> >>>>> ==> >>>>> 
> >>>>> 
> >>>>> With those patches we got API leaks fixed (I hope,
brief tests show
> >>>> 
> >>>> that) and
> >>>> 
> >>>>> got rid of "kernel notifier loop terminated"
message.
> >>>> 
> >>>> Nevertheless, FUSE
> >>>> 
> >>>>> client still leaks.
> >>>>> 
> >>>>> I have several test
> >>>> 
> >>>> volumes with several million of small files (100K?2M in
> >>>> 
> >>>>> average). I
> >>>> 
> >>>> do 2 types of FUSE client testing:
> >>>>> 1) find /mnt/volume -type d
> >>>>> 2)
> >>>> 
> >>>> rsync -av -H /mnt/source_volume/* /mnt/target_volume/
> >>>> 
> >>>>> And most
> >>>> 
> >>>> up-to-date results are shown below:
> >>>>> === find /mnt/volume -type d
> >>>> 
> >>>> ==> >>>> 
> >>>>> Memory consumption: ~4G
> >>>> 
> >>>>> Statedump:
> >>>> https://gist.github.com/10cde83c63f1b4f1dd7a
> >>>> 
> >>>>> Valgrind:
> >>>> https://gist.github.com/097afb01ebb2c5e9e78d
> >>>> 
> >>>>> I guess,
> >>>> 
> >>>> fuse-bridge/fuse-resolve. related.
> >>>> 
> >>>>> === rsync -av -H
> >>>> 
> >>>> /mnt/source_volume/* /mnt/target_volume/ ==>
>>>>
> >>>>> Memory consumption:
> >>>> ~3.3...4G
> >>>> 
> >>>>> Statedump (target volume):
> >>>> https://gist.github.com/31e43110eaa4da663435
> >>>> 
> >>>>> Valgrind (target volume):
> >>>> https://gist.github.com/f8e0151a6878cacc9b1a
> >>>> 
> >>>>> I guess,
> >>>> 
> >>>> DHT-related.
> >>>> 
> >>>>> Give me more patches to test :).
> >>>> 
> >>>> _______________________________________________
> >>>> 
> >>>>> Gluster-devel mailing
> >>>> 
> >>>> list
> >>>> 
> >>>>> Gluster-devel at gluster.org
> >>>> 
> >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> >>> 
> >>> _______________________________________________
> >>> Gluster-devel mailing list
> >>> Gluster-devel at gluster.org
> >>> http://www.gluster.org/mailman/listinfo/gluster-devel

Raghavendra G

2016-Feb-04 08:46 UTC

head link

[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I

On Mon, Feb 1, 2016 at 2:24 PM, Soumya Koduri <skoduri at redhat.com>
wrote:
>
>
> On 02/01/2016 01:39 PM, Oleksandr Natalenko wrote:
>
>> Wait. It seems to be my bad.
>>
>> Before unmounting I do drop_caches (2), and glusterfs process CPU usage
>> goes to 100% for a while. I haven't waited for it to drop to 0%,
and
>> instead perform unmount. It seems glusterfs is purging inodes and
that's
>> why it uses 100% of CPU. I've re-tested it, waiting for CPU usage
to
>> become normal, and got no leaks.
>>
>> Will verify this once again and report more.
>>
>> BTW, if that works, how could I limit inode cache for FUSE client? I do
>> not want it to go beyond 1G, for example, even if I have 48G of RAM on
>> my server.
>>
>
> Its hard-coded for now. For fuse the lru limit (of the inodes which are
> not active) is (32*1024).
> One of the ways to address this (which we were discussing earlier) is to
> have an option to configure inode cache limit.

We cannot set a limit on inode cache in fuse-bridge. As long as kernel is
aware of an inode (through a lookup), fuse-client is _forced_ to keep that
inode in inode table. The reason is we pass the address of inode object as
nodeid to kernel. We cannot send a gfid as nodeid since gfid is 128 bit and
nodeid is 64 bit. This is the reason behind setting an infinite lru limit.
However, this problem is not there for inode table management in server as,
client can communicate with server using 128 bit gfids.

> If that sounds good, we can then check on if it has to be
> global/volume-level, client/server/both.
>
> Thanks,
> Soumya
>
>
>
>> 01.02.2016 09:54, Soumya Koduri ???????:
>>
>>> On 01/31/2016 03:05 PM, Oleksandr Natalenko wrote:
>>>
>>>> Unfortunately, this patch doesn't help.
>>>>
>>>> RAM usage on "find" finish is ~9G.
>>>>
>>>> Here is statedump before drop_caches: https://gist.github.com/
>>>> fc1647de0982ab447e20
>>>>
>>>
>>> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
>>> size=706766688
>>> num_allocs=2454051
>>>
>>>
>>>> And after drop_caches:
https://gist.github.com/5eab63bc13f78787ed19
>>>>
>>>
>>> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
>>> size=550996416
>>> num_allocs=1913182
>>>
>>> There isn't much significant drop in inode contexts. One of the
>>> reasons could be because of dentrys holding a refcount on the
inodes
>>> which shall result in inodes not getting purged even after
>>> fuse_forget.
>>>
>>>
>>> pool-name=fuse:dentry_t
>>> hot-count=32761
>>>
>>> if  '32761' is the current active dentry count, it still
doesn't seem
>>> to match up to inode count.
>>>
>>> Thanks,
>>> Soumya
>>>
>>>>
>>>> And here is Valgrind output:
>>>> https://gist.github.com/2490aeac448320d98596
>>>>
>>>> On ??????, 30 ????? 2016 ?. 22:56:37 EET Xavier Hernandez
wrote:
>>>>
>>>>> There's another inode leak caused by an incorrect
counting of
>>>>> lookups on directory reads.
>>>>>
>>>>> Here's a patch that solves the problem for
>>>>> 3.7:
>>>>>
>>>>> http://review.gluster.org/13324
>>>>>
>>>>> Hopefully with this patch the
>>>>> memory leaks should disapear.
>>>>>
>>>>> Xavi
>>>>>
>>>>> On 29.01.2016 19:09, Oleksandr
>>>>>
>>>>> Natalenko wrote:
>>>>>
>>>>>> Here is intermediate summary of current memory
>>>>>>
>>>>>
>>>>> leaks in FUSE client
>>>>>
>>>>> investigation.
>>>>>>
>>>>>> I use GlusterFS v3.7.6
>>>>>>
>>>>>
>>>>> release with the following patches:
>>>>>
>>>>>> ==>>>>>>
>>>>>
>>>>> Kaleb S KEITHLEY (1):
>>>>>>
>>>>> fuse: use-after-free fix in fuse-bridge, revisited
>>>>>
>>>>> Pranith Kumar K
>>>>>>
>>>>>
>>>>> (1):
>>>>>
>>>>>> mount/fuse: Fix use-after-free crash
>>>>>>
>>>>>
>>>>> Soumya Koduri (3):
>>>>>>
>>>>> gfapi: Fix inode nlookup counts
>>>>>
>>>>> inode: Retire the inodes from the lru
>>>>>>
>>>>>
>>>>> list in inode_table_destroy
>>>>>
>>>>> upcall: free the xdr* allocations
>>>>>> ==>>>>>>
>>>>>>
>>>>>> With those patches we got API leaks fixed (I hope,
brief tests show
>>>>>>
>>>>>
>>>>> that) and
>>>>>
>>>>> got rid of "kernel notifier loop terminated"
message.
>>>>>>
>>>>>
>>>>> Nevertheless, FUSE
>>>>>
>>>>> client still leaks.
>>>>>>
>>>>>> I have several test
>>>>>>
>>>>>
>>>>> volumes with several million of small files (100K?2M in
>>>>>
>>>>> average). I
>>>>>>
>>>>>
>>>>> do 2 types of FUSE client testing:
>>>>>
>>>>>> 1) find /mnt/volume -type d
>>>>>> 2)
>>>>>>
>>>>>
>>>>> rsync -av -H /mnt/source_volume/* /mnt/target_volume/
>>>>>
>>>>> And most
>>>>>>
>>>>>
>>>>> up-to-date results are shown below:
>>>>>
>>>>>> === find /mnt/volume -type d
>>>>>>
>>>>>
>>>>> ==>>>>>
>>>>> Memory consumption: ~4G
>>>>>>
>>>>>
>>>>> Statedump:
>>>>>>
>>>>> https://gist.github.com/10cde83c63f1b4f1dd7a
>>>>>
>>>>> Valgrind:
>>>>>>
>>>>> https://gist.github.com/097afb01ebb2c5e9e78d
>>>>>
>>>>> I guess,
>>>>>>
>>>>>
>>>>> fuse-bridge/fuse-resolve. related.
>>>>>
>>>>> === rsync -av -H
>>>>>>
>>>>>
>>>>> /mnt/source_volume/* /mnt/target_volume/
==>>>>>
>>>>> Memory consumption:
>>>>>>
>>>>> ~3.3...4G
>>>>>
>>>>> Statedump (target volume):
>>>>>>
>>>>> https://gist.github.com/31e43110eaa4da663435
>>>>>
>>>>> Valgrind (target volume):
>>>>>>
>>>>> https://gist.github.com/f8e0151a6878cacc9b1a
>>>>>
>>>>> I guess,
>>>>>>
>>>>>
>>>>> DHT-related.
>>>>>
>>>>> Give me more patches to test :).
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>>
>>>>> Gluster-devel mailing
>>>>>>
>>>>>
>>>>> list
>>>>>
>>>>> Gluster-devel at gluster.org
>>>>>>
>>>>>
>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>


-- 
Raghavendra G
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160204/4107c264/attachment.html>

Gluster users - Feb 2016 - [Gluster-devel] GlusterFS FUSE client leaks summary — part I

[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I

[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I

[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I

[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I