Oleksandr Natalenko
2016-Jan-29 18:09 UTC
[Gluster-users] GlusterFS FUSE client leaks summary — part I
Here is intermediate summary of current memory leaks in FUSE client
investigation.
I use GlusterFS v3.7.6 release with the following patches:
==Kaleb S KEITHLEY (1):
fuse: use-after-free fix in fuse-bridge, revisited
Pranith Kumar K (1):
mount/fuse: Fix use-after-free crash
Soumya Koduri (3):
gfapi: Fix inode nlookup counts
inode: Retire the inodes from the lru list in inode_table_destroy
upcall: free the xdr* allocations
==
With those patches we got API leaks fixed (I hope, brief tests show that) and
got rid of "kernel notifier loop terminated" message. Nevertheless,
FUSE
client still leaks.
I have several test volumes with several million of small files (100K?2M in
average). I do 2 types of FUSE client testing:
1) find /mnt/volume -type d
2) rsync -av -H /mnt/source_volume/* /mnt/target_volume/
And most up-to-date results are shown below:
=== find /mnt/volume -type d ==
Memory consumption: ~4G
Statedump: https://gist.github.com/10cde83c63f1b4f1dd7a
Valgrind: https://gist.github.com/097afb01ebb2c5e9e78d
I guess, fuse-bridge/fuse-resolve. related.
=== rsync -av -H /mnt/source_volume/* /mnt/target_volume/ ==
Memory consumption: ~3.3...4G
Statedump (target volume): https://gist.github.com/31e43110eaa4da663435
Valgrind (target volume): https://gist.github.com/f8e0151a6878cacc9b1a
I guess, DHT-related.
Give me more patches to test :).
Vijay Bellur
2016-Jan-29 21:11 UTC
[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I
On 01/29/2016 01:09 PM, Oleksandr Natalenko wrote:> Here is intermediate summary of current memory leaks in FUSE client > investigation. > > I use GlusterFS v3.7.6 release with the following patches: > > ==> Kaleb S KEITHLEY (1): > fuse: use-after-free fix in fuse-bridge, revisited > > Pranith Kumar K (1): > mount/fuse: Fix use-after-free crash > > Soumya Koduri (3): > gfapi: Fix inode nlookup counts > inode: Retire the inodes from the lru list in inode_table_destroy > upcall: free the xdr* allocations > ==> > With those patches we got API leaks fixed (I hope, brief tests show that) and > got rid of "kernel notifier loop terminated" message. Nevertheless, FUSE > client still leaks. > > I have several test volumes with several million of small files (100K?2M in > average). I do 2 types of FUSE client testing: > > 1) find /mnt/volume -type d > 2) rsync -av -H /mnt/source_volume/* /mnt/target_volume/ > > And most up-to-date results are shown below: > > === find /mnt/volume -type d ==> > Memory consumption: ~4G > Statedump: https://gist.github.com/10cde83c63f1b4f1dd7a > Valgrind: https://gist.github.com/097afb01ebb2c5e9e78d > > I guess, fuse-bridge/fuse-resolve. related. > > === rsync -av -H /mnt/source_volume/* /mnt/target_volume/ ==> > Memory consumption: ~3.3...4G > Statedump (target volume): https://gist.github.com/31e43110eaa4da663435 > Valgrind (target volume): https://gist.github.com/f8e0151a6878cacc9b1a > > I guess, DHT-related. > > Give me more patches to test :).Thank you as ever for your detailed reports! This patch should help the dht leaks observed as part of dht_do_rename() in valgrind logs of target volume. http://review.gluster.org/#/c/13322/ Can you please verify if this indeed helps? Regards, Vijay
Xavier Hernandez
2016-Jan-30 21:56 UTC
[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I
There's another inode leak caused by an incorrect counting of lookups on directory reads. Here's a patch that solves the problem for 3.7: http://review.gluster.org/13324 Hopefully with this patch the memory leaks should disapear. Xavi On 29.01.2016 19:09, Oleksandr Natalenko wrote:> Here is intermediate summary of current memoryleaks in FUSE client> investigation. > > I use GlusterFS v3.7.6release with the following patches:> > ==> Kaleb S KEITHLEY (1): >fuse: use-after-free fix in fuse-bridge, revisited> > Pranith Kumar K(1):> mount/fuse: Fix use-after-free crash > > Soumya Koduri (3): >gfapi: Fix inode nlookup counts> inode: Retire the inodes from the lrulist in inode_table_destroy> upcall: free the xdr* allocations > ==>> With those patches we got API leaks fixed (I hope, brief tests showthat) and> got rid of "kernel notifier loop terminated" message.Nevertheless, FUSE> client still leaks. > > I have several testvolumes with several million of small files (100K?2M in> average). Ido 2 types of FUSE client testing:> > 1) find /mnt/volume -type d > 2)rsync -av -H /mnt/source_volume/* /mnt/target_volume/> > And mostup-to-date results are shown below:> > === find /mnt/volume -type d==>> Memory consumption: ~4G > Statedump:https://gist.github.com/10cde83c63f1b4f1dd7a> Valgrind:https://gist.github.com/097afb01ebb2c5e9e78d> > I guess,fuse-bridge/fuse-resolve. related.> > === rsync -av -H/mnt/source_volume/* /mnt/target_volume/ ==>> Memory consumption:~3.3...4G> Statedump (target volume):https://gist.github.com/31e43110eaa4da663435> Valgrind (target volume):https://gist.github.com/f8e0151a6878cacc9b1a> > I guess,DHT-related.> > Give me more patches to test :). >_______________________________________________> Gluster-devel mailinglist> Gluster-devel at gluster.org >http://www.gluster.org/mailman/listinfo/gluster-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160130/c1b51822/attachment.html>
Oleksandr Natalenko
2016-Feb-03 07:37 UTC
[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I
Here goes the report on DHT-related leaks patch ("rsync" test).
RAM usage before drop_caches: [1]
Statedump before drop_caches: [2]
RAM usage after drop_caches: [3]
Statedump after drop_caches: [4]
Statedumps diff: [5]
Valgrind output: [6]
[1] https://gist.github.com/ca8d56834c14c4bfa98e
[2] https://gist.github.com/06dc910d7261750d486c
[3] https://gist.github.com/c482b170848a21b6e5f3
[4] https://gist.github.com/ed7f56336b4cbf39f7e8
[5] https://gist.github.com/f8597f34b56d949f7dcb
[6] https://gist.github.com/102fc2d2dfa2d2d179fa
I guess, the patch works.
29.01.2016 23:11, Vijay Bellur ???????:> On 01/29/2016 01:09 PM, Oleksandr Natalenko wrote:
>> Here is intermediate summary of current memory leaks in FUSE client
>> investigation.
>>
>> I use GlusterFS v3.7.6 release with the following patches:
>>
>> ==>> Kaleb S KEITHLEY (1):
>> fuse: use-after-free fix in fuse-bridge, revisited
>>
>> Pranith Kumar K (1):
>> mount/fuse: Fix use-after-free crash
>>
>> Soumya Koduri (3):
>> gfapi: Fix inode nlookup counts
>> inode: Retire the inodes from the lru list in
>> inode_table_destroy
>> upcall: free the xdr* allocations
>> ==>>
>> With those patches we got API leaks fixed (I hope, brief tests show
>> that) and
>> got rid of "kernel notifier loop terminated" message.
Nevertheless,
>> FUSE
>> client still leaks.
>>
>> I have several test volumes with several million of small files
>> (100K?2M in
>> average). I do 2 types of FUSE client testing:
>>
>> 1) find /mnt/volume -type d
>> 2) rsync -av -H /mnt/source_volume/* /mnt/target_volume/
>>
>> And most up-to-date results are shown below:
>>
>> === find /mnt/volume -type d ==>>
>> Memory consumption: ~4G
>> Statedump: https://gist.github.com/10cde83c63f1b4f1dd7a
>> Valgrind: https://gist.github.com/097afb01ebb2c5e9e78d
>>
>> I guess, fuse-bridge/fuse-resolve. related.
>>
>> === rsync -av -H /mnt/source_volume/* /mnt/target_volume/ ==>>
>> Memory consumption: ~3.3...4G
>> Statedump (target volume):
>> https://gist.github.com/31e43110eaa4da663435
>> Valgrind (target volume): https://gist.github.com/f8e0151a6878cacc9b1a
>>
>> I guess, DHT-related.
>>
>> Give me more patches to test :).
>
> Thank you as ever for your detailed reports!
>
> This patch should help the dht leaks observed as part of
> dht_do_rename() in valgrind logs of target volume.
>
> http://review.gluster.org/#/c/13322/
>
> Can you please verify if this indeed helps?
>
> Regards,
> Vijay