Oleksandr Natalenko
2016-Jan-29 18:09 UTC
[Gluster-users] GlusterFS FUSE client leaks summary — part I
Here is intermediate summary of current memory leaks in FUSE client investigation. I use GlusterFS v3.7.6 release with the following patches: ==Kaleb S KEITHLEY (1): fuse: use-after-free fix in fuse-bridge, revisited Pranith Kumar K (1): mount/fuse: Fix use-after-free crash Soumya Koduri (3): gfapi: Fix inode nlookup counts inode: Retire the inodes from the lru list in inode_table_destroy upcall: free the xdr* allocations == With those patches we got API leaks fixed (I hope, brief tests show that) and got rid of "kernel notifier loop terminated" message. Nevertheless, FUSE client still leaks. I have several test volumes with several million of small files (100K?2M in average). I do 2 types of FUSE client testing: 1) find /mnt/volume -type d 2) rsync -av -H /mnt/source_volume/* /mnt/target_volume/ And most up-to-date results are shown below: === find /mnt/volume -type d == Memory consumption: ~4G Statedump: https://gist.github.com/10cde83c63f1b4f1dd7a Valgrind: https://gist.github.com/097afb01ebb2c5e9e78d I guess, fuse-bridge/fuse-resolve. related. === rsync -av -H /mnt/source_volume/* /mnt/target_volume/ == Memory consumption: ~3.3...4G Statedump (target volume): https://gist.github.com/31e43110eaa4da663435 Valgrind (target volume): https://gist.github.com/f8e0151a6878cacc9b1a I guess, DHT-related. Give me more patches to test :).
Vijay Bellur
2016-Jan-29 21:11 UTC
[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I
On 01/29/2016 01:09 PM, Oleksandr Natalenko wrote:> Here is intermediate summary of current memory leaks in FUSE client > investigation. > > I use GlusterFS v3.7.6 release with the following patches: > > ==> Kaleb S KEITHLEY (1): > fuse: use-after-free fix in fuse-bridge, revisited > > Pranith Kumar K (1): > mount/fuse: Fix use-after-free crash > > Soumya Koduri (3): > gfapi: Fix inode nlookup counts > inode: Retire the inodes from the lru list in inode_table_destroy > upcall: free the xdr* allocations > ==> > With those patches we got API leaks fixed (I hope, brief tests show that) and > got rid of "kernel notifier loop terminated" message. Nevertheless, FUSE > client still leaks. > > I have several test volumes with several million of small files (100K?2M in > average). I do 2 types of FUSE client testing: > > 1) find /mnt/volume -type d > 2) rsync -av -H /mnt/source_volume/* /mnt/target_volume/ > > And most up-to-date results are shown below: > > === find /mnt/volume -type d ==> > Memory consumption: ~4G > Statedump: https://gist.github.com/10cde83c63f1b4f1dd7a > Valgrind: https://gist.github.com/097afb01ebb2c5e9e78d > > I guess, fuse-bridge/fuse-resolve. related. > > === rsync -av -H /mnt/source_volume/* /mnt/target_volume/ ==> > Memory consumption: ~3.3...4G > Statedump (target volume): https://gist.github.com/31e43110eaa4da663435 > Valgrind (target volume): https://gist.github.com/f8e0151a6878cacc9b1a > > I guess, DHT-related. > > Give me more patches to test :).Thank you as ever for your detailed reports! This patch should help the dht leaks observed as part of dht_do_rename() in valgrind logs of target volume. http://review.gluster.org/#/c/13322/ Can you please verify if this indeed helps? Regards, Vijay
Xavier Hernandez
2016-Jan-30 21:56 UTC
[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I
There's another inode leak caused by an incorrect counting of lookups on directory reads. Here's a patch that solves the problem for 3.7: http://review.gluster.org/13324 Hopefully with this patch the memory leaks should disapear. Xavi On 29.01.2016 19:09, Oleksandr Natalenko wrote:> Here is intermediate summary of current memoryleaks in FUSE client> investigation. > > I use GlusterFS v3.7.6release with the following patches:> > ==> Kaleb S KEITHLEY (1): >fuse: use-after-free fix in fuse-bridge, revisited> > Pranith Kumar K(1):> mount/fuse: Fix use-after-free crash > > Soumya Koduri (3): >gfapi: Fix inode nlookup counts> inode: Retire the inodes from the lrulist in inode_table_destroy> upcall: free the xdr* allocations > ==>> With those patches we got API leaks fixed (I hope, brief tests showthat) and> got rid of "kernel notifier loop terminated" message.Nevertheless, FUSE> client still leaks. > > I have several testvolumes with several million of small files (100K?2M in> average). Ido 2 types of FUSE client testing:> > 1) find /mnt/volume -type d > 2)rsync -av -H /mnt/source_volume/* /mnt/target_volume/> > And mostup-to-date results are shown below:> > === find /mnt/volume -type d==>> Memory consumption: ~4G > Statedump:https://gist.github.com/10cde83c63f1b4f1dd7a> Valgrind:https://gist.github.com/097afb01ebb2c5e9e78d> > I guess,fuse-bridge/fuse-resolve. related.> > === rsync -av -H/mnt/source_volume/* /mnt/target_volume/ ==>> Memory consumption:~3.3...4G> Statedump (target volume):https://gist.github.com/31e43110eaa4da663435> Valgrind (target volume):https://gist.github.com/f8e0151a6878cacc9b1a> > I guess,DHT-related.> > Give me more patches to test :). >_______________________________________________> Gluster-devel mailinglist> Gluster-devel at gluster.org >http://www.gluster.org/mailman/listinfo/gluster-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160130/c1b51822/attachment.html>
Oleksandr Natalenko
2016-Feb-03 07:37 UTC
[Gluster-users] [Gluster-devel] GlusterFS FUSE client leaks summary — part I
Here goes the report on DHT-related leaks patch ("rsync" test). RAM usage before drop_caches: [1] Statedump before drop_caches: [2] RAM usage after drop_caches: [3] Statedump after drop_caches: [4] Statedumps diff: [5] Valgrind output: [6] [1] https://gist.github.com/ca8d56834c14c4bfa98e [2] https://gist.github.com/06dc910d7261750d486c [3] https://gist.github.com/c482b170848a21b6e5f3 [4] https://gist.github.com/ed7f56336b4cbf39f7e8 [5] https://gist.github.com/f8597f34b56d949f7dcb [6] https://gist.github.com/102fc2d2dfa2d2d179fa I guess, the patch works. 29.01.2016 23:11, Vijay Bellur ???????:> On 01/29/2016 01:09 PM, Oleksandr Natalenko wrote: >> Here is intermediate summary of current memory leaks in FUSE client >> investigation. >> >> I use GlusterFS v3.7.6 release with the following patches: >> >> ==>> Kaleb S KEITHLEY (1): >> fuse: use-after-free fix in fuse-bridge, revisited >> >> Pranith Kumar K (1): >> mount/fuse: Fix use-after-free crash >> >> Soumya Koduri (3): >> gfapi: Fix inode nlookup counts >> inode: Retire the inodes from the lru list in >> inode_table_destroy >> upcall: free the xdr* allocations >> ==>> >> With those patches we got API leaks fixed (I hope, brief tests show >> that) and >> got rid of "kernel notifier loop terminated" message. Nevertheless, >> FUSE >> client still leaks. >> >> I have several test volumes with several million of small files >> (100K?2M in >> average). I do 2 types of FUSE client testing: >> >> 1) find /mnt/volume -type d >> 2) rsync -av -H /mnt/source_volume/* /mnt/target_volume/ >> >> And most up-to-date results are shown below: >> >> === find /mnt/volume -type d ==>> >> Memory consumption: ~4G >> Statedump: https://gist.github.com/10cde83c63f1b4f1dd7a >> Valgrind: https://gist.github.com/097afb01ebb2c5e9e78d >> >> I guess, fuse-bridge/fuse-resolve. related. >> >> === rsync -av -H /mnt/source_volume/* /mnt/target_volume/ ==>> >> Memory consumption: ~3.3...4G >> Statedump (target volume): >> https://gist.github.com/31e43110eaa4da663435 >> Valgrind (target volume): https://gist.github.com/f8e0151a6878cacc9b1a >> >> I guess, DHT-related. >> >> Give me more patches to test :). > > Thank you as ever for your detailed reports! > > This patch should help the dht leaks observed as part of > dht_do_rename() in valgrind logs of target volume. > > http://review.gluster.org/#/c/13322/ > > Can you please verify if this indeed helps? > > Regards, > Vijay