[Gluster-users] Fuse memleaks, all versions

Yannick Perret
Tue, 02 Aug 2016 09:22:20 -0000

Le 02/08/2016 ? 05:11, Pranith Kumar Karampuri a ?crit :
>
>
> On Mon, Aug 1, 2016 at 3:40 PM, Yannick Perret
> <yannick.perret at liris.cnrs.fr <mailto:yannick.perret at
liris.cnrs.fr>>
> wrote:
>
>     Le 29/07/2016 ? 18:39, Pranith Kumar Karampuri a ?crit :
>>
>>
>>     On Fri, Jul 29, 2016 at 2:26 PM, Yannick Perret
>>     <yannick.perret at liris.cnrs.fr
>>     <mailto:yannick.perret at liris.cnrs.fr>> wrote:
>>
>>         Ok, last try:
>>         after investigating more versions I found that FUSE client
>>         leaks memory on all of them.
>>         I tested:
>>         - 3.6.7 client on debian 7 32bit and on debian 8 64bit (with
>>         3.6.7 serveurs on debian 8 64bit)
>>         - 3.6.9 client on debian 7 32bit and on debian 8 64bit (with
>>         3.6.7 serveurs on debian 8 64bit)
>>         - 3.7.13 client on debian 8 64bit (with 3.8.1 serveurs on
>>         debian 8 64bit)
>>         - 3.8.1 client on debian 8 64bit (with 3.8.1 serveurs on
>>         debian 8 64bit)
>>         In all cases compiled from sources, appart for 3.8.1 where
>>         .deb were used (due to a configure runtime error).
>>         For 3.7 it was compiled with --disable-tiering. I also tried
>>         to compile with --disable-fusermount (no change).
>>
>>         In all of these cases the memory (resident & virtual) of
>>         glusterfs process on client grows on each activity and never
>>         reach a max (and never reduce).
>>         "Activity" for these tests is cp -Rp and ls -lR.
>>         The client I let grows the most overreached ~4Go RAM. On
>>         smaller machines it ends by OOM killer killing glusterfs
>>         process or glusterfs dying due to allocation error.
>>
>>         In 3.6 mem seems to grow continusly, whereas in 3.8.1 it
>>         grows by "steps" (430400 ko ? 629144 (~1min) ? 762324
(~1min)
>>         ? 827860?).
>>
>>         All tests performed on a single test volume used only by my
>>         test client. Volume in a basic x2 replica. The only
>>         parameters I changed on this volume (without any effect) are
>>         diagnostics.client-log-level set to ERROR and
>>         network.inode-lru-limit set to 1024.
>>
>>
>>     Could you attach statedumps of your runs?
>>     The following link has steps to capture
>>
this(https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/
>>     ). We basically need to see what are the memory types that are
>>     increasing. If you could help find the issue, we can send the
>>     fixes for your workload. There is a 3.8.2 release in around 10
>>     days I think. We can probably target this issue for that?
>     Here are statedumps.
>     Steps:
>     1. mount -t glusterfs ldap1.my.domain:SHARE /root/MNT/ (here VSZ
>     and RSS are 381896 35828)
>     2. take a dump with kill -USR1 <pid-of-glusterfs-process> (file
>     glusterdump.n1.dump.1470042769)
>     3. perform a 'ls -lR /root/MNT | wc -l' (btw result of wc -l is
>     518396 :)) and a 'cp -Rp /usr/* /root/MNT/boo' (VSZ/RSS are
>     1301536/711992 at end of these operations)
>     4. take a dump with kill -USR1 <pid-of-glusterfs-process> (file
>     glusterdump.n2.dump.1470043929)
>     5. do 'cp -Rp * /root/MNT/toto/', so on an other directory
>     (VSZ/RSS are 1432608/909968 at end of this operation)
>     6. take a dump with kill -USR1 <pid-of-glusterfs-process> (file
>     glusterdump.n3.dump.)
>
>
> Hey,
>       Thanks a lot for providing this information. Looking at these
> steps, I don't see any problem for the increase in memory. Both ls -lR
> and cp -Rp commands you did in the step-3 will add new inodes in
> memory which increase the memory. What happens is as long as the
> kernel thinks these inodes need to be in memory gluster keeps them in
> memory. Once kernel doesn't think the inode is necessary, it sends
> 'inode-forgets'. At this point the memory starts reducing. So it
kind
> of depends on the memory pressure kernel is under. But you said it
> lead to OOM-killers on smaller machines which means there could be
> some leaks. Could you modify the steps as follows to check to confirm
> there are leaks? Please do this test on those smaller machines which
> lead to OOM-killers.
>
Thanks for your feedback. I will send these statedumps today.
--
Y.

> Steps:
> 1. mount -t glusterfs ldap1.my.domain:SHARE /root/MNT/ (here VSZ and
> RSS are 381896 35828)
> 2. perform a 'ls -lR /root/MNT | wc -l' (btw result of wc -l is
518396
> :)) and a 'cp -Rp /usr/* /root/MNT/boo' (VSZ/RSS are 1301536/711992
at
> end of these operations)
> 3. do 'cp -Rp * /root/MNT/toto/', so on an other directory (VSZ/RSS
> are 1432608/909968 at end of this operation)
> 4. Delete all the files and directories you created in steps 2, 3 above
> 5. Take statedump with kill -USR1 <pid-of-glusterfs-process>
> 6. Repeat steps from 2-5
>
> Attach these two statedumps. I think the statedumps will be even more
> affective if the mount does not have any data when you start the
> experiment.
>
> HTH
>
>
>     Dump files are gzip'ed because they are very large.
>     Dump files are here (too big for email):
>     http://wikisend.com/download/623430/glusterdump.n1.dump.1470042769.gz
>     http://wikisend.com/download/771220/glusterdump.n2.dump.1470043929.gz
>     http://wikisend.com/download/428752/glusterdump.n3.dump.1470045181.gz
>     (I keep the files if someone whats them in an other format)
>
>     Client and servers are installed from .deb files
>     (glusterfs-client_3.8.1-1_amd64.deb and
>     glusterfs-common_3.8.1-1_amd64.deb on client side).
>     They are all Debian 8 64bit. Servers are test machines that serve
>     only one volume to this sole client. Volume is a simple x2
>     replica. I just changed for test network.inode-lru-limit value to
>     1024. Mount point /root/MNT is only used for these tests.
>
>     --
>     Y.
>
>
>
>
>
> --
> Pranith


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160802/4ca5f786/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3369 bytes
Desc: Signature cryptographique S/MIME
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160802/4ca5f786/attachment.p7s>