Gluster users - Mar 2010 - Old story - glusterfs memory usage

I upgraded today from 2.0.8 to 3.0.3.
The first sad impression was that unify no longer works, but it is not that
important. Not a surprise. Switched to dht. Tried my old "du" test:
du /home
(/home is a glusterfs filesystem, of course ;) and saw again the bad old
unlimited increase of glusterfs clients memory usage. I hoped that the
problem went to quick-read, but it didn't.
So switched back to 2.0.10rc1, to restore unify.
And then did a simple test, to be sure that no translators are to be blamed.
I created the following, trivial setup:

file loopback.vol:
volume loopback
 type storage/posix
 option directory /root/loopback
end-volume

And then mounted /root/loopback on /root/loop-test:

mount -t glusterfs loopback.vol /root/loop-test

Do you see? No translators, even no server and networking. Just a loopback
to a directory on the local machine.

I copied then the /usr directory to /root/loop-test (c.a. 160000 files).
And then ran "du /root/loop-test".
Memory usage of respective glusterfs process went up from 16 MB to 50 MB.
I could reproduce it perfectly by umounting /root/loop-test, mounting it
again and re-running "du".
More files touched mean more memory used by glusterfs.
This is not a memory leak. Repeating this "du" does not cause memory
usage go even a single byte up. Glusterfs client keeps somewhere an 
information about _every file touched_, and keeps it _forever_. As the situation
did not improve since the times of glusterfs 1.3, I assume this behavior 
to be a part of its design. IMHO a wrong one. This is not a feature,
this is a bug. It costs memory and probably performace too, if it is a kind
of cache (with millions of entries, potentially).
I filed a bug report almost one year ago - no change. Dear other users of
glusterfs - are you at least able to reproduce this memory consumption
problem? If it occurs on my site only, then the reason is somewhere in my
system setup, otherwise - let us do something about it, finally.
It is not in translators, it is in the core of glusterfs.
With regards
Krzysztof Strasburger

Hi,

Does your mount point has large number of files? Did you try droping caches
(using echo 3 > /proc/sys/vm/drop_caches) after doing du? If yes, what were
the results?

regards,
On Thu, Mar 25, 2010 at 2:38 PM, Krzysztof Strasburger <
strasbur at chkw386.ch.pwr.wroc.pl> wrote:
> I upgraded today from 2.0.8 to 3.0.3.
> The first sad impression was that unify no longer works, but it is not that
> important. Not a surprise. Switched to dht. Tried my old "du"
test:
> du /home
> (/home is a glusterfs filesystem, of course ;) and saw again the bad old
> unlimited increase of glusterfs clients memory usage. I hoped that the
> problem went to quick-read, but it didn't.
> So switched back to 2.0.10rc1, to restore unify.
> And then did a simple test, to be sure that no translators are to be
> blamed.
> I created the following, trivial setup:
>
> file loopback.vol:
> volume loopback
>  type storage/posix
>  option directory /root/loopback
> end-volume
>
> And then mounted /root/loopback on /root/loop-test:
>
> mount -t glusterfs loopback.vol /root/loop-test
>
> Do you see? No translators, even no server and networking. Just a loopback
> to a directory on the local machine.
>
> I copied then the /usr directory to /root/loop-test (c.a. 160000 files).
> And then ran "du /root/loop-test".
> Memory usage of respective glusterfs process went up from 16 MB to 50 MB.
> I could reproduce it perfectly by umounting /root/loop-test, mounting it
> again and re-running "du".
> More files touched mean more memory used by glusterfs.
> This is not a memory leak. Repeating this "du" does not cause
memory
> usage go even a single byte up. Glusterfs client keeps somewhere an
> information about _every file touched_, and keeps it _forever_. As the
> situation
> did not improve since the times of glusterfs 1.3, I assume this behavior
> to be a part of its design. IMHO a wrong one. This is not a feature,
> this is a bug. It costs memory and probably performace too, if it is a kind
> of cache (with millions of entries, potentially).
> I filed a bug report almost one year ago - no change. Dear other users of
> glusterfs - are you at least able to reproduce this memory consumption
> problem? If it occurs on my site only, then the reason is somewhere in my
> system setup, otherwise - let us do something about it, finally.
> It is not in translators, it is in the core of glusterfs.
> With regards
> Krzysztof Strasburger
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>


-- 
Raghavendra G

On Thursday 25 March 2010 06:38:00 am Krzysztof Strasburger
wrote:
> Dear other users of
> glusterfs - are you at least able to reproduce this memory consumption
> problem?
I had a problem like this with 3.0.2 and the Quickread translator. That is a 
known issue. Do you have the same problem with the latest version of Gluster 
and no performance translators? I think your email says you went back to 2.x 
before removing the translators.

-- 
Benjamin Long

Hi,

I copied then the /usr directory to /root/loop-test (c.a. 160000
files).
> And then ran "du /root/loop-test".
> Memory usage of respective glusterfs process went up from 16 MB to 50 MB.
>
Ok,

> I could reproduce it perfectly by umounting /root/loop-test, mounting it
> again and re-running "du".
> More files touched mean more memory used by glusterfs.
> This is not a memory leak. Repeating this "du" does not cause
memory
> usage go even a single byte up.

This is expected.

> * Glusterfs client keeps somewhere an
> information about _every file touched_,
>
* and keeps it _forever_.


Both comments made above are wrong.

GlusterFS keeps the inode table entries (ie, dentries) keeping 1:1 map
between what kernel VFS has in its memory.

It gets free'd up when kernel sends 'forget()' on inode (it sends
forget for
each and every inode it has in memory). It does it automatically as and when
memory usage increases.

To send forceful forgets to glusterfs, do the following.

bash# echo 3 > /proc/sys/vm/drop_caches

and see the memory usage after this.

As the situation
> did not improve since the times of glusterfs 1.3, I assume this behavior
> to be a part of its design.

As explained above, its not part of design, but a filesystem implementation
requirement.

-Amar

Krzysztof Strasburger wrote:
> I upgraded today from 2.0.8 to 3.0.3.
> The first sad impression was that unify no longer works, but it is not that
> important. Not a surprise. Switched to dht. Tried my old "du"
test:
> du /home
> (/home is a glusterfs filesystem, of course ;) and saw again the bad old
> unlimited increase of glusterfs clients memory usage. I hoped that the
> problem went to quick-read, but it didn't.
This is a known problem. See a previous email on the devel list about it 
at:
http://article.gmane.org/gmane.comp.file-systems.gluster.devel/1469

A bug is filed is at:
http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=545

For more info on the drop_caches mentioned in that gluster-devel thread, 
see:
http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

Do let us know if dropping caches as shown in that thread helps.

Thanks
-Shehjar
> So switched back to 2.0.10rc1, to restore unify.
> And then did a simple test, to be sure that no translators are to be
blamed.
> I created the following, trivial setup:
> 
> file loopback.vol:
> volume loopback
>  type storage/posix
>  option directory /root/loopback
> end-volume
> 
> And then mounted /root/loopback on /root/loop-test:
> 
> mount -t glusterfs loopback.vol /root/loop-test
> 
> Do you see? No translators, even no server and networking. Just a loopback
> to a directory on the local machine.
> 
> I copied then the /usr directory to /root/loop-test (c.a. 160000 files).
> And then ran "du /root/loop-test".
> Memory usage of respective glusterfs process went up from 16 MB to 50 MB.
> I could reproduce it perfectly by umounting /root/loop-test, mounting it
> again and re-running "du".
> More files touched mean more memory used by glusterfs.
> This is not a memory leak. Repeating this "du" does not cause
memory
> usage go even a single byte up. Glusterfs client keeps somewhere an 
> information about _every file touched_, and keeps it _forever_. As the
situation
> did not improve since the times of glusterfs 1.3, I assume this behavior 
> to be a part of its design. IMHO a wrong one. This is not a feature,
> this is a bug. It costs memory and probably performace too, if it is a kind
> of cache (with millions of entries, potentially).
> I filed a bug report almost one year ago - no change. Dear other users of
> glusterfs - are you at least able to reproduce this memory consumption
> problem? If it occurs on my site only, then the reason is somewhere in my
> system setup, otherwise - let us do something about it, finally.
> It is not in translators, it is in the core of glusterfs.
> With regards
> Krzysztof Strasburger
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

On Mon, Mar 29, 2010 at 12:35:32PM +0530, Shehjar Tikoo
wrote:
> Krzysztof Strasburger wrote:
> >On Mon, Mar 29, 2010 at 11:48:44AM +0530, Shehjar Tikoo wrote:
> >>This is a known problem. See a previous email on the devel list
about it
> >>at:
> >>http://article.gmane.org/gmane.comp.file-systems.gluster.devel/1469
> >>
> >>A bug is filed is at:
> >>http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=545
> >>
> >>For more info on the drop_caches mentioned in that gluster-devel
thread,
> >>see:
> >>http://www.linuxinsight.com/proc_sys_vm_drop_caches.html
> >>
> >>Do let us know if dropping caches as shown in that thread helps.
> >Dear Shehjar,
> >thank you, but drop_caches really is not the Holy Grail of this
problem.
> >It simply does not change anything, to set it to something != 0.
> >Amon Ott figured it out, that forget() is called, but memory is not
freed
> >anyway. 
> 
> Of course, just doing drop_caches is not the silver bullet here. In some 
> cases, it needs to be preceded by a sync. What is your experience with
that?
> 
> In any case, if Ott is right, then we might have a memory leak. The best 
> step is to file a bug.
Shehjar - I have sent the response to your private mail; sorry for that.
Syncing is IMHO irrelevant here, as there are no dirty buffers to be
written out to fs. The files are only opened, stated and closed.
I filed related bug report in ancient days, when the bug database was hosted on
savannah. If I remember correctly, it has been closed in the meantime. 
Should I file a new report, or find the old one and reopen it (if it is closed)?
Krzysztof