Hi I am using Glusterfs 2.0.6 and I am experiencing some memory issues on the clients. The clients will grow their memory usage to several GB's over one or two days. This causes my app to run out of memroy and I need to restart my app and unmount the glusterfs volumes. I have tried adjusting the io-cache to max 512MB but this does not seem to have any effect. My clients use replicate and distribute.>From my glusterfs.vol file:volume dfs type cluster/distribute option lookup-unhashed no option min-free-disk 5% subvolumes repl-000-001-01 repl-000-001-02 repl-000-001-03 repl-000-001-04 repl-002-003-01 repl-002-003-02 repl-002-003-03 repl-002-003-04 end-volume # Enable write-behind to decrease write latency volume wb type performance/write-behind option flush-behind off option cache-size 64MB subvolumes dfs end-volume volume cache type performance/io-cache option cache-size 512MB subvolumes wb end-volume Is this a memory leak or is there a way to limit the memory usage on the clients? I am running this on Ubuntu 9.04. Regards -- Roland Rabben Founder & CEO Jotta AS Cell: +47 90 85 85 39 Phone: +47 21 04 29 00 Email: roland at jotta.no
Hi Roland, Does doing "echo 3 > /proc/sys/vm/drop_caches" bring down the memory usage? regards, On Fri, Jan 1, 2010 at 2:47 PM, Roland Rabben <roland at jotta.no> wrote:> Hi > I am using Glusterfs 2.0.6 and I am experiencing some memory issues on the > clients. The clients will grow their memory usage to several GB's over one > or two days. > This causes my app to run out of memroy and I need to restart my app and > unmount the glusterfs volumes. > > I have tried adjusting the io-cache to max 512MB but this does not seem to > have any effect. > My clients use replicate and distribute. > > From my glusterfs.vol file: > > volume dfs > type cluster/distribute > option lookup-unhashed no > option min-free-disk 5% > subvolumes repl-000-001-01 repl-000-001-02 repl-000-001-03 > repl-000-001-04 repl-002-003-01 repl-002-003-02 repl-002-003-03 > repl-002-003-04 > end-volume > > # Enable write-behind to decrease write latency > volume wb > type performance/write-behind > option flush-behind off > option cache-size 64MB > subvolumes dfs > end-volume > > volume cache > type performance/io-cache > option cache-size 512MB > subvolumes wb > end-volume > > Is this a memory leak or is there a way to limit the memory usage on the > clients? > > I am running this on Ubuntu 9.04. > > Regards > > > -- > Roland Rabben > Founder & CEO Jotta AS > Cell: +47 90 85 85 39 > Phone: +47 21 04 29 00 > Email: roland at jotta.no > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > >-- Raghavendra G
Thanks, Raghavendra. Roland, I would be very interested to know if you plan to move to a newer version - that might solve some of your problems. If you do not plan to upgrade, then I would also like to know if there are certain features in 2.0.6 which you feel do not allow you to upgrade. In the ideal case, we would like to have to support a fewer set of recent versions hence the question :-). Regards, Tejas. ----- Original Message ----- From: "Raghavendra G" <raghavendra at gluster.com> To: "Roland Rabben" <roland at jotta.no> Cc: "gluster-users" <gluster-users at gluster.org> Sent: Friday, January 1, 2010 8:41:32 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Subject: Re: [Gluster-users] Memory problems / leak in 2.0.6? Hi Roland, Does doing "echo 3 > /proc/sys/vm/drop_caches" bring down the memory usage? regards, On Fri, Jan 1, 2010 at 2:47 PM, Roland Rabben <roland at jotta.no> wrote:> Hi > I am using Glusterfs 2.0.6 and I am experiencing some memory issues on the > clients. The clients will grow their memory usage to several GB's over one > or two days. > This causes my app to run out of memroy and I need to restart my app and > unmount the glusterfs volumes. > > I have tried adjusting the io-cache to max 512MB but this does not seem to > have any effect. > My clients use replicate and distribute. > > From my glusterfs.vol file: > > volume dfs > type cluster/distribute > option lookup-unhashed no > option min-free-disk 5% > subvolumes repl-000-001-01 repl-000-001-02 repl-000-001-03 > repl-000-001-04 repl-002-003-01 repl-002-003-02 repl-002-003-03 > repl-002-003-04 > end-volume > > # Enable write-behind to decrease write latency > volume wb > type performance/write-behind > option flush-behind off > option cache-size 64MB > subvolumes dfs > end-volume > > volume cache > type performance/io-cache > option cache-size 512MB > subvolumes wb > end-volume > > Is this a memory leak or is there a way to limit the memory usage on the > clients? > > I am running this on Ubuntu 9.04. > > Regards > > > -- > Roland Rabben > Founder & CEO Jotta AS > Cell: +47 90 85 85 39 > Phone: +47 21 04 29 00 > Email: roland at jotta.no > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > >-- Raghavendra G _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Krzysztof Strasburger
2010-Jan-04 05:24 UTC
[Gluster-users] Memory problems / leak in 2.0.6?
The memory leak you discuss here seems to be the same I reported months ago. Reproduction: run anything, that just touches a lot of files (open/stat/close), for example ls -lR or du on a big directory tree. After reading the recent correspondence, I hope that the problem has been moved to the quick-read translator in 3.0, but it is still to be checked on my site. I will report the results ASAP. With regards Krzysztof