Hello I''m trying to control the amount of cache memory used by Lustre clients with Lustre 2.0 There is a /proc tuning called "max_cache_mb" in llite which is documented in Lustre manual. Unfortunately, after testing it and checking the source code it seems this variable is present but it not used anymore in Lustre code. This is not the case in 1.8 I did not find if this was removed or this was partially included in Lustre 2.0. What''s the current status of this and how can I tell to my client to avoid caching too many data? Those clients do a lot of read and write in Lustre filesystems but thoses files will not be re-read soon, so it is useless to fill memory with it. Moreover, when the client memory is full, Lustre performance are really impacted. Thanks in advance Aur?lien Degr?mont
On 2011-03-17, at 6:37 AM, DEGREMONT Aurelien wrote:> I''m trying to control the amount of cache memory used by Lustre clients > with Lustre 2.0 > There is a /proc tuning called "max_cache_mb" in llite which is > documented in Lustre manual. > Unfortunately, after testing it and checking the source code it seems > this variable is present but it not used anymore in Lustre code. > This is not the case in 1.8. > > I did not find if this was removed or this was partially included in > Lustre 2.0. > What''s the current status of this and how can I tell to my client to > avoid caching too many data?The client VM usage was one of the areas that was completely rewritten by Nikita for 2.0, so it was likely this functionality was lost at that time. I don''t have any idea at this time how hard it would be to restore.> Those clients do a lot of read and write in Lustre filesystems but > thoses files will not be re-read soon, so it is useless to fill memory > with it. Moreover, when the client memory is full, Lustre performance > are really impacted.It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc.
Hello! On Mar 17, 2011, at 5:44 PM, Andreas Dilger wrote:>> I did not find if this was removed or this was partially included in >> Lustre 2.0. >> What''s the current status of this and how can I tell to my client to >> avoid caching too many data? > The client VM usage was one of the areas that was completely rewritten by Nikita for 2.0, so it was likely this functionality was lost at that time. I don''t have any idea at this time how hard it would be to restore.I wonder if the generic vm tunables could be used instead? I think that was the original plan.>> Those clients do a lot of read and write in Lustre filesystems but >> thoses files will not be re-read soon, so it is useless to fill memory >> with it. Moreover, when the client memory is full, Lustre performance >> are really impacted. > It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use.Yes, that would totally make sense to do regardless of other methods. Bye, Oleg
Oleg Drokin a ?crit :> Hello! > > On Mar 17, 2011, at 5:44 PM, Andreas Dilger wrote: > >> The client VM usage was one of the areas that was completely rewritten by Nikita for 2.0, so it was likely this functionality was lost at that time. I don''t have any idea at this time how hard it would be to restore. >> > > I wonder if the generic vm tunables could be used instead? I think that was the original plan. >So the issue is that ''max_cached_mb'' tunable should be removed if it makes no sense. The default with the generic tunable is that you could really control how they behave per filesystem.>>> Those clients do a lot of read and write in Lustre filesystems but >>> thoses files will not be re-read soon, so it is useless to fill memory >>> with it. Moreover, when the client memory is full, Lustre performance >>> are really impacted. >>> >> It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use. >> > Yes, that would totally make sense to do regardless of other methods. >Hmm... I do not want to patch ''cp'' or ''dd'' :) Aur?lien
On Mar 18, 2011, at 1:07 AM, DEGREMONT Aurelien wrote:>>>> Those clients do a lot of read and write in Lustre filesystems but >>>> thoses files will not be re-read soon, so it is useless to fill memory >>>> with it. Moreover, when the client memory is full, Lustre performance >>>> are really impacted. >>>> >>> It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use. >>> >> Yes, that would totally make sense to do regardless of other methods. >> > > Hmm... I do not want to patch ''cp'' or ''dd'' :)You might want to have a look at this: http://code.google.com/p/pagecache-mangagement/ We use it when copying crash dumps off of our OSSes, as we don''t want to disturb cached metadata. Jason -- Jason Rappleye System Administrator NASA Advanced Supercomputing Division NASA Ames Research Center Moffett Field, CA 94035
Jason Rappleye a ?crit :> On Mar 18, 2011, at 1:07 AM, DEGREMONT Aurelien wrote: > >>> Yes, that would totally make sense to do regardless of other methods. >>> >> Hmm... I do not want to patch ''cp'' or ''dd'' :) >> > > You might want to have a look at this: > > http://code.google.com/p/pagecache-mangagement/ > > We use it when copying crash dumps off of our OSSes, as we don''t want to disturb cached metadata. >Interesting project. I will have a look. Thank you. Aur?lien
After checking 2.6.35 kernel source code, POSIX_FADV_NOREUSE actually doesn''t do anything. So I don''t know how it helps. Probably we should do POSIX_FADV_DONTNEED after reading? Jay On Mar 17, 2011, at 5:43 PM, Oleg Drokin wrote:> Hello! > > On Mar 17, 2011, at 5:44 PM, Andreas Dilger wrote: >>> I did not find if this was removed or this was partially included in >>> Lustre 2.0. >>> What''s the current status of this and how can I tell to my client to >>> avoid caching too many data? >> The client VM usage was one of the areas that was completely rewritten by Nikita for 2.0, so it was likely this functionality was lost at that time. I don''t have any idea at this time how hard it would be to restore. > > I wonder if the generic vm tunables could be used instead? I think that was the original plan. > >>> Those clients do a lot of read and write in Lustre filesystems but >>> thoses files will not be re-read soon, so it is useless to fill memory >>> with it. Moreover, when the client memory is full, Lustre performance >>> are really impacted. >> It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use. > > Yes, that would totally make sense to do regardless of other methods. > > Bye, > Oleg > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
We used POSIX_FADV_DONTNEED in distributed iozone test to clear cache on slave client after initial write and before client reads back same data. It helped to see real data rates instead of unrealistically high read rate due do cacheing. Perhaps it was non-lustre (NFS) file server. If you do scripting, small executable like http://www.citi.umich.edu/projects/asci/benchmarks.html (scroll down to Clearcache) can be called after "cp" or "dd." Alex. On Mar 19, 2011, at 12:47 AM, Jay wrote:> After checking 2.6.35 kernel source code, POSIX_FADV_NOREUSE > actually doesn''t do anything. So I don''t know how it helps. Probably > we should do POSIX_FADV_DONTNEED after reading? > > Jay> On Mar 18, 2011, at 1:07 AM, DEGREMONT Aurelien wrote: > >>> ... snip... >> Hmm... I do not want to patch ''cp'' or ''dd'' :)