Hello list, I''d like to implement sliding window and hight water mark deletion on a lustre filesystem. For that I need information from all files. A first idea was to recursivly traverse the file tree and to a `stat filename` for every file to collect the meta data. Now since in lustre the meta data is stored on the mds, I though that maybe there is a more efficient way of collecting the information. So the question is: is there a way that I could query the mds for the information that stat would give me(I need access, modify, creation time plus the file size). And if so, can someone guess the performance increase of quering the mds directly over using `stat` ? We are using lustre 1.6.something. Isaac
On Sep 16, 2009 11:56 +0200, Isaac Hailperin wrote:> I''d like to implement sliding window and hight water mark deletion on a > lustre filesystem. For that I need information from all files. A first > idea was to recursivly traverse the file tree and to a `stat filename` > for every file to collect the meta data. > > Now since in lustre the meta data is stored on the mds, I though that > maybe there is a more efficient way of collecting the information. > > So the question is: is there a way that I could query the mds for the > information that stat would give me(I need access, modify, creation time > plus the file size). And if so, can someone guess the performance > increase of quering the mds directly over using `stat` ?This is what e2scan is for, in the Lustre e2fsprogs. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Have you looked at Robinhood? http://sourceforge.net/projects/robinhood It does high water mark, and provides a nice interface to query Lustre metadata information as well. Todd On Wed, Sep 16, 2009 at 2:56 AM, Isaac Hailperin <hailperin at zib.de> wrote:> > Hello list, > > I''d like to implement sliding window and hight water mark deletion on a > lustre filesystem. For that I need information from all files. A first > idea was to recursivly traverse the file tree and to a `stat filename` > for every file to collect the meta data. > > Now since in lustre the meta data is stored on the mds, I though that > maybe there is a more efficient way of collecting the information. > > So the question is: is there a way that I could query the mds for the > information that stat would give me(I need access, modify, creation time > plus the file size). And if so, can someone guess the performance > increase of quering the mds directly over using `stat` ? > > We are using lustre 1.6.something. > > Isaac > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
On Sep 16, 2009 08:54 -0700, Todd Hartmann wrote:> Have you looked at Robinhood? http://sourceforge.net/projects/robinhood > > It does high water mark, and provides a nice interface to query Lustre > metadata information as well.FYI, Robin Hood is also what we are planning to use in 2.x with HSM, and it is being integrated into the 2.0 changelogs so that it can avoid full filesystem scanning.> Todd > > On Wed, Sep 16, 2009 at 2:56 AM, Isaac Hailperin <hailperin at zib.de> wrote: > > > > Hello list, > > > > I''d like to implement sliding window and hight water mark deletion on a > > lustre filesystem. For that I need information from all files. A first > > idea was to recursivly traverse the file tree and to a `stat filename` > > for every file to collect the meta data. > > > > Now since in lustre the meta data is stored on the mds, I though that > > maybe there is a more efficient way of collecting the information. > > > > So the question is: is there a way that I could query the mds for the > > information that stat would give me(I need access, modify, creation time > > plus the file size). And if so, can someone guess the performance > > increase of quering the mds directly over using `stat` ? > > > > We are using lustre 1.6.something. > > > > Isaac > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Andreas Dilger wrote:> On Sep 16, 2009 08:54 -0700, Todd Hartmann wrote: >> Have you looked at Robinhood? http://sourceforge.net/projects/robinhood >> >> It does high water mark, and provides a nice interface to query Lustre >> metadata information as well.Currently the website is not responding, but it sounds interesting. Thank you for pointing this out.> FYI, Robin Hood is also what we are planning to use in 2.x with HSM, > and it is being integrated into the 2.0 changelogs so that it can > avoid full filesystem scanning. >That is good news. Isaac -- Isaac Hailperin tel: +49 30 84185 160 ZIB fax: +49 30 84185 311 Takustr. 7 D-14195 Berlin