Lustre discuss - Aug 2009 - applications question

Since we have been using lustre for a while now we would like to port
some of them to use I/O features of lustre. Currently, all of the
applications are very memory intensive, meaning they take in the
entire dataset at 1 time load it into memory (64GB) and process. I was
wondering if there are some techniques people are using to
enhance,optimize, and use less memory their applications  by using
Lustre. Are there any white papers, algorithms or tricks I can use to
reduce memory footprint (with a slight loss of performance)?

TIA

On Aug 12, 2009  07:13 -0400, Mag Gam wrote:
> Since we have been using lustre for a while now we would like to port
> some of them to use I/O features of lustre. Currently, all of the
> applications are very memory intensive, meaning they take in the
> entire dataset at 1 time load it into memory (64GB) and process. I was
> wondering if there are some techniques people are using to
> enhance,optimize, and use less memory their applications  by using
> Lustre. Are there any white papers, algorithms or tricks I can use to
> reduce memory footprint (with a slight loss of performance)?
I would steer you away from using memory-mapped IO.  This is functional
with Lustre, but is not at all the fastest way to do IO with Lustre.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.