Lustre discuss - Sep 2009 - Working with large amounts of files

Hello,

I was wondering what ideas are in Lustre''s roadmap to address the
issues
with serialization and management that appear when dealing with large
amounts of files in parallel.
Clusters are getting more and more cores, with some of them having hundreds
of thousands already, file systems, however, continue with the paradigm of
individual files, even on massively parallel contexts. Wouldn''t it be
better
for parallel file system like Lustre to somehow bundle files together for a
group of tasks? If I recall correctly, there is a proposal to extend POSIX
to achieve some of this.

I would love to hear some insight about the topic.


Thanks,
Alvaro.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090924/65c128ab/attachment.html

On Sep 24, 2009  22:56 +0200, Alvaro Aguilera wrote:
> I was wondering what ideas are in Lustre''s roadmap to address the
issues
> with serialization and management that appear when dealing with large
> amounts of files in parallel.
> Clusters are getting more and more cores, with some of them having hundreds
> of thousands already, file systems, however, continue with the paradigm of
> individual files, even on massively parallel contexts. Wouldn''t it
be better
> for parallel file system like Lustre to somehow bundle files together for a
> group of tasks? If I recall correctly, there is a proposal to extend POSIX
> to achieve some of this.
In many cases, applications are starting to move to large shared file
IO models, because that provides for an easier way to manage the data
once the IO is complete.  It can also (depending on the application)
make it easier to partition the workload onto a different number of
cores.

Some sites have used loopback mounts of Lustre-backed files with an
ext2 filesystem inside in order to provide very efficient access to
a large number of small files.  This is of course not coherent over
multiple nodes, but in some workloads that is acceptable.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.