Hello Stephen
The issue of the ADIO driver for MPI/IO and optimizing it was discussed
at the LUG. We told the attendees that we were interested in working
with the lustre.org (design issues are on lustre-devel@clusterfs.com)
community to improve this.
We are certainly very interested in your contributions to this effort -
because nothing will affect the results more than the evaluation of
applications, as you are proposing.
The research team from DK Panda from Ohio State University has
volunteered to take the lead on this effort.
I see a large number of opportunities:
- Lustre traditionally has relied on plain good POSIX performance. At
a parallel I/O tutorial at SC 2005 a number of weaknesses were
identified through a study by Argonne. We can summarize these
weaknesses, and people can propose and implement fixes for them.
- the Lustre ADIO driver, written by HP for the path forward, is
attached - this should be packaged better
- Profile applications and improve weak links in the Lustre protocol
- Lustre has working hooks to set striping on a file or directory
- Lustre has hooks to disable locking, we should verify that these
still work
- Lustre has hooks to do direct non-cached I/O
- Lustre has an un-published mechanism to set the stripe pattern on a
file on an extent basis, e.g.
First MB of the file is not striped, remainder is striped over all
storage targets
This "join" feature, funded by CEA, is fairly unique and could be
used for an exemplary HDF5 implementation.
The join feature lacks some recovery aspects but these would not
interfere with a serious exploration of the feature for scientific
computing. I will send the design considerations for this feature to
this mailing list. It is present in Lustre 1.4.6.
- Sandia has architected and proposed a Lustre feature for "group
locks" which enable interesting opportunities for multiple jobs to share
or not share a particular file.
>From here I would like to hand the discussion over to DK.
Best wishes,
- Peter -
-------------- next part --------------
A non-text attachment was scrubbed...
Name: adio.patch
Type: application/octet-stream
Size: 66261 bytes
Desc: adio.patch
Url :
http://mail.clusterfs.com/pipermail/lustre-devel/attachments/20060426/4801e236/adio-0001.obj