OpenSFS has identified a need for a Lustre roadmap with clear architectural and
feature delivery targets. Before initiating any feature development, OpenSFS
wants to be sure the architecture meets the requirements of the HPC community.
Most of our pain points center around the metadata server and backend storage.
To motivate the architecture discussion, we are gathering requirements for our
near term (within the next year) and long term (3+ yrs out) needs.
The list below is a start. We recognize that it is somewhat arbitrary to
specify performance requirements in the absence of information about the
platform on which the file system will run.
Please send comments and suggestions to <discuss at lists.opensfs.org>.
This mailing list is open to the public. All interested in contributing to this
discussion are encouraged to subscribe at <http://lists.opensfs.org>. We
will have open community concalls starting Thursday 2/10 to work on the
architecture and roadmap to meet these requirements.
* metadata performance
GOAL: improve file system scalability and interactive
performance
requirements: Q2 2012 Q1 2014
-------------- -----------------
- # files in file system 100 billion 1 trillion
- # files in directory 50 million 10 billion
- aggregate file creates/s 100 thousand ?
- single file creates/s ? 30 thousand
- directory listings/s ? ?
- open files per process - 100 thousand
- file system capacity 30 PB 100 PB
- # clients 30 thousand ?
- ...
* backend storage
GOAL: provide reliable, scalable backing store for
Lustre servers
requirements:
- large LUNs (min?, max?)
- end-to-end data integrity
(ie provide resiliency that T10 PI gives local file systems)
- low performance impact for file system repair
- direct I/O mode
- ...