On Dec 08, 2005 22:26 -0800, Jeffrey W. Baker wrote:> On Thu, 2005-12-08 at 22:08 -0800, Phil Schwan wrote: > > A well-tuned but modest metadata server can sustain on the order of > > 5,000 metadata operations per second, and in our experience this > > usually corresponds to many more NFSops -- but indeed, you''ll > > probably have to test it out to be sure. > > Could you elaborate on the client parallelism required to achieve 5,000 > metadata ops/s? Or put another way, roughly how many ops might you > expect with a single client and an unloaded MDS? I would expect network > latency to be the controlling factor when parallelism is low.This is exactly the case. Because client operations are single-threaded by the POSIX API (there are no asynchronous POSIX APIs for metadata) a Lustre client is currently stuck behind a network-latency RPC for many metadata operation. For non-modifying operations (e.g. getattr) we do have coherent caches on the clients for repeat-use items. In this regard Lustre has the interesting behaviour that the aggregate performance improves as clients are added, compared to many systems where the performance degrades as clients are added. Just looking at a recent regression test result for "open(O_CREAT)", a single client can do about 1000/s average in a single directory (over 300k items in 300s test). Two clients can do double that. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Hi Dan-- On Dec 1, 2005, at 13:05, Daniel Hanks wrote:> > I''ve been reading what I can about Lustre. I work for a large web > hosting company, and currently we use NetApps for some of our storage > needs. On one of our filers we''re currently pushing over 20k NFS > operations/second from around 100 client machines. > > I''m curious to know if you think that Lustre could conceivably be a > cheaper replacement for such a setup? Our traffic patterns are large > numbers of small (a few kilobytes to a few megabytes) reads and writes > (more read-heavy though). Would we overwhelm a singe MDS with this > amount of I/O operations? I imagine a lot would depend on the ratio of > reads/writes to lookup operations.That''s true -- it''s very difficult to compare NFSops to I/O or metadata ops. A well-tuned but modest metadata server can sustain on the order of 5,000 metadata operations per second, and in our experience this usually corresponds to many more NFSops -- but indeed, you''ll probably have to test it out to be sure. In the past we''ve tried a large variety of metadata-intensive benchmarks to demonstrate that the single MDS is not a problem for today''s loads. For all the benchmarks we ran, Lustre was at least as good -- and usually quite a bit better -- than the enterprise NFS server that we compared it to. Hope this helps, -Phil
On Thu, 2005-12-08 at 22:08 -0800, Phil Schwan wrote:> A well-tuned but modest metadata server can sustain on the order of > 5,000 metadata operations per second, and in our experience this > usually corresponds to many more NFSops -- but indeed, you''ll > probably have to test it out to be sure.Hi Phil, Could you elaborate on the client parallelism required to achieve 5,000 metadata ops/s? Or put another way, roughly how many ops might you expect with a single client and an unloaded MDS? I would expect network latency to be the controlling factor when parallelism is low. -jwb
Hi folks, I''ve been reading what I can about Lustre. I work for a large web hosting company, and currently we use NetApps for some of our storage needs. On one of our filers we''re currently pushing over 20k NFS operations/second from around 100 client machines. I''m curious to know if you think that Lustre could conceivably be a cheaper replacement for such a setup? Our traffic patterns are large numbers of small (a few kilobytes to a few megabytes) reads and writes (more read-heavy though). Would we overwhelm a singe MDS with this amount of I/O operations? I imagine a lot would depend on the ratio of reads/writes to lookup operations. I like the idea of a storage cluster that can be expanded by simply adding more (relatively) inexpensive machines/storage. Expanding a NetApp requires expensive disk shelves and disks, and eventually you reach limits in the filer heads, requiring more filer heads (much more expense), expensive clustering/mirroring options, etc, etc. Thanks for any input you can provide. Thanks, -- Dan Hanks