Lustre discuss - May 2008 - Benchmarking small file performance

We build a Lustre appliance that comes fully configured and tuned. A
number of customers have asked us about small file performance. We are
looking for feedback on how people "quantify" this type of performance
with their Lustre based systems. In most cases we are looking at
applications that are reading and writing files in the 1K to 20M size
range. 
 
Based on the advice of the Lustre development team (thanks), we are
currently working with PIOS which is described in the Lustre manual. Are
there other tools / suites that people use to quantify this kind of
performance?
 
Thanks,
 
Rick
 
*************************************************
Rick Friedman
Terascala
 
Lustre for the rest of us...  www.terascala.com
 
 
 
 
 
 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080515/68d58a2b/attachment.html

Hi Rick,


On Thu, May 15, 2008 at 04:21:56PM -0400, Rick Friedman
wrote:
> We build a Lustre appliance that comes fully configured and tuned. A
> number of customers have asked us about small file performance. We are
> looking for feedback on how people "quantify" this type of
performance
> with their Lustre based systems. In most cases we are looking at
> applications that are reading and writing files in the 1K to 20M size
> range. 
One tool commonly used to compare file system operation performance for
local filesystems and NFS clients is bonnie++
(http://www.coker.com.au/bonnie++/). It''s not really designed for
paralell filesystems but at least it will quantify single-client lustre
performance in a way that your customers can relate to. 

Note that since bonnie++ only runs on a single machine at the time you
need some kind of wrapper to run it on a multi-client lustre cluster.
There is a very primitive one available at
http://ragnark.vestdata.no/download/dbonnie++.txt, but what you should
do is make a version of bonnie++ that use MPI for syncronization instead
of semaphores, so that it can easily be run on clusters.



-- 
Ragnar Kj?rstad

On Sat, 17 May 2008, Ragnar Kj?rstad wrote:
Hi,

     Postmark is a good benchmark for small file peformance. I have tried 
it with gfs and lustre before for comparison. Lustre doesn''t shine well
in
that benchmark. But when the file size is set a little high, lustre does pick 
up quite a bit.

Regards
Balagopal

> Hi Rick,
> 
> 
> On Thu, May 15, 2008 at 04:21:56PM -0400, Rick Friedman wrote:
> > We build a Lustre appliance that comes fully configured and tuned. A
> > number of customers have asked us about small file performance. We are
> > looking for feedback on how people "quantify" this type of
performance
> > with their Lustre based systems. In most cases we are looking at
> > applications that are reading and writing files in the 1K to 20M size
> > range. 
> 
> One tool commonly used to compare file system operation performance for
> local filesystems and NFS clients is bonnie++
> (http://www.coker.com.au/bonnie++/). It''s not really designed for
> paralell filesystems but at least it will quantify single-client lustre
> performance in a way that your customers can relate to. 
> 
> Note that since bonnie++ only runs on a single machine at the time you
> need some kind of wrapper to run it on a multi-client lustre cluster.
> There is a very primitive one available at
> http://ragnark.vestdata.no/download/dbonnie++.txt, but what you should
> do is make a version of bonnie++ that use MPI for syncronization instead
> of semaphores, so that it can easily be run on clusters.
> 
> 
> 
>

On May 17, 2008  13:47 -0300, Balagopal Pillai wrote:
> On Sat, 17 May 2008, Ragnar Kj?rstad wrote:
>      Postmark is a good benchmark for small file peformance. I have tried 
> it with gfs and lustre before for comparison. Lustre doesn''t shine
well in
> that benchmark. But when the file size is set a little high, lustre does
pick
> up quite a bit.
Do you have numbers at what file size Lustre does well with postmark?
Given HTML email with large attachments, the days that average email
size for many users is getting quite large.
> > On Thu, May 15, 2008 at 04:21:56PM -0400, Rick Friedman wrote:
> > > We build a Lustre appliance that comes fully configured and
tuned. A
> > > number of customers have asked us about small file performance.
We are
> > > looking for feedback on how people "quantify" this type
of performance
> > > with their Lustre based systems. In most cases we are looking at
> > > applications that are reading and writing files in the 1K to 20M
size
> > > range. 
> > 
> > One tool commonly used to compare file system operation performance
for
> > local filesystems and NFS clients is bonnie++
> > (http://www.coker.com.au/bonnie++/). It''s not really designed
for
> > paralell filesystems but at least it will quantify single-client
lustre
> > performance in a way that your customers can relate to. 
> > 
> > Note that since bonnie++ only runs on a single machine at the time you
> > need some kind of wrapper to run it on a multi-client lustre cluster.
> > There is a very primitive one available at
> > http://ragnark.vestdata.no/download/dbonnie++.txt, but what you should
> > do is make a version of bonnie++ that use MPI for syncronization
instead
> > of semaphores, so that it can easily be run on clusters.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

On Sat, 17 May 2008, Andreas Dilger wrote:
I don''t have the numbers anymore, i did the benchmark almost an year
ago
to decide between gfs and lustre. I remember when the filesize was close 
to 1 MB, the performance of lustre started picking up and became closer 
to gfs and nfs. The default file size of postmark is quite small, more in line
with a regular
maildir setup 
and text only emails. The lru size also had an effect and once that is 
bumped up, as mentioned in the manual and wiki, the number of files that 
is used by postmark can be increased without a big penalty. In the 
default settings for very small files, the results are worse than nfs.
Lustre could still be a good choice for mail servers that use mbox, but 
for maildir i am not so sure.

Regards
Balagopal 

> On May 17, 2008  13:47 -0300, Balagopal Pillai wrote:
> > On Sat, 17 May 2008, Ragnar Kj?rstad wrote:
> >      Postmark is a good benchmark for small file peformance. I have
tried
> > it with gfs and lustre before for comparison. Lustre doesn''t
shine well in
> > that benchmark. But when the file size is set a little high, lustre
does pick
> > up quite a bit.
> 
> Do you have numbers at what file size Lustre does well with postmark?
> Given HTML email with large attachments, the days that average email
> size for many users is getting quite large.
> 
> > > On Thu, May 15, 2008 at 04:21:56PM -0400, Rick Friedman wrote:
> > > > We build a Lustre appliance that comes fully configured and
tuned. A
> > > > number of customers have asked us about small file
performance. We are
> > > > looking for feedback on how people "quantify" this
type of performance
> > > > with their Lustre based systems. In most cases we are
looking at
> > > > applications that are reading and writing files in the 1K to
20M size
> > > > range. 
> > > 
> > > One tool commonly used to compare file system operation
performance for
> > > local filesystems and NFS clients is bonnie++
> > > (http://www.coker.com.au/bonnie++/). It''s not really
designed for
> > > paralell filesystems but at least it will quantify single-client
lustre
> > > performance in a way that your customers can relate to. 
> > > 
> > > Note that since bonnie++ only runs on a single machine at the
time you
> > > need some kind of wrapper to run it on a multi-client lustre
cluster.
> > > There is a very primitive one available at
> > > http://ragnark.vestdata.no/download/dbonnie++.txt, but what you
should
> > > do is make a version of bonnie++ that use MPI for syncronization
instead
> > > of semaphores, so that it can easily be run on clusters.
> 
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>

Rick,

I would suggest "fileop" benchmark in the iozone benchmark suite to
measure metadata performance. In addition, iozone comes with parallel
version to run multiple instances across clients.

http://iozone.org/

Spew is another good benchmark to measure small and large file I/O
performance
http://spew.berlios.de/

HTH,
-Kums


On Thu, 2008-05-15 at 16:21 -0400, Rick Friedman wrote:
>  
> We build a Lustre appliance that comes fully configured and tuned. A
> number of customers have asked us about small file performance. We are
> looking for feedback on how people "quantify" this type of
performance
> with their Lustre based systems. In most cases we are looking at
> applications that are reading and writing files in the 1K to 20M size
> range. 
>  
> Based on the advice of the Lustre development team (thanks), we are
> currently working with PIOS which is described in the Lustre manual.
> Are there other tools / suites that people use to quantify this kind
> of performance?
>  
> Thanks,
>  
> Rick
>  
> *************************************************
> Rick Friedman
> Terascala
>  
> Lustre for the rest of us...  www.terascala.com
>  
>  
>  
>  
>  
>  
>  
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss