Hello all, I have a user at the Naval Research Laboratory running Lustre on a Cray XD1. I''ll let his email to me tell the story, but he doesn''t want to know if the numbers he''s observing are reasonable. What do you say out there? Can anyone out there tell me if his numbers make sense? TIA, --garland Begin forwarded message:> From: > Date: April 17, 2008 1:27:11 PM EDT > To: > Cc: > Subject: I/O tests on Lustre > > > I ran the io test that I ran on the XD1 Lustre on the ice > machine. This program writes data from > an increasing number of processors ( 1,2,4,8,16,32,64) using MPI > for parallelization and then > reads it back. Enough data is written so that when I read the > data has not been cached somewhere. > Data block sizes are large so my programs should be giving me > peak bandwidths. > > My last set of results were > Numbers of Processes Write Rate Read Rate > MBs/sec MBs/sec > 1 763.7 506.4 > 2 1248.8 968.7 > 4 1298.7 782.3 > 8 1292.5 900.6 > 16 1298.1 820.7 > 32 1296.7 809.3 > 64 1296.3 878.9 > > > The thirty two process failed with a time out but everything else > worked ok. > > My question is Do these rates look reasonable based on your > Lustre experience? > > Also any ideas on why I might be timing out on an occasional run? > > wendell > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080514/1ac8b00c/attachment.html
On May 14, 2008 16:14 -0400, Garland Jackson wrote:> I have a user at the Naval Research Laboratory running Lustre on a Cray > XD1. > I''ll let his email to me tell the story, but he doesn''t want to know if the > numbers he''s observing are reasonable. > > What do you say out there? Can anyone out there tell me if his numbers > make sense?It''s completely dependent upon the hardware configuration (OSTs, disks, etc). Some customers have 50GB/s filesystems, while I have a 40MB/s Lustre filesystem at home.> Begin forwarded message: > >> From: >> Date: April 17, 2008 1:27:11 PM EDT >> To: >> Cc: >> Subject: I/O tests on Lustre >> >> >> I ran the io test that I ran on the XD1 Lustre on the ice machine. This >> program writes data from >> an increasing number of processors ( 1,2,4,8,16,32,64) using MPI for >> parallelization and then >> reads it back. Enough data is written so that when I read the data has >> not been cached somewhere. >> Data block sizes are large so my programs should be giving me peak >> bandwidths. >> >> My last set of results were >> Numbers of Processes Write Rate Read Rate >> MBs/sec MBs/sec >> 1 763.7 506.4 >> 2 1248.8 968.7 >> 4 1298.7 782.3 >> 8 1292.5 900.6 >> 16 1298.1 820.7 >> 32 1296.7 809.3 >> 64 1296.3 878.9 >> >> >> The thirty two process failed with a time out but everything else worked >> ok. >> >> My question is Do these rates look reasonable based on your Lustre >> experience? >> >> Also any ideas on why I might be timing out on an occasional run? >> >> wendell >> >> >> _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
I concur with Andreas; this question comes around regularly, but the all the details of the Lustre FS deployment, as well as host cluster configuration and implementation, have inordinate significance. You can refer to the thread "Off-topic: largest existing Lustre file system?" back in January of this year. With the sole intention of finding a very high maximal rate, I ran benchmark MPI executions with exceptional parameters and got up to 28 GB/s. However, when I moderated any dynamic, the byte-rate performance often fell off quickly. In general, your numbers seem reasonable to me. Marty Andreas Dilger wrote:> On May 14, 2008 16:14 -0400, Garland Jackson wrote: > >> I have a user at the Naval Research Laboratory running Lustre on a Cray >> XD1. >> I''ll let his email to me tell the story, but he doesn''t want to know if the >> numbers he''s observing are reasonable. >> >> What do you say out there? Can anyone out there tell me if his numbers >> make sense? >> > > It''s completely dependent upon the hardware configuration (OSTs, disks, > etc). Some customers have 50GB/s filesystems, while I have a 40MB/s > Lustre filesystem at home. > > >> Begin forwarded message: >> >> >>> From: >>> Date: April 17, 2008 1:27:11 PM EDT >>> To: >>> Cc: >>> Subject: I/O tests on Lustre >>> >>> >>> I ran the io test that I ran on the XD1 Lustre on the ice machine. This >>> program writes data from >>> an increasing number of processors ( 1,2,4,8,16,32,64) using MPI for >>> parallelization and then >>> reads it back. Enough data is written so that when I read the data has >>> not been cached somewhere. >>> Data block sizes are large so my programs should be giving me peak >>> bandwidths. >>> >>> My last set of results were >>> Numbers of Processes Write Rate Read Rate >>> MBs/sec MBs/sec >>> 1 763.7 506.4 >>> 2 1248.8 968.7 >>> 4 1298.7 782.3 >>> 8 1292.5 900.6 >>> 16 1298.1 820.7 >>> 32 1296.7 809.3 >>> 64 1296.3 878.9 >>> >>> >>> The thirty two process failed with a time out but everything else worked >>> ok. >>> >>> My question is Do these rates look reasonable based on your Lustre >>> experience? >>> >>> Also any ideas on why I might be timing out on an occasional run? >>> >>> wendell >>> >>> >>> > > >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080515/a62cdf1e/attachment.html