I am seeing the following odd behavior. I have several OSSes, each with a 7T RAID 5. If I use a single client to create a single 1T file which is striped to a single OST, the performance starts off at about 400 MB/s (which is typical for my HW), then gradually decreases, until it reaches 250 MB/s. I''ve seen this with both IOZone and dd. As an experiment, I wrote a script that creates two hundred 10G files, without removing them. Again, the performance starts off at 400 MB/s. But, the performance stays nearly the same throughout the test. The only difference between these tests is that in the second case, there are lots of opens & closes, and in the first case, just a single open and close. Can anyone explain what is happening here, and how to possible fix it? Thanks. Roger Spellman Sr. Staff Engineer Terascala, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080530/afda1d3d/attachment.html
Hi There, I was wondering if there was a plan to merge the patches to kernel by Lustre to the regular Kernel.org path? If so, what is the plan? Thanks a lot. Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080530/4f891afe/attachment-0001.html
Kumaran Rajaram
2008-May-30 21:00 UTC
[Lustre-discuss] Performance Drop creating big files
Roger, I would suspect the Lustre client side caching influencing your write performance. 10GB is not adequate, try atleast 4x of the RAM size for file-sizes. Try doing the same tests with O_DIRECT flag as it''d truly measure your disk I/O performance bypassing FS + buffer cache. HTH, -Kums On Fri, 2008-05-30 at 16:25 -0400, Roger Spellman wrote:> I am seeing the following odd behavior. I have several OSSes, each > with a 7T RAID 5. > > > > If I use a single client to create a single 1T file which is striped > to a single OST, the performance starts off at about 400 MB/s (which > is typical for my HW), then gradually decreases, until it reaches 250 > MB/s. I?ve seen this with both IOZone and dd. > > > As an experiment, I wrote a script that creates two hundred 10G files, > without removing them. Again, the performance starts off at 400 MB/s. > But, the performance stays nearly the same throughout the test. > > > > The only difference between these tests is that in the second case, > there are lots of opens & closes, and in the first case, just a single > open and close. > > > > Can anyone explain what is happening here, and how to possible fix it? > > > > Thanks. > > > > Roger Spellman > > Sr. Staff Engineer > > Terascala, Inc. > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Lundgren, Andrew
2008-May-30 21:03 UTC
[Lustre-discuss] Performance Drop creating big files
Are you suggesting he use machines with 40G of RAM to work with 10G files? We have many 800-900G files... I am not sure that is a realistic number. -- Andrew> -----Original Message----- > From: lustre-discuss-bounces at lists.lustre.org > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of > Kumaran Rajaram > Sent: Friday, May 30, 2008 3:01 PM > To: Roger Spellman > Cc: lustre-discuss at lists.lustre.org > Subject: Re: [Lustre-discuss] Performance Drop creating big files > > Roger, > > I would suspect the Lustre client side caching influencing your write > performance. 10GB is not adequate, try atleast 4x of the RAM size for > file-sizes. Try doing the same tests with O_DIRECT flag as it''d truly > measure your disk I/O performance bypassing FS + buffer cache. > > HTH, > -Kums > > On Fri, 2008-05-30 at 16:25 -0400, Roger Spellman wrote: > > I am seeing the following odd behavior. I have several OSSes, each > > with a 7T RAID 5. > > > > > > > > If I use a single client to create a single 1T file which is striped > > to a single OST, the performance starts off at about 400 MB/s (which > > is typical for my HW), then gradually decreases, until it > reaches 250 > > MB/s. I''ve seen this with both IOZone and dd. > > > > > > As an experiment, I wrote a script that creates two hundred > 10G files, > > without removing them. Again, the performance starts off > at 400 MB/s. > > But, the performance stays nearly the same throughout the test. > > > > > > > > The only difference between these tests is that in the second case, > > there are lots of opens & closes, and in the first case, > just a single > > open and close. > > > > > > > > Can anyone explain what is happening here, and how to > possible fix it? > > > > > > > > Thanks. > > > > > > > > Roger Spellman > > > > Sr. Staff Engineer > > > > Terascala, Inc. > > > > > > > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Kumaran Rajaram
2008-May-30 21:07 UTC
[Lustre-discuss] Performance Drop creating big files
I meant use file-size that are 4x the RAM size :-). So if your RAM size is 4GB, use atleast 16GB file-sizes for the benchmarks. On Fri, 2008-05-30 at 15:03 -0600, Lundgren, Andrew wrote:> Are you suggesting he use machines with 40G of RAM to work with 10G files? > > We have many 800-900G files... I am not sure that is a realistic number. > > -- > Andrew > > > -----Original Message----- > > From: lustre-discuss-bounces at lists.lustre.org > > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of > > Kumaran Rajaram > > Sent: Friday, May 30, 2008 3:01 PM > > To: Roger Spellman > > Cc: lustre-discuss at lists.lustre.org > > Subject: Re: [Lustre-discuss] Performance Drop creating big files > > > > Roger, > > > > I would suspect the Lustre client side caching influencing your write > > performance. 10GB is not adequate, try atleast 4x of the RAM size for > > file-sizes. Try doing the same tests with O_DIRECT flag as it''d truly > > measure your disk I/O performance bypassing FS + buffer cache. > > > > HTH, > > -Kums > > > > On Fri, 2008-05-30 at 16:25 -0400, Roger Spellman wrote: > > > I am seeing the following odd behavior. I have several OSSes, each > > > with a 7T RAID 5. > > > > > > > > > > > > If I use a single client to create a single 1T file which is striped > > > to a single OST, the performance starts off at about 400 MB/s (which > > > is typical for my HW), then gradually decreases, until it > > reaches 250 > > > MB/s. I''ve seen this with both IOZone and dd. > > > > > > > > > As an experiment, I wrote a script that creates two hundred > > 10G files, > > > without removing them. Again, the performance starts off > > at 400 MB/s. > > > But, the performance stays nearly the same throughout the test. > > > > > > > > > > > > The only difference between these tests is that in the second case, > > > there are lots of opens & closes, and in the first case, > > just a single > > > open and close. > > > > > > > > > > > > Can anyone explain what is happening here, and how to > > possible fix it? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > Roger Spellman > > > > > > Sr. Staff Engineer > > > > > > Terascala, Inc. > > > > > > > > > > > > > > > _______________________________________________ > > > Lustre-discuss mailing list > > > Lustre-discuss at lists.lustre.org > > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
Lundgren, Andrew
2008-May-30 21:07 UTC
[Lustre-discuss] Performance Drop creating big files
I get it. Thanks!> -----Original Message----- > From: Kumaran Rajaram [mailto:krajaram at sgi.com] > Sent: Friday, May 30, 2008 3:07 PM > To: Lundgren, Andrew > Cc: Roger Spellman; lustre-discuss at lists.lustre.org > Subject: RE: [Lustre-discuss] Performance Drop creating big files > > > I meant use file-size that are 4x the RAM size :-). So if > your RAM size > is 4GB, use atleast 16GB file-sizes for the benchmarks. > > On Fri, 2008-05-30 at 15:03 -0600, Lundgren, Andrew wrote: > > Are you suggesting he use machines with 40G of RAM to work > with 10G files? > > > > We have many 800-900G files... I am not sure that is a > realistic number. > > > > -- > > Andrew > > > > > -----Original Message----- > > > From: lustre-discuss-bounces at lists.lustre.org > > > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of > > > Kumaran Rajaram > > > Sent: Friday, May 30, 2008 3:01 PM > > > To: Roger Spellman > > > Cc: lustre-discuss at lists.lustre.org > > > Subject: Re: [Lustre-discuss] Performance Drop creating big files > > > > > > Roger, > > > > > > I would suspect the Lustre client side caching > influencing your write > > > performance. 10GB is not adequate, try atleast 4x of the > RAM size for > > > file-sizes. Try doing the same tests with O_DIRECT flag > as it''d truly > > > measure your disk I/O performance bypassing FS + buffer cache. > > > > > > HTH, > > > -Kums > > > > > > On Fri, 2008-05-30 at 16:25 -0400, Roger Spellman wrote: > > > > I am seeing the following odd behavior. I have several > OSSes, each > > > > with a 7T RAID 5. > > > > > > > > > > > > > > > > If I use a single client to create a single 1T file > which is striped > > > > to a single OST, the performance starts off at about > 400 MB/s (which > > > > is typical for my HW), then gradually decreases, until it > > > reaches 250 > > > > MB/s. I''ve seen this with both IOZone and dd. > > > > > > > > > > > > As an experiment, I wrote a script that creates two hundred > > > 10G files, > > > > without removing them. Again, the performance starts off > > > at 400 MB/s. > > > > But, the performance stays nearly the same throughout the test. > > > > > > > > > > > > > > > > The only difference between these tests is that in the > second case, > > > > there are lots of opens & closes, and in the first case, > > > just a single > > > > open and close. > > > > > > > > > > > > > > > > Can anyone explain what is happening here, and how to > > > possible fix it? > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > Roger Spellman > > > > > > > > Sr. Staff Engineer > > > > > > > > Terascala, Inc. > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Lustre-discuss mailing list > > > > Lustre-discuss at lists.lustre.org > > > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > _______________________________________________ > > > Lustre-discuss mailing list > > > Lustre-discuss at lists.lustre.org > > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > >
To refer to your original discussion, how were you writing your 1TB file? I''m mainly wondering about the blocksize of the individual writes. Marty Barnaby Lundgren, Andrew wrote:> I get it. > > Thanks! > > >> -----Original Message----- >> From: Kumaran Rajaram [mailto:krajaram at sgi.com] >> Sent: Friday, May 30, 2008 3:07 PM >> To: Lundgren, Andrew >> Cc: Roger Spellman; lustre-discuss at lists.lustre.org >> Subject: RE: [Lustre-discuss] Performance Drop creating big files >> >> >> I meant use file-size that are 4x the RAM size :-). So if >> your RAM size >> is 4GB, use atleast 16GB file-sizes for the benchmarks. >> >> On Fri, 2008-05-30 at 15:03 -0600, Lundgren, Andrew wrote: >> >>> Are you suggesting he use machines with 40G of RAM to work >>> >> with 10G files? >> >>> We have many 800-900G files... I am not sure that is a >>> >> realistic number. >> >>> -- >>> Andrew >>> >>> >>>> -----Original Message----- >>>> From: lustre-discuss-bounces at lists.lustre.org >>>> [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of >>>> Kumaran Rajaram >>>> Sent: Friday, May 30, 2008 3:01 PM >>>> To: Roger Spellman >>>> Cc: lustre-discuss at lists.lustre.org >>>> Subject: Re: [Lustre-discuss] Performance Drop creating big files >>>> >>>> Roger, >>>> >>>> I would suspect the Lustre client side caching >>>> >> influencing your write >> >>>> performance. 10GB is not adequate, try atleast 4x of the >>>> >> RAM size for >> >>>> file-sizes. Try doing the same tests with O_DIRECT flag >>>> >> as it''d truly >> >>>> measure your disk I/O performance bypassing FS + buffer cache. >>>> >>>> HTH, >>>> -Kums >>>> >>>> On Fri, 2008-05-30 at 16:25 -0400, Roger Spellman wrote: >>>> >>>>> I am seeing the following odd behavior. I have several >>>>> >> OSSes, each >> >>>>> with a 7T RAID 5. >>>>> >>>>> >>>>> >>>>> If I use a single client to create a single 1T file >>>>> >> which is striped >> >>>>> to a single OST, the performance starts off at about >>>>> >> 400 MB/s (which >> >>>>> is typical for my HW), then gradually decreases, until it >>>>> >>>> reaches 250 >>>> >>>>> MB/s. I''ve seen this with both IOZone and dd. >>>>> >>>>> >>>>> As an experiment, I wrote a script that creates two hundred >>>>> >>>> 10G files, >>>> >>>>> without removing them. Again, the performance starts off >>>>> >>>> at 400 MB/s. >>>> >>>>> But, the performance stays nearly the same throughout the test. >>>>> >>>>> >>>>> >>>>> The only difference between these tests is that in the >>>>> >> second case, >> >>>>> there are lots of opens & closes, and in the first case, >>>>> >>>> just a single >>>> >>>>> open and close. >>>>> >>>>> >>>>> >>>>> Can anyone explain what is happening here, and how to >>>>> >>>> possible fix it? >>>> >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> >>>>> Roger Spellman >>>>> >>>>> Sr. Staff Engineer >>>>> >>>>> Terascala, Inc. >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>> >>>> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080530/6f849482/attachment.html
Hi Roger, This sounds terribly familiar. The likely cause can be found by reading through the following thread: http://lists.lustre.org/pipermail/lustre-discuss/2008-March/007024.html Note that running dumpe2fs on each device, as suggested by one of the messages in the thread, did *not* result in a performance improvement. j On May 30, 2008, at 1:25 PM, Roger Spellman wrote:> I am seeing the following odd behavior. I have several OSSes, each > with a 7T RAID 5. > > > > If I use a single client to create a single 1T file which is > striped to a single OST, the performance starts off at about 400 MB/ > s (which is typical for my HW), then gradually decreases, until it > reaches 250 MB/s. I?ve seen this with both IOZone and dd. > > > As an experiment, I wrote a script that creates two hundred 10G > files, without removing them. Again, the performance starts off at > 400 MB/s. But, the performance stays nearly the same throughout > the test. > > > > The only difference between these tests is that in the second case, > there are lots of opens & closes, and in the first case, just a > single open and close. > > > > Can anyone explain what is happening here, and how to possible fix it? > > > > Thanks. > > > > Roger Spellman > > Sr. Staff Engineer > > Terascala, Inc. > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- Jason Rappleye System Administrator NASA Advanced Supercomputing Division NASA Ames Research Center Moffett Field, CA 94035 jason.rappleye at nasa.gov
> I would suspect the Lustre client side caching influencing your write > performance. 10GB is not adequate, try atleast 4x of the RAM size for > file-sizes. > HTH, > -KumsKums, thanks for the suggestion. I have 4G of RAM on my client and 4G of RAM on my server. I typically create 20G files, but I''ve found that 10G files are nearly the same performance. The performance degradation that I see starts occurring at about 100G. So, I don''t think that it is caching that is causing the problems. Roger Spellman Sr. Staff Engineer Terascala, Inc.