Hi All, I am currently working on a 3 node oss cluster on centos 6.3 with lustre 2.3 with 12 osts on qdr infiniband. This is just a test bed cluster at this point and will be further scaled out. Currently my single file, single client write speed is about 840MB/s (tested with dd). I''m looking to get this up to 1300-1500MB/s with a multithreaded app to a single file from a single client. The file is currently striped across all osts. Does anyone have any tuning pointers to achieve this kind of performance for this setup? Thanks Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130506/de5ea949/attachment.html
The simplest method is to add multiple dd sessions writing to different files. After that you start getting into multi-client benchmarking utilities such as IOR and iozone which have more overhead to get your first run, but will give you better results. Also, don''t take the tool''s word for it, but measure the FS performance using various freely available tools for monitoring Lustre performance (or watch writes going to disk or over the IB fabric, or ...) -Ben Evans ________________________________ From: lustre-discuss-bounces at lists.lustre.org [lustre-discuss-bounces at lists.lustre.org] on behalf of Mike Ware [charnobyl3000 at gmail.com] Sent: Monday, May 06, 2013 2:27 PM To: lustre-discuss at lists.lustre.org Subject: [Lustre-discuss] Single file write performance tuning Hi All, I am currently working on a 3 node oss cluster on centos 6.3 with lustre 2.3 with 12 osts on qdr infiniband. This is just a test bed cluster at this point and will be further scaled out. Currently my single file, single client write speed is about 840MB/s (tested with dd). I''m looking to get this up to 1300-1500MB/s with a multithreaded app to a single file from a single client. The file is currently striped across all osts. Does anyone have any tuning pointers to achieve this kind of performance for this setup? Thanks Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130506/c79c89ac/attachment.html
Christopher J. Morrone
2013-May-06 20:53 UTC
[Lustre-discuss] Single file write performance tuning
Mike, That is going to be very challenging. There are known implementation problems in the Lustre 2.X client code that are holding back performance when you have multiple threads trying to write to the same file on the same lustre client node. See ticket: https://jira.hpdd.intel.com/browse/LU-1669 The problem is well understood, but so far no one has had time to spend on implementing the solution. Chris On 05/06/2013 11:27 AM, Mike Ware wrote:> Hi All, > I am currently working on a 3 node oss cluster on centos 6.3 with lustre > 2.3 with 12 osts on qdr infiniband. This is just a test bed cluster at > this point and will be further scaled out. Currently my single file, > single client write speed is about 840MB/s (tested with dd). I''m looking > to get this up to 1300-1500MB/s with a multithreaded app to a single > file from a single client. The file is currently striped across all > osts. Does anyone have any tuning pointers to achieve this kind of > performance for this setup? > > Thanks > Mike > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Jeremy Filizetti
2013-May-08 12:18 UTC
[Lustre-discuss] Single file write performance tuning
If your need is just 1300-1500 MB/s you might want to use Lustre 1.8.9. Your 840 MB/s seems consistent with the numbers I''ve seen for 2.3.61 but my 1.8.9 clients could do around 1.1-1.4 GB/s for the same dd. You''ll need checksums turned off on the 1.8 client. Single threaded IO is being tracked at: https://jira.hpdd.intel.com/browse/LU-1056 Jeremy On Mon, May 6, 2013 at 2:27 PM, Mike Ware <charnobyl3000 at gmail.com> wrote:> Hi All, > I am currently working on a 3 node oss cluster on centos 6.3 with lustre > 2.3 with 12 osts on qdr infiniband. This is just a test bed cluster at > this point and will be further scaled out. Currently my single file, > single client write speed is about 840MB/s (tested with dd). I''m looking to > get this up to 1300-1500MB/s with a multithreaded app to a single file from > a single client. The file is currently striped across all osts. Does anyone > have any tuning pointers to achieve this kind of performance for this setup? > > Thanks > Mike > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130508/aecf164f/attachment.html
Parinay Kondekar
2013-May-09 09:29 UTC
[Lustre-discuss] Single file write performance tuning
You might want to take a look at - http://wiki.lustre.org/manual/LustreManual20_HTML/LustreProc.html , (Lustre IO tunables) if not already. On 6 May 2013 23:57, Mike Ware <charnobyl3000 at gmail.com> wrote:> Hi All, > I am currently working on a 3 node oss cluster on centos 6.3 with lustre > 2.3 with 12 osts on qdr infiniband. This is just a test bed cluster at > this point and will be further scaled out. Currently my single file, > single client write speed is about 840MB/s (tested with dd). I''m looking to > get this up to 1300-1500MB/s with a multithreaded app to a single file from > a single client. The file is currently striped across all osts. Does anyone > have any tuning pointers to achieve this kind of performance for this setup? > > Thanks > Mike > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20130509/4b2139b8/attachment.html
Henwood, Richard
2013-May-09 14:51 UTC
[Lustre-discuss] Single file write performance tuning
On Thu, 2013-05-09 at 14:59 +0530, Parinay Kondekar wrote:> You might want to take a look at > - http://wiki.lustre.org/manual/LustreManual20_HTML/LustreProc.html , > (Lustre IO tunables) if not already. > >Given you''re using 2.3, the Manual maintained at: http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml maybe a better reference for you. ... as pointed out by Chris on this thread, however, there is an open issue (LU-1669) that will probably prove more of a restriction than adjusting turntables. cheers, Richard> > > On 6 May 2013 23:57, Mike Ware <charnobyl3000 at gmail.com> wrote: > Hi All, > I am currently working on a 3 node oss cluster on centos 6.3 > with lustre 2.3 with 12 osts on qdr infiniband. This is just > a test bed cluster at this point and will be further scaled > out. Currently my single file, single client write speed is > about 840MB/s (tested with dd). I''m looking to get this up to > 1300-1500MB/s with a multithreaded app to a single file from a > single client. The file is currently striped across all osts. > Does anyone have any tuning pointers to achieve this kind of > performance for this setup? > > > Thanks > Mike > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-- Richard.Henwood at intel.com High Performance Data Division