Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: 7stripe.png Type: image/png Size: 5387 bytes Desc: not available Url : http://mail.clusterfs.com/pipermail/lustre-devel/attachments/20061114/fbe79d25/7stripe.png -------------- next part -------------- A non-text attachment was scrubbed... Name: 8stripe.png Type: image/png Size: 5276 bytes Desc: not available Url : http://mail.clusterfs.com/pipermail/lustre-devel/attachments/20061114/fbe79d25/8stripe.png
Hi Peter, On Tue, 14 Nov 2006, Peter J Braam wrote:> We are not seeing this everywhere, and I am sending this message to > provoke sufficient response from the community to hopefully lead to some > understanding.That sounds like a very interesting problem, however we have no idea what workload and system were used to give these figures. Could you tell us a bit more? -- Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net
Some key parameters are at the top: - single shared file - striped over 7 and 8 OST''s - liblustre clients (i.e. no caching, server side locking) - application is doing 4MB IO''s It would be good to know that is the stripe_size (default is 4 or 8 MB I think), and the precise IOR parameters used to create this run. - Peter -> -----Original Message----- > From: Jean-Marc Saffroy [mailto:jean-marc.saffroy@ext.bull.net] > Sent: Wednesday, November 15, 2006 6:55 AM > To: Peter J Braam > Cc: lustre-devel@clusterfs.com > Subject: Re: [Lustre-devel] Wide striping puzzles > > Hi Peter, > > On Tue, 14 Nov 2006, Peter J Braam wrote: > > > We are not seeing this everywhere, and I am sending this message to > > provoke sufficient response from the community to hopefully lead to > > some understanding. > > That sounds like a very interesting problem, however we have > no idea what workload and system were used to give these figures. > > Could you tell us a bit more? > > > -- > Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net
Jean-Marc points out I only replied to him first try :) On Wed, 2006-11-15 at 14:54 +0100, Jean-Marc Saffroy wrote:> Hi Peter, > > On Tue, 14 Nov 2006, Peter J Braam wrote: > > > We are not seeing this everywhere, and I am sending this message to > > provoke sufficient response from the community to hopefully lead to some > > understanding. > > That sounds like a very interesting problem, however we have no idea what > workload and system were used to give these figures. > > Could you tell us a bit more?That was my first thought as well. If this is the result of IOR testing I''d be interested in the actual script used as well as all the environmental information: o client hardware o storage hardware o networking model o Lustre version o kernel version o non-standard patches o tunings Time permitting perhaps we can run some comparison tests around here. Cheers, Andrew
Lee, Do you happen to know what exactly was done? - Peter -> -----Original Message----- > From: lustre-devel-bounces@clusterfs.com > [mailto:lustre-devel-bounces@clusterfs.com] On Behalf Of > Andrew Uselton > Sent: Wednesday, November 15, 2006 8:51 AM > To: lustre-devel@clusterfs.com > Subject: Re: [Lustre-devel] Wide striping puzzles > > Jean-Marc points out I only replied to him first try :) > > On Wed, 2006-11-15 at 14:54 +0100, Jean-Marc Saffroy wrote: > > Hi Peter, > > > > On Tue, 14 Nov 2006, Peter J Braam wrote: > > > > > We are not seeing this everywhere, and I am sending this > message to > > > provoke sufficient response from the community to > hopefully lead to > > > some understanding. > > > > That sounds like a very interesting problem, however we > have no idea > > what workload and system were used to give these figures. > > > > Could you tell us a bit more? > > That was my first thought as well. If this is the result of > IOR testing I''d be interested in the actual script used as > well as all the environmental information: > o client hardware > o storage hardware > o networking model > o Lustre version > o kernel version > o non-standard patches > o tunings > > Time permitting perhaps we can run some comparison tests around here. > Cheers, > Andrew > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel
Some additional info on the IOR runs:> o client hardwareAMD opteron, cray xt3 compute node, liblustre client> o storage hardwareDDN s2a8500> o networking modelcray portals> o Lustre version1.4.6 + cray patches, mostly from release 1.4.35> o kernel version (on OSS and MDS)Linux rsclogin01 2.6.5-7.276-ss #7 Tue Sep 26 18:27:08 PDT 2006 x86_64 x86_64 x86_64 GNU/Linux (on clients the OS is catamount)> o non-standard patchescray xt3 build, from release 1.4.35> o tuningsno tuning on test cluster, other than to turn off debugging on the clients>The I/O system on that machine contains 8 OSTs, 2 per OSS, MDS on a separate node. Those are Linux nodes. IOR args for 7stripe plot: yod -sz $n /home/user/utils/IOR -g -k -v -a POSIX -C -i 1 -o /scratch1/user/7-stripe/$n/ior_test -b 7168m -w -t 8m yod -sz $n /home/user/utils/IOR -g -v -a POSIX -C -i 1 -o /scratch1/user/7-stripe/$n/ior_test -b 7168m -r -t 8m for 8stripe plot: yod -sz $n /home/user/utils/IOR -g -k -v -a POSIX -C -i 1 -o /scratch1/user/8-stripe/$n/ior_test -b 8192m -w -t 8m yod -sz $n /home/user/utils/IOR -k -g -v -a POSIX -C -i 1 -o /scratch1/user/8-stripe/$n/ior_test -b 8192m -r -t 8m n in [1,24] The striping on the directories are set prior to the run: lfs setstripe /scratch1/user/7-stripe 2097152 -1 7 lfs setstripe /scratch1/user/8-stripe 2097152 -1 7 Ruth On Wed, 2006-11-15 at 13:58 -0700, Peter J. Braam wrote:> Some key parameters are at the top: > - single shared file > - striped over 7 and 8 OST''s > - liblustre clients (i.e. no caching, server side locking) > - application is doing 4MB IO''s > > It would be good to know that is the stripe_size (default is 4 or 8 MB > I > think), and the precise IOR parameters used to create this run. > > - Peter - > > > -----Original Message----- > > From: Jean-Marc Saffroy [mailto:jean-marc.saffroy@ext.bull.net] > > Sent: Wednesday, November 15, 2006 6:55 AM > > To: Peter J Braam > > Cc: lustre-devel@clusterfs.com > > Subject: Re: [Lustre-devel] Wide striping puzzles > > > > Hi Peter, > > > > On Tue, 14 Nov 2006, Peter J Braam wrote: > > > > > We are not seeing this everywhere, and I am sending this message > to > > > provoke sufficient response from the community to hopefully lead > to > > > some understanding. > > > > That sounds like a very interesting problem, however we have > > no idea what workload and system were used to give these figures. > > > > Could you tell us a bit more? > > > > > > -- > > Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > > >
On Mon, 2006-11-20 at 10:27 -0700, Ruth Klundt wrote:> Some additional info on the IOR runs: > > > o client hardware > AMD opteron, cray xt3 compute node, liblustre client > > o storage hardware > DDN s2a8500 > > o networking model > cray portals > > o Lustre version > 1.4.6 + cray patches, mostly from release 1.4.35 > > o kernel version (on OSS and MDS) > Linux rsclogin01 2.6.5-7.276-ss #7 Tue Sep 26 18:27:08 PDT 2006 x86_64 > x86_64 x86_64 GNU/Linux > (on clients the OS is catamount) > > o non-standard patches > cray xt3 build, from release 1.4.35 > > o tunings > no tuning on test cluster, other than to turn off debugging on the > clients > > > > The I/O system on that machine contains 8 OSTs, 2 per OSS, MDS on a > separate node. Those are Linux nodes. > > IOR args for 7stripe plot: > > yod -sz $n /home/user/utils/IOR -g -k -v -a POSIX -C -i 1 > -o /scratch1/user/7-stripe/$n/ior_test -b 7168m -w -t 8m > yod -sz $n /home/user/utils/IOR -g -v -a POSIX -C -i 1 > -o /scratch1/user/7-stripe/$n/ior_test -b 7168m -r -t 8m > > for 8stripe plot: > yod -sz $n /home/user/utils/IOR -g -k -v -a POSIX -C -i 1 > -o /scratch1/user/8-stripe/$n/ior_test -b 8192m -w -t 8m > yod -sz $n /home/user/utils/IOR -k -g -v -a POSIX -C -i 1 > -o /scratch1/user/8-stripe/$n/ior_test -b 8192m -r -t 8m > > n in [1,24] > > The striping on the directories are set prior to the run: > lfs setstripe /scratch1/user/7-stripe 2097152 -1 7 > lfs setstripe /scratch1/user/8-stripe 2097152 -1 7That''s 8, not 7...> > Ruth > > > On Wed, 2006-11-15 at 13:58 -0700, Peter J. Braam wrote: > > Some key parameters are at the top: > > - single shared file > > - striped over 7 and 8 OST''s > > - liblustre clients (i.e. no caching, server side locking) > > - application is doing 4MB IO''s > > > > It would be good to know that is the stripe_size (default is 4 or 8 MB > > I > > think), and the precise IOR parameters used to create this run. > > > > - Peter - > > > > > -----Original Message----- > > > From: Jean-Marc Saffroy [mailto:jean-marc.saffroy@ext.bull.net] > > > Sent: Wednesday, November 15, 2006 6:55 AM > > > To: Peter J Braam > > > Cc: lustre-devel@clusterfs.com > > > Subject: Re: [Lustre-devel] Wide striping puzzles > > > > > > Hi Peter, > > > > > > On Tue, 14 Nov 2006, Peter J Braam wrote: > > > > > > > We are not seeing this everywhere, and I am sending this message > > to > > > > provoke sufficient response from the community to hopefully lead > > to > > > > some understanding. > > > > > > That sounds like a very interesting problem, however we have > > > no idea what workload and system were used to give these figures. > > > > > > Could you tell us a bit more? > > > > > > > > > -- > > > Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net > > > > _______________________________________________ > > Lustre-devel mailing list > > Lustre-devel@clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > > > > > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > >
On Mon, 2006-11-20 at 10:27 -0700, Ruth Klundt wrote:> Some additional info on the IOR runs: > > > o client hardware > AMD opteron, cray xt3 compute node, liblustre client > > o storage hardware > DDN s2a8500 > > o networking model > cray portals > > o Lustre version > 1.4.6 + cray patches, mostly from release 1.4.35 > > o kernel version (on OSS and MDS) > Linux rsclogin01 2.6.5-7.276-ss #7 Tue Sep 26 18:27:08 PDT 2006 x86_64 > x86_64 x86_64 GNU/Linux > (on clients the OS is catamount) > > o non-standard patches > cray xt3 build, from release 1.4.35 > > o tunings > no tuning on test cluster, other than to turn off debugging on the > clients > > > > The I/O system on that machine contains 8 OSTs, 2 per OSS, MDS on a > separate node. Those are Linux nodes. > > IOR args for 7stripe plot: > > yod -sz $n /home/user/utils/IOR -g -k -v -a POSIX -C -i 1 > -o /scratch1/user/7-stripe/$n/ior_test -b 7168m -w -t 8m > yod -sz $n /home/user/utils/IOR -g -v -a POSIX -C -i 1 > -o /scratch1/user/7-stripe/$n/ior_test -b 7168m -r -t 8m > > for 8stripe plot: > yod -sz $n /home/user/utils/IOR -g -k -v -a POSIX -C -i 1 > -o /scratch1/user/8-stripe/$n/ior_test -b 8192m -w -t 8m > yod -sz $n /home/user/utils/IOR -k -g -v -a POSIX -C -i 1 > -o /scratch1/user/8-stripe/$n/ior_test -b 8192m -r -t 8m > > n in [1,24] > > The striping on the directories are set prior to the run: > lfs setstripe /scratch1/user/7-stripe 2097152 -1 7 > lfs setstripe /scratch1/user/8-stripe 2097152 -1 7On the 8-stripe run, you meant ... -1 8, yes? --Lee> > Ruth > > > On Wed, 2006-11-15 at 13:58 -0700, Peter J. Braam wrote: > > Some key parameters are at the top: > > - single shared file > > - striped over 7 and 8 OST''s > > - liblustre clients (i.e. no caching, server side locking) > > - application is doing 4MB IO''s > > > > It would be good to know that is the stripe_size (default is 4 or 8 MB > > I > > think), and the precise IOR parameters used to create this run. > > > > - Peter - > > > > > -----Original Message----- > > > From: Jean-Marc Saffroy [mailto:jean-marc.saffroy@ext.bull.net] > > > Sent: Wednesday, November 15, 2006 6:55 AM > > > To: Peter J Braam > > > Cc: lustre-devel@clusterfs.com > > > Subject: Re: [Lustre-devel] Wide striping puzzles > > > > > > Hi Peter, > > > > > > On Tue, 14 Nov 2006, Peter J Braam wrote: > > > > > > > We are not seeing this everywhere, and I am sending this message > > to > > > > provoke sufficient response from the community to hopefully lead > > to > > > > some understanding. > > > > > > That sounds like a very interesting problem, however we have > > > no idea what workload and system were used to give these figures. > > > > > > Could you tell us a bit more? > > > > > > > > > -- > > > Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net > > > > _______________________________________________ > > Lustre-devel mailing list > > Lustre-devel@clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > > > > > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > >
On Mon, 2006-11-20 at 10:30 -0700, Lee Ward wrote:> On Mon, 2006-11-20 at 10:27 -0700, Ruth Klundt wrote: > > Some additional info on the IOR runs: > > > > > o client hardware > > AMD opteron, cray xt3 compute node, liblustre client > > > o storage hardware > > DDN s2a8500 > > > o networking model > > cray portals > > > o Lustre version > > 1.4.6 + cray patches, mostly from release 1.4.35 > > > o kernel version (on OSS and MDS) > > Linux rsclogin01 2.6.5-7.276-ss #7 Tue Sep 26 18:27:08 PDT 2006 x86_64 > > x86_64 x86_64 GNU/Linux > > (on clients the OS is catamount) > > > o non-standard patches > > cray xt3 build, from release 1.4.35 > > > o tunings > > no tuning on test cluster, other than to turn off debugging on the > > clients > > > > > > > The I/O system on that machine contains 8 OSTs, 2 per OSS, MDS on a > > separate node. Those are Linux nodes. > > > > IOR args for 7stripe plot: > > > > yod -sz $n /home/user/utils/IOR -g -k -v -a POSIX -C -i 1 > > -o /scratch1/user/7-stripe/$n/ior_test -b 7168m -w -t 8m > > yod -sz $n /home/user/utils/IOR -g -v -a POSIX -C -i 1 > > -o /scratch1/user/7-stripe/$n/ior_test -b 7168m -r -t 8m > > > > for 8stripe plot: > > yod -sz $n /home/user/utils/IOR -g -k -v -a POSIX -C -i 1 > > -o /scratch1/user/8-stripe/$n/ior_test -b 8192m -w -t 8m > > yod -sz $n /home/user/utils/IOR -k -g -v -a POSIX -C -i 1 > > -o /scratch1/user/8-stripe/$n/ior_test -b 8192m -r -t 8m > > > > n in [1,24] > > > > The striping on the directories are set prior to the run: > > lfs setstripe /scratch1/user/7-stripe 2097152 -1 7 > > lfs setstripe /scratch1/user/8-stripe 2097152 -1 7 > > On the 8-stripe run, you meant ... -1 8, yes? >Right> --Lee > > > > > Ruth > > > > > > On Wed, 2006-11-15 at 13:58 -0700, Peter J. Braam wrote: > > > Some key parameters are at the top: > > > - single shared file > > > - striped over 7 and 8 OST''s > > > - liblustre clients (i.e. no caching, server side locking) > > > - application is doing 4MB IO''s > > > > > > It would be good to know that is the stripe_size (default is 4 or 8 MB > > > I > > > think), and the precise IOR parameters used to create this run. > > > > > > - Peter - > > > > > > > -----Original Message----- > > > > From: Jean-Marc Saffroy [mailto:jean-marc.saffroy@ext.bull.net] > > > > Sent: Wednesday, November 15, 2006 6:55 AM > > > > To: Peter J Braam > > > > Cc: lustre-devel@clusterfs.com > > > > Subject: Re: [Lustre-devel] Wide striping puzzles > > > > > > > > Hi Peter, > > > > > > > > On Tue, 14 Nov 2006, Peter J Braam wrote: > > > > > > > > > We are not seeing this everywhere, and I am sending this message > > > to > > > > > provoke sufficient response from the community to hopefully lead > > > to > > > > > some understanding. > > > > > > > > That sounds like a very interesting problem, however we have > > > > no idea what workload and system were used to give these figures. > > > > > > > > Could you tell us a bit more? > > > > > > > > > > > > -- > > > > Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net > > > > > > _______________________________________________ > > > Lustre-devel mailing list > > > Lustre-devel@clusterfs.com > > > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > > > > > > > > > > > > > _______________________________________________ > > Lustre-devel mailing list > > Lustre-devel@clusterfs.com > > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > > > > > >