Hello, Recently we installed 6 OSS pairs with 8 OST per pair. Total 48 OST''s. Each OST is with 3.7 TB. In all it is 177 TB file system. Lustre version installed is 1.8.1.1 and currently using client based on RHEL 5U2 which is 1.6.x. When running the individual OST test from performance perspecitve we are able to get around 17.5 GB performance. Out target is to cross 10 GBPS write performance using single file w/o -F option avoiding client side cache. I have reached max to 7.5GB for write performance , but not going beyond. I tried using stripe count as 48 for a single file along with default stripe size which is 1MB. But not able to cross 10 GBPS. Command line used for running the IOR as follow /opt/intel/mpi/bin64/mpirun --totalnum=96 --file=$PBS_NODEFILE --rsh=/usr/bin/ssh -1 --ordered --verbose -l -machinefile $PBS_NODEFILE -np 96 /newScratch/IOR/src/C/IOR.mpiio -a MPIIO -b 22G -C -i 3 -k -t 1m -w -r -R -W -x -N 96 -o /newScratch/hp.stripeC48/IOR.dat We have used lustre_config to create the file system. Appriciate your help. Regards SP
On 9/14/10 5:57 PM, satish patil wrote:> Hello, > > Recently we installed 6 OSS pairs with 8 OST per pair. Total 48 OST''s. Each OST is with 3.7 TB. In all it is 177 TB file system. Lustre version installed is 1.8.1.1 and currently using client based on RHEL 5U2 which is 1.6.x. When running the individual OST test from performance perspecitve we are able to get around 17.5 GB performance. Out target is to cross 10 GBPS write performance using single file w/o -F option avoiding client side cache. I have reached max to 7.5GB for write performance , but not going beyond. I tried using stripe count as 48 for a single file along with default stripe size which is 1MB. But not able to cross 10 GBPS.Can you give a detailed description for your system topology? We have met customer with more large theory bandwidth, but worse performance, because of the unexpected back-end storage performance in parallel. For I/O performance testing, full stripe maybe not the best choice. Using single stripe files, and spreading these relative small files to all OSTs evenly, maybe give better result.> Command line used for running the IOR as follow > /opt/intel/mpi/bin64/mpirun --totalnum=96 --file=$PBS_NODEFILE --rsh=/usr/bin/ssh -1 --ordered --verbose -l -machinefile $PBS_NODEFILE -np 96 /newScratch/IOR/src/C/IOR.mpiio -a MPIIO -b 22G -C -i 3 -k -t 1m -w -r -R -W -x -N 96 -o /newScratch/hp.stripeC48/IOR.dat > > We have used lustre_config to create the file system.On the other hand, lustre provides basic I/O performance utils (under lustre-iokit). You can use them step by step for the basic elements performance (like back-end storage, obdfilter, and network), which can help you to locate where the performance issues are. Cheers, Nasf> Appriciate your help. > > Regards > SP > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Thanks for your feedback. Back end storage P2000 G3 which is SAS based - 8 Gbps SAN using 450GB-15K. It is clients requirement to have performanace with Single file using all OST''s. Regards SP --- On Tue, 9/14/10, Fan Yong <yong.fan at whamcloud.com> wrote:> From: Fan Yong <yong.fan at whamcloud.com> > Subject: Re: [Lustre-discuss] IOR performance - Need help > To: lustre-discuss at lists.lustre.org > Date: Tuesday, September 14, 2010, 4:24 PM > ? On 9/14/10 5:57 PM, satish > patil wrote: > > Hello, > > > > Recently we installed 6 OSS pairs with 8 OST per pair. > Total 48 OST''s. Each OST is with 3.7 TB. In all it is 177 TB > file system. Lustre version installed is 1.8.1.1 and > currently using client based on RHEL 5U2 which is 1.6.x. > When running the individual OST test from performance > perspecitve we are able to get around 17.5 GB performance. > Out target is to cross 10 GBPS write performance using > single file w/o -F option avoiding client side cache.? > I have reached max to 7.5GB for write performance , but not > going beyond. I tried using stripe count as 48 for a single > file along with default stripe size which is 1MB. But not > able to cross 10 GBPS. > Can you give a detailed description for your system > topology? We have > met customer with more large theory bandwidth, but worse > performance, > because of the unexpected back-end storage performance in > parallel. > > For I/O performance testing, full stripe maybe not the best > choice. > Using single stripe files, and spreading these relative > small files to > all OSTs evenly, maybe give better result. > > Command line used for running the IOR as follow > > /opt/intel/mpi/bin64/mpirun --totalnum=96 > --file=$PBS_NODEFILE --rsh=/usr/bin/ssh -1 --ordered > --verbose -l -machinefile $PBS_NODEFILE -np 96 > /newScratch/IOR/src/C/IOR.mpiio -a MPIIO -b 22G -C -i 3 -k > -t 1m -w -r -R -W -x -N 96 -o > /newScratch/hp.stripeC48/IOR.dat > > > > We have used lustre_config to create the file system. > On the other hand, lustre provides basic I/O performance > utils (under > lustre-iokit). You can use them step by step for the basic > elements > performance (like back-end storage, obdfilter, and > network), which can > help you to locate where the performance issues are. > > > Cheers, > Nasf > > Appriciate your help. > > > > Regards > > SP > > > > > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On Tue, 2010-09-14 at 04:20 -0700, satish patil wrote:> Thanks for your feedback. Back end storage P2000 G3 which is SAS based - 8 Gbps SAN using 450GB-15K. It is clients requirement to have performanace with Single file using all OST''s.As fanyong said, you need to step back from doing lustre end-to-end testing and verify the performance of your components with the iokit. The lustre-iokit has various benchmark tools in it to test first your raw storage as individual units. Once you have done that and verified that your storage hardware is capable of the performance you are looking for, you can move on to the next benchmark which will test your storage in aggregate to reveal any bottlenecks. Once that has met your expectations you can benchmark the network->storage pipeline with a different tool. It is only through this methodological, step-by-step process that you are going to figure out if your hardware is truly capable of the performance you desire and if not, find out exactly where your bottleneck is. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100914/4e5b8832/attachment-0001.bin
On 9/14/10 7:20 PM, satish patil wrote:> Thanks for your feedback. Back end storage P2000 G3 which is SAS based - 8 Gbps SAN using 450GB-15K. It is clients requirement to have performanace with Single file using all OST''s. >It is quite necessary to verify that the raw system (without lustre) can achieve the parallel I/O performance more than 10GB/s as your expected. The real test result is more convincing than any nominal parallel I/O performance, especially for SAN infrastructure based storage. Cheers, Nasf> Regards > SP > > --- On Tue, 9/14/10, Fan Yong<yong.fan at whamcloud.com> wrote: > >> From: Fan Yong<yong.fan at whamcloud.com> >> Subject: Re: [Lustre-discuss] IOR performance - Need help >> To: lustre-discuss at lists.lustre.org >> Date: Tuesday, September 14, 2010, 4:24 PM >> On 9/14/10 5:57 PM, satish >> patil wrote: >>> Hello, >>> >>> Recently we installed 6 OSS pairs with 8 OST per pair. >> Total 48 OST''s. Each OST is with 3.7 TB. In all it is 177 TB >> file system. Lustre version installed is 1.8.1.1 and >> currently using client based on RHEL 5U2 which is 1.6.x. >> When running the individual OST test from performance >> perspecitve we are able to get around 17.5 GB performance. >> Out target is to cross 10 GBPS write performance using >> single file w/o -F option avoiding client side cache. >> I have reached max to 7.5GB for write performance , but not >> going beyond. I tried using stripe count as 48 for a single >> file along with default stripe size which is 1MB. But not >> able to cross 10 GBPS. >> Can you give a detailed description for your system >> topology? We have >> met customer with more large theory bandwidth, but worse >> performance, >> because of the unexpected back-end storage performance in >> parallel. >> >> For I/O performance testing, full stripe maybe not the best >> choice. >> Using single stripe files, and spreading these relative >> small files to >> all OSTs evenly, maybe give better result. >>> Command line used for running the IOR as follow >>> /opt/intel/mpi/bin64/mpirun --totalnum=96 >> --file=$PBS_NODEFILE --rsh=/usr/bin/ssh -1 --ordered >> --verbose -l -machinefile $PBS_NODEFILE -np 96 >> /newScratch/IOR/src/C/IOR.mpiio -a MPIIO -b 22G -C -i 3 -k >> -t 1m -w -r -R -W -x -N 96 -o >> /newScratch/hp.stripeC48/IOR.dat >>> We have used lustre_config to create the file system. >> On the other hand, lustre provides basic I/O performance >> utils (under >> lustre-iokit). You can use them step by step for the basic >> elements >> performance (like back-end storage, obdfilter, and >> network), which can >> help you to locate where the performance issues are. >> >> >> Cheers, >> Nasf >>> Appriciate your help. >>> >>> Regards >>> SP >>> >>> >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > >