Hi Leandro-- There are two important things to note about the test: #1. If you are trying to utilize multiple OSTs for a test involving a single client writing to a single file, then be sure to stripe the file properly. There are two times when striping will help you: if you want to create a file bigger than will fit on one OST, or if you need more performance for a SINGLE FILE than can be provided by one OST. It sounds like your test falls into the latter category. We encourage most users to use a default stripe count of 1, because it tends to be that a single OST can by itself provide better performance than a single client. If you''re using one IDE disk per OST, however, that''s not really the case. So make sure that you''re striping appropriately. You can also use "lfs setstripe" to change the default striping for new files created in a directory. #2. Lustre 1.4.x has many significant performance improvements, especially for read In tests that we conducted with Lustre 1.4.x, read performs on par with write, instead of way behind. I hope that helps. -Phil On Sep 16, 2005, at 14:28, Leandro Tavares Carneiro wrote:> > I''m doing some tests in a lustre filesystem, using lustre 1.2.4, > and iozone show me a slow read/reread performance. > I''m using now 8 OSS with one OST each and a MDS with one MDT. All > disks are local to the OSS, using common IDE disks. I know, thats > bad, but is what i have now to do the tests. > The write performance is good, and the network bandwidth > utilization is higher on the write test than the read test. > The iozone comand i used: > > /opt/iozone/bin/iozone -b /root/report -e -t 4 -F test0 test1 test2 > test3 -i 0 -i 1 -k 0 -k 1 -M -r 256k -j 8 -P 0 -R -s 1g -T > > I''m using a very simple configuration. My script is very similar to > the one found on the HOWTO. I have tried with a stripe_size of 1M > and 4M, and it wont made too much difference.
Hi Phil, My main concern is with a production test i have made and the performance was terrible. In this test, the network wasn''t the ideal, but i expected more. This is why i have looked for a tool to test the individual performance and, after that, try to test with a lot of machines in parallel, some writing and others reading. The application i want to run read much more than write. All the process reads the same file, and some files are read and reread. This files can be small as some Megabytes or large as 50 Gigabytes. What i need is scalability, so i create a lustre file system with 16 OSS, each with one OST, to test a job with 500 CPUs running on this file system. But the job wont run. The process can''t open a main file, with 50Gb. I know the supported version is better. I will test it, and want to do that this week. -- Leandro Tavares Carneiro Petrobras TI/TI-E&P/STEP Suporte Tecnico de E&P Av Chile, 65 sala 1501 EDISE - Rio de Janeiro / RJ Tel: (0xx21) 3224-1427 Em Sex, 2005-09-16 às 15:57 -0400, Phil Schwan escreveu:> Hi Leandro-- > > There are two important things to note about the test: > > #1. If you are trying to utilize multiple OSTs for a test involving a > single client writing to a single file, then be sure to stripe the > file properly. > > There are two times when striping will help you: if you want to > create a file bigger than will fit on one OST, or if you need more > performance for a SINGLE FILE than can be provided by one OST. It > sounds like your test falls into the latter category. > > We encourage most users to use a default stripe count of 1, because > it tends to be that a single OST can by itself provide better > performance than a single client. If you''re using one IDE disk per > OST, however, that''s not really the case. > > So make sure that you''re striping appropriately. You can also use > "lfs setstripe" to change the default striping for new files created > in a directory. > > #2. Lustre 1.4.x has many significant performance improvements, > especially for read > > In tests that we conducted with Lustre 1.4.x, read performs on par > with write, instead of way behind. > > I hope that helps. > > -Phil > > On Sep 16, 2005, at 14:28, Leandro Tavares Carneiro wrote: > > > > I''m doing some tests in a lustre filesystem, using lustre 1.2.4, > > and iozone show me a slow read/reread performance. > > I''m using now 8 OSS with one OST each and a MDS with one MDT. All > > disks are local to the OSS, using common IDE disks. I know, thats > > bad, but is what i have now to do the tests. > > The write performance is good, and the network bandwidth > > utilization is higher on the write test than the read test. > > The iozone comand i used: > > > > /opt/iozone/bin/iozone -b /root/report -e -t 4 -F test0 test1 test2 > > test3 -i 0 -i 1 -k 0 -k 1 -M -r 256k -j 8 -P 0 -R -s 1g -T > > > > I''m using a very simple configuration. My script is very similar to > > the one found on the HOWTO. I have tried with a stripe_size of 1M > > and 4M, and it wont made too much difference.
I''m doing some tests in a lustre filesystem, using lustre 1.2.4, and iozone show me a slow read/reread performance. I''m using now 8 OSS with one OST each and a MDS with one MDT. All disks are local to the OSS, using common IDE disks. I know, thats bad, but is what i have now to do the tests. The write performance is good, and the network bandwidth utilization is higher on the write test than the read test. The iozone comand i used: /opt/iozone/bin/iozone -b /root/report -e -t 4 -F test0 test1 test2 test3 -i 0 -i 1 -k 0 -k 1 -M -r 256k -j 8 -P 0 -R -s 1g -T I''m using a very simple configuration. My script is very similar to the one found on the HOWTO. I have tried with a stripe_size of 1M and 4M, and it wont made too much difference. The nodes i''m using are Dual Opteron 246 with 4Gb RAM. All network interconnect is gigabit ethernet. I''m running a "generic RHEL", CentOS 3.4 on all nodes. In other test we do here, if gpfs, the read performance was better than the write! And 10 times faster than lustre. Please, if anoyone have some advice to me, to make it perform better, will help. I have plans to test the supported version of lustre sometime in the next week. Thanks in advance, -- Leandro Tavares Carneiro Petrobras TI/TI-E&P/STEP Suporte Tecnico de E&P Av Chile, 65 sala 1501 EDISE - Rio de Janeiro / RJ Tel: (0xx21) 3224-1427
Hi, We are testing lustre 1.4.5 release to public recently and the read performance was not good as expected, at least to me. I''m a little lost on what i can do to make it better. The test we do was using 25 OSS, each one with one OST and a MDS. This numbers means i have 2 OSS per rack, on a 13 rack cluster. Each rack have 66 nodes and all nodes have the same hardware, with 2 CPUs Opteron 246. The local disk are a regular ATA. I am using the pre-compiled kernel 2.4.21-32 and lustre packages, running on CentOS 3.4 (RHEL 3 Update 4 compiled from sources by the community). It runs without errors, but i see an warning in the messages and i don''t know what it means: Dec 6 15:04:39 ost1 kernel: LustreError: 2673:0: (pack_generic.c:359:lustre_msg_buf()) msg 0000010008c52140 buffer[1] size 0 too small (required 1) This message appears on all the OSTs when i get up the MDS. I don''t know how can i correct or improve it. If anyone can give to me some help on this, it will be great. Thanks in advance, -- Leandro Tavares Carneiro Petrobras TI/TI-E&P/STEP Suporte T?cnico de E&P Av Chile, 65 sala 1501 EDISE - Rio de Janeiro / RJ Tel: (0xx21) 3224-1427
On Dec 08, 2005 09:23 -0200, Leandro Tavares Carneiro wrote:> We are testing lustre 1.4.5 release to public recently and the read > performance was not good as expected, at least to me.Could you provide more detail on what test you are running, what performance you are getting, and what you think you should be getting? It may be a simple fix (e.g. change of default striping parameters), but I can''t comment until I know more about what you are testing.> I''m a little lost on what i can do to make it better. The test we do was > using 25 OSS, each one with one OST and a MDS. This numbers means i have > 2 OSS per rack, on a 13 rack cluster. Each rack have 66 nodes and all > nodes have the same hardware, with 2 CPUs Opteron 246. The local disk > are a regular ATA.The OSTs are using the local ATA disks, or some other disks?> It runs without errors, but i see an warning in the messages: > > Dec 6 15:04:39 ost1 kernel: LustreError: 2673:0: > (pack_generic.c:359:lustre_msg_buf()) msg 0000010008c52140 buffer[1] > size 0 too small (required 1)This is a harmless message, and can be ignored (it is fixed in a later release), sorry. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.