>>>>> "Andreas" == Andreas Dilger <adilger@clusterfs.com> writes:Goswin> On Mar 08, 2006 00:02 +0100, Goswin von Brederlow wrote: Goswin> The setup is pretty simple. I have one server doing meta Goswin> data and object storage and a second server just object Goswin> storage. Files are one stripe each. All nodes are on a Goswin> Gigabit switch. I''ve run bonnie with 1-4 clinets in Goswin> parallel (see below. Goswin> Goswin> The write speed goes from 100 MiB/s to nearly 180 MiB/s Goswin> which is close enough to GBit speeds and I''m very pleased Goswin> with that. But read performance looks low (34-64 MiB/s) Goswin> and rewrite even lower (10-29 Mib/s) and fluctuates Goswin> widely. Andreas> If you have only a single OST with a single GigE adapter, There are two OSTs with one GigE adapter each, it''s just that the MDS is sitting together with one of the OSTs on the same machine. But that shouldn''t impact streaming I/O performance. Andreas> the peak network bandwidth is about 110MB/s. Getting Andreas> throughput of 180MB/s is nice, but would at least seem to Andreas> indicate there is a backlog of cached data on the client Andreas> (maximum would be 32MB/client/OSC by default). Even Andreas> then, this would only be a small fraction of the total Andreas> file size. How many OSTs and GigE adapters do you have? see above Andreas> I suspect that the read speed is impacted by flushing the Andreas> dirty write data from cache. Instead, the test should be Andreas> doing an fsync after the write phase, to reduce the write Andreas> rate to sane levels, and to fairly record the read rate. We''ll try iozone, to see whether this is the case. Roland >> Version 1.03 ------Sequential Output------ --Sequential Input- >> --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP >> K/sec %CP /sec %CP beo-41 8G 38182 28 6884 95 9248 99 394.0 4 >> beo-42 8G 51270 37 8283 94 21358 97 315.5 3 beo-43 8G 38431 27 >> 6139 96 7879 99 388.6 4 beo-44 8G 51016 38 7805 96 27675 97 >> 365.1 4 >> >> beo-41,8G,,,38182,28,6884,95,,,9248,99,394.0,4,,,,,,,,,,,,, >> beo-42,8G,,,51270,37,8283,94,,,21358,97,315.5,3,,,,,,,,,,,,, >> beo-43,8G,,,38431,27,6139,96,,,7879,99,388.6,4,,,,,,,,,,,,, >> beo-44,8G,,,51016,38,7805,96,,,27675,97,365.1,4,,,,,,,,,,,,,
"Iozone" <capps@iozone.org> writes:> ----- Original Message ----- > From: "Andreas Dilger" <adilger@clusterfs.com> > To: "Goswin von Brederlow" <brederlo@informatik.uni-tuebingen.de> > Cc: <lustre-discuss@clusterfs.com> > Sent: Tuesday, March 07, 2006 5:53 PM > Subject: Re: [Lustre-discuss] Is this normal?Hmm, I didn''t get that mail yet it seems.>>> >> I suspect that the read speed is impacted by flushing the dirty write >> data from cache. Instead, the test should be doing an fsync after the >> write phase, to reduce the write rate to sane levels, and to fairly >> record the read rate.Bonnie does a fsync() after every test or the results would be totaly scewed by the systems cache. Running bonnie with one 1G file for example still gives only ~100MiB/s write speed but a read speed of over 1GiB/s since the file is completly cached. By default bonnie also uses twice the amount of ram as size of the test so the read test can''t use cached data. And most certainly the 32MB max dirty buffers of lustre will have 0 effect compared to an 8GB read even if they weren''t fsync()ed.> If one would have used Iozone, instead of Bonnie, the fsync(fd) > would have been done in between the write, rewrite, read, and > read tests, so you would not see the reader being punished by > unflushed dirty write data. > > But then again, I do have a certain bias... :-)Added to the list. The more the merrier.> Enjoy, > Don Capps > >> Cheers, AndreasMfG Goswin
On Mar 08, 2006 00:02 +0100, Goswin von Brederlow wrote:> The setup is pretty simple. I have one server doing meta data and > object storage and a second server just object storage. Files are one > stripe each. All nodes are on a Gigabit switch. I''ve run bonnie with > 1-4 clinets in parallel (see below. > > The write speed goes from 100 MiB/s to nearly 180 MiB/s which is close > enough to GBit speeds and I''m very pleased with that. But read > performance looks low (34-64 MiB/s) and rewrite even lower (10-29 > Mib/s) and fluctuates widely.If you have only a single OST with a single GigE adapter, the peak network bandwidth is about 110MB/s. Getting throughput of 180MB/s is nice, but would at least seem to indicate there is a backlog of cached data on the client (maximum would be 32MB/client/OSC by default). Even then, this would only be a small fraction of the total file size. How many OSTs and GigE adapters do you have? I suspect that the read speed is impacted by flushing the dirty write data from cache. Instead, the test should be doing an fsync after the write phase, to reduce the write rate to sane levels, and to fairly record the read rate.> Version 1.03 ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > beo-41 8G 38182 28 6884 95 9248 99 394.0 4 > beo-42 8G 51270 37 8283 94 21358 97 315.5 3 > beo-43 8G 38431 27 6139 96 7879 99 388.6 4 > beo-44 8G 51016 38 7805 96 27675 97 365.1 4 > > beo-41,8G,,,38182,28,6884,95,,,9248,99,394.0,4,,,,,,,,,,,,, > beo-42,8G,,,51270,37,8283,94,,,21358,97,315.5,3,,,,,,,,,,,,, > beo-43,8G,,,38431,27,6139,96,,,7879,99,388.6,4,,,,,,,,,,,,, > beo-44,8G,,,51016,38,7805,96,,,27675,97,365.1,4,,,,,,,,,,,,,Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
----- Original Message ----- From: "Andreas Dilger" <adilger@clusterfs.com> To: "Goswin von Brederlow" <brederlo@informatik.uni-tuebingen.de> Cc: <lustre-discuss@clusterfs.com> Sent: Tuesday, March 07, 2006 5:53 PM Subject: Re: [Lustre-discuss] Is this normal?>> > I suspect that the read speed is impacted by flushing the dirty write > data from cache. Instead, the test should be doing an fsync after the > write phase, to reduce the write rate to sane levels, and to fairly > record the read rate.If one would have used Iozone, instead of Bonnie, the fsync(fd) would have been done in between the write, rewrite, read, and read tests, so you would not see the reader being punished by unflushed dirty write data. But then again, I do have a certain bias... :-) Enjoy, Don Capps> > Cheers, Andreas > -- > Andreas Dilger > Principal Software Engineer > Cluster File Systems, Inc. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
Hi, I don''t have a clue if this performance pattern is normal but I thought it looked odd. So I ask. :) The setup is pretty simple. I have one server doing meta data and object storage and a second server just object storage. Files are one stripe each. All nodes are on a Gigabit switch. I''ve run bonnie with 1-4 clinets in parallel (see below. The write speed goes from 100 MiB/s to nearly 180 MiB/s which is close enough to GBit speeds and I''m very pleased with that. But read performance looks low (34-64 MiB/s) and rewrite even lower (10-29 Mib/s) and fluctuates widely. Is that normal behaviour? MfG Goswin PS: Bonnie reads/writes linear and uses 8 1GB files per client. Rewrite changes one byte per 4K block I believe. ---------------------------------------------------------------------- Linux beo-31 2.6.15.6 #1 SMP Mon Mar 6 15:48:34 CET 2006 x86_64 GNU/Linux Lustre 1.4.6 (+ patch for 2.6.15) ---------------------------------------------------------------------- One client: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP beo-41 8G 100462 74 10321 99 34880 98 445.9 5 beo-41,8G,,,100462,74,10321,99,,,34880,98,445.9,5,,,,,,,,,,,,, ---------------------------------------------------------------------- Two clients: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP beo-41 8G 74827 56 10620 95 23832 93 413.7 4 beo-42 8G 82556 62 10882 97 23319 93 421.4 4 beo-41,8G,,,74827,56,10620,95,,,23832,93,413.7,4,,,,,,,,,,,,, beo-42,8G,,,82556,62,10882,97,,,23319,93,421.4,4,,,,,,,,,,,,, ---------------------------------------------------------------------- Three clients: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP beo-41 8G 44810 32 7948 95 29064 93 399.6 4 beo-42 8G 52912 39 7909 96 31005 92 404.8 4 beo-43 8G 59893 45 6923 97 12653 99 409.4 3 beo-41,8G,,,44810,32,7948,95,,,29064,93,399.6,4,,,,,,,,,,,,, beo-42,8G,,,52912,39,7909,96,,,31005,92,404.8,4,,,,,,,,,,,,, beo-43,8G,,,59893,45,6923,97,,,12653,99,409.4,3,,,,,,,,,,,,, ---------------------------------------------------------------------- Four clients: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP beo-41 8G 38182 28 6884 95 9248 99 394.0 4 beo-42 8G 51270 37 8283 94 21358 97 315.5 3 beo-43 8G 38431 27 6139 96 7879 99 388.6 4 beo-44 8G 51016 38 7805 96 27675 97 365.1 4 beo-41,8G,,,38182,28,6884,95,,,9248,99,394.0,4,,,,,,,,,,,,, beo-42,8G,,,51270,37,8283,94,,,21358,97,315.5,3,,,,,,,,,,,,, beo-43,8G,,,38431,27,6139,96,,,7879,99,388.6,4,,,,,,,,,,,,, beo-44,8G,,,51016,38,7805,96,,,27675,97,365.1,4,,,,,,,,,,,,,