In the process of investigating performance in another area I happened to be measuring sequential cached reads (in a fairly basic manner): $ dd if=/dev/zero of=/tmp/file bs=8k count=100000 # create file 819200000 bytes transferred in 4.849394 secs (168928321 bytes/sec) $ dd of=/dev/null if=/tmp/file bs=8k # read it 819200000 bytes transferred in 2.177922 secs (376138354 bytes/sec) $ dd of=/dev/null if=/tmp/file bs=8k # read again 819200000 bytes transferred in 2.178407 secs (376054620 bytes/sec) $ dd of=/dev/null if=/tmp/file bs=32k # read it 819200000 bytes transferred in 1.801944 secs (454620117 bytes/sec) I ran vmstat to check there really was no read access to the filesystem. Now I had no idea whether this was the sort of performance to be expected or not, so checked on an *identical* cpu, memory, mobo machine running Gentoo - this gets 620MB/s for 8k blocks and 700MB/s for 32k ones. I'm gonna quickly add - it's not my intention to start a Linux vs BSD war here, the Gentoo data is mentioned to suggest that "Well - looks like I should see if I can get the BSD box to do more than 433MB/s!". The system is 2x1.26Ghz PIII, Supermicro P3TDER Serverworks HE-SL (dual channel), 2x1G PC133 ECC DIMMS FreeBSD setup is: FreeBSD 6.2-PRERELEASE #7: Mon Nov 27 19:32:33 NZDT 2006, with kernel based on GENERIC + SMP. I have /etc/malloc.conf -> >aj Any suggestions/ideas about how to improve this? Cheers Mark P.s: Someone's gonna say this "why does this unrealistic test matter?". Well, I work with databases a lot (usually postgres) and a great deal of effort goes into caching table/relation data as much as possible - so reading (and writing - but I'm not testing that here!) it as fast as possible (sequentially or random) is clearly important!
On 20/12/2006 12:05 PM, Mark Kirkwood wrote:> In the process of investigating performance in another area I happened > to be measuring sequential cached reads (in a fairly basic manner): > > $ dd if=/dev/zero of=/tmp/file bs=8k count=100000 # create file > 819200000 bytes transferred in 4.849394 secs (168928321 bytes/sec) > > $ dd of=/dev/null if=/tmp/file bs=8k # read it > 819200000 bytes transferred in 2.177922 secs (376138354 bytes/sec) > $ dd of=/dev/null if=/tmp/file bs=8k # read again > 819200000 bytes transferred in 2.178407 secs (376054620 bytes/sec) > $ dd of=/dev/null if=/tmp/file bs=32k # read it > 819200000 bytes transferred in 1.801944 secs (454620117 bytes/sec) > > I ran vmstat to check there really was no read access to the filesystem. > > Now I had no idea whether this was the sort of performance to be > expected or not, so checked on an *identical* cpu, memory, mobo machine > running Gentoo - this gets 620MB/s for 8k blocks and 700MB/s for 32k ones. > > ... > > The system is 2x1.26Ghz PIII, Supermicro P3TDER Serverworks HE-SL (dual > channel), 2x1G PC133 ECC DIMMSWhat does the memory-related stats from "top" show you? Did you have any other memory intensive applications running at the time? A random example from one of my systems (1GB RAM): Mem: 478M Active, 317M Inact, 150M Wired, 36M Cache, 111M Buf, 16M Free Glancing at the 'top' man page, "150M Wired" seems to be the data file cache, while "111M Buf" is block-level caching... and "36M Cache" is related to the VM. See the end of this email for the same figures after some testing - the "Wired" figure goes up while "Cache" disappears. That should give you an idea as to how much RAM is being used for the buffer/block IO cache ("111M Buf" in the above example, as I understand it), and the VM disk cache ("36M Cache" in the above example). You might also want to look at: sysctl vfs. and see whether or not there is anything there that may affect it. For instance, whether there is a maximum size in terms of files that will be cached...? Someone with more VFS/etc knowledge than I may be able to better advise you there... It might be worthwhile trying with a series of different file size to determine if there is a point where the caching performance drops... I just did a few quick tests on a relatively old machine (2x P3-933Mhz, 1GB RAM)... in this case, /tmp is on a 3ware SATA RAID controller (8xxx?) running RAID1 on two 160gb SATA disks)... First, with an 8MB file: $ dd if=/dev/zero of=/tmp/zero bs=8k count=1000 1000+0 records in 1000+0 records out 8192000 bytes transferred in 0.238275 secs (34380470 bytes/sec) $ dd if=/tmp/zero of=/dev/null bs=8k 1000+0 records in 1000+0 records out 8192000 bytes transferred in 0.022824 secs (358919664 bytes/sec) $ dd if=/tmp/zero of=/dev/null bs=8k 1000+0 records in 1000+0 records out 8192000 bytes transferred in 0.022845 secs (358590033 bytes/sec) Next, with an 80MB file: $ dd if=/dev/zero of=/tmp/zero bs=8k count=10000 10000+0 records in 10000+0 records out 81920000 bytes transferred in 2.549876 secs (32127050 bytes/sec) $ dd if=/tmp/zero of=/dev/null bs=8k 10000+0 records in 10000+0 records out 81920000 bytes transferred in 0.226559 secs (361583258 bytes/sec) $ dd if=/tmp/zero of=/dev/null bs=8k 10000+0 records in 10000+0 records out 81920000 bytes transferred in 0.232528 secs (352301702 bytes/sec) Then with an 800MB file, which based on the results (~360mb/sec down to ~42mb/sec) presumably blows the cache: $ dd if=/dev/zero of=/tmp/zero bs=8k count=100000 100000+0 records in 100000+0 records out 819200000 bytes transferred in 26.029121 secs (31472442 bytes/sec) $ dd if=/tmp/zero of=/dev/null bs=8k 100000+0 records in 100000+0 records out 819200000 bytes transferred in 19.463309 secs (42089451 bytes/sec) $ dd if=/tmp/zero of=/dev/null bs=8k 100000+0 records in 100000+0 records out 819200000 bytes transferred in 19.224657 secs (42611944 bytes/sec) Trying with something in between, I tried a 200mb file: $ dd if=/dev/zero of=/tmp/zero bs=8k count=25000 25000+0 records in 25000+0 records out 204800000 bytes transferred in 6.517742 secs (31421925 bytes/sec) $ dd if=/tmp/zero of=/dev/null bs=8k 25000+0 records in 25000+0 records out 204800000 bytes transferred in 0.866951 secs (236230194 bytes/sec) $ dd if=/tmp/zero of=/dev/null bs=8k 25000+0 records in 25000+0 records out 204800000 bytes transferred in 0.849929 secs (240961277 bytes/sec) So here we are somewhere in between -- around 240mb/sec... Looking at "top" now, I am seeing: Mem: 479M Active, 282M Inact, 199M Wired, 111M Buf, 36M Free compared with the earlier figures: Mem: 478M Active, 317M Inact, 150M Wired, 36M Cache, 111M Buf, 16M Free Hopefully all this means something and points you in the right direction...!!! --Antony
On Tuesday 19 December 2006 22:05, Mark Kirkwood wrote:> In the process of investigating performance in another area I happened > to be measuring sequential cached reads (in a fairly basic manner): > > $ dd if=/dev/zero of=/tmp/file bs=8k count=100000 # create file > 819200000 bytes transferred in 4.849394 secs (168928321 bytes/sec) > > $ dd of=/dev/null if=/tmp/file bs=8k # read it > 819200000 bytes transferred in 2.177922 secs (376138354 bytes/sec) > $ dd of=/dev/null if=/tmp/file bs=8k # read again > 819200000 bytes transferred in 2.178407 secs (376054620 bytes/sec) > $ dd of=/dev/null if=/tmp/file bs=32k # read it > 819200000 bytes transferred in 1.801944 secs (454620117 bytes/sec) >hum, look my releng_6: # dd of=/dev/null if=/c/c1/file bs=8k 100000+0 records in 100000+0 records out 819200000 bytes transferred in 1.017492 secs (805116851 bytes/sec) # dd of=/dev/null if=/c/c1/file bs=8k 100000+0 records in 100000+0 records out 819200000 bytes transferred in 1.015790 secs (806465915 bytes/sec) # dd of=/dev/null if=/c/c1/file bs=32k 25000+0 records in 25000+0 records out 819200000 bytes transferred in 0.896507 secs (913768635 bytes/sec) -- Jo?o A mensagem foi scaneada pelo sistema de e-mail e pode ser considerada segura. Service fornecido pelo Datacenter Matik https://datacenter.matik.com.br
JoaoBR wrote:> hum, look my releng_6: > > # dd of=/dev/null if=/c/c1/file bs=8k > 100000+0 records in > 100000+0 records out > 819200000 bytes transferred in 1.017492 secs (805116851 bytes/sec) > # dd of=/dev/null if=/c/c1/file bs=8k > 100000+0 records in > 100000+0 records out > 819200000 bytes transferred in 1.015790 secs (806465915 bytes/sec) > # dd of=/dev/null if=/c/c1/file bs=32k > 25000+0 records in > 25000+0 records out > 819200000 bytes transferred in 0.896507 secs (913768635 bytes/sec)Similar here (RELENG_6, a fairly slow P4/2Ghz):> dd if=file of=/dev/null bs=8k7690+1 records in 7690+1 records out 63003776 bytes transferred in 0.086171 secs (731146977 bytes/sec)> dd if=file of=/dev/null bs=32k1922+1 records in 1922+1 records out 63003776 bytes transferred in 0.070524 secs (893366745 bytes/sec)
JoaoBR wrote:> On Tuesday 19 December 2006 22:05, Mark Kirkwood wrote: >> In the process of investigating performance in another area I happened >> to be measuring sequential cached reads (in a fairly basic manner): >> >> $ dd if=/dev/zero of=/tmp/file bs=8k count=100000 # create file >> 819200000 bytes transferred in 4.849394 secs (168928321 bytes/sec)hartmann: dd if=/dev/zero of=/tmp/file bs=8k count=100000 100000+0 records in 100000+0 records out 819200000 bytes transferred in 18.540244 secs (44184964 bytes/sec) hartmann: dd if=/dev/zero of=/tmp/file bs=8k count=100000 100000+0 records in 100000+0 records out 819200000 bytes transferred in 18.047627 secs (45391009 bytes/sec)>> >> $ dd of=/dev/null if=/tmp/file bs=8k # read it >> 819200000 bytes transferred in 2.177922 secs (376138354 bytes/sec)hartmann: dd of=/dev/null if=/tmp/file bs=8k 100000+0 records in 100000+0 records out 819200000 bytes transferred in 1.898039 secs (431603332 bytes/sec)>> $ dd of=/dev/null if=/tmp/file bs=8k # read again>> 819200000 bytes transferred in 2.178407 secs (376054620 bytes/sec) hartmann: dd of=/dev/null if=/tmp/file bs=8k 100000+0 records in 100000+0 records out 819200000 bytes transferred in 1.328635 secs (616572658 bytes/sec)>> 819200000 bytes transferred in 2.178407 secs (376054620 bytes/sec) >> $ dd of=/dev/null if=/tmp/file bs=32k # read it >> 819200000 bytes transferred in 1.801944 secs (454620117 bytes/sec)hartmann: dd of=/dev/null if=/tmp/file bs=32k 25000+0 records in 25000+0 records out 819200000 bytes transferred in 1.037108 secs (789888848 bytes/sec) These are my numbers, but the box is not configured to be benchmarked and is a P4/3.0 GHz, HTT enabled, SMP kernel (most recent RELENG_6) with SCHED_ULE enabled. HD interface is ATA100, slightly aged, I guess ... Regards, Oliver
Hi mark, i just happened to lurk around and read the thread. Did you try to run iozone on both systems? Benchmarks like this are designed to test performance of filesystems on a rather wide domain of all related values (block size, file size, etc...) It also produces graphs so that someone can easily see some potential problems -- Achilleas Mantzios