Hi,
Just thought I'd share some test results of mine in case anyone is
interested.
Basically the tests are siumulating what our product does with files -
although the tests do it a lot quicker (not as many files though).
The test is to create 1 million files (each containing the text of the
file number) spread over a number of directories. The files are then
removed in the same manner as they are created. I added tests later to
overwrite the existing files to see if there is any additional
performance hit.
All results in seconds, in kernel order 2.4.20 (pre11), 2.4.21, 2.4.21 +
dir_index
Files spread over 100 directories:
Create : 1764, N/A, 1078
Remove : 611, N/A, 1310
Ok so results are not that great - better to use more directories.
Files spread over 1000 directories:
Create: 409, 305, 284
Remove: 138, 96, 141
Create: 377, 312, 276
Overwrite: 317, 246, 268
Overwrite2: 321, 241, 265
I was interested in seeing if the htree patches would give me a
performance increase - although I guess the sequential nature of the
file creation/deletion probably will not benefit - although I was
suprised that adding dir_index hit the overwrite and removal speed -
probably based on this testing I won't use this feature (yet).
Cpu info:
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 3
model name : AMD Duron(tm) Processor
stepping : 1
cpu MHz : 1016.400
cache size : 64 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips : 2025.06
hdparm -i /dev/hda
/dev/hda:
Model=Maxtor 6Y080L0, FwRev=YAR41BW0, SerialNo=Y2M0FH0E
Config={ Fixed }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=57
BuffType=3(DualPortCache), BuffSize=2048kB, MaxMultSect=16, MultSect=16
DblWordIO=no, OldPIO=2, DMA=yes, OldDMA=0
CurCHS=4047/16/255, CurSects=-217054981, LBA=yes, LBAsects=160086528
tDMA={min:120,rec:120}, DMA modes: mword0 mword1 mword2
IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4
UDMA modes: mode0 mode1 mode2 mode3 mode4 *mode5 mode6
[ct_sys@installed TEST]# hdparm /dev/hda
/dev/hda:
multcount = 16 (on)
I/O support = 0 (default 16-bit)
unmaskirq = 0 (off)
using_dma = 1 (on)
keepsettings = 0 (off)
nowerr = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 9964/255/63, sectors = 160086528, start = 0
Kernel 2.4.20 & 21 both output same disk settings.
Glen
Hi, On Wed, 2003-06-18 at 12:02, glen wrote:> Files spread over 100 directories: > > Create : 1764, N/A, 1078 > Remove : 611, N/A, 1310 > > Ok so results are not that great - better to use more directories. > > Files spread over 1000 directories: > > Create: 409, 305, 284 > Remove: 138, 96, 141Why use more directories? That just means that you spend more time updating more metadata for more directories. The point of htree is to let the fs do this work itself, as efficiently as possible: if you try with _1_ directory, you should see the best benefit from htree. However, the performance still doesn't look that good, especially on the 100 dirs case (10,000 files per dir) --- are you sure you set the dir-index flag on the filesystem to enable htree operation? Cheers, Stephen
Where abouts would one get those patches? I am using 2.4.21 patched with extfs-update-2.4.21rc8. Steve Andreas Dilger wrote:>On Jun 18, 2003 09:55 -0600, kwijibo@zianet.com wrote: > > >>I am seeing about a 10 percent improvement with htree and >>these are without any further tweaking which I hope to improve >>a bit more. I am real curious to see how it performs over NFS >>which is in my queue of tests I want to run. >> >> > >Note that for safe NFS use of htree you should have the htree NFS >patches from tytso applied. This avoids bad behaviour (e.g. duplicate >directory entries) if one process is reading from the directory at >exactly the same time another one is updating it. > >Cheers, Andreas >-- >Andreas Dilger >http://sourceforge.net/projects/ext2resize/ >http://www-mddsp.enel.ucalgary.ca/People/adilger/ > > > > >
Andreas Dilger wrote:>On Jun 18, 2003 09:55 -0600, kwijibo@zianet.com wrote: > > >>I am seeing about a 10 percent improvement with htree and >>these are without any further tweaking which I hope to improve >>a bit more. I am real curious to see how it performs over NFS >>which is in my queue of tests I want to run. >> >> > >Note that for safe NFS use of htree you should have the htree NFS >patches from tytso applied. This avoids bad behaviour (e.g. duplicate >directory entries) if one process is reading from the directory at >exactly the same time another one is updating it. > >Cheers, Andreas >-- >Andreas Dilger >http://sourceforge.net/projects/ext2resize/ >http://www-mddsp.enel.ucalgary.ca/People/adilger/ > >Now that I think about it I thought these fixes were in the latest patch. Am I wrong in this assumption? Steve