thr3ads.net - Ext3 users - with dir_index ls is slower than without? [Mar 2008]

If this information is useful, please help other people find it:
Share via:

Sebastian Reitenbach

2008-Mar-31 06:36 UTC

with dir_index ls is slower than without?

Hi,

I try to tune a ext3 filesystem, as I've heard, that when enabling dir_index
option, then ls -l or find will be a lot faster than before. So I did.

I created 2 partition on the harddisc, each 20GB:

installhost2:~ # fdisk -l /dev/sda

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         192     1542208+  82  Linux swap / Solaris
/dev/sda2   *         193        2803    20972857+  83  Linux
/dev/sda3            2804        5236    19543072+  83  Linux
/dev/sda4            5237        7669    19543072+  83  Linux

/dev/sda3 was formatted with the dir_index option enabled, /dev/sda4 with 
dir_index disabled:

installhost2:/ # tune2fs -l /dev/sda3
tune2fs 1.38 (30-Jun-2005)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          d90ccbb9-f45a-4304-87d8-805fce775c23
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal dir_index filetype needs_recovery 
sparse_super
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              2443200
Block count:              4885768
Reserved block count:     244288
Free blocks:              4273422
Free inodes:              1943188
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16288
Inode blocks per group:   509
Filesystem created:       Thu Mar 27 17:14:40 2008
Last mount time:          Fri Mar 28 11:39:47 2008
Last write time:          Fri Mar 28 11:39:47 2008
Mount count:              7
Maximum mount count:      28
Last checked:             Thu Mar 27 17:14:40 2008
Check interval:           15552000 (6 months)
Next check after:         Tue Sep 23 18:14:40 2008
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      306a3c58-3cbb-4f4a-856a-e48ae3006a07
Journal backup:           inode blocks

installhost2:/ # tune2fs -l /dev/sda4
tune2fs 1.38 (30-Jun-2005)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          2bb124a4-f7c7-4cac-b0c1-16aa8afc67eb
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery sparse_super
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              2443200
Block count:              4885768
Reserved block count:     244288
Free blocks:              4274331
Free inodes:              1943188
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16288
Inode blocks per group:   509
Filesystem created:       Thu Mar 27 17:15:03 2008
Last mount time:          Fri Mar 28 11:39:47 2008
Last write time:          Fri Mar 28 11:39:47 2008
Mount count:              7
Maximum mount count:      23
Last checked:             Thu Mar 27 17:15:03 2008
Check interval:           15552000 (6 months)
Next check after:         Tue Sep 23 18:15:03 2008
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      1cfc2290-e289-4c49-a57f-2b2e3b9e91c4
Journal backup:           inode blocks

The partitions are mounted: 
/dev/sda3 on /mnt/index type ext3 (rw)
/dev/sda4 on /mnt/noindex type ext3 (rw)

If I create 500000 files, each 1kB from /dev/urandom, the ls -la command 
needs 
a lot of time on the partition with dir_index enabled (the wc -l is to 
eleminate the slow terminal :), the files were created on one partition and 
rsynced to the other: 

installhost2:~ # time ls -la /mnt/index/ | wc -l
 500005
 
 real 2m41.015s
 user 0m4.568s
 sys 0m6.520s
 
 
 installhost2:~ # time ls -la /mnt/noindex/ | wc -l
 500005
 
 real 0m10.792s
 user 0m3.172s
 sys 0m6.000s

I expected the dir_index should speedup this a little bit?
I assume I'm still missing sth?

I'm on SLES10sp1, kernel 2.6.16.46 x86_64.

kind regards
Sebastian

Nicolas KOWALSKI

2008-Mar-31 08:36 UTC

head link

with dir_index ls is slower than without?

"Sebastian Reitenbach" <sebastia at l00-bugdead-prods.de>
writes:
> installhost2:~ # time ls -la /mnt/index/ | wc -l
>  500005
>  
>  real 2m41.015s
>  user 0m4.568s
>  sys 0m6.520s
>  
>  
>  installhost2:~ # time ls -la /mnt/noindex/ | wc -l
>  500005
>  
>  real 0m10.792s
>  user 0m3.172s
>  sys 0m6.000s
>
> I expected the dir_index should speedup this a little bit?
> I assume I'm still missing sth?
I think the point of dir_index is "only" to quickly find in a large
directory a file when you _already_ have its name.

The performance of listing is not its purpose, and as you noted it,
even makes performance worse.

-- 
Nicolas

Sebastian Reitenbach

2008-Mar-31 11:18 UTC

head link

Re: with dir_index ls is slower than without?

Hi Nicolas,

Nicolas KOWALSKI <niko@petole.dyndns.org> wrote: > "Sebastian Reitenbach" <sebastia@l00-bugdead-prods.de>
writes:
> 
> > installhost2:~ # time ls -la /mnt/index/ | wc -l
> >  500005
> >  
> >  real 2m41.015s
> >  user 0m4.568s
> >  sys 0m6.520s
> >  
> >  
> >  installhost2:~ # time ls -la /mnt/noindex/ | wc -l
> >  500005
> >  
> >  real 0m10.792s
> >  user 0m3.172s
> >  sys 0m6.000s
> >
> > I expected the dir_index should speedup this a little bit?
> > I assume I'm still missing sth?
> 
> I think the point of dir_index is "only" to quickly find in a
large
> directory a file when you _already_ have its name.
> 
> The performance of listing is not its purpose, and as you noted it,
> even makes performance worse.
ah, that would explain what I've seen here. 

after reading your answer, I found this older mail in the archives:
http://osdir.com/ml/file-systems.ext3.user/2004-09/msg00029.html

So everything seems to depend on how the application is using the
filesystem. 
Picking a single given file might be faster than with a plain ext3, but 
scanning and opening all files in a directory might become slower. I wanted 
to use the dir_index for some partitions, like for cyrus imap server, and 
for some other applications. I think I have to benchmark the applications, 
to see whether they get a speed gain of the dir_index or not.

kind regards
Sebastian>

Ext3 users - Mar 2008 - with dir_index ls is slower than without?

with dir_index ls is slower than without?

with dir_index ls is slower than without?

Re: with dir_index ls is slower than without?