Ext3 users - Dec 2007 - Ancient Very slow directory traversal thread

On Mon, 2007-12-03 at 14:37 -0500, Rashkae wrote:
> I just came across your message to a mailing list here:
[message concerned it taking hours to go through directories in a mail
spool on ext3]
> 
> https://www.redhat.com/archives/ext3-users/2007-October/msg00019.html
> 
> This might be a problem you resolved for yourself a long time ago, but I 
> thought you might be interested to know that Theodore's spd_readdir 
> library works great with star (even though I also cannot get it to work 
> with tar or even du).
That's interesting.  Since I couldn't get it to work with tar, got no
response, and wasn't sure how or if to get it to work with the daemon to
that really needed it, I haven't made and progress.

I wonder what determines whether the library helps or
hurts.
> 
> star with spd_readdir is what I use to back up maildir spools, and 
> something I consider absolutely necessary for any such storage on ext3 
> or reiserfs filesystems.
I hadn't noticed this was an issue with reiser.

I also just noticed that e2fsck has an option, -D, to optimize
directories.  The man page says this will reindex directories.  Does
anyone know if that could help?

Ross

On Dec 04, 2007  10:52 -0800, Ross Boylan wrote:
> On Mon, 2007-12-03 at 14:37 -0500, Rashkae wrote:
> > I just came across your message to a mailing list here:
> [message concerned it taking hours to go through directories in a mail
> spool on ext3]
> > 
> > https://www.redhat.com/archives/ext3-users/2007-October/msg00019.html
> > 
> > This might be a problem you resolved for yourself a long time ago, but
I
> > thought you might be interested to know that Theodore's
spd_readdir
> > library works great with star (even though I also cannot get it to
work
> > with tar or even du).
>
> That's interesting.  Since I couldn't get it to work with tar, got
no
> response, and wasn't sure how or if to get it to work with the daemon
to
> that really needed it, I haven't made and progress.
>
> I wonder what determines whether the library helps or hurts.
Maybe it depends if the app is using normal readdir() calls, or is
maybe implementing the directory traversal itself?
> I also just noticed that e2fsck has an option, -D, to optimize
> directories.  The man page says this will reindex directories.  Does
> anyone know if that could help?
That will compress empty space from directories and rebuild the hash
table for directories that are not indexed (e.g. older dirs created
before DIR_INDEX feature was enabled in fs).

It will keep them in hash order so it won't help this issue.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.