Ext3 users - Aug 2006 - can ext3 directory entries be overwritten? -- Re: extremely slow "ls" on a cleared fatty ext3 directory on FC4/5

Hi, all,

 I'm a newbie to ext3 file system, but what a pity if
ext3 could not shrink after containing files and
subdirectories get deleted.

 If the ext3 directory could not shrink, then another
natural question is: can the deleted directory entries
be overwritten by new files/subdirs? The following is
an example to detail my question:

Suppose a directory named myDir hold 3 files a, b, c.
then the ext3 file system will create 5 entries under
myDir, ".", "..", 'a', 'b', 'c'.

lets say I remove file a and b, then the entries on
et3 will looks as:

 '.', '..', '#a', '#b', 'c'. with
'#?' means the file
is delteded.

at last I added three files d ,e,f, which entries
should the ext3 file system create for myDir? case A,
or case B? it looks like case A has better performance
and is not hard to implement. But I'm not sure how
ext3 picks.

case A, '#a', '#b' entries are reused, so only one
more entries are created.

'.', '..', 'd', 'e', 'c', 'f'.

case B, '#a', '#b' are not reused, instead, 3 more
entries are created. so total entries are:

'.', '..', '#a', '#b', 'c', 'd',
'e', 'f'.

Please help, I'll be more than appreciated if any one
can point out ext3 howtos and introduction links.
Thanks a lot.

--- John Wendel <jwendel10 at comcast.net> wrote:
> Robinson Tiemuqinke wrote:
> >  Hi,
> >
> >  A stupid flat directory /tmp holding 5 millon
> files, the directory
> >  locates on a ext3 file system with dir_index
> feature turned on. The
> >  running Linux are FC4 and FC5.
> >
> >  The files are just directly under /tmp, not in
> any subdirectories --
> >  they are results of mis-operations of users.
> >
> >  Then a 'ls' or 'find' command will take one hour
> to finish, a lot of
> >  other applications on the computer boxes are
> affected.
> >
> >  I managed to have deleted the files one by one
> with a 'find . |xargs
> >  rm -rf' similar command in about 10 hours. but
> after a file system
> >  sync, it still take me 20 minutes to list the
> cleaned /tmp directory
> >  again -- even now the directory holds only 8
> files total.
> >
> >  so I try to 'ls' the directory itself (not any
> files and
> >  subdirectories on it) and find that its size is
> stupidly large (it is
> >  131M even after deletion) compared with 4K for
> normal directories.
> >
> >  -bash-3.00# ls -alFdh /tmp* drwxrwxrwt  4 root 
> staff 4.0K Aug 12
> >  23:17 new_tmp/ drwxrwxrwt  4 root  staff 131M Aug
> 12 20:30 tmp/
> >
> >  Anyone know why the former fatty directory still
> looks unchanged and
> >  takes hours to traverse even after 99.999999%
> files got removed?
> >
> >  If there are any ways to fix this kind of problem
> without rebooting
> >  machine? I'm afraid of the commands "rsync -avHn
> /tmp/ /new_tmp/; rm
> >  -rf /tmp/ && mv /new_tmp/ /tmp" because other
> applications are
> >  accessing /tmp/ as well.
> >
> >  Please help. Thanks a lot.
> 
> EXT3 directories grow but they don't shrink.
> Rebooting won't fix this 
> problem.
> The only fix that I know is to delete the old
> directory and create a new 
> one.
> BTW, XFS automatically shrinks directories (but has
> its own set of 
> problems).
> 
> Regards,
> 
> John
> 
> -- 
> fedora-list mailing list
> fedora-list at redhat.com
> To unsubscribe:
> https://www.redhat.com/mailman/listinfo/fedora-list
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

On Sun, Aug 13, 2006 at 11:57:41AM -0700, Robinson Tiemuqinke
wrote:
> Hi, all,
> 
>  I'm a newbie to ext3 file system, but what a pity if
> ext3 could not shrink after containing files and
> subdirectories get deleted.
> 
>  If the ext3 directory could not shrink, then another
> natural question is: can the deleted directory entries
> be overwritten by new files/subdirs? The following is
> an example to detail my question:
Yes, of course.  If a directory contains deleted directory entries,
they do o fcourse get reused.  So in the case where the number of
files remains more or less at a steady state, ext3 will do fine.  What
ext3 does not currently handle is the case where someone creates 131
megabytes worth of directories (which is enough for 4-8 million
files!), and then deletes them all.  The directory will not shrink
back down to its original size.

Likewise, most modern filesystems implement on-line resizing, where
the total size of the filesystem can be extended if you are using an
LVM and/or RAID system, and can expand the underlying
volume/partition.  However, no filesystems that I know of implement an
on-link shrink operation.  Some have an off-line shrink operations;
some don't even have that.

						- Ted

13.08.06 ? 22:57 Robinson Tiemuqinke ? ?????? ?????(?):
> Hi, all,
>
>  I'm a newbie to ext3 file system, but what a pity if
> ext3 could not shrink after containing files and
> subdirectories get deleted.
>
>  If the ext3 directory could not shrink, then another
> natural question is: can the deleted directory entries
> be overwritten by new files/subdirs? The following is
> an example to detail my question:
>
> Suppose a directory named myDir hold 3 files a, b, c.
> then the ext3 file system will create 5 entries under
> myDir, ".", "..", 'a', 'b',
'c'.
>
> lets say I remove file a and b, then the entries on
> et3 will looks as:
>
>  '.', '..', '#a', '#b', 'c'. with
'#?' means the file
> is delteded.
>
> at last I added three files d ,e,f, which entries
> should the ext3 file system create for myDir? case A,
> or case B? it looks like case A has better performance
> and is not hard to implement. But I'm not sure how
> ext3 picks.
>
> case A, '#a', '#b' entries are reused, so only one
> more entries are created.
>
> '.', '..', 'd', 'e', 'c',
'f'.
>
> case B, '#a', '#b' are not reused, instead, 3 more
> entries are created. so total entries are:
>
> '.', '..', '#a', '#b', 'c',
'd', 'e', 'f'.
>
> Please help, I'll be more than appreciated if any one
> can point out ext3 howtos and introduction links.
> Thanks a lot.
>
> --- John Wendel <jwendel10 at comcast.net> wrote:
>
>> Robinson Tiemuqinke wrote:
>> >  Hi,
>> >
>> >  A stupid flat directory /tmp holding 5 millon
>> files, the directory
>> >  locates on a ext3 file system with dir_index
>> feature turned on. The
>> >  running Linux are FC4 and FC5.
>> >
>> >  The files are just directly under /tmp, not in
>> any subdirectories --
>> >  they are results of mis-operations of users.
>> >
>> >  Then a 'ls' or 'find' command will take one hour
>> to finish, a lot of
>> >  other applications on the computer boxes are
>> affected.
>> >
>> >  I managed to have deleted the files one by one
>> with a 'find . |xargs
>> >  rm -rf' similar command in about 10 hours. but
>> after a file system
>> >  sync, it still take me 20 minutes to list the
>> cleaned /tmp directory
>> >  again -- even now the directory holds only 8
>> files total.
>> >
>> >  so I try to 'ls' the directory itself (not any
>> files and
>> >  subdirectories on it) and find that its size is
>> stupidly large (it is
>> >  131M even after deletion) compared with 4K for
>> normal directories.
>> >
>> >  -bash-3.00# ls -alFdh /tmp* drwxrwxrwt  4 root
>> staff 4.0K Aug 12
>> >  23:17 new_tmp/ drwxrwxrwt  4 root  staff 131M Aug
>> 12 20:30 tmp/
>> >
>> >  Anyone know why the former fatty directory still
>> looks unchanged and
>> >  takes hours to traverse even after 99.999999%
>> files got removed?
>> >
>> >  If there are any ways to fix this kind of problem
>> without rebooting
>> >  machine? I'm afraid of the commands "rsync -avHn
>> /tmp/ /new_tmp/; rm
>> >  -rf /tmp/ && mv /new_tmp/ /tmp" because other
>> applications are
>> >  accessing /tmp/ as well.
>> >
>> >  Please help. Thanks a lot.
>>
>> EXT3 directories grow but they don't shrink.
>> Rebooting won't fix this
>> problem.
>> The only fix that I know is to delete the old
>> directory and create a new
>> one.
>> BTW, XFS automatically shrinks directories (but has
>> its own set of
>> problems).
Use option -D in e2fsck for optimize directories.

        -D     Optimize  directories  in filesystem.  This option causes  
e2fsck to try to optimize
               all directories, either by reindexing them if  the   
filesystem  supports  directory
               indexing,   or  by  sorting and compressing directories for  
smaller directories, or
               for filesystems using traditional linear directories.

-- 
Igor A. Valcov