Ext3 users - May 2008 - ext3_dx_add_entry: Directory index full!

On Sun, May 18, 2008 at 05:38:58PM +0200, Bernie Innocenti
wrote:
> Bernie Innocenti wrote:
>> On 2.6.24.4-64.fc8, I createed and mounted a filesystem like this:
>>   mke2fs -m0 -b 1024 -R stride=64 -I 128 -i 2048 -j  -L mail -O 
>> dir_index,sparse_super -v /dev/sdc1
>
> I cannot reproduce it any more if I reformat omitting "-b 1024".
> Maybe it would reappear with 200K * 4 = 800K files?
Using a filesystem with 4k blocks, and assuming the filenames are of
the same average length, you should be able to get approximately 200k
* (4**3) = 12.8 million files in a single directory.  If you use a 2k
block filesystem, the limit will be approximately 200k * (2**3) = 1.6
million files in a single directory.

Regards,

						- Ted

P.S.  Past a certain point, you really don't want to have that many
files in a Maildir directory; if the user is never going to be
deleting his SPAM, then you should seriously think about using a Unix
mbox style storage scheme.  Even with a 1k block filesystem, at 12
million files you'll be wasting 6 gigabytes of disk space of slack
space that is totally being wasted since the whole point of using
Maildir is to make it easy to delete or replace individual mail
messages.  If you want to archive all of your SPAM, why use a Maildir
format mbox at all?

On Sun, 2008-05-18 at 17:04 -0400, Theodore Tso wrote:
> P.S.  Past a certain point, you really don't want to have that many
> files in a Maildir directory; if the user is never going to be
> deleting his SPAM, then you should seriously think about using a Unix
> mbox style storage scheme.  Even with a 1k block filesystem, at 12
> million files you'll be wasting 6 gigabytes of disk space of slack
> space that is totally being wasted since the whole point of using
> Maildir is to make it easy to delete or replace individual mail
> messages.  If you want to archive all of your SPAM, why use a Maildir
> format mbox at all?
> 
Cyrus, which I am using on ext3, has a maildir like format in which each
message is a separate file.  (It might even be maildir, but I think
not).  This is what has led to my very slow directory traversal times on
backup.

Cyrus does not offer a choice of formats in the sense of switching to
something like mbox, and it is intended to be for large scale use.
Functionally, it is a mail database.

I suspect many other systems, even those offering a choice of format,
won't let you mix different formats.  So if you like maildir for some
stuff, you may need to use it for all.

So it seems to me it would be useful if the filesystem supported such
useage patterns well.

Ross Boylan

[I cut out most of the distribution list.]