Hi, I have an application that deals with 100,000 to 1,000,000 image files. I initially structured it to use multiple directories, so that file 123456 would be stored in /12/34/123456. I'm now wondering if that's pointless, as it would simplify things to simply store the file in /123456. Can anyone indicate whether I'm gaining anything by using smaller directories in ext3/ext4? Thanks. Mitch -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/ext3-users/attachments/20100308/8b6625ca/attachment.htm>
On 03/08/2010 08:23 PM, Mitch Trachtenberg wrote:> Hi, > > I have an application that deals with 100,000 to 1,000,000 image files. > > I initially structured it to use multiple directories, so that file > 123456 would be stored in /12/34/123456. I'm now wondering if that's > pointless, as it would simplify things to simply store the file in /123456. > > Can anyone indicate whether I'm gaining anything by using smaller > directories in ext3/ext4? Thanks. > > Mitch >I think that breaking up your files into subdirectories makes it easier to navigate the tree and find files from a human point of view. Even better if the bytes reflect something like year/month/day/hour/min (assuming your pathname has a date based guid or similar encoding). You can have a million files in one large directory, but be careful to iterate and copy them in a sorted order (sorted by inode) to avoid nasty performance issues that are side effects of the way we hash file names in ext3/4. Good luck! Ric
Mitch Trachtenberg wrote:> Hi, > > I have an application that deals with 100,000 to 1,000,000 image files. > > I initially structured it to use multiple directories, so that file > 123456 would be stored in /12/34/123456. I'm now wondering if that's > pointless, as it would simplify things to simply store the file in /123456. > > Can anyone indicate whether I'm gaining anything by using smaller > directories in ext3/ext4? Thanks. > > Mitch >If you have one file per dir, that's a lot of dirs, and the time to search for new dir inode locations can get rather expensive as the fs fills, in my experience. You may also want to toy with setting the "topdir" flag on a dir; new directories -under- that topdir get spread around the block groups. New dirs under a non-topdir tend to stay closer to the parent. Finally, remember that ext2/3 has a limit of 32000 or so files per dir. ext4 lifts this restriction. -Eric