Ok, I'm running a vBulletin forum (3.8.4) and found that all user attachments go into 1 single directory for each user. For each attached file in the forum, there's 2 files on disk (*.attach and *.thumb), for pictures that is. One user already has over 100,000 attachements, thus, over 200,000 files in his attach directory. Someone recently told me to 'keep an eye on it' because certain setups can't hold more than X number of files in a single directory. Yet someone else said I could have over 1 trillion files in a single directory if the HDD was large enough... Here's my setup: linux version: 2.6.18-92.1.10.el5 php: 5.1.6 mySQL: 5.0.45 File system: ext3 vB support has told me any limitation there might be, will not be the result of vB, so now I'm looking at either Linux and the way it handles files, or the ext3 file system. Does anyone know if I can just keep going with putting files into one directory? (there will be over 1million probably by year's end. Hopefully not more than 5 million ever). And, will having so many files in a single directory cause any performance problems? (ie: slowdowns) My only option is to hire a coder to somehow have it split the 1M+ files into several subdirs, say 50,000 per subdir. But even though it's messy, if it really doesn't make a difference in the end whether they're in 50 subdirs, or just 1 dir, then I won't bother (and can sigh a breath of relief) Thanks in advance!! z0diac is offline Looking for Linux Hosting? Click Here. -- View this message in context: http://www.nabble.com/How-many-files-can-I-have-safely-in-a-subdirectory--tp25212801p25212801.html Sent from the Ext3 - User mailing list archive at Nabble.com.
Greetings, I'm not sure if it's still the case, but there used to be a limit to how many subdirectories a directory can have. 32k, to be exact. We ended up creating our own (application level) directory hashing algorithm to work around it several years ago. This might only be a kernel 2.4 thing though. I'm unaware of any limit to number of files. However once the number of files in a directory gets above about 64k, filesystem performance will significantly decrease unless the filesystem has the dir_index option. dir_index can be specified at filesystem creation or added later using tune2fs (an fsck is required). Charles Charles Riley eRAD, Inc. ----- Original Message ----- From: "z0diac" <web2009 at zeroreality.com> To: ext3-users at redhat.com Sent: Sunday, August 30, 2009 12:00:12 PM GMT -05:00 US/Canada Eastern Subject: How many files can I have safely in a subdirectory? Ok, I'm running a vBulletin forum (3.8.4) and found that all user attachments go into 1 single directory for each user. For each attached file in the forum, there's 2 files on disk (*.attach and *.thumb), for pictures that is. One user already has over 100,000 attachements, thus, over 200,000 files in his attach directory. Someone recently told me to 'keep an eye on it' because certain setups can't hold more than X number of files in a single directory. Yet someone else said I could have over 1 trillion files in a single directory if the HDD was large enough... Here's my setup: linux version: 2.6.18-92.1.10.el5 php: 5.1.6 mySQL: 5.0.45 File system: ext3 vB support has told me any limitation there might be, will not be the result of vB, so now I'm looking at either Linux and the way it handles files, or the ext3 file system. Does anyone know if I can just keep going with putting files into one directory? (there will be over 1million probably by year's end. Hopefully not more than 5 million ever). And, will having so many files in a single directory cause any performance problems? (ie: slowdowns) My only option is to hire a coder to somehow have it split the 1M+ files into several subdirs, say 50,000 per subdir. But even though it's messy, if it really doesn't make a difference in the end whether they're in 50 subdirs, or just 1 dir, then I won't bother (and can sigh a breath of relief) Thanks in advance!! z0diac is offline Looking for Linux Hosting? Click Here. -- View this message in context: http://www.nabble.com/How-many-files-can-I-have-safely-in-a-subdirectory--tp25212801p25212801.html Sent from the Ext3 - User mailing list archive at Nabble.com. _______________________________________________ Ext3-users mailing list Ext3-users at redhat.com https://www.redhat.com/mailman/listinfo/ext3-users
Stephen Samuel (gmail)
2009-Sep-01 17:50 UTC
How many files can I have safely in a subdirectory?
Well, if you presume the possibility of running into bugs when the directory gets over 2GB, and directory entries averaging under 20 bytes, then you might see a problem at around 100million entries. You can probably expect performance issues before that point. If you expect these directories to keep growing year after year, you might want to consider doing directory hashing... If nothing else, it could get ugly if someone decides to do an 'ls' on a directory with 10million entries. On Sun, Aug 30, 2009 at 9:00 AM, z0diac<web2009 at zeroreality.com> wrote:> > Ok, I'm running a vBulletin forum (3.8.4) and found that all user attachments > go into 1 single directory for each user. For each attached file in the > forum, there's 2 files on disk (*.attach and *.thumb), for pictures that is. > > One user already has over 100,000 attachements, thus, over 200,000 files in > his attach directory. > > Someone recently told me to 'keep an eye on it' because certain setups can't > hold more than X number of files in a single directory. Yet someone else > said I could have over 1 trillion files in a single directory if the HDD was > large enough... > > Here's my setup: > > linux version: 2.6.18-92.1.10.el5 > php: 5.1.6 > mySQL: 5.0.45 > File system: ext3 > > vB support has told me any limitation there might be, will not be the result > of vB, so now I'm looking at either Linux and the way it handles files, or > the ext3 file system. > > Does anyone know if I can just keep going with putting files into one > directory? (there will be over 1million probably by year's end. Hopefully > not more than 5 million ever). > > And, will having so many files in a single directory cause any performance > problems? (ie: slowdowns) > > My only option is to hire a coder to somehow have it split the 1M+ files > into several subdirs, say 50,000 per subdir. But even though it's messy, if > it really doesn't make a difference in the end whether they're in 50 > subdirs, or just 1 dir, then I won't bother (and can sigh a breath of > relief) > > > Thanks in advance!! > z0diac is offline > Looking for Linux Hosting? Click Here. > > -- > View this message in context: http://www.nabble.com/How-many-files-can-I-have-safely-in-a-subdirectory--tp25212801p25212801.html > Sent from the Ext3 - User mailing list archive at Nabble.com. > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users >-- Stephen Samuel http://www.bcgreen.com Software, like love, 778-861-7641 grows when you give it away
On Sun, Aug 30, 2009 at 09:00:12 -0700, z0diac <web2009 at zeroreality.com> wrote:> > Someone recently told me to 'keep an eye on it' because certain setups can't > hold more than X number of files in a single directory. Yet someone else > said I could have over 1 trillion files in a single directory if the HDD was > large enough...When I have directories in the few million range doing mass changes gets extremely slow. Besides some the other other things mentioned, you need to worry about the inode limit on the file system. The default now, is lower than it used to be. This bit me once when I was moving a directory with lots of files to another system with a similar size partition when i was expecting a similar inode limit.
Stephen Samuel (gmail)
2009-Sep-02 00:42 UTC
How many files can I have safely in a subdirectory?
The 2GB worry isn't about the contents of files in the directory, but rather what happens when the directory 'file' itself gets to be over 2GB in size. If something's likely to break, then it's either there or at 4GB (( i.e. if somebody made the mistake of using a 32bit pointer in the wrong place )). There are probably not a whole lot of examples of directories (as opposed to data files) getting larger than 2GB, so I'd consider it relatively uncharted territory. On Tue, Sep 1, 2009 at 12:00 PM, Marc<marc_n at zeroreality.com> wrote:> Thank you for the reply. ?(it seemed to come only via email as your reply > and one other to my post, aren't showing up in the thread). > > Yes there is definitely over 2gb of files in that directory. ?My' inodes' > are only at 3% of the 590M or whatever it was, that were created at the time > the disc structure was created. ?I just wasn't sure if there was a limit > with ext3 to the # of files that could reside in a single directory. ?I > guess I will have to start adding new files under a new user account (which > will put them in that user's attachment subdir). > > Thanks.! > > At 01:50 PM 9/1/2009, you wrote: >> >> Well, if you presume the possibility of running into bugs when the >> directory gets over 2GB, >> and directory entries averaging ?under 20 bytes, then you might see a >> problem at around >> 100million entries. >> You can probably expect performance issues before that point. >> If you expect these directories to keep growing year after year, >> you might want to consider doing directory hashing... >> If nothing else, it could get ugly if someone decides to do an 'ls' >> on a directory with 10million entries. >> >> >> On Sun, Aug 30, 2009 at 9:00 AM, z0diac<web2009 at zeroreality.com> wrote: >> > >> > Ok, I'm running a vBulletin forum (3.8.4) and found that all user >> > attachments >> > go into 1 single directory for each user. For each attached file in the >> > forum, there's 2 files on disk (*.attach and *.thumb), for pictures that >> > is. >> > >> > One user already has over 100,000 attachements, thus, over 200,000 files >> > in >> > his attach directory. >> > >> > Someone recently told me to 'keep an eye on it' because certain setups >> > can't >> > hold more than X number of files in a single directory. Yet someone else >> > said I could have over 1 trillion files in a single directory if the HDD >> > was >> > large enough... >> > >> > Here's my setup: >> > >> > linux version: 2.6.18-92.1.10.el5 >> > php: 5.1.6 >> > mySQL: 5.0.45 >> > File system: ext3 >> > >> > vB support has told me any limitation there might be, will not be the >> > result >> > of vB, so now I'm looking at either Linux and the way it handles files, >> > or >> > the ext3 file system. >> > >> > Does anyone know if I can just keep going with putting files into one >> > directory? (there will be over 1million probably by year's end. >> > Hopefully >> > not more than 5 million ever). >> > >> > And, will having so many files in a single directory cause any >> > performance >> > problems? (ie: slowdowns) >> > >> > My only option is to hire a coder to somehow have it split the 1M+ files >> > into several subdirs, say 50,000 per subdir. But even though it's messy, >> > if >> > it really doesn't make a difference in the end whether they're in 50 >> > subdirs, or just 1 dir, then I won't bother (and can sigh a breath of >> > relief) >> > >> > >> > Thanks in advance!! >> > z0diac is offline >> > Looking for Linux Hosting? Click Here. >> > >> > -- >> > View this message in context: >> > http://www.nabble.com/How-many-files-can-I-have-safely-in-a-subdirectory--tp25212801p25212801.html >> > Sent from the Ext3 - User mailing list archive at Nabble.com. >> > >> > _______________________________________________ >> > Ext3-users mailing list >> > Ext3-users at redhat.com >> > https://www.redhat.com/mailman/listinfo/ext3-users >> > >> >> >> >> -- >> Stephen Samuel http://www.bcgreen.com ?Software, like love, >> 778-861-7641 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?grows when you give it away > >-- Stephen Samuel http://www.bcgreen.com Software, like love, 778-861-7641 grows when you give it away