I am building a few clusters, one with 10 TB, and one with 100 TB of OST. I will eventually build a 500 TB cluster as well. Can anyone recommend how much storage I should have for my MDS? How many bytes are stored per file on the MDS? Roger Spellman Tera Scala, Inc. www.terascala.com _________________________________________________________________ Is your PC infected? Get a FREE online computer virus scan from McAfee® Security. http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963
The MDS storage depends on the number of files + stripe of each file. Lustre uses 512bytes (not striped) - 4KB (heavily striped) per file to store file metadata + extended file attributes. Cheers, -Kums>>> "RS RS" <rs2006ts@hotmail.com> 7/18/2006 9:28 AM >>>I am building a few clusters, one with 10 TB, and one with 100 TB of OST. I will eventually build a 500 TB cluster as well. Can anyone recommend how much storage I should have for my MDS? How many bytes are stored per file on the MDS? Roger Spellman Tera Scala, Inc. www.terascala.com _________________________________________________________________ Is your PC infected? Get a FREE online computer virus scan from McAfee? Security. http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963 _______________________________________________ Lustre-discuss mailing list Lustre-discuss@clusterfs.com https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
On Jul 18, 2006 11:28 -0400, RS RS wrote:> I am building a few clusters, one with 10 TB, and one with 100 TB of OST. > I will eventually build a 500 TB cluster as well. > > Can anyone recommend how much storage I should have for my MDS? > > How many bytes are stored per file on the MDS?A safe rule of thumb is 4kB per file + 400MB for journal, * 2. Actual size needed depends on average file size in filesystem, but in general it doesn''t need to be gigantic. A 2TB MDS could hold up to 500M files. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Andreas, I''m not sure what the *2 (times two) is doing in the formula. If it is 4kB per file, then indeed, a 2TB MDS can hold 500M file (ignoring the journal for the time being). But, if it is 8kB per file, then a 2TB MDS can hold "only" :-) 250M file. Can you clarify the *2? Also, is there a way to control the journal size? For example, would we get better performance having a larger journal? Thanks. Roger Spellman Tera Scala, Inc. www.terascala.com> >A safe rule of thumb is 4kB per file + 400MB for journal, * 2. Actual size >needed depends on average file size in filesystem, but in general it >doesn''t >need to be gigantic. A 2TB MDS could hold up to 500M files. > >Cheers, Andreas_________________________________________________________________ On the road to retirement? Check out MSN Life Events for advice on how to get there! http://lifeevents.msn.com/category.aspx?cid=Retirement
On Jul 19, 2006 10:26 -0400, RS RS wrote:> I''m not sure what the *2 (times two) is doing in the formula. If it is 4kB > per file, then indeed, a 2TB MDS can hold 500M file (ignoring the journal > for the time being). But, if it is 8kB per file, then a 2TB MDS can hold > "only" :-) 250M file. > > Can you clarify the *2?This is just margin for the number of files you anticipate creating, leeway in case the average file size is smaller than expecting, or if you add more OSTs and increase the size of the filesystem size.> Also, is there a way to control the journal size? For example, would we > get better performance having a larger journal?If you use an external journal then it can be larger, and if you have enough RAM in the MDS this will improve performance. Note that the journal can pin an amount of kernel memory equal to the journal size.> Roger Spellman > Tera Scala, Inc. > www.terascala.com > > > > >A safe rule of thumb is 4kB per file + 400MB for journal, * 2. Actual size > >needed depends on average file size in filesystem, but in general it > >doesn''t > >need to be gigantic. A 2TB MDS could hold up to 500M files. > > > >Cheers, Andreas > > _________________________________________________________________ > On the road to retirement? Check out MSN Life Events for advice on how to > get there! http://lifeevents.msn.com/category.aspx?cid=Retirement > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.