Hello I am looking for advice or suggestions on specking the MDS and OSS for a new file system I am building. The IO patterns as have been explained to me are jobs that either create a big mpeg file or thousands of about 100kb frames for video. This could grow to hudreds of thousands of 100k files. I have specked 2 MDS for a cluster 8x300GB SAS 15k drives for the MDT in a raid 10(I used raid 10 to reduce the chance of the MDT becomming a bottleneck with many small files) 2x300GB SAS 15k drives for the MGS in a raid 1 1xHostpare 32gb 800mhz ram 3.0 or 2.5 dual quad core procs 4 OSS servers 16GB 667mhz ram 2.5 ghz dual quad core procs 6x1TB SATA 3 7.2k disks 5TB OST in a raid 5 (possibly running software raid) expandable to 2 5TB OSTs Per OSS 1xHotspare Since this is the first Lustre file system I am building. I noticed the sizing page: http://manual.lustre.org/manual/LustreManual16_HTML/SystemLimits.html#50548887_46859 The document mentions only dual core procs in regards to the MDS. The document also mentions a much lower amount of system RAM required for the OST. Am I wasting resources here by over-specking these boxes or is it a good idea to over spec a bit? Am I on the right track based on the IO pattern as I understand it? Sebastian
Sebastion, you are correct on the OST Memory, you probably wont need the 16GB, and if you can save money on using only 4GB, and buy bigger/more drives, you will probably be better off. Evan On 3/5/09 6:29 PM, "Sebastian Gutierrez" <sebastian.gutierrez at cs.stanford.edu> wrote:> Hello > > I am looking for advice or suggestions on specking the MDS and OSS for a > new file system I am building. > > The IO patterns as have been explained to me are jobs that either create a > big mpeg file or thousands of about 100kb frames for video. > This could grow to hudreds of thousands of 100k files. > > I have specked > 2 MDS for a cluster > 8x300GB SAS 15k drives for the MDT in a raid 10(I used raid 10 to reduce > the chance of the MDT becomming a bottleneck with many small files) > 2x300GB SAS 15k drives for the MGS in a raid 1 > 1xHostpare > 32gb 800mhz ram > 3.0 or 2.5 dual quad core procs > > 4 OSS servers > 16GB 667mhz ram > 2.5 ghz dual quad core procs > 6x1TB SATA 3 7.2k disks 5TB OST in a raid 5 (possibly running software raid) > expandable to 2 5TB OSTs Per OSS > 1xHotspare > > Since this is the first Lustre file system I am building. I noticed the > sizing page: > http://manual.lustre.org/manual/LustreManual16_HTML/SystemLimits.html#50548887 > _46859 > > The document mentions only dual core procs in regards to the MDS. > > The document also mentions a much lower amount of system RAM required for > the OST. > > Am I wasting resources here by over-specking these boxes or is it a good > idea to over spec a bit? > > Am I on the right track based on the IO pattern as I understand it? > > Sebastian > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
On Mar 06, 2009 09:37 -0800, Evan Felix wrote:> Sebastion, you are correct on the OST Memory, you probably wont need the > 16GB, and if you can save money on using only 4GB, and buy bigger/more > drives, you will probably be better off.In the 1.8 release the OSS nodes will cache reads with little/no performance impact even if your app doesn''t use them, but of course performance improvement if you app DOES re-read files commonly, so that should be taken into account.> On 3/5/09 6:29 PM, "Sebastian Gutierrez" > <sebastian.gutierrez at cs.stanford.edu> wrote: > > > Hello > > > > I am looking for advice or suggestions on specking the MDS and OSS for a > > new file system I am building. > > > > The IO patterns as have been explained to me are jobs that either create a > > big mpeg file or thousands of about 100kb frames for video. > > This could grow to hudreds of thousands of 100k files. > > > > I have specked > > 2 MDS for a cluster > > 8x300GB SAS 15k drives for the MDT in a raid 10(I used raid 10 to reduce > > the chance of the MDT becomming a bottleneck with many small files) > > 2x300GB SAS 15k drives for the MGS in a raid 1The MGS needs virtually no disk IO, so using smallest/cheapest SATA drives is fine.> > 4 OSS servers > > 16GB 667mhz ram > > 2.5 ghz dual quad core procs > > 6x1TB SATA 3 7.2k disks 5TB OST in a raid 5 (possibly running software raid) > > expandable to 2 5TB OSTs Per OSS > > 1xHotspare > > > > Since this is the first Lustre file system I am building. I noticed the > > sizing page: > > http://manual.lustre.org/manual/LustreManual16_HTML/SystemLimits.html#50548887 > > _46859 > > > > The document mentions only dual core procs in regards to the MDS.That''s because this section is old. If you want more MDS performance (up to a limit) adding more cores + RAM is useful.> > The document also mentions a much lower amount of system RAM required for > > the OST. > > > > Am I wasting resources here by over-specking these boxes or is it a good > > idea to over spec a bit? > > > > Am I on the right track based on the IO pattern as I understand it? > > > > Sebastian > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discussCheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.