Forgive my novice understanding of lustre. Here is my current config: 1 combined MGS/MDT, 4 OSS''s attached to DDN controller/SAN. We have 32 1Tb OSTs (8 on each OSS) and because of the nature of our jobs, the files written to lustre vary greatly (5MB ~ 400GB size files). I''m not sure what algorithym lustre uses to balance writes to the fs but we have very high inconsistencies in disk space usage. One OST maybe only 40% full while another 94% full. This is problematic because if one of our OST''s fills up, jobs begin to fail with out of disk space errors even though there are terabytes of space unused. If someone can suggest a solution I would be eteranally grateful. The only solution I see is increasing each OST from 1TB to maybe 2 or 3TB? So this lead me to my second question - what type of performance degradation (if any) am I looking at if increase the size of the OSTs? and is there a limit to the size of OSTs? Any input is appreciated! Syed -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100218/dbd0abd5/attachment.html
On 2010-02-18, at 10:19, syed haider wrote:> 1 combined MGS/MDT, 4 OSS''s attached to DDN controller/SAN > We have 32 1Tb OSTs (8 on each OSS) and because of the nature of our > jobs, the files written to lustre vary greatly (5MB ~ 400GB size > files). > we have very high inconsistencies in disk space usage. One OST maybe > only 40% full while another 94% full.If your files are up to 40% of the OST size, then only 1 + fraction files are needed to hit a 50% imbalance between OSTs. I would really suggest to use 8TB OSTs to minimize this effect.> The only solution I see is increasing each OST from 1TB to maybe 2 > or 3TB? So this lead me to my second question - what type of > performance degradation (if any) am I looking at if increase the > size of the OSTs? and is there a limit to the size of OSTs? Any > input is appreciated!Also, with the DDN it doesn''t make any sense to split an 8TB tier of disks into 8 separate 1TB LUNs. That is just causing extra contention on the disks shared between these 8 LUNs, and hurting your performance. Most DDN installations use the 8TB LUN size, which is the maximum size until 1.8.2 where it goes to 16TB (enough for a tier of 2TB disks). Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.
Thank you Andreas, you''ve answered my question. 4 OST''s sounds a lot better than 32! On Thu, Feb 18, 2010 at 4:47 PM, Andreas Dilger <adilger at sun.com> wrote:> On 2010-02-18, at 10:19, syed haider wrote: > >> 1 combined MGS/MDT, 4 OSS''s attached to DDN controller/SAN >> We have 32 1Tb OSTs (8 on each OSS) and because of the nature of our jobs, >> the files written to lustre vary greatly (5MB ~ 400GB size files). >> we have very high inconsistencies in disk space usage. One OST maybe only >> 40% full while another 94% full. >> > > If your files are up to 40% of the OST size, then only 1 + fraction files > are needed to hit a 50% imbalance between OSTs. I would really suggest to > use 8TB OSTs to minimize this effect. > > > The only solution I see is increasing each OST from 1TB to maybe 2 or 3TB? >> So this lead me to my second question - what type of performance degradation >> (if any) am I looking at if increase the size of the OSTs? and is there a >> limit to the size of OSTs? Any input is appreciated! >> > > Also, with the DDN it doesn''t make any sense to split an 8TB tier of disks > into 8 separate 1TB LUNs. That is just causing extra contention on the > disks shared between these 8 LUNs, and hurting your performance. Most DDN > installations use the 8TB LUN size, which is the maximum size until 1.8.2 > where it goes to 16TB (enough for a tier of 2TB disks). > > Cheers, Andreas > -- > Andreas Dilger > Sr. Staff Engineer, Lustre Group > Sun Microsystems of Canada, Inc. > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100218/134b3f04/attachment.html