I recall that there is an 8T limit on the size of an OST. Is that just for Lustre 1.6? Or also 1.8 and 2.0? Now that we are starting to get 2T drives, is there any way to exceed 8T? Thanks. Roger Spellman Staff Engineer Terascala, Inc. 508-588-1501 www.terascala.com http://www.terascala.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100126/e8fe0f04/attachment.html
On Tue, 2010-01-26 at 09:18 -0500, Roger Spellman wrote:> I recall that there is an 8T limit on the size of an OST. Is that > just for Lustre 1.6? Or also 1.8 and 2.0? > > Now that we are starting to get 2T drives, is there any way to exceed > 8T?I believe 16T is coming in 1.8.2 for distros where we can use ext4. IIRC, that''s RHEL5 at least. Unfortunately the ChangeLog(s) seem to be lacking details yet. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100126/31e9256d/attachment.bin
Yes, that is correct. So far we have only tested on RHEL5, but I believe that this should also work in theory on SLES11. However, we have so far been unable to find a volunteer able to help with SLES11 testing to confirm that the practice matches up with the theory. I have been talking with a couple of our OEM partners about this possibility so hopefully we can get this done in the near future. A couple of sites have been testing 16TB LUNs on RHEL with a pre-release version of 1.8.2 and the feedback so far has been positive. Brian J. Murrell wrote:> On Tue, 2010-01-26 at 09:18 -0500, Roger Spellman wrote: > >> I recall that there is an 8T limit on the size of an OST. Is that >> just for Lustre 1.6? Or also 1.8 and 2.0? >> >> Now that we are starting to get 2T drives, is there any way to exceed >> 8T? >> > > I believe 16T is coming in 1.8.2 for distros where we can use ext4. > IIRC, that''s RHEL5 at least. Unfortunately the ChangeLog(s) seem to be > lacking details yet. > > b. > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On Tue, Jan 26, 2010 at 09:15:13AM -0800, Peter Jones wrote:> Yes, that is correct. So far we have only tested on RHEL5To be clear, 16TB support is only for rhel5 using ext4-based ldiskfs. The default for rhel5 is still to use ext3. We will actually provide 2 sets of rpms, one with ext3-based ldiskfs (for people who do not need >8TB support) and one with ext4-based ldiskfs. For those who want to compile their own rpms, ext4 can be selected at configure time with the --enable-ext4 option. Johann
On Tuesday 26 January 2010, Johann Lombardi wrote:> On Tue, Jan 26, 2010 at 09:15:13AM -0800, Peter Jones wrote: > > Yes, that is correct. So far we have only tested on RHEL5 > > To be clear, 16TB support is only for rhel5 using ext4-based ldiskfs.Why is there a 16TiB limit? Ext4 has no such limit (except for 16TiB _file_ size). /Peter> The default for rhel5 is still to use ext3. We will actually provide > 2 sets of rpms, one with ext3-based ldiskfs (for people who do not > need >8TB support) and one with ext4-based ldiskfs. > For those who want to compile their own rpms, ext4 can be selected > at configure time with the --enable-ext4 option. > > Johann-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100126/3eac42f3/attachment.bin
> --- On Tue, 1/26/10, Johann Lombardi <johann at sun.com> wrote > > To be clear, 16TB support is only for rhel5 using ext4-based ldiskfs. > The default for rhel5 is still to use ext3. We will actually provide > 2 sets of rpms, one with ext3-based ldiskfs (for people who do not > need >8TB support) and one with ext4-based ldiskfs. > For those who want to compile their own rpms, ext4 can be selected > at configure time with the --enable-ext4 option.Hmm, nice to know. Where I can find more information about differences between ext3/4 based ldiskfs? I''d assume that 1.6 ext3 based lustre fs cannot be upgraded to ext4 based? How about quota support? T
On Tue, Jan 26, 2010 at 08:30:32PM +0100, Peter Kjellstrom wrote:> Why is there a 16TiB limit? Ext4 has no such limit (except for 16TiB _file_ > size).Because we have only tested & validated up to 16TB. You are of course free to try with bigger device (use the force_over_16tb mount option), but at your own risk. Johann
On Tue, Jan 26, 2010 at 11:41:40AM -0800, Tommi T wrote:> Hmm, nice to know. Where I can find more information about differences > between ext3/4 based ldiskfs?Actually, you won''t see any change at the lustre level, except bigger target support. Rebasing ldiskfs on ext4 allows us to reduce the number of patches we have to apply (e.g. extents, mballoc, ...) and to take advantage of fixes that went into the mainline and were not in our ldiskfs patches.> I''d assume that 1.6 ext3 based lustre fs cannot be upgraded to ext4 based?Yes, you can upgrade from ext3 to ext4.> How about quota support?No change. Johann
Am Dienstag 26 Januar 2010 18:15:13 schrieb Peter Jones: Any chance for 16TB+ with ext4 on vanilla kernels ? If yes, which kernel version ? Regards Heiko> Yes, that is correct. So far we have only tested on RHEL5, but I believe > that this should also work in theory on SLES11. However, we have so far > been unable to find a volunteer able to help with SLES11 testing to > confirm that the practice matches up with the theory. I have been > talking with a couple of our OEM partners about this possibility so > hopefully we can get this done in the near future. A couple of sites > have been testing 16TB LUNs on RHEL with a pre-release version of 1.8.2 > and the feedback so far has been positive. > > Brian J. Murrell wrote: > > On Tue, 2010-01-26 at 09:18 -0500, Roger Spellman wrote: > > > >> I recall that there is an 8T limit on the size of an OST. Is that > >> just for Lustre 1.6? Or also 1.8 and 2.0? > >> > >> Now that we are starting to get 2T drives, is there any way to exceed > >> 8T? > >> > > > > I believe 16T is coming in 1.8.2 for distros where we can use ext4. > > IIRC, that''s RHEL5 at least. Unfortunately the ChangeLog(s) seem to be > > lacking details yet. > > > > b.
> I recall that there is an 8T limit on the size of an OST. Is > that just for Lustre 1.6? Or also 1.8 and 2.0? Now that we > are starting to get 2T drives, is there any way to exceed 8T? > Thanks. [ ... ]While I have seen Lustre developers recommend having few large OSTs, queries like this seem to me to be based on assuming that ''fsck'' (or "resilvering" in the futurea) of the OSTs is never needed. Good luck with that. One of the advantages of Lustre, which applies regardless of the clustering aspect, is that OSTs can be checked in parallel. There is a good argument that for ''fsck'' purposes file systems should not be larger than one disk; because ''fsck''ing, unlike large or parallel reading or writing, does not scale well on RAID, and the ratio of disc arms per TB is ever shrinking. Sure, OSTs have a significant fixed overhead in Lustre, yet Lustre implementors should consider carefully how long their clients are prepared to wait for a damaged OST to be checked or reloaded from backup.