Forgive me if this is already covered in a FAQ. I am trying out Lustre. So far, so good. I am testing the ability to add new OSTs. I have found that it''s very easy to add new ones, especially in v. 1.6. However I was uncertain if, after adding a new OST if data would still be written to the older OSTs. For instance, I have a set of 5 OSTs. I find that they are running low on memory so I add another 5 OSTs. I see that now all 10 OSTs are being written to. What will happen when the 5 older OSTs run out of space? Will Lustre prevent new data from being added or will it automatically allocate the writes to the to the 5 new OSTs? Is there a way to tell Lustre to ONLY write to certain OSTs?
https://bugzilla.lustre.org/show_bug.cgi?id=7208 I already found my answer. So I can add OSTs and I can be assured that newly created files will have room to grow. As far as older files, I can do a data migration for more even distribution. Cool. David Bernick wrote:> Forgive me if this is already covered in a FAQ. > > I am trying out Lustre. So far, so good. > > I am testing the ability to add new OSTs. I have found that it''s very > easy to add new ones, especially in v. 1.6. However I was uncertain if, > after adding a new OST if data would still be written to the older OSTs. > > For instance, I have a set of 5 OSTs. I find that they are running low > on memory so I add another 5 OSTs. I see that now all 10 OSTs are being > written to. > > What will happen when the 5 older OSTs run out of space? Will Lustre > prevent new data from being added or will it automatically allocate the > writes to the to the 5 new OSTs? > > Is there a way to tell Lustre to ONLY write to certain OSTs? > > >-- David Bernick, CISSP | IT Manager Lextranet | 107 Union Wharf | Boston, MA 02109 http://www.lextranet.com (617)227-4469x219 You will pay for your sins. If you have already paid, please disregard this message. THE INFORMATION IN THIS MESSAGE IS INTENDED ONLY FOR THE PERSONAL AND CONFIDENTIAL USE OF THE DESIGNATED RECIPIENTS NAMED ABOVE AND MAY CONTAIN LEGALLY PRIVILEGED INFORMATION. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error, and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by telephone at 617-227-4469 Ext. 219. Thank you. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 191 bytes Desc: OpenPGP digital signature Url : http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20061011/6a5eefb4/signature.bin
Indeed. Lustre 1.6 includes a weighting function to try to fill less-used OSTs preferentially. You can tell Lustre to assign a file to a particular set of OSTs using "lfs setstripe" (stripe start). You can "setstripe" a directory and everything (new) in that dir will be written using those stripe parameters. David Bernick wrote:> https://bugzilla.lustre.org/show_bug.cgi?id=7208 > > I already found my answer. So I can add OSTs and I can be assured that > newly created files will have room to grow. > > As far as older files, I can do a data migration for more even > distribution. Cool. > > David Bernick wrote: > >> Forgive me if this is already covered in a FAQ. >> >> I am trying out Lustre. So far, so good. >> >> I am testing the ability to add new OSTs. I have found that it''s very >> easy to add new ones, especially in v. 1.6. However I was uncertain if, >> after adding a new OST if data would still be written to the older OSTs. >> >> For instance, I have a set of 5 OSTs. I find that they are running low >> on memory so I add another 5 OSTs. I see that now all 10 OSTs are being >> written to. >> >> What will happen when the 5 older OSTs run out of space? Will Lustre >> prevent new data from being added or will it automatically allocate the >> writes to the to the 5 new OSTs? >> >> Is there a way to tell Lustre to ONLY write to certain OSTs? >> >> >> >> > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
Nathaniel Rutman wrote:> Indeed. Lustre 1.6 includes a weighting function to try to fill > less-used OSTs preferentially. > You can tell Lustre to assign a file to a particular set of OSTs using > "lfs setstripe" (stripe start). > You can "setstripe" a directory and everything (new) in that dir will be > written using those stripe > parameters.Is this the weighting function present in Lustre 1.5.95? Or will it be added to 1.6 further down the line? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 191 bytes Desc: OpenPGP digital signature Url : http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20061012/f577c9f8/signature.bin
David Bernick wrote:> Nathaniel Rutman wrote: > >> Indeed. Lustre 1.6 includes a weighting function to try to fill >> less-used OSTs preferentially. >> You can tell Lustre to assign a file to a particular set of OSTs using >> "lfs setstripe" (stripe start). >> You can "setstripe" a directory and everything (new) in that dir will be >> written using those stripe >> parameters. >> > > Is this the weighting function present in Lustre 1.5.95? Or will it be > added to 1.6 further down the line? > >It''s in all the betas, bug 9862 in ChangeLog.
Oops. The free-space stripe weighting (what we call "stripe QOS"), although present in all the 1.6 betas, was inadvertently set to give a priority of "0" to the free space (versus trying to place the stripes "widely" -- nicely distributed across OSSs and OSTs to maximize network balancing). This priority can be adjusted via the proc file /proc/fs/lustre/lov/lustre-mdtlov/qos_prio_free The default in the future will be 90%. You can set this permanently on existing betas with this command on the MGS: lctl conf_param <fsname>-MDT0000.lov.qos_prio_free=90 Note that setting the priority to 100% just means that OSS distribution doesn''t count in the weighting, but the stripe assignment is still done via a weighting -- if OST2 has twice as much free space as OST1, it will be twice as likely to be used, _not_ guaranteed to be used. Also note that stripe QOS doesn''t kick in until two OSTs are imbalanced by more than 20%. Until then, a faster round-robin stripe allocater is used. (The new round-robin order also maximizes network balancing.) Nathaniel Rutman wrote:> David Bernick wrote: >> Nathaniel Rutman wrote: >> >>> Indeed. Lustre 1.6 includes a weighting function to try to fill >>> less-used OSTs preferentially. >>> You can tell Lustre to assign a file to a particular set of OSTs using >>> "lfs setstripe" (stripe start). >>> You can "setstripe" a directory and everything (new) in that dir >>> will be >>> written using those stripe >>> parameters. >>> >> >> Is this the weighting function present in Lustre 1.5.95? Or will it be >> added to 1.6 further down the line? >> >> > It''s in all the betas, bug 9862 in ChangeLog. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >