Lustre community - Feb 2012 - Create directory where files get striped over the same set of OSTs

Hello,
My lustre installation has a total of 64 OSTs and I have created a 
directory that has been configured over 8 OSTs.
However, files in this directory get striped over different OSTs. For 
example,
file 1 might get striped over OSTs 0 thru 7, whereas file 2 might get 
striped over OSTs 8 thru 15.

How can I configure a directory so that files in it get striped across 
the *same* set of OSTs?

Thanks,
Kshitij Mehta
PhD candidate
Parallel Software Technologies Lab (pstl.cs.uh.edu)
Dept. of Computer Science
University of Houston
Houston, Texas, USA

Moving to lustre-discuss instead of lustre-community, which is not for
user discussions.

On 2012-02-09, at 4:24 PM, Kshitij Mehta wrote:
> My lustre installation has a total of 64 OSTs and I have created a 
> directory that has been configured over 8 OSTs.
> However, files in this directory get striped over different OSTs. For 
> example,
> file 1 might get striped over OSTs 0 thru 7, whereas file 2 might get 
> striped over OSTs 8 thru 15.
> 
> How can I configure a directory so that files in it get striped across 
> the *same* set of OSTs?
You can use OST pools.  See the Lustre manual, section 19.2.

Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Engineer                   http://www.whamcloud.com/

We have Lustre v 1.6.7, so I don''t think we have OST pools. 
We have an application that generates a few files. We wanted to test the
performance over a subset of OSTs. However, we end up hitting all OSTs,
since the different files get created over a different set of OSTs.  

Thanks for your reply,

Kshitij Mehta
PhD Candidate
Parallel Software Technologies Lab (pstl.cs.uh.edu)
Dept. of Computer Science
University of Houston
Houston, Texas, USA


-----Original Message-----
From: Wynkoop, John Louis [mailto:jwynkoop at utk.edu] 
Sent: Thursday, February 09, 2012 7:39 PM
To: Kshitij Mehta
Subject: Re: [Lustre-community] Create directory where files get striped
over the same set of OSTs

Kshitij,

You can consider using stripe offsets (IE: lfs setstripe -i 0), this will
configure Lustre to start the striping on the same OST each time.   To
keep the files confined to a certain set of OSTs, you should look into OST
pools.  You didn''t mention your version of Lustre, but anything 1.8 or
later
should have support for pools.  Just out of curiosity, why do you want to
limit which OSTs you are using?  That will greatly affect the usage balance
across your OSTs and reduce your total throughput.  Hope this helps.

--
John Wynkoop
HPC System Administrator
National Institute for Computational Sciences The University of Tennessee,
Knoxville
email: jwynkoop at utk.edu




On 2/9/12 6:24 PM, "Kshitij Mehta" <kmehta at cs.uh.edu> wrote:
>Hello,
>My lustre installation has a total of 64 OSTs and I have created a
>directory that has been configured over 8 OSTs.
>However, files in this directory get striped over different OSTs. For
>example,
>file 1 might get striped over OSTs 0 thru 7, whereas file 2 might get
>striped over OSTs 8 thru 15.
>
>How can I configure a directory so that files in it get striped across
>the *same* set of OSTs?
>
>Thanks,
>Kshitij Mehta
>PhD candidate
>Parallel Software Technologies Lab (pstl.cs.uh.edu)
>Dept. of Computer Science
>University of Houston
>Houston, Texas, USA
>_______________________________________________
>Lustre-community mailing list
>Lustre-community at lists.lustre.org
>http://lists.lustre.org/mailman/listinfo/lustre-community
>

More appropriate mail list for this question.

Cheers,
Kevin

On 2/9/12 3:24 PM, "Kshitij Mehta" <kmehta at cs.uh.edu> wrote:
>Hello,
>My lustre installation has a total of 64 OSTs and I have created a
>directory that has been configured over 8 OSTs.
>However, files in this directory get striped over different OSTs. For
>example,
>file 1 might get striped over OSTs 0 thru 7, whereas file 2 might get
>striped over OSTs 8 thru 15.
>
>How can I configure a directory so that files in it get striped across
>the *same* set of OSTs?
>
>Thanks,
>Kshitij Mehta
>PhD candidate
>Parallel Software Technologies Lab (pstl.cs.uh.edu)
>Dept. of Computer Science
>University of Houston
>Houston, Texas, USA
>_______________________________________________
>Lustre-community mailing list
>Lustre-community at lists.lustre.org
>http://lists.lustre.org/mailman/listinfo/lustre-community

Hi Kshitij,

If you''re using  a reasonably modern Lustre you can use the OST pools 
feature. This allows you to assign certain OST''s to certain 
files/directories.

see ''OST Pools'' in the operations manual for more information.

-cf

On 02/13/2012 09:56 AM, Kevin Canady wrote:
> More appropriate mail list for this question.
>
> Cheers,
> Kevin
>
> On 2/9/12 3:24 PM, "Kshitij Mehta"<kmehta at cs.uh.edu> 
wrote:
>
>> Hello,
>> My lustre installation has a total of 64 OSTs and I have created a
>> directory that has been configured over 8 OSTs.
>> However, files in this directory get striped over different OSTs. For
>> example,
>> file 1 might get striped over OSTs 0 thru 7, whereas file 2 might get
>> striped over OSTs 8 thru 15.
>>
>> How can I configure a directory so that files in it get striped across
>> the *same* set of OSTs?
>>
>> Thanks,
>> Kshitij Mehta
>> PhD candidate
>> Parallel Software Technologies Lab (pstl.cs.uh.edu)
>> Dept. of Computer Science
>> University of Houston
>> Houston, Texas, USA
>> _______________________________________________
>> Lustre-community mailing list
>> Lustre-community at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-community
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Hi Colin,
We have Lustre 1.6.7, so OST pools is not available on it.
I had the following suggestion from someone which I am going to try out:
> In that case, you can set the striping on the directory in question to
> start at OST index 0 with a stripe_count of 8 and create your files in
> that directory.  This should This will stripe all the files over 8 OSTs,
> but it will probably get you close to what you want for testing purposes.
>
> Here''s a quick guide:
> http://www.nics.tennessee.edu/node/311
>
>
> You''ll want something like this:
> lfs setstripe<directory>  -s<stripe_size>  -i 0 -c 8
>
> The round robin allocated should keep you in the first 8 OSTs but be
> careful that you don''t switch to the weighted allocator when you
start
> filling up your OSTs.
Thanks,

On 02/13/2012 11:15 AM, Colin Faber wrote:
> Hi Kshitij,
>
> If you''re using  a reasonably modern Lustre you can use the OST
pools
> feature. This allows you to assign certain OST''s to certain
> files/directories.
>
> see ''OST Pools'' in the operations manual for more
information.
>
> -cf
>
> On 02/13/2012 09:56 AM, Kevin Canady wrote:
>> More appropriate mail list for this question.
>>
>> Cheers,
>> Kevin
>>
>> On 2/9/12 3:24 PM, "Kshitij Mehta"<kmehta at cs.uh.edu>
wrote:
>>
>>> Hello,
>>> My lustre installation has a total of 64 OSTs and I have created a
>>> directory that has been configured over 8 OSTs.
>>> However, files in this directory get striped over different OSTs.
For
>>> example,
>>> file 1 might get striped over OSTs 0 thru 7, whereas file 2 might
get
>>> striped over OSTs 8 thru 15.
>>>
>>> How can I configure a directory so that files in it get striped
across
>>> the *same* set of OSTs?
>>>
>>> Thanks,
>>> Kshitij Mehta
>>> PhD candidate
>>> Parallel Software Technologies Lab (pstl.cs.uh.edu)
>>> Dept. of Computer Science
>>> University of Houston
>>> Houston, Texas, USA
>>> _______________________________________________
>>> Lustre-community mailing list
>>> Lustre-community at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-community
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> ______________________________________________________________________
> This email may contain privileged or confidential information, which should
only be used for the purpose for which it was sent by Xyratex. No further rights
or licenses are granted to use such information. If you are not the intended
recipient of this message, please notify the sender by return and delete it. You
may not use, copy, disclose or rely on the information contained in it.
>
> Internet email is susceptible to data corruption, interception and
unauthorised amendment for which Xyratex does not accept liability. While we
have taken reasonable precautions to ensure that this email is free of viruses,
Xyratex does not accept liability for the presence of any computer viruses in
this email, nor for any losses caused as a result of viruses.
>
> Xyratex Technology Limited (03134912), Registered in England&  Wales,
Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.
>
> The Xyratex group of companies also includes, Xyratex Ltd, registered in
Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia)
Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in
The People''s Republic of China and Xyratex Japan Limited registered in
Japan.
> ______________________________________________________________________
>
>
-- 
Kshitij Mehta
PhD Candidate
Parallel Software Technologies Lab (http://pstl.cs.uh.edu)
Dept. of Computer Science
University of Houston
Texas, USA

W dniu 10.02.2012 00:24, Kshitij Mehta pisze:
> Hello,
Hi,
> How can I configure a directory so that files in it get striped across 
> the *same* set of OSTs?
Define and use Lustre OST Pools.
Check http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools

Best regards,
Przemys?aw Dubaniewicz
WCSS