Lustre discuss - Feb 2010 - More Efficient Method of Messuring Usage?

Howdy,

I just ran a disk usage report for our Lustre (v1.8.1.1) scratch space (6TB used
of 40TB) that ended up taking 25 hours to complete. Yikes.

I''m wondering if anyone can suggest a more efficient way to get disk
usage data?

The disk usage script in question works as follows
1. builds an array of directories to check (/scratch/user1 /scratch/user2 /
scratch/user3 and so on)
2. loops through the array running the Linux ''du'' command on
each user directory storing the results in a new array
3. Once the loop completes a report is generated and emailed using the results
array

Steps 1 and 3 run very quick, so step 2 is the long straw. Traditionally, this
script has been used on our local home directories and will complete in
approximately 15 minutes (for ~ 1TB of usage).

I should mention that my 25 hour test was run from a Lustre client. We have one
MDS and two OSS (each serving 3 OSTs). Also, we do not have any quota enabled.

Thanks for any suggestions,

Mike

================================Mike Hanby
mhanby at uab.edu
Information Systems Specialist II
IT HPCS / Research Computing

Mike Hanby wrote:
> Howdy,
> 
> I just ran a disk usage report for our Lustre (v1.8.1.1) scratch space (6TB
used of 40TB) that ended up taking 25 hours to complete. Yikes.
> 
> I''m wondering if anyone can suggest a more efficient way to get
disk usage data?
Hi Mike,

If you enable quotas, you can get an instantaneous readout per user.

Cheers,

Guy


-- 
Dr. Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
Tel: +44 (0)1223 834244 x 6925
Fax: +44 (0)1223 496802


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

As Guy suggested, quotas are probably the quickest path to solution.

If you don''t want to enable them for some reason, RobinHood might do  
the trick for you:

http://robinhood.sourceforge.net/

You can use it to monitor utilization without using the purge  
functionality. I''ve looked into using it for this purpose, but I  
haven''t tried it yet.

j

On Feb 17, 2010, at 8:23 AM, Guy Coates wrote:
> Mike Hanby wrote:
>> Howdy,
>>
>> I just ran a disk usage report for our Lustre (v1.8.1.1) scratch  
>> space (6TB used of 40TB) that ended up taking 25 hours to complete.  
>> Yikes.
>>
>> I''m wondering if anyone can suggest a more efficient way to
get
>> disk usage data?
>
> Hi Mike,
>
> If you enable quotas, you can get an instantaneous readout per user.
>
> Cheers,
>
> Guy
>
>
> -- 
> Dr. Guy Coates,  Informatics System Group
> The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
> Tel: +44 (0)1223 834244 x 6925
> Fax: +44 (0)1223 496802
>
>
> -- 
> The Wellcome Trust Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a
> company registered in England with number 2742969, whose registered
> office is 215 Euston Road, London, NW1 2BE.
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
--
Jason Rappleye
System Administrator
NASA Advanced Supercomputing Division
NASA Ames Research Center
Moffett Field, CA 94035

Jason / Guy thanks for the suggestions.

We may go ahead and enable quota since we''ll need them at some point.

Robinhood looks interesting since the next logical step is what do you do when
the disk report shows certain users abusing the FS. And again the traditional
tools don''t scale well to TB worth of storage.

Thanks for sharing that link,

Mike

-----Original Message-----
From: Jason Rappleye [mailto:jason.rappleye at nasa.gov] 
Sent: Wednesday, February 17, 2010 11:56 AM
To: Mike Hanby
Cc: lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] More Efficient Method of Messuring Usage?


As Guy suggested, quotas are probably the quickest path to solution.

If you don''t want to enable them for some reason, RobinHood might do  
the trick for you:

http://robinhood.sourceforge.net/

You can use it to monitor utilization without using the purge  
functionality. I''ve looked into using it for this purpose, but I  
haven''t tried it yet.

j

On Feb 17, 2010, at 8:23 AM, Guy Coates wrote:
> Mike Hanby wrote:
>> Howdy,
>>
>> I just ran a disk usage report for our Lustre (v1.8.1.1) scratch  
>> space (6TB used of 40TB) that ended up taking 25 hours to complete.  
>> Yikes.
>>
>> I''m wondering if anyone can suggest a more efficient way to
get
>> disk usage data?
>
> Hi Mike,
>
> If you enable quotas, you can get an instantaneous readout per user.
>
> Cheers,
>
> Guy
>
>
> -- 
> Dr. Guy Coates,  Informatics System Group
> The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
> Tel: +44 (0)1223 834244 x 6925
> Fax: +44 (0)1223 496802
>
>
> -- 
> The Wellcome Trust Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a
> company registered in England with number 2742969, whose registered
> office is 215 Euston Road, London, NW1 2BE.
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
--
Jason Rappleye
System Administrator
NASA Advanced Supercomputing Division
NASA Ames Research Center
Moffett Field, CA 94035