I''ve been reading up on setting up quotas and looks like luster needs to be shut down for that as it scans the entire filesystem. The thing is we already have ours up and running and with quite a bit of data on it. So any idea on how to estimate how long it would be to setup quotas on lustre? David -- Personally, I liked the university. They gave us money and facilities, we didn''t have to produce anything! You''ve never been out of college! You don''t know what it''s like out there! I''ve worked in the private sector. They expect results. -Ray Ghostbusters
On Thu, Mar 10, 2011 at 11:51:44AM -0600, David Noriega wrote:> I''ve been reading up on setting up quotas and looks like luster needs > to be shut down for that as it scans the entire filesystem. The thingThe problem is that accounting can be wrong if files/blocks are allocated/freed during the scan.> is we already have ours up and running and with quite a bit of data on > it. So any idea on how to estimate how long it would be to setup > quotas on lustre?quotacheck has been greatly improved in 1.8.2 (see bugzilla ticket 19763 for more information). As an example, quotacheck takes approximately 5min to complete when run against a 3.4TB filesystem (2 OSTs) which is 87% full. Cheers, Johann
Well we are running lustre 1.8.4, so thats great to hear. Thanks On Thu, Mar 10, 2011 at 12:15 PM, Johann Lombardi <johann at whamcloud.com> wrote:> On Thu, Mar 10, 2011 at 11:51:44AM -0600, David Noriega wrote: >> I''ve been reading up on setting up quotas and looks like luster needs >> to be shut down for that as it scans the entire filesystem. The thing > > The problem is that accounting can be wrong if files/blocks are allocated/freed during the scan. > >> is we already have ours up and running and with quite a bit of data on >> it. So any idea on how to estimate how long it would be to setup >> quotas on lustre? > > quotacheck has been greatly improved in 1.8.2 (see bugzilla ticket 19763 for more information). As an example, quotacheck takes approximately 5min to complete when run against a 3.4TB filesystem (2 OSTs) which is 87% full. > > Cheers, > Johann >-- Personally, I liked the university. They gave us money and facilities, we didn''t have to produce anything! You''ve never been out of college! You don''t know what it''s like out there! I''ve worked in the private sector. They expect results. -Ray Ghostbusters
Sebastien Piechurski
2011-Mar-11 12:29 UTC
[Lustre-discuss] Setting up quotas after the fact
Just to confirm what Johann said, I ran 2 weeks ago a complete quotacheck on a 600TB filesystem with ~50 millions inodes in approximately 5 minutes too. The filesystem is distributed across 28 OSS (168 OSTs of 3.6TB each). So you should be able to run your quotacheck quite quickly. Version was 1.8.5.> -----Original Message----- > From: lustre-discuss-bounces at lists.lustre.org > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of > David Noriega > Sent: jeudi 10 mars 2011 19:31 > To: lustre-discuss at lists.lustre.org > Subject: Re: [Lustre-discuss] Setting up quotas after the fact > > Well we are running lustre 1.8.4, so thats great to hear. Thanks > > On Thu, Mar 10, 2011 at 12:15 PM, Johann Lombardi > <johann at whamcloud.com> wrote: > > On Thu, Mar 10, 2011 at 11:51:44AM -0600, David Noriega wrote: > >> I''ve been reading up on setting up quotas and looks like > luster needs > >> to be shut down for that as it scans the entire > filesystem. The thing > > > > The problem is that accounting can be wrong if files/blocks > are allocated/freed during the scan. > > > >> is we already have ours up and running and with quite a > bit of data on > >> it. So any idea on how to estimate how long it would be to setup > >> quotas on lustre? > > > > quotacheck has been greatly improved in 1.8.2 (see bugzilla > ticket 19763 for more information). As an example, quotacheck > takes approximately 5min to complete when run against a 3.4TB > filesystem (2 OSTs) which is 87% full. > > > > Cheers, > > Johann > > > > > > -- > Personally, I liked the university. They gave us money and facilities, > we didn''t have to produce anything! You''ve never been out of college! > You don''t know what it''s like out there! I''ve worked in the private > sector. They expect results. -Ray Ghostbusters > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
That''s very good news. Thanks for the feedback S?bastien. Johann On Fri, Mar 11, 2011 at 06:29:11AM -0600, Sebastien Piechurski wrote:> Just to confirm what Johann said, I ran 2 weeks ago a complete > quotacheck on a 600TB filesystem with ~50 millions inodes in > approximately 5 minutes too. The filesystem is distributed across 28 OSS > (168 OSTs of 3.6TB each). > So you should be able to run your quotacheck quite quickly.-- Johann Lombardi Whamcloud, Inc. www.whamcloud.com
That sounds to me like it is only hitting the MDT and not the OSTs. which would mean if you have a lustre system that has been upgraded from an older version, most of the file sizes reported on the MDT will be zero and this must affect the validity of the quotacheck... GREG Johann Lombardi wrote:> That''s very good news. Thanks for the feedback S?bastien. > > Johann > > On Fri, Mar 11, 2011 at 06:29:11AM -0600, Sebastien Piechurski wrote: >> Just to confirm what Johann said, I ran 2 weeks ago a complete >> quotacheck on a 600TB filesystem with ~50 millions inodes in >> approximately 5 minutes too. The filesystem is distributed across 28 OSS >> (168 OSTs of 3.6TB each). >> So you should be able to run your quotacheck quite quickly. >-- Greg Matthews 01235 778658 Senior Computer Systems Administrator Diamond Light Source, Oxfordshire, UK -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
Sebastien Piechurski
2011-Mar-11 17:13 UTC
[Lustre-discuss] Setting up quotas after the fact
Gregory, Lustre does not (yet) store file sizes on the MDT (there is a project called SOM for Size On MDT, which is currently suspended I think). Instead, when a file size is requested, each OST holding objects for this file are queried for these objects size, which are then summed up. So, the quotacheck actually hits the OSTs.> -----Original Message----- > From: lustre-discuss-bounces at lists.lustre.org > [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of > Gregory Matthews > Sent: vendredi 11 mars 2011 17:52 > To: lustre-discuss at lists.lustre.org > Subject: Re: [Lustre-discuss] Setting up quotas after the fact > > That sounds to me like it is only hitting the MDT and not the OSTs. > which would mean if you have a lustre system that has been > upgraded from > an older version, most of the file sizes reported on the MDT will be > zero and this must affect the validity of the quotacheck... > > GREG > > Johann Lombardi wrote: > > That''s very good news. Thanks for the feedback S?bastien. > > > > Johann > > > > On Fri, Mar 11, 2011 at 06:29:11AM -0600, Sebastien > Piechurski wrote: > >> Just to confirm what Johann said, I ran 2 weeks ago a complete > >> quotacheck on a 600TB filesystem with ~50 millions inodes in > >> approximately 5 minutes too. The filesystem is distributed > across 28 OSS > >> (168 OSTs of 3.6TB each). > >> So you should be able to run your quotacheck quite quickly. > > > > > -- > Greg Matthews 01235 778658 > Senior Computer Systems Administrator > Diamond Light Source, Oxfordshire, UK > > > -- > This e-mail and any attachments may contain confidential, > copyright and or privileged material, and are for the use of > the intended addressee only. If you are not the intended > addressee or an authorised recipient of the addressee please > notify us of receipt by returning the e-mail and do not use, > copy, retain, distribute or disclose the information in or > attached to the e-mail. > Any opinions expressed within this e-mail are those of the > individual and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail > or any attachments are free from viruses and we cannot accept > liability for any damage which you may sustain as a result of > software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). > Registered in England and Wales with its registered office at > Diamond House, Harwell Science and Innovation Campus, Didcot, > Oxfordshire, OX11 0DE, United Kingdom > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
On Fri, Mar 11, 2011 at 04:52:21PM +0000, Gregory Matthews wrote:> That sounds to me like it is only hitting the MDT and not the OSTs.No, it hits both. Fortunately, quotacheck is run in parallel on the MDT and the OSTs, that''s why it scales well. Johann -- Johann Lombardi Whamcloud, Inc. www.whamcloud.com
Johann Lombardi wrote:> On Fri, Mar 11, 2011 at 04:52:21PM +0000, Gregory Matthews wrote: >> That sounds to me like it is only hitting the MDT and not the OSTs. >> > > No, it hits both. Fortunately, quotacheck is run in parallel on the > MDT and the OSTs, that''s why it scales well.ah ok. thanks for the clarification. It still sounds very fast! GREG> > Johann >-- Greg Matthews 01235 778658 Senior Computer Systems Administrator Diamond Light Source, Oxfordshire, UK -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom