Lustre discuss - May 2008 - quota persistence over filesystem shutdown

Hi all,

I have run into a problem with the quota system not being re-activated over a
filesystem shutdown.


I initially activated quotas on the lustre filesystem by running:

#quotacheck ug /mnt/data1

and then setting a user quota with:

#lfs -u gmpc 307200 309200 10000 11000 /mnt/data1


quota-ing  works as expected. If I umount and remount the filesystem on the
clients, the quota information and behaviour is preserved.


However, if I un-mount the clients and then restart the MDS/OSTs, the quota
system is not active once the filesystem is remounted on the client:


# lfs quota -u gmpc /mnt/data1
Disk quotas for user gmpc (uid xxxx):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
quota failed: No such process


If I run:

#lfs quotaon  /mnt/data1

on the client, the quota system is re-activated and everything works as
expected.


# lfs quota -u gmpc /mnt/data1
Disk quotas for user gmpc (uid 10795):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
     /mnt/data1   58308  307200  309200            1302   10000   11000
data1-MDT0000_UUID
                    432       0  102400            1302       0    5000
data1-OST0000_UUID
                   3272       0  102400
data1-OST0001_UUID
                  50500       0  102400
data1-OST0002_UUID
                   2052*      0       1
data1-OST0003_UUID
                   2052*      0       1


I am running lustre 1.6.4.3 / kernel 2.6.18 on debian ia32. The lquota module
has been loaded, and there are no error messages being generated on the
client/ost/mds.

Cheers,

Guy

-- 
Dr. Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
Tel: +44 (0)1223 834244 x 6925
Fax: +44 (0)1223 496802


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

You might want to use mkfs.lustre or tunefs.lustre with quota_type=ug parameter

Andrew.

On Wednesday 21 May 2008 20:32:22 Guy Coates wrote:
> Hi all,
> 
> I have run into a problem with the quota system not being re-activated over
a
> filesystem shutdown.
> 
> 
> I initially activated quotas on the lustre filesystem by running:
> 
> #quotacheck ug /mnt/data1
> 
> and then setting a user quota with:
> 
> #lfs -u gmpc 307200 309200 10000 11000 /mnt/data1
> 
> 
> quota-ing  works as expected. If I umount and remount the filesystem on the
> clients, the quota information and behaviour is preserved.
> 
> 
> However, if I un-mount the clients and then restart the MDS/OSTs, the quota
> system is not active once the filesystem is remounted on the client:
> 
> 
> # lfs quota -u gmpc /mnt/data1
> Disk quotas for user gmpc (uid xxxx):
>      Filesystem  kbytes   quota   limit   grace   files   quota   limit  
grace
> quota failed: No such process
> 
> 
> If I run:
> 
> #lfs quotaon  /mnt/data1
> 
> on the client, the quota system is re-activated and everything works as
expected.
> 
> 
> # lfs quota -u gmpc /mnt/data1
> Disk quotas for user gmpc (uid 10795):
>      Filesystem  kbytes   quota   limit   grace   files   quota   limit  
grace
>      /mnt/data1   58308  307200  309200            1302   10000   11000
> data1-MDT0000_UUID
>                     432       0  102400            1302       0    5000
> data1-OST0000_UUID
>                    3272       0  102400
> data1-OST0001_UUID
>                   50500       0  102400
> data1-OST0002_UUID
>                    2052*      0       1
> data1-OST0003_UUID
>                    2052*      0       1
> 
> 
> I am running lustre 1.6.4.3 / kernel 2.6.18 on debian ia32. The lquota
module
> has been loaded, and there are no error messages being generated on the
> client/ost/mds.
> 
> Cheers,
> 
> Guy
> 
> -- 
> Dr. Guy Coates,  Informatics System Group
> The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
> Tel: +44 (0)1223 834244 x 6925
> Fax: +44 (0)1223 496802
> 
>

On Wed, May 21, 2008 at 11:25:29PM +0400, Andrew Perepechko
wrote:
> You might want to use mkfs.lustre or tunefs.lustre with quota_type=ug
parameter
Huh? The manual doesn''t say anything about mkfs.lustre or
tunefs.lustre, but
all is to be done using lfs
(http://manual.lustre.org/manual/LustreManual16_HTML/ConfiguringQuotas.html)

Also in lustre/utils nothing points to mkfs.lustre or tunefs.lustre on
grepping for quotas.


Cheers,
Bernd

> 
> Andrew.
> 
> On Wednesday 21 May 2008 20:32:22 Guy Coates wrote:
> > Hi all,
> > 
> > I have run into a problem with the quota system not being re-activated
over a
> > filesystem shutdown.
> > 
> > 
> > I initially activated quotas on the lustre filesystem by running:
> > 
> > #quotacheck ug /mnt/data1
> > 
> > and then setting a user quota with:
> > 
> > #lfs -u gmpc 307200 309200 10000 11000 /mnt/data1
> > 
> > 
> > quota-ing  works as expected. If I umount and remount the filesystem
on the
> > clients, the quota information and behaviour is preserved.
> > 
> > 
> > However, if I un-mount the clients and then restart the MDS/OSTs, the
quota
> > system is not active once the filesystem is remounted on the client:
> > 
> > 
> > # lfs quota -u gmpc /mnt/data1
> > Disk quotas for user gmpc (uid xxxx):
> >      Filesystem  kbytes   quota   limit   grace   files   quota  
limit   grace
> > quota failed: No such process
> > 
> > 
> > If I run:
> > 
> > #lfs quotaon  /mnt/data1
> > 
> > on the client, the quota system is re-activated and everything works
as expected.
> > 
> > 
> > # lfs quota -u gmpc /mnt/data1
> > Disk quotas for user gmpc (uid 10795):
> >      Filesystem  kbytes   quota   limit   grace   files   quota  
limit   grace
> >      /mnt/data1   58308  307200  309200            1302   10000  
11000
> > data1-MDT0000_UUID
> >                     432       0  102400            1302       0   
5000
> > data1-OST0000_UUID
> >                    3272       0  102400
> > data1-OST0001_UUID
> >                   50500       0  102400
> > data1-OST0002_UUID
> >                    2052*      0       1
> > data1-OST0003_UUID
> >                    2052*      0       1
> > 
> > 
> > I am running lustre 1.6.4.3 / kernel 2.6.18 on debian ia32. The lquota
module
> > has been loaded, and there are no error messages being generated on
the
> > client/ost/mds.
> > 
> > Cheers,
> > 
> > Guy
> > 
> > -- 
> > Dr. Guy Coates,  Informatics System Group
> > The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
> > Tel: +44 (0)1223 834244 x 6925
> > Fax: +44 (0)1223 496802
> > 
> > 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

Andrew Perepechko wrote:
> You might want to use mkfs.lustre or tunefs.lustre with quota_type=ug
parameter

I tried:

tunefs.lustre  --param="quota_type=ug" /dev/XXX

on the MDS and OSTs, but still I still have the same problem.




tunefs.lustre  --param="quota_type=ug" /dev/mds1/data
checking for existing Lustre data: found CONFIGS/mountdata
Reading CONFIGS/mountdata

   Read previous values:
Target:     data1-MDT0000
Index:      0
Lustre FS:  data1
Mount type: ldiskfs
Flags:      0x405
              (MDT MGS )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: quota_type=ug


   Permanent disk data:
Target:     data1-MDT0000
Index:      0
Lustre FS:  data1
Mount type: ldiskfs
Flags:      0x405
              (MDT MGS )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: quota_type=ug

tunefs.lustre: according to /etc/mtab /dev/mds1/data is already mounted on
/mnt/mds1

Cheers,

Guy


Dr. Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
Tel: +44 (0)1223 834244 x 6925
Fax: +44 (0)1223 496802


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

On Thu, May 22, 2008 at 12:31:42AM +0200, Bernd Schubert
wrote:
> Huh?
See bug 13359.
> The manual doesn''t say anything about mkfs.lustre or
tunefs.lustre, but
> all is to be done using lfs
(http://manual.lustre.org/manual/LustreManual16_HTML/ConfiguringQuotas.html)
It is a lack in the documentation, we are going to fix this.

Johann

On Thu, May 22, 2008 at 10:59:00AM +0100, Guy Coates
wrote:
> on the MDS and OSTs, but still I still have the same problem.
You ran the tunefs.lustre command, stopped/started the MDT/OSTs and quotas
were not automatically turned on, right?
Did you run quotacheck on the filesystem at least once?
Please advise.

Cheers,
Johann

Johann Lombardi wrote:
> On Thu, May 22, 2008 at 10:59:00AM +0100, Guy Coates wrote:
>> on the MDS and OSTs, but still I still have the same problem.
> 
> You ran the tunefs.lustre command, stopped/started the MDT/OSTs and quotas
> were not automatically turned on, right?
> Did you run quotacheck on the filesystem at least once?
> Please advise.

I originally tried

quota_type=ug

but that isn''t a recognised parameter; I''m in the process of
using

xxt.quota_type=ug as described in 13359.

Thanks,

Guy

-- 
Dr. Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
Tel: +44 (0)1223 834244 x 6925
Fax: +44 (0)1223 496802


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

On Friday 23 May 2008 10:09:52 Johann Lombardi wrote:
> On Thu, May 22, 2008 at 12:31:42AM +0200, Bernd Schubert wrote:
> > Huh?
>
> See bug 13359.
>
> > The manual doesn''t say anything about mkfs.lustre or
tunefs.lustre, but
> > all is to be done using lfs
> >
(http://manual.lustre.org/manual/LustreManual16_HTML/ConfiguringQuotas.ht
> >ml)
>
> It is a lack in the documentation, we are going to fix this.
Ah, thanks! Hmm, running tunefs.lustre on a filesystem in production is not
always easy.
Will lctl also do the job? Something like 

lctl fsname-MDT000.mdt.quota_type=ug

for i in `lctl dl | awk ''{print $4}'' | sed -e
''s/-osc$//''`; do lctl ${i}.mdt.quota_type=ug; done

on the MGS? 


Thanks,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Bernd Schubert wrote:
> On Friday 23 May 2008 10:09:52 Johann Lombardi wrote:
>> On Thu, May 22, 2008 at 12:31:42AM +0200, Bernd Schubert wrote:
>>> Huh?
>> See bug 13359.
Humm,

The quotaing is still not persistent.



I''ve created a new filesystem and ran:

 tunefs.lustre    --param="xxt.quota_type=ug"

on the MDS and OSTs. However, when I started the MDS/OSTs I got the following
error:


May 23 11:31:31 bc-1-1-01 kernel: Lustre: Ignoring unrecognized param
''xxt.quota_type=ug''

Confusingly, that error message was only see once; on subsequent remounts the
error message isn''t displayed.


Guy

-- 
Dr. Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
Tel: +44 (0)1223 834244 x 6925
Fax: +44 (0)1223 496802


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

On Friday 23 May 2008 13:00:02 Guy Coates wrote:
> Bernd Schubert wrote:
> > On Friday 23 May 2008 10:09:52 Johann Lombardi wrote:
> >> On Thu, May 22, 2008 at 12:31:42AM +0200, Bernd Schubert wrote:
> >>> Huh?
> >>
> >> See bug 13359.
>
> Humm,
>
> The quotaing is still not persistent.
>
>
>
> I''ve created a new filesystem and ran:
>
>  tunefs.lustre    --param="xxt.quota_type=ug"
>
> on the MDS and OSTs. However, when I started the MDS/OSTs I got the
> following error:
>
>
> May 23 11:31:31 bc-1-1-01 kernel: Lustre: Ignoring unrecognized param
> ''xxt.quota_type=ug''
>
> Confusingly, that error message was only see once; on subsequent remounts
> the error message isn''t displayed.
I think you misunderstood ''xxt'', xxt={mdt,ost}. So on the MDT
you have to
use "mdt.quota_type=ug" and on the OST "ost.quota_type=ug".


Hope it helps,
Bernd

-- 
Bernd Schubert
Q-Leap Networks GmbH

Bernd Schubert wrote:
> On Friday 23 May 2008 13:00:02 Guy Coates wrote:
>> Bernd Schubert wrote:
>>> On Friday 23 May 2008 10:09:52 Johann Lombardi wrote:
>>>> On Thu, May 22, 2008 at 12:31:42AM +0200, Bernd Schubert wrote:
>>>>> Huh?
>>>> See bug 13359.
>> Humm,
>>
>> The quotaing is still not persistent.
>>
>>
>>
>> I''ve created a new filesystem and ran:
>>
>>  tunefs.lustre    --param="xxt.quota_type=ug"
>>
>> on the MDS and OSTs. However, when I started the MDS/OSTs I got the
>> following error:
>>
>>
>> May 23 11:31:31 bc-1-1-01 kernel: Lustre: Ignoring unrecognized param
>> ''xxt.quota_type=ug''
>>
>> Confusingly, that error message was only see once; on subsequent
remounts
>> the error message isn''t displayed.
> 
> I think you misunderstood ''xxt'', xxt={mdt,ost}. So on the
MDT you have to
> use "mdt.quota_type=ug" and on the OST
"ost.quota_type=ug".
> 
> 
> Hope it helps,
> Bernd
> 
That did the trick!

Thanks all.

Cheers,

Guy

-- 
Dr. Guy Coates,  Informatics System Group
The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
Tel: +44 (0)1223 834244 x 6925
Fax: +44 (0)1223 496802


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

In the fall we plan to roll out our 1.6.5 release in production and if
all goes well in testing, enable quotas with no limits for block/inode
usage reporting.

I''m adding the tunefs commands to our rollout notes.

Will we need to re-run quotacheck in our startup scripts when the MDS 
restarts uncleanly?

Thanks,

Jim

On Fri, May 23, 2008 at 01:13:59PM +0100, Guy Coates
wrote:
> Bernd Schubert wrote:
> > On Friday 23 May 2008 13:00:02 Guy Coates wrote:
> >> Bernd Schubert wrote:
> >>> On Friday 23 May 2008 10:09:52 Johann Lombardi wrote:
> >>>> On Thu, May 22, 2008 at 12:31:42AM +0200, Bernd Schubert
wrote:
> >>>>> Huh?
> >>>> See bug 13359.
> >> Humm,
> >>
> >> The quotaing is still not persistent.
> >>
> >>
> >>
> >> I''ve created a new filesystem and ran:
> >>
> >>  tunefs.lustre    --param="xxt.quota_type=ug"
> >>
> >> on the MDS and OSTs. However, when I started the MDS/OSTs I got
the
> >> following error:
> >>
> >>
> >> May 23 11:31:31 bc-1-1-01 kernel: Lustre: Ignoring unrecognized
param
> >> ''xxt.quota_type=ug''
> >>
> >> Confusingly, that error message was only see once; on subsequent
remounts
> >> the error message isn''t displayed.
> > 
> > I think you misunderstood ''xxt'', xxt={mdt,ost}. So
on the MDT you have to
> > use "mdt.quota_type=ug" and on the OST
"ost.quota_type=ug".
> > 
> > 
> > Hope it helps,
> > Bernd
> > 
> That did the trick!
> 
> Thanks all.
> 
> Cheers,
> 
> Guy
> 
> -- 
> Dr. Guy Coates,  Informatics System Group
> The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK
> Tel: +44 (0)1223 834244 x 6925
> Fax: +44 (0)1223 496802
> 
> 
> -- 
>  The Wellcome Trust Sanger Institute is operated by Genome Research 
>  Limited, a charity registered in England with number 1021457 and a 
>  company registered in England with number 2742969, whose registered 
>  office is 215 Euston Road, London, NW1 2BE. 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

On Fri, May 23, 2008 at 11:00:49AM -0700, Jim Garlick
wrote:
> Will we need to re-run quotacheck in our startup scripts when the MDS 
> restarts uncleanly?
Yes, quotacheck is supposed to be run if the underlying ldiskfs filesystem
has not been unmounted cleanly (due to a crash for instance). The accounting
must be fixed to keep quotas accurate.
That being said, we now use journaled quota, so quotacheck is no longer needed
after an unclean shutdown with lustre 1.6.5/1.4.12 and above (see bug 14443).

Cheers,
Johann

On Fri, May 23, 2008 at 11:16:09AM +0200, Bernd Schubert
wrote:
> Ah, thanks! Hmm, running tunefs.lustre on a filesystem in production is not
always easy.
> Will lctl also do the job? Something like 
Yes, ''lctl conf_param fsname-OSTxxxx.ost.quota_type=ug'' for
both OSTs and MDT
(OST/ost should be replaced by MDT/mdt) should do the trick.

HTH
Johann

On Monday 26 May 2008 16:28:51 Johann Lombardi wrote:
> On Fri, May 23, 2008 at 11:16:09AM +0200, Bernd Schubert wrote:
> > Ah, thanks! Hmm, running tunefs.lustre on a filesystem in production
is
> > not always easy. Will lctl also do the job? Something like
>
> Yes, ''lctl conf_param fsname-OSTxxxx.ost.quota_type=ug''
for both OSTs and
> MDT (OST/ost should be replaced by MDT/mdt) should do the trick.

Great and thanks a lot for your help!


-- 
Bernd Schubert
Q-Leap Networks GmbH