Lustre discuss - Oct 2009 - Is there a way to set lru_size and have it stick?

Is there a way to set the lru_size to a fixed value and have it stay that way
across mounts?

I know it can be set using:
$ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
But that isn''t retained across a reboot.

Thank you.

--
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091008/6690643e/attachment.html

On Friday 09 October 2009, Lundgren, Andrew wrote:
> Is there a way to set the lru_size to a fixed value and have it stay that
>  way across mounts?
> 
> I know it can be set using:
> $ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
> But that isn''t retained across a reboot.
Even worse, if for some reason, e.g. evictions connection to OSTs get lost, it 
will also also reset to default. We are for now compiling our packages LRU 
disabled.

-- 
Bernd Schubert
DataDirect Networks

On 8-Oct-09, at 22:28, Lundgren, Andrew wrote:
> Is there a way to set the lru_size to a fixed value and have it stay  
> that way across mounts?
>
> I know it can be set using:
> $ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
> But that isn?t retained across a reboot.

"lctl set_param" is only for temporary tunable setting.  You can use
"lctl conf_param" to set a permanent tunable.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

On Saturday 10 October 2009, Andreas Dilger wrote:
> On 8-Oct-09, at 22:28, Lundgren, Andrew wrote:
> > Is there a way to set the lru_size to a fixed value and have it stay
> > that way across mounts?
> >
> > I know it can be set using:
> > $ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
> > But that isn?t retained across a reboot.
> 
> "lctl set_param" is only for temporary tunable setting.  You can
use
> "lctl conf_param" to set a permanent tunable.
Would you mind to provide an example line? I never understood the logic of 
lctl conf_param. This fails:

lctl conf_param ldlm.namespaces.testfs-OST0002-osc.lru_size=800
error: conf_param: Invalid argument



Thanks,
Bernd

On Saturday 10 October 2009, Andreas Dilger wrote:
> On 8-Oct-09, at 22:28, Lundgren, Andrew wrote:
> > Is there a way to set the lru_size to a fixed value and have it stay
> > that way across mounts?
> >
> > I know it can be set using:
> > $ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
> > But that isn?t retained across a reboot.
> 
> "lctl set_param" is only for temporary tunable setting.  You can
use
> "lctl conf_param" to set a permanent tunable.
Would you mind to provide an example line? I never understood the logic of 
lctl conf_param. This fails:

lctl conf_param ldlm.namespaces.testfs-OST0002-osc.lru_size=800
error: conf_param: Invalid argument


And this as well

lctl conf_param testfs-MDT0000.ldlm.namespaces.testfs-OST0002-osc.lru_size=800


Thanks,
Bernd

-- 
Bernd Schubert
DataDirect Networks

I have tried using:

# lctl conf_param content-MDT0000.osc.lru_size=800

Seen this in the log: 
Oct 12 18:35:36 abcd0202 kernel: Lustre: Modifying parameter
content-MDT0000-mdc.osc.lru_size in log content-client
Oct 12 18:35:36 abcd0202 kernel: Lustre: Skipped 1 previous similar message

But then on the clients, the lru_size doesn''t seem to change:

OSS # cat ./fs/lustre/ldlm/namespaces/*/lru_size
33
0
0
0
1
0
1
200

I have also set it for the OST individually from the MDS.  It doesn''t
seem to do anything for the other machines.

Is this a permanently tunable parameter, or am I just specifying the wrong
setting?
> -----Original Message-----
> From: Bernd Schubert [mailto:bs_lists at aakef.fastmail.fm]
> Sent: Monday, October 12, 2009 11:21 AM
> To: lustre-discuss at lists.lustre.org
> Cc: Andreas Dilger; Lundgren, Andrew
> Subject: Re: [Lustre-discuss] Is there a way to set lru_size and have
> it stick?
> 
> On Saturday 10 October 2009, Andreas Dilger wrote:
> > On 8-Oct-09, at 22:28, Lundgren, Andrew wrote:
> > > Is there a way to set the lru_size to a fixed value and have it
> stay
> > > that way across mounts?
> > >
> > > I know it can be set using:
> > > $ lctl set_param ldlm.namespaces.*osc*.lru_size=$((NR_CPU*100))
> > > But that isn?t retained across a reboot.
> >
> > "lctl set_param" is only for temporary tunable setting.  You
can use
> > "lctl conf_param" to set a permanent tunable.
> 
> Would you mind to provide an example line? I never understood the logic
> of
> lctl conf_param. This fails:
> 
> lctl conf_param ldlm.namespaces.testfs-OST0002-osc.lru_size=800
> error: conf_param: Invalid argument
> 
> 
> And this as well
> 
> lctl conf_param testfs-MDT0000.ldlm.namespaces.testfs-OST0002-
> osc.lru_size=800
> 
> 
> Thanks,
> Bernd
> 
> --
> Bernd Schubert
> DataDirect Networks

On 12-Oct-09, at 12:11, Lundgren, Andrew wrote:
> I have tried using:
>
> # lctl conf_param content-MDT0000.osc.lru_size=800
>
> Seen this in the log:
> Oct 12 18:35:36 abcd0202 kernel: Lustre: Modifying parameter content- 
> MDT0000-mdc.osc.lru_size in log content-client
> Oct 12 18:35:36 abcd0202 kernel: Lustre: Skipped 1 previous similar  
> message
>
> But then on the clients, the lru_size doesn''t seem to change:
>
> OSS # cat ./fs/lustre/ldlm/namespaces/*/lru_size
> 33
> 0
> 0
> 0
> 1
> 0
> 1
> 200
>
> I have also set it for the OST individually from the MDS.  It  
> doesn''t seem to do anything for the other machines.
>
> Is this a permanently tunable parameter, or am I just specifying the  
> wrong setting?

My apologies.  Any parameter settable in a /proc/fs/lustre/ file can  
usually
be specified as
<obd|fsname>.<obdtype>.<proc_file_name>=<value>, e.g.:

   * tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
   * lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
   * lctl conf_param testfs.llite.max_read_ahead_mb=16
   * ... testfs-MDT0000.lov.stripesize=2M
   * ... testfs-OST0000.osc.max_dirty_mb=29.15
   * ... testfs-OST0000.ost.client_cache_seconds=15
   * ... testfs.sys.timeout=40

However, it isn''t currently possible to specify a conf_param tunable  
for ldlm
settings, since they do not have their own OBD device and the tunable  
code is
(unfortunately) slightly different than other parts of the Lustre proc  
tunables.

Ages and ages ago, this was done because the externally-contributed  
lprocfs code
was very buggy and we wanted to make sure that the ldlm /proc tunables  
(which
were, at the time, the only ones that were actually required for Lustre
functionality) would continue working while lprocfs was disabled until  
fixed.

Until now, there was no reason to change that code, but it makes sense  
to fix
that now...  Could you file a bug on this?

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

On Tuesday 13 October 2009, Andreas Dilger wrote:
> On 12-Oct-09, at 12:11, Lundgren, Andrew wrote:
> > I have tried using:
> >
> > # lctl conf_param content-MDT0000.osc.lru_size=800
> >
> > Seen this in the log:
> > Oct 12 18:35:36 abcd0202 kernel: Lustre: Modifying parameter content-
> > MDT0000-mdc.osc.lru_size in log content-client
> > Oct 12 18:35:36 abcd0202 kernel: Lustre: Skipped 1 previous similar
> > message
> >
> > But then on the clients, the lru_size doesn''t seem to change:
> >
> > OSS # cat ./fs/lustre/ldlm/namespaces/*/lru_size
> > 33
> > 0
> > 0
> > 0
> > 1
> > 0
> > 1
> > 200
> >
> > I have also set it for the OST individually from the MDS.  It
> > doesn''t seem to do anything for the other machines.
> >
> > Is this a permanently tunable parameter, or am I just specifying the
> > wrong setting?
> 
> My apologies.  Any parameter settable in a /proc/fs/lustre/ file can
> usually
> be specified as
<obd|fsname>.<obdtype>.<proc_file_name>=<value>, e.g.:

Thanks, I think this should go into the the man page of lctl.

> 
>    * tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
>    * lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
>    * lctl conf_param testfs.llite.max_read_ahead_mb=16
>    * ... testfs-MDT0000.lov.stripesize=2M
>    * ... testfs-OST0000.osc.max_dirty_mb=29.15
>    * ... testfs-OST0000.ost.client_cache_seconds=15
>    * ... testfs.sys.timeout=40
> 
> However, it isn''t currently possible to specify a conf_param
tunable
> for ldlm
> settings, since they do not have their own OBD device and the tunable
> code is
> (unfortunately) slightly different than other parts of the Lustre proc
> tunables.
> 
> Ages and ages ago, this was done because the externally-contributed
> lprocfs code
> was very buggy and we wanted to make sure that the ldlm /proc tunables
> (which
> were, at the time, the only ones that were actually required for Lustre
> functionality) would continue working while lprocfs was disabled until
> fixed.
> 
> Until now, there was no reason to change that code, but it makes sense
> to fix
> that now...  Could you file a bug on this?
Done, bug 21084

Cheers,
Bernd

-- 
Bernd Schubert
DataDirect Networks

Thanks Bernd.

For a work around, we are doing a cron every 5 minutes for now to force it down
after unmount/remounts.

--
Andrew

-----Original Message-----
From: Bernd Schubert [mailto:bs_lists at aakef.fastmail.fm] 
Sent: Tuesday, October 13, 2009 4:15 AM
To: Andreas Dilger
Cc: Lundgren, Andrew; lustre-discuss at lists.lustre.org
Subject: Re: [Lustre-discuss] Is there a way to set lru_size and have it stick?

On Tuesday 13 October 2009, Andreas Dilger wrote:
> On 12-Oct-09, at 12:11, Lundgren, Andrew wrote:
> > I have tried using:
> >
> > # lctl conf_param content-MDT0000.osc.lru_size=800
> >
> > Seen this in the log:
> > Oct 12 18:35:36 abcd0202 kernel: Lustre: Modifying parameter content-
> > MDT0000-mdc.osc.lru_size in log content-client
> > Oct 12 18:35:36 abcd0202 kernel: Lustre: Skipped 1 previous similar
> > message
> >
> > But then on the clients, the lru_size doesn''t seem to change:
> >
> > OSS # cat ./fs/lustre/ldlm/namespaces/*/lru_size
> > 33
> > 0
> > 0
> > 0
> > 1
> > 0
> > 1
> > 200
> >
> > I have also set it for the OST individually from the MDS.  It
> > doesn''t seem to do anything for the other machines.
> >
> > Is this a permanently tunable parameter, or am I just specifying the
> > wrong setting?
> 
> My apologies.  Any parameter settable in a /proc/fs/lustre/ file can
> usually
> be specified as
<obd|fsname>.<obdtype>.<proc_file_name>=<value>, e.g.:

Thanks, I think this should go into the the man page of lctl.

> 
>    * tunefs.lustre --param mdt.group_upcall=NONE /dev/sda1
>    * lctl conf_param testfs-MDT0000.mdt.group_upcall=NONE
>    * lctl conf_param testfs.llite.max_read_ahead_mb=16
>    * ... testfs-MDT0000.lov.stripesize=2M
>    * ... testfs-OST0000.osc.max_dirty_mb=29.15
>    * ... testfs-OST0000.ost.client_cache_seconds=15
>    * ... testfs.sys.timeout=40
> 
> However, it isn''t currently possible to specify a conf_param
tunable
> for ldlm
> settings, since they do not have their own OBD device and the tunable
> code is
> (unfortunately) slightly different than other parts of the Lustre proc
> tunables.
> 
> Ages and ages ago, this was done because the externally-contributed
> lprocfs code
> was very buggy and we wanted to make sure that the ldlm /proc tunables
> (which
> were, at the time, the only ones that were actually required for Lustre
> functionality) would continue working while lprocfs was disabled until
> fixed.
> 
> Until now, there was no reason to change that code, but it makes sense
> to fix
> that now...  Could you file a bug on this?
Done, bug 21084

Cheers,
Bernd

-- 
Bernd Schubert
DataDirect Networks