Lustre discuss - Dec 2013 - Number of OSS threads

How many OSS threads should I be running on an OSS with one OST with 12
disks in RAID6?

http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.50438272_55226

talks about optimising the number of threads on an OSS. As a starting
point it suggests the number of active spindles (so for our RAID6 over
12 disks, I make that 10).

The default on a machine with 12/24 Gig RAM and 8 cpu cores (16 threads)
is 128 - with a maximum of 512. That''s a rather big difference!!!

At the moment, I''m talking about Lustre 1.8.9, but I assume it
shouldn''t
vary too much between Lustre releases[1].

Chris

Christopher,

An easy answer, and often the best answer, is to let Lustre decide.  If you set
an "initial" thread count, the threads will auto tune up (not beyond
4x the initial, I believe).  I also believe there''s a max setting also,
but you probably want to let it auto tune up so that you can see what the
workload causes Lustre to do.  Then you can set an appropriate initial and upper
level.

Note also that there are 3 types of tunable threads on the OSS''s,
AFAIK.

There''s more to it, but the Lustre manual should cover this topic
pretty well.  If not, please feel free to ask some more.

--
Brett Lee
Sr. Systems Engineer
Intel High Performance Data Division

> -----Original Message-----
> From: lustre-discuss-bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org
[mailto:lustre-discuss-
> bounces-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org] On Behalf Of Christopher
J. Walker
> Sent: Wednesday, December 04, 2013 12:28 PM
> To: lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org
> Subject: [Lustre-discuss] Number of OSS threads
> 
> How many OSS threads should I be running on an OSS with one OST with 12
> disks in RAID6?
> 
> http://build.whamcloud.com/job/lustre-
> manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.50438272_
> 55226
> 
> talks about optimising the number of threads on an OSS. As a starting point
it
> suggests the number of active spindles (so for our RAID6 over
> 12 disks, I make that 10).
> 
> The default on a machine with 12/24 Gig RAM and 8 cpu cores (16 threads) is
> 128 - with a maximum of 512. That''s a rather big difference!!!
> 
> At the moment, I''m talking about Lustre 1.8.9, but I assume it
shouldn''t vary too
> much between Lustre releases[1].
> 
> Chris
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

On 2013/12/04 12:27 PM, "Christopher J. Walker"
<C.J.Walker-aUOafQfIvX+1Qrn1Bg8BZw@public.gmane.org>
wrote:
>How many OSS threads should I be running on an OSS with one OST with 12
>disks in RAID6?
>
>http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/
>lustre_manual.xhtml#dbdoclet.50438272_55226
>
>talks about optimising the number of threads on an OSS. As a starting
>point it suggests the number of active spindles (so for our RAID6 over
>12 disks, I make that 10).
>
>The default on a machine with 12/24 Gig RAM and 8 cpu cores (16 threads)
>is 128 - with a maximum of 512. That''s a rather big difference!!!
>
>At the moment, I''m talking about Lustre 1.8.9, but I assume it
shouldn''t
>vary too much between Lustre releases[1].
My general rule of thumb is 32 threads per OST.  That isn''t always
right,
but it is a good starting point for keeping the disks busy.  To be honest,
I think 1 thread per spindle is probably too low.

Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division