Lustre discuss - Jun 2011 - Performance versus allocation balance(?)

Hi all,

(many thanks to J. Lombardi directing our intention to this feature)
we noticed that a filesytem based on

  2 x  (4 x OSS + 1 SFA 10000)
  SLES11 + Lustre 1.8.4

slowed down drastically from 19.2 GB/s write throughput to 2.9 GB/s.
Of course the environment wasn''t changed. ;-) Measurements were taken
on
a nearly empty file system (19.2GB/s) and the current state 70% full
(2.9GB/s). All measurements were performed exclusively during system
maintenance. I.e. no other application using the cluster nor the storage
devices were active.

When trying to find a solution to reach the ''old'' value again
a test
changing the parameter ''qos_threshold_rr'' to 100% leads to the
desired
performance.

Upon checking the performance counters on the OS and the SFA side we
noticed that for the default setting for qos_threshold (16%) the
''distribution'' of the objects was not equal. A large number of
OST were
completely inactive. We found as expected an equal load distribution for
100%.

Does anyone can confirm this observation?

Is this really a ''feature'' of Lustre and if so are there plans
to
''re-design'' the object allocation part, so that

-a- full bandwidth can be reached with help of RR
-b- Allocation balancing is done with help of a ''background''
thread
    shifting objects accordingly to the striping policy of the file(s)
    some time later to OSTs with lower object allocation?

Many thanks in advance

Cheers

-Frank Heckes


------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Besuchen Sie uns auf unserem neuen Webauftritt unter www.fz-juelich.de

On 2011-06-27, at 3:25 AM, Frank Heckes wrote:
> (many thanks to J. Lombardi directing our intention to this feature)
> we noticed that a filesytem based on
> 
>  2 x  (4 x OSS + 1 SFA 10000)
>  SLES11 + Lustre 1.8.4
> 
> slowed down drastically from 19.2 GB/s write throughput to 2.9 GB/s.
> Of course the environment wasn''t changed. ;-) Measurements were
taken on
> a nearly empty file system (19.2GB/s) and the current state 70% full
> (2.9GB/s). All measurements were performed exclusively during system
> maintenance. I.e. no other application using the cluster nor the storage
> devices were active.
> 
> When trying to find a solution to reach the ''old'' value
again a test
> changing the parameter ''qos_threshold_rr'' to 100% leads
to the desired
> performance.
> 
> Upon checking the performance counters on the OS and the SFA side we
> noticed that for the default setting for qos_threshold (16%) the
> ''distribution'' of the objects was not equal. A large
number of OST were
> completely inactive. We found as expected an equal load distribution for
> 100%.
This dynamic load balancing is intended to even out OST usage in case
the free space is imbalanced between OSTs.  I haven''t heard it being
such a significant impact on performance in the past, however.

Do you have imbalanced free space on the OSTs?  If yes, how much and do
you know why it got imbalanced in the past?
> Does anyone can confirm this observation?
> 
> Is this really a ''feature'' of Lustre and if so are there
plans to
> ''re-design'' the object allocation part, so that
> 
> -a- full bandwidth can be reached with help of RR
There is a bug open that discusses how to improve the OST space balancing
using the round-robin allocator to avoid the dramatic performance loss
that you are seeing.  In https://bugzilla.lustre.org/show_bug.cgi?id=18547
there is a discussion of the issues and possible solutions, but work has
not progressed recently.
> -b- Allocation balancing is done with help of a
''background'' thread
>    shifting objects accordingly to the striping policy of the file(s)
>    some time later to OSTs with lower object allocation?
This is a separate issue, and depends on at least part of the HSM feature
landing in order to provide the "layout lock" functionality to allow
files
to change the OST objects over which they are striped in a safe way.  The
HSM policy engine is also well suited to doing background space management.

It is always better to do proper layout in the first place ("a" above)
to
avoid the need to move data around afterward, so I think both approaches
are useful to implement.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.

Hi Andreas,

many thanks for your detailed answer.

> Do you have imbalanced free space on the OSTs?  If yes, how much and do
> you know why it got imbalanced in the past?
>
The allocation varies from 65 - 70% usage per OST. The mean was 67%. So
definitely no difference > 25%.

> > Does anyone can confirm this observation?
> >
> > Is this really a ''feature'' of Lustre and if so are
there plans to
> > ''re-design'' the object allocation part, so that
> >
> > -a- full bandwidth can be reached with help of RR
>
> There is a bug open that discusses how to improve the OST space balancing
> using the round-robin allocator to avoid the dramatic performance loss
> that you are seeing.  In https://bugzilla.lustre.org/show_bug.cgi?id=18547
> there is a discussion of the issues and possible solutions, but work has
> not progressed recently.
>
I''ll check the bugzilla ticket, to find out whether there''s a
solution
for our problem.

Many thanks.

Cheers

-Frank



------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Besuchen Sie uns auf unserem neuen Webauftritt unter www.fz-juelich.de