llvm dev - Sep 2015 - [RFC] LNT Feature: tracking performance changes

I’d like to propose a new LNT feature that I plan to start working on
immediately: formal performance change tracking.

Motivation: LNT produces a lot of data, but it is not very actionable.  Unless
you are watching the nightly reports, or checking LNT regularly, performance
changes can happen and be missed.  Oh no!  Once commits have been in tree for a
long time, it is harder to fix them even if they cause a negative performance
impact. Finding performance changes early is important.

Solution: ease the burden of finding and verifying performance changes, with a
new set of interfaces to assist the user.  I’ll build a new area in LNT for
tracking performance changes that are identified as significant based on our
current techniques for finding performance changes.  Changes will be shown in a
“triage page” for review and be grouped together into “regressions” using
heuristics. The database entry for a regression will be used to track its
current status and resolution. The regression can be associated with a separate
bugzilla report, and provides a way of linking between the bug report and the
raw live performance results. Changes in performance will also be able to be
annotated in the performance graphs by referring to the regression information.

My intent is that with a small amount of user input it will be possible to
find/track/verify performance changes.  Use of this feature will be optional,
and probably off by default.

I plan to make 4 changes to the LNT interface: the triage page, a regression
list page, a regression details page, and changes to the graph page to add
regression annotations.

Under the hood, I will be making some other big changes, including,
authentication and user accounts (so that only known users can edit regression
details), form validation, adding a mechanism for large offline computations,
and replacing the guts of the graphing system to load data dynamically.

I will be developing some heuristics for combining changes. For instance several
benchmarks that all regress at the same run are probably caused by the same
change. Likewise, changes of the same benchmark on different architecture or
optimization levels.

Even if you won’t be using the tracking directly these features should speed up
the rest of the LNT interface.

----- Original Message -----
> From: "Chris Matthews via llvm-dev" <llvm-dev at
lists.llvm.org>
> To: "llvm-dev" <llvm-dev at lists.llvm.org>
> Sent: Monday, September 21, 2015 6:15:44 PM
> Subject: [llvm-dev] [RFC] LNT Feature: tracking performance changes
> 
> I’d like to propose a new LNT feature that I plan to start working on
> immediately: formal performance change tracking.
> 
> Motivation: LNT produces a lot of data, but it is not very
> actionable.  Unless you are watching the nightly reports, or
> checking LNT regularly, performance changes can happen and be
> missed.  Oh no!  Once commits have been in tree for a long time, it
> is harder to fix them even if they cause a negative performance
> impact. Finding performance changes early is important.
> 
> Solution: ease the burden of finding and verifying performance
> changes, with a new set of interfaces to assist the user.  I’ll
> build a new area in LNT for tracking performance changes that are
> identified as significant based on our current techniques for
> finding performance changes.  Changes will be shown in a “triage
> page” for review and be grouped together into “regressions” using
> heuristics. The database entry for a regression will be used to
> track its current status and resolution. The regression can be
> associated with a separate bugzilla report, and provides a way of
> linking between the bug report and the raw live performance results.
> Changes in performance will also be able to be annotated in the
> performance graphs by referring to the regression information.
> 
> My intent is that with a small amount of user input it will be
> possible to find/track/verify performance changes.  Use of this
> feature will be optional, and probably off by default.
> 
> I plan to make 4 changes to the LNT interface: the triage page, a
> regression list page, a regression details page, and changes to the
> graph page to add regression annotations.
> 
> Under the hood, I will be making some other big changes, including,
> authentication and user accounts (so that only known users can edit
> regression details), form validation, adding a mechanism for large
> offline computations, and replacing the guts of the graphing system
> to load data dynamically.
> 
> I will be developing some heuristics for combining changes. For
> instance several benchmarks that all regress at the same run are
> probably caused by the same change. Likewise, changes of the same
> benchmark on different architecture or optimization levels.
> 
> Even if you won’t be using the tracking directly these features
> should speed up the rest of the LNT interface.
This all sounds really nice, and linking the performance regressions to specific
bug reports sounds useful. However, I think that unless we're able to e-mail
commit authors who cause regressions like we do for buildbot failures, as well
as others who are architecture performance experts on the relevant systems, this
won't see the use it needs.

 -Hal
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

I agree.  I plan to implement that next!
> On Sep 21, 2015, at 4:25 PM, Hal Finkel <hfinkel at anl.gov> wrote:
> 
>  However, I think that unless we're able to e-mail commit authors who
cause regressions like we do for buildbot failures, as well as others who are
architecture performance experts on the relevant systems, this won't see the
use it needs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150921/6d44b09e/attachment.html>

Sounds like a great plan :)
> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Chris Matthews via llvm-dev
> Sent: 22 September 2015 00:16
> To: llvm-dev
> Subject: [llvm-dev] [RFC] LNT Feature: tracking performance changes
> 
> The regression can be associated with a separate bugzilla
> report, and provides a way of linking between the bug report and the raw
> live performance results. Changes in performance will also be able to be
> annotated in the performance graphs by referring to the regression
> information.
I'm assuming that the implementation will also allow to link to other bug
tracking systems, not just bugzilla? That'd be great as we're also using
LNT internally to track performance and we're using a different bug tracker
internally.
> My intent is that with a small amount of user input it will be possible
> to find/track/verify performance changes.  Use of this feature will be
> optional, and probably off by default.
I don't see yet why this needs to be off by default, but I'm assuming
I'll understand once a few more details are described.
> I plan to make 4 changes to the LNT interface: the triage page, a
> regression list page, a regression details page, and changes to the
> graph page to add regression annotations.
Our use of LNT to track performance so far has been to manually inspect the
daily report page - making sure that every single day has been inspected to try
and not miss performance deltas. This has been error-prone indeed.
After this lands, should we aim to write up some "best practice to do
performance tracking with LNT"-documentation?
 
> Under the hood, I will be making some other big changes, including,
> authentication and user accounts (so that only known users can edit
> regression details), form validation, adding a mechanism for large
> offline computations, and replacing the guts of the graphing system to
> load data dynamically.
Really nice - hopefully this will also alleviate some of the stability and
performance issues we've seen on llvm.org/perf.
> I will be developing some heuristics for combining changes. For instance
> several benchmarks that all regress at the same run are probably caused
> by the same change. Likewise, changes of the same benchmark on different
> architecture or optimization levels.
> 
> Even if you won’t be using the tracking directly these features should
> speed up the rest of the LNT interface.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev