llvm dev - Sep 2015 - [LNT][RFC] Providing useful Order context in LNT

I’d like some feedback on how people are using Orders in LNT, and how we can
improve that.  Abstractly, Orders are the version of the thing LNT is testing.
On llvm.org/perf, Orders are the SVN revision of the compiler.  The Order is
used to combine runs together, as well as to provide the x-axis of LNT’s graphs.
Other things I have seen used as Orders: git tags, git distance, unix time
stamps, letters or numbers.

What I would really like in LNT is more information about Orders.  Data I
(personally) would like:
 - blame list
 - change list with commit messages
 - links to more information

It would be great to be able to show change lists between Orders so we know what
the Order is actually capturing, including in the performance graphs and run
pages.  For SVN based system, that is easy, it is the change list between the
two revisions. I’d also like to be able to link to more information (like veiwvc
in the case of SVN), I think that can be accomplished with templates URLs.  Long
term, it would be nice to formally blame performance regressions, so a real
blame list is needed for that.

I think the ideal interface for everything I have been thinking about is just
given two Orders, show the changes.  Concretely for the SVN case I think we’d
need to have LNT keep a mirror of the SVN repo of interest.  That might be too
much to formally reside in LNT, so maybe a new external service that collects
and caches Order information.  Does anyone have any input on what and how we
should do this?

Hi Chris,

We (ab)use Orders by setting them to be buildbot build numbers.

Internally, we have a buildbot master and an LNT instance running. We have
the ability to do "try" builds (for A/B testing) and these also submit
to
LNT. This means two things:
  1. The try builds shouldn't be mixed with the daily/weekendly builds to
avoid polluting the database and making it useless for regression tracking.
  2. Being able to locate a specific buildbot build result inside LNT is
required.

To handle both of these, we ensure there is a bijective mapping between LNT
results and buildbot builds:
  buildbot builder -> LNT machine
  buildbot buildnumber -> LNT order

This makes it easy to work out which LNT run corresponds to which buildbot
run, which is a good thing. It does mean we abuse Orders horribly though,
and so date ordering doesn't work sometimes. We have a specific A/B
buildbot builder, so these show up in a different LNT machine to the
dailies.

I think for us to use orders properly, we'd need two things to happen:
  1. The ability to mark a submitted run as "polluting" or somesuch -
it
shouldn't be compared against by default and it shouldn't show up on
performance trend graphs unless explicitly asked to. This handles A/B
testing.
  2. The ability to "tag" an LNT run with some sort of label, and then
to
search for that label. That allows us to tag LNT runs with the builder name
and number, and we can reuse Orders to mean the right thing (SVN revision).

I think for your changes, that would mean that your order information
should also be able to contain information about whether a run was built
using a patched compiler, or different command line flags.

Any other/better ideas about how we can converge our workflows appreciated!

Cheers,

James

On Tue, 29 Sep 2015 at 00:49 Chris Matthews via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I’d like some feedback on how people are using Orders in LNT, and how we
> can improve that.  Abstractly, Orders are the version of the thing LNT is
> testing. On llvm.org/perf, Orders are the SVN revision of the compiler.
> The Order is used to combine runs together, as well as to provide the
> x-axis of LNT’s graphs. Other things I have seen used as Orders: git tags,
> git distance, unix time stamps, letters or numbers.
>
> What I would really like in LNT is more information about Orders.  Data I
> (personally) would like:
>  - blame list
>  - change list with commit messages
>  - links to more information
>
> It would be great to be able to show change lists between Orders so we
> know what the Order is actually capturing, including in the performance
> graphs and run pages.  For SVN based system, that is easy, it is the change
> list between the two revisions. I’d also like to be able to link to more
> information (like veiwvc in the case of SVN), I think that can be
> accomplished with templates URLs.  Long term, it would be nice to formally
> blame performance regressions, so a real blame list is needed for that.
>
> I think the ideal interface for everything I have been thinking about is
> just given two Orders, show the changes.  Concretely for the SVN case I
> think we’d need to have LNT keep a mirror of the SVN repo of interest.
> That might be too much to formally reside in LNT, so maybe a new external
> service that collects and caches Order information.  Does anyone have any
> input on what and how we should do this?
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150929/7a8bf6c5/attachment.html>