llvm dev - Feb 2020 - LLVM compile-time regression tracking?

Hi,

Does the LLVM project perform any kind of tracking for commit-by-commit
compile-time changes? It looks like LNT only tracks run-time performance
(and to be honest I wasn't able to make heads or tails of the results even
for that -- the interface was pretty unintuitive to me.)

While it is "normal" that each new LLVM release regresses compile-time
by
2-3%, LLVM 10 seems to be going for a new record here, with optimized
builds regressing by around 5-15%, and that's already after mitigating the
largest regression:
https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c

It does not seem like these regressions are specific to Rust either, see
for example https://bugs.llvm.org/show_bug.cgi?id=44408.

The Rust project tracks compile-time results for each commit (see link
above), and thanks to that can quickly spot and revert changes that
negatively affect performance. I think it would be really great if
something similar would exist for LLVM...

Regards,
Nikita
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200208/abd5c9ab/attachment.html>

Hi,


On Sat, Feb 8, 2020 at 5:17 AM Nikita Popov via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> Does the LLVM project perform any kind of tracking for commit-by-commit
> compile-time changes? It looks like LNT only tracks run-time performance
> (and to be honest I wasn't able to make heads or tails of the results
even
> for that -- the interface was pretty unintuitive to me.)
>
At some point Apple was feeding compile-time data to LNT, but it seems like
the jobs aren't active anymore:
http://green.lab.llvm.org/green/view/Compile%20Time/

The main difficulty is acting on the regressions though...

-- 
Mehdi



>
> While it is "normal" that each new LLVM release regresses
compile-time by
> 2-3%, LLVM 10 seems to be going for a new record here, with optimized
> builds regressing by around 5-15%, and that's already after mitigating
the
> largest regression:
>
https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c
>
> It does not seem like these regressions are specific to Rust either, see
> for example https://bugs.llvm.org/show_bug.cgi?id=44408.
>
> The Rust project tracks compile-time results for each commit (see link
> above), and thanks to that can quickly spot and revert changes that
> negatively affect performance. I think it would be really great if
> something similar would exist for LLVM...
>
> Regards,
> Nikita
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200208/49dbe09b/attachment.html>

Hi,
> On 9 Feb 2020, at 03:28, Mehdi AMINI via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi,
> 
> 
> On Sat, Feb 8, 2020 at 5:17 AM Nikita Popov via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi,
> 
> Does the LLVM project perform any kind of tracking for commit-by-commit
compile-time changes? It looks like LNT only tracks run-time performance (and to
be honest I wasn't able to make heads or tails of the results even for that
-- the interface was pretty unintuitive to me.)
> 
> At some point Apple was feeding compile-time data to LNT, but it seems like
the jobs aren't active anymore:
http://green.lab.llvm.org/green/view/Compile%20Time/
<http://green.lab.llvm.org/green/view/Compile%20Time/>
Yes, it looks like the matrix view is broken. One of the relevant jobs is
http://104.154.54.203/db_default/v4/nts/machine/1353
<http://104.154.54.203/db_default/v4/nts/machine/1353> which has a fairly
recent submission http://104.154.54.203/db_default/v4/nts/131539
<http://104.154.54.203/db_default/v4/nts/131539>

But it looks like we need to adjust the ‘Run order’. It is currently using git
commit hashes, which messes up the ordering of submissions. And submissions by
the bot seem to be very irregular, due to various failures.


> While it is "normal" that each new LLVM release regresses
compile-time by 2-3%, LLVM 10 seems to be going for a new record here, with
optimized builds regressing by around 5-15%, and that's already after
mitigating the largest regression:
https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c
<https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c>
> 
> It does not seem like these regressions are specific to Rust either, see
for example https://bugs.llvm.org/show_bug.cgi?id=44408
<https://bugs.llvm.org/show_bug.cgi?id=44408>.
> 
> The Rust project tracks compile-time results for each commit (see link
above), and thanks to that can quickly spot and revert changes that negatively
affect performance. I think it would be really great if something similar would
exist for LLVM...
> 
The Rust tracking looks awesome! It’s great that we get feedback from there,
thanks for keeping an eye on that.

In theory, the bots on green dragon should submit CTMark numbers regularly for
CTMark on a few configurations (although not for every commit), but as mentioned
earlier there are a few issues that crept in. I’ll try to reach out to the right
people to have a look.

Cheers,
Florian
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200213/50dd696a/attachment-0001.html>

On Sat, Feb 8, 2020 at 2:17 PM Nikita Popov <nikita.ppv at gmail.com>
wrote:
> Hi,
>
> Does the LLVM project perform any kind of tracking for commit-by-commit
> compile-time changes? It looks like LNT only tracks run-time performance
> (and to be honest I wasn't able to make heads or tails of the results
even
> for that -- the interface was pretty unintuitive to me.)
>
> While it is "normal" that each new LLVM release regresses
compile-time by
> 2-3%, LLVM 10 seems to be going for a new record here, with optimized
> builds regressing by around 5-15%, and that's already after mitigating
the
> largest regression:
>
https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c
>
> It does not seem like these regressions are specific to Rust either, see
> for example https://bugs.llvm.org/show_bug.cgi?id=44408.
>
> The Rust project tracks compile-time results for each commit (see link
> above), and thanks to that can quickly spot and revert changes that
> negatively affect performance. I think it would be really great if
> something similar would exist for LLVM...
>
FTR I ended up writing a service for this myself. Pretty graphs available
at: http://llvm-compile-time-tracker.com/graphs.php.

We're on a downwards trend right now, let's keep it that way :)

Regards,
Nikita
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200323/24bb48cd/attachment.html>