llvm dev - May 2021 - Statistics for Effectiveness of Passes on Reference Workloads

Hi,

Has anyone gathered statistics on reference workloads (*) for
(transformation) passes
that are enabled / run by default (e.g., -O3) but most of the
time _don't_ do any effective transformation?
Even better if we also have such statistics for passes that are _not_
enabled by default
(e.g., loop fusion, distribution, interchange, NewGVN).

And yet even better if people have some idea / data for the reason for
ineffectiveness.
Bad heuristics / decision-making? Are some of these transformations useless
most of the time?
Or maybe they are useful but their implementation in LLVM is not powerful
enough. Or maybe
they incur a significant compile-time overhead.

If not, it would also be helpful if anyone who has tried gathering similar
statistics has any
advice on how to approach it (my rough idea is initially
use --print-changed / --print-after-all on these workloads for -O3 and then
try to
slide in passes that are not enabled by default; although that's harder to
do it right).

@Hideto: In the last LLVM meeting you gave a related talk [1]. Do you maybe
have
the full statistics and / or ways to reproduce them?

Best,
Stefanos

(*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
Processing, Graph Processing, Web browsers, databases like sqlite etc.

[1] https://www.youtube.com/watch?v=QvF68tOt_w8
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210524/e70b6e60/attachment.html>

I think optimization remarks is a good framework for measuring whether a Pass
does any work.
But unfortunately it is not widely adopted by Passes outside loop
transformations. And even for the Passes that use it, my impression was that not
every predicates are annotated with optimization remarks.

-Min
> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
> 
> Hi,
> 
> Has anyone gathered statistics on reference workloads (*) for
(transformation) passes
> that are enabled / run by default (e.g., -O3) but most of the
> time _don't_ do any effective transformation?
> Even better if we also have such statistics for passes that are _not_
enabled by default
> (e.g., loop fusion, distribution, interchange, NewGVN).
> 
> And yet even better if people have some idea / data for the reason for
ineffectiveness.
> Bad heuristics / decision-making? Are some of these transformations useless
most of the time?
> Or maybe they are useful but their implementation in LLVM is not powerful
enough. Or maybe
> they incur a significant compile-time overhead.
> 
> If not, it would also be helpful if anyone who has tried gathering similar
statistics has any
> advice on how to approach it (my rough idea is initially
> use --print-changed / --print-after-all on these workloads for -O3 and then
try to
> slide in passes that are not enabled by default; although that's harder
to do it right).
> 
> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you maybe
have
> the full statistics and / or ways to reproduce them?
> 
> Best,
> Stefanos
> 
> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
Processing, Graph Processing, Web browsers, databases like sqlite etc.
> 
> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
<https://www.youtube.com/watch?v=QvF68tOt_w8>_______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210524/d14c6f2c/attachment.html>