llvm dev - May 2021 - Statistics for Effectiveness of Passes on Reference Workloads

I think optimization remarks is a good framework for measuring whether a Pass
does any work.
But unfortunately it is not widely adopted by Passes outside loop
transformations. And even for the Passes that use it, my impression was that not
every predicates are annotated with optimization remarks.

-Min
> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
> 
> Hi,
> 
> Has anyone gathered statistics on reference workloads (*) for
(transformation) passes
> that are enabled / run by default (e.g., -O3) but most of the
> time _don't_ do any effective transformation?
> Even better if we also have such statistics for passes that are _not_
enabled by default
> (e.g., loop fusion, distribution, interchange, NewGVN).
> 
> And yet even better if people have some idea / data for the reason for
ineffectiveness.
> Bad heuristics / decision-making? Are some of these transformations useless
most of the time?
> Or maybe they are useful but their implementation in LLVM is not powerful
enough. Or maybe
> they incur a significant compile-time overhead.
> 
> If not, it would also be helpful if anyone who has tried gathering similar
statistics has any
> advice on how to approach it (my rough idea is initially
> use --print-changed / --print-after-all on these workloads for -O3 and then
try to
> slide in passes that are not enabled by default; although that's harder
to do it right).
> 
> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you maybe
have
> the full statistics and / or ways to reproduce them?
> 
> Best,
> Stefanos
> 
> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
Processing, Graph Processing, Web browsers, databases like sqlite etc.
> 
> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
<https://www.youtube.com/watch?v=QvF68tOt_w8>_______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210524/d14c6f2c/attachment.html>

Hi Min,

Honestly, I don't think optimization remarks are good to check _whether_ a
pass did any work. A far better option, to the best of my knowledge, is, as
I mentioned, the --print-after-all family [1]
Optimization remarks may help on the last question I posed and that is to
check _why_ a transformation was not applied. Although, if one reaches this
point, they will need to use way more
tools than remarks.

 And even for the Passes that use it, my impression was that not
every
> predicates are annotated with optimization remarks.

Definitely not and a lot of remarks are not that descriptive either.

Best,
Stefanos

[1] https://godbolt.org/z/58Ms7qW4s

Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at
uci.edu>
έγραψε:
> I think optimization remarks is a good framework for measuring whether a
> Pass does any work.
> But unfortunately it is not widely adopted by Passes outside loop
> transformations. And even for the Passes that use it, my impression was
> that not every predicates are annotated with optimization remarks.
>
> -Min
>
> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi,
>
> Has anyone gathered statistics on reference workloads (*) for
> (transformation) passes
> that are enabled / run by default (e.g., -O3) but most of the
> time _don't_ do any effective transformation?
> Even better if we also have such statistics for passes that are _not_
> enabled by default
> (e.g., loop fusion, distribution, interchange, NewGVN).
>
> And yet even better if people have some idea / data for the reason for
> ineffectiveness.
> Bad heuristics / decision-making? Are some of these transformations
> useless most of the time?
> Or maybe they are useful but their implementation in LLVM is not powerful
> enough. Or maybe
> they incur a significant compile-time overhead.
>
> If not, it would also be helpful if anyone who has tried gathering similar
> statistics has any
> advice on how to approach it (my rough idea is initially
> use --print-changed / --print-after-all on these workloads for -O3 and
> then try to
> slide in passes that are not enabled by default; although that's harder
to
> do it right).
>
> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you
> maybe have
> the full statistics and / or ways to reproduce them?
>
> Best,
> Stefanos
>
> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
> Processing, Graph Processing, Web browsers, databases like sqlite etc.
>
> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210524/19e30dc1/attachment.html>