llvm dev - May 2021 - Statistics for Effectiveness of Passes on Reference Workloads

Hi Min,

Honestly, I don't think optimization remarks are good to check _whether_ a
pass did any work. A far better option, to the best of my knowledge, is, as
I mentioned, the --print-after-all family [1]
Optimization remarks may help on the last question I posed and that is to
check _why_ a transformation was not applied. Although, if one reaches this
point, they will need to use way more
tools than remarks.

 And even for the Passes that use it, my impression was that not
every
> predicates are annotated with optimization remarks.

Definitely not and a lot of remarks are not that descriptive either.

Best,
Stefanos

[1] https://godbolt.org/z/58Ms7qW4s

Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at
uci.edu>
έγραψε:
> I think optimization remarks is a good framework for measuring whether a
> Pass does any work.
> But unfortunately it is not widely adopted by Passes outside loop
> transformations. And even for the Passes that use it, my impression was
> that not every predicates are annotated with optimization remarks.
>
> -Min
>
> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi,
>
> Has anyone gathered statistics on reference workloads (*) for
> (transformation) passes
> that are enabled / run by default (e.g., -O3) but most of the
> time _don't_ do any effective transformation?
> Even better if we also have such statistics for passes that are _not_
> enabled by default
> (e.g., loop fusion, distribution, interchange, NewGVN).
>
> And yet even better if people have some idea / data for the reason for
> ineffectiveness.
> Bad heuristics / decision-making? Are some of these transformations
> useless most of the time?
> Or maybe they are useful but their implementation in LLVM is not powerful
> enough. Or maybe
> they incur a significant compile-time overhead.
>
> If not, it would also be helpful if anyone who has tried gathering similar
> statistics has any
> advice on how to approach it (my rough idea is initially
> use --print-changed / --print-after-all on these workloads for -O3 and
> then try to
> slide in passes that are not enabled by default; although that's harder
to
> do it right).
>
> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you
> maybe have
> the full statistics and / or ways to reproduce them?
>
> Best,
> Stefanos
>
> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
> Processing, Graph Processing, Web browsers, databases like sqlite etc.
>
> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210524/19e30dc1/attachment.html>

For the new pass manager, `opt --print-changed(=quiet)` is very nice for
filtering out passes that didn't do anything. You could add something to
StandardInstrumentations to do something besides just printing the IR out
when it changes.

On Mon, May 24, 2021 at 1:52 PM Stefanos Baziotis via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi Min,
>
> Honestly, I don't think optimization remarks are good to check
_whether_ a
> pass did any work. A far better option, to the best of my knowledge, is, as
> I mentioned, the --print-after-all family [1]
> Optimization remarks may help on the last question I posed and that is to
> check _why_ a transformation was not applied. Although, if one reaches this
> point, they will need to use way more
> tools than remarks.
>
>  And even for the Passes that use it, my impression was that not every
>> predicates are annotated with optimization remarks.
>
>
> Definitely not and a lot of remarks are not that descriptive either.
>
> Best,
> Stefanos
>
> [1] https://godbolt.org/z/58Ms7qW4s
>
> Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at
uci.edu>
> έγραψε:
>
>> I think optimization remarks is a good framework for measuring whether
a
>> Pass does any work.
>> But unfortunately it is not widely adopted by Passes outside loop
>> transformations. And even for the Passes that use it, my impression was
>> that not every predicates are annotated with optimization remarks.
>>
>> -Min
>>
>> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> Hi,
>>
>> Has anyone gathered statistics on reference workloads (*) for
>> (transformation) passes
>> that are enabled / run by default (e.g., -O3) but most of the
>> time _don't_ do any effective transformation?
>> Even better if we also have such statistics for passes that are _not_
>> enabled by default
>> (e.g., loop fusion, distribution, interchange, NewGVN).
>>
>> And yet even better if people have some idea / data for the reason for
>> ineffectiveness.
>> Bad heuristics / decision-making? Are some of these transformations
>> useless most of the time?
>> Or maybe they are useful but their implementation in LLVM is not
powerful
>> enough. Or maybe
>> they incur a significant compile-time overhead.
>>
>> If not, it would also be helpful if anyone who has tried gathering
>> similar statistics has any
>> advice on how to approach it (my rough idea is initially
>> use --print-changed / --print-after-all on these workloads for -O3 and
>> then try to
>> slide in passes that are not enabled by default; although that's
harder
>> to do it right).
>>
>> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you
>> maybe have
>> the full statistics and / or ways to reproduce them?
>>
>> Best,
>> Stefanos
>>
>> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
>> Processing, Graph Processing, Web browsers, databases like sqlite etc.
>>
>> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210524/17680e4a/attachment.html>

Hideto (cc'ed) did look into this last year and he has some data.
You can find a quick summary in his lighting talk last year [1] and
also read a little bit about it in our paper draft! (attached),
especially Section 2.2 and Figure 3 are interesting here.

Not to say that we should not build some functionality upstream
to do this regularly :)

~ Johannes

[1] https://www.youtube.com/watch?v=nxfew3hsMFM&t=1435s


On 5/24/21 3:51 PM, Stefanos Baziotis via llvm-dev
wrote:
> Hi Min,
>
> Honestly, I don't think optimization remarks are good to check
_whether_ a
> pass did any work. A far better option, to the best of my knowledge, is, as
> I mentioned, the --print-after-all family [1]
> Optimization remarks may help on the last question I posed and that is to
> check _why_ a transformation was not applied. Although, if one reaches this
> point, they will need to use way more
> tools than remarks.
>
>   And even for the Passes that use it, my impression was that not every
>> predicates are annotated with optimization remarks.
>
> Definitely not and a lot of remarks are not that descriptive either.
>
> Best,
> Stefanos
>
> [1] https://godbolt.org/z/58Ms7qW4s
>
> Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at
uci.edu>
> έγραψε:
>
>> I think optimization remarks is a good framework for measuring whether
a
>> Pass does any work.
>> But unfortunately it is not widely adopted by Passes outside loop
>> transformations. And even for the Passes that use it, my impression was
>> that not every predicates are annotated with optimization remarks.
>>
>> -Min
>>
>> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> Hi,
>>
>> Has anyone gathered statistics on reference workloads (*) for
>> (transformation) passes
>> that are enabled / run by default (e.g., -O3) but most of the
>> time _don't_ do any effective transformation?
>> Even better if we also have such statistics for passes that are _not_
>> enabled by default
>> (e.g., loop fusion, distribution, interchange, NewGVN).
>>
>> And yet even better if people have some idea / data for the reason for
>> ineffectiveness.
>> Bad heuristics / decision-making? Are some of these transformations
>> useless most of the time?
>> Or maybe they are useful but their implementation in LLVM is not
powerful
>> enough. Or maybe
>> they incur a significant compile-time overhead.
>>
>> If not, it would also be helpful if anyone who has tried gathering
similar
>> statistics has any
>> advice on how to approach it (my rough idea is initially
>> use --print-changed / --print-after-all on these workloads for -O3 and
>> then try to
>> slide in passes that are not enabled by default; although that's
harder to
>> do it right).
>>
>> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you
>> maybe have
>> the full statistics and / or ways to reproduce them?
>>
>> Best,
>> Stefanos
>>
>> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
>> Processing, Graph Processing, Web browsers, databases like sqlite etc.
>>
>> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name:
Towards_Compile_Time_Reducing_Compiler_Optimization_Selection_via_Machine_Learning.pdf
Type: application/pdf
Size: 1947927 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210524/c9a4b7ec/attachment-0001.pdf>

Hi Johannes,

Yes, I CC'd Hideto on the opening message, I should've CC'd all the
contributors. If the full data is available, that would be a great start!
And thanks for the paper! I didn't know about it.

Not to say that we should not build some functionality
upstream
> to do this regularly :)

I have a (bad) feeling that I'll need to do a lot of drudgework to get this
data, so, I might as well help to set something up if other people are
interested.
The closest thing I know is a compile-time-measurement website (?) Nikita
Popov (CC'd) had set up.

Best,
Stefanos

Στις Τρί, 25 Μαΐ 2021 στις 12:01 π.μ., ο/η Johannes Doerfert <
johannesdoerfert at gmail.com> έγραψε:
> Hideto (cc'ed) did look into this last year and he has some data.
> You can find a quick summary in his lighting talk last year [1] and
> also read a little bit about it in our paper draft! (attached),
> especially Section 2.2 and Figure 3 are interesting here.
>
> Not to say that we should not build some functionality upstream
> to do this regularly :)
>
> ~ Johannes
>
> [1] https://www.youtube.com/watch?v=nxfew3hsMFM&t=1435s
>
>
> On 5/24/21 3:51 PM, Stefanos Baziotis via llvm-dev wrote:
> > Hi Min,
> >
> > Honestly, I don't think optimization remarks are good to check
_whether_
> a
> > pass did any work. A far better option, to the best of my knowledge,
is,
> as
> > I mentioned, the --print-after-all family [1]
> > Optimization remarks may help on the last question I posed and that is
to
> > check _why_ a transformation was not applied. Although, if one reaches
> this
> > point, they will need to use way more
> > tools than remarks.
> >
> >   And even for the Passes that use it, my impression was that not
every
> >> predicates are annotated with optimization remarks.
> >
> > Definitely not and a lot of remarks are not that descriptive either.
> >
> > Best,
> > Stefanos
> >
> > [1] https://godbolt.org/z/58Ms7qW4s
> >
> > Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at
uci.edu>
> > έγραψε:
> >
> >> I think optimization remarks is a good framework for measuring
whether a
> >> Pass does any work.
> >> But unfortunately it is not widely adopted by Passes outside loop
> >> transformations. And even for the Passes that use it, my
impression was
> >> that not every predicates are annotated with optimization remarks.
> >>
> >> -Min
> >>
> >> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
> >> llvm-dev at lists.llvm.org> wrote:
> >>
> >> Hi,
> >>
> >> Has anyone gathered statistics on reference workloads (*) for
> >> (transformation) passes
> >> that are enabled / run by default (e.g., -O3) but most of the
> >> time _don't_ do any effective transformation?
> >> Even better if we also have such statistics for passes that are
_not_
> >> enabled by default
> >> (e.g., loop fusion, distribution, interchange, NewGVN).
> >>
> >> And yet even better if people have some idea / data for the reason
for
> >> ineffectiveness.
> >> Bad heuristics / decision-making? Are some of these
transformations
> >> useless most of the time?
> >> Or maybe they are useful but their implementation in LLVM is not
> powerful
> >> enough. Or maybe
> >> they incur a significant compile-time overhead.
> >>
> >> If not, it would also be helpful if anyone who has tried gathering
> similar
> >> statistics has any
> >> advice on how to approach it (my rough idea is initially
> >> use --print-changed / --print-after-all on these workloads for -O3
and
> >> then try to
> >> slide in passes that are not enabled by default; although
that's harder
> to
> >> do it right).
> >>
> >> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do
you
> >> maybe have
> >> the full statistics and / or ways to reproduce them?
> >>
> >> Best,
> >> Stefanos
> >>
> >> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment,
Image
> >> Processing, Graph Processing, Web browsers, databases like sqlite
etc.
> >>
> >> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>
> >>
> >>
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210525/17547abf/attachment.html>