Stefanos Baziotis via llvm-dev
2021-May-24 21:12 UTC
[llvm-dev] Statistics for Effectiveness of Passes on Reference Workloads
Hi Johannes, Yes, I CC'd Hideto on the opening message, I should've CC'd all the contributors. If the full data is available, that would be a great start! And thanks for the paper! I didn't know about it. Not to say that we should not build some functionality upstream> to do this regularly :)I have a (bad) feeling that I'll need to do a lot of drudgework to get this data, so, I might as well help to set something up if other people are interested. The closest thing I know is a compile-time-measurement website (?) Nikita Popov (CC'd) had set up. Best, Stefanos Στις Τρί, 25 Μαΐ 2021 στις 12:01 π.μ., ο/η Johannes Doerfert < johannesdoerfert at gmail.com> έγραψε:> Hideto (cc'ed) did look into this last year and he has some data. > You can find a quick summary in his lighting talk last year [1] and > also read a little bit about it in our paper draft! (attached), > especially Section 2.2 and Figure 3 are interesting here. > > Not to say that we should not build some functionality upstream > to do this regularly :) > > ~ Johannes > > [1] https://www.youtube.com/watch?v=nxfew3hsMFM&t=1435s > > > On 5/24/21 3:51 PM, Stefanos Baziotis via llvm-dev wrote: > > Hi Min, > > > > Honestly, I don't think optimization remarks are good to check _whether_ > a > > pass did any work. A far better option, to the best of my knowledge, is, > as > > I mentioned, the --print-after-all family [1] > > Optimization remarks may help on the last question I posed and that is to > > check _why_ a transformation was not applied. Although, if one reaches > this > > point, they will need to use way more > > tools than remarks. > > > > And even for the Passes that use it, my impression was that not every > >> predicates are annotated with optimization remarks. > > > > Definitely not and a lot of remarks are not that descriptive either. > > > > Best, > > Stefanos > > > > [1] https://godbolt.org/z/58Ms7qW4s > > > > Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at uci.edu> > > έγραψε: > > > >> I think optimization remarks is a good framework for measuring whether a > >> Pass does any work. > >> But unfortunately it is not widely adopted by Passes outside loop > >> transformations. And even for the Passes that use it, my impression was > >> that not every predicates are annotated with optimization remarks. > >> > >> -Min > >> > >> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev < > >> llvm-dev at lists.llvm.org> wrote: > >> > >> Hi, > >> > >> Has anyone gathered statistics on reference workloads (*) for > >> (transformation) passes > >> that are enabled / run by default (e.g., -O3) but most of the > >> time _don't_ do any effective transformation? > >> Even better if we also have such statistics for passes that are _not_ > >> enabled by default > >> (e.g., loop fusion, distribution, interchange, NewGVN). > >> > >> And yet even better if people have some idea / data for the reason for > >> ineffectiveness. > >> Bad heuristics / decision-making? Are some of these transformations > >> useless most of the time? > >> Or maybe they are useful but their implementation in LLVM is not > powerful > >> enough. Or maybe > >> they incur a significant compile-time overhead. > >> > >> If not, it would also be helpful if anyone who has tried gathering > similar > >> statistics has any > >> advice on how to approach it (my rough idea is initially > >> use --print-changed / --print-after-all on these workloads for -O3 and > >> then try to > >> slide in passes that are not enabled by default; although that's harder > to > >> do it right). > >> > >> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you > >> maybe have > >> the full statistics and / or ways to reproduce them? > >> > >> Best, > >> Stefanos > >> > >> (*) SPEC, Polybench, Cryptographic libraries, Genome alignment, Image > >> Processing, Graph Processing, Web browsers, databases like sqlite etc. > >> > >> [1] https://www.youtube.com/watch?v=QvF68tOt_w8 > >> _______________________________________________ > >> LLVM Developers mailing list > >> llvm-dev at lists.llvm.org > >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >> > >> > >> > > > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210525/17547abf/attachment.html>
Hideto Ueno via llvm-dev
2021-May-25 01:45 UTC
[llvm-dev] Statistics for Effectiveness of Passes on Reference Workloads
Hi all,> Yes, I CC'd Hideto on the opening message, I should've CC'd all thecontributors. If the full data is available, that would be a great start! And thanks for the paper! I didn't know about it. What I did in the last year is to dump results to temporary files at PasManager.h[0], and then look over all results. I don't think it is a good way to accumulate the results so it is necessary to consider another way if we want the functionality in the trunk. [0] https://github.com/llvm/llvm-project/blob/d881319cc5606baa7668405a296d0960a83a1e4c/llvm/include/llvm/IR/PassManager.h#L509 On Tue, May 25, 2021 at 6:12 AM Stefanos Baziotis < stefanos.baziotis at gmail.com> wrote:> Hi Johannes, > > Yes, I CC'd Hideto on the opening message, I should've CC'd all the > contributors. If the full data is available, that would be a great start! > And thanks for the paper! I didn't know about it. > > Not to say that we should not build some functionality upstream >> to do this regularly :) > > > I have a (bad) feeling that I'll need to do a lot of drudgework to get > this data, so, I might as well help to set something up if other people are > interested. > The closest thing I know is a compile-time-measurement website (?) Nikita > Popov (CC'd) had set up. > > Best, > Stefanos > > Στις Τρί, 25 Μαΐ 2021 στις 12:01 π.μ., ο/η Johannes Doerfert < > johannesdoerfert at gmail.com> έγραψε: > >> Hideto (cc'ed) did look into this last year and he has some data. >> You can find a quick summary in his lighting talk last year [1] and >> also read a little bit about it in our paper draft! (attached), >> especially Section 2.2 and Figure 3 are interesting here. >> >> Not to say that we should not build some functionality upstream >> to do this regularly :) >> >> ~ Johannes >> >> [1] https://www.youtube.com/watch?v=nxfew3hsMFM&t=1435s >> >> >> On 5/24/21 3:51 PM, Stefanos Baziotis via llvm-dev wrote: >> > Hi Min, >> > >> > Honestly, I don't think optimization remarks are good to check >> _whether_ a >> > pass did any work. A far better option, to the best of my knowledge, >> is, as >> > I mentioned, the --print-after-all family [1] >> > Optimization remarks may help on the last question I posed and that is >> to >> > check _why_ a transformation was not applied. Although, if one reaches >> this >> > point, they will need to use way more >> > tools than remarks. >> > >> > And even for the Passes that use it, my impression was that not every >> >> predicates are annotated with optimization remarks. >> > >> > Definitely not and a lot of remarks are not that descriptive either. >> > >> > Best, >> > Stefanos >> > >> > [1] https://godbolt.org/z/58Ms7qW4s >> > >> > Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at uci.edu >> > >> > έγραψε: >> > >> >> I think optimization remarks is a good framework for measuring whether >> a >> >> Pass does any work. >> >> But unfortunately it is not widely adopted by Passes outside loop >> >> transformations. And even for the Passes that use it, my impression was >> >> that not every predicates are annotated with optimization remarks. >> >> >> >> -Min >> >> >> >> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev < >> >> llvm-dev at lists.llvm.org> wrote: >> >> >> >> Hi, >> >> >> >> Has anyone gathered statistics on reference workloads (*) for >> >> (transformation) passes >> >> that are enabled / run by default (e.g., -O3) but most of the >> >> time _don't_ do any effective transformation? >> >> Even better if we also have such statistics for passes that are _not_ >> >> enabled by default >> >> (e.g., loop fusion, distribution, interchange, NewGVN). >> >> >> >> And yet even better if people have some idea / data for the reason for >> >> ineffectiveness. >> >> Bad heuristics / decision-making? Are some of these transformations >> >> useless most of the time? >> >> Or maybe they are useful but their implementation in LLVM is not >> powerful >> >> enough. Or maybe >> >> they incur a significant compile-time overhead. >> >> >> >> If not, it would also be helpful if anyone who has tried gathering >> similar >> >> statistics has any >> >> advice on how to approach it (my rough idea is initially >> >> use --print-changed / --print-after-all on these workloads for -O3 and >> >> then try to >> >> slide in passes that are not enabled by default; although that's >> harder to >> >> do it right). >> >> >> >> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you >> >> maybe have >> >> the full statistics and / or ways to reproduce them? >> >> >> >> Best, >> >> Stefanos >> >> >> >> (*) SPEC, Polybench, Cryptographic libraries, Genome alignment, Image >> >> Processing, Graph Processing, Web browsers, databases like sqlite etc. >> >> >> >> [1] https://www.youtube.com/watch?v=QvF68tOt_w8 >> >> _______________________________________________ >> >> LLVM Developers mailing list >> >> llvm-dev at lists.llvm.org >> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >> >> >> >> >> > >> > _______________________________________________ >> > LLVM Developers mailing list >> > llvm-dev at lists.llvm.org >> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210525/6b584770/attachment.html>