llvm dev - Jan 2021 - bugpoint and the new pass manager

Forked the thread and changed the subject to talk about bugpoint specifically.
CCed folks on https://bugs.llvm.org/show_bug.cgi?id=48813

On 2021-01-26, Arthur Eubanks via llvm-dev wrote:
>Hi all,
>
>We've been fixing the various remaining issues in order to turn on the
new
>pass manager for the optimization pipeline, and it's about time to turn
it
>on. (Thanks to everyone who has helped with testing and fixing the new pass
>manager!)
>
>https://reviews.llvm.org/D95380 is the change that would happen, which sets
>the CMake flag -DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=ON by default. This
>affects anything that uses the LLVM_ENABLE_NEW_PASS_MANAGER macro, which
>includes opt's handling of the `opt -instcombine` syntax, clang, and
>ThinLTO in lld drivers. This does not affect the backend target-specific
>codegen pipeline since that's mostly not been ported to use the new PM
>infrastructure yet.
>
>Here <https://bugs.llvm.org/show_bug.cgi?id=46649> is the umbrella bug
for
>turning on the new PM with blockers. The main one is loop unswitching on
>divergent loop conditions is unsafe
><https://bugs.llvm.org/show_bug.cgi?id=48819>, which is being looked
into.
>There's also the LLVM C API and bugpoint that still use the legacy PM,
>which can be ported later on and only block the removal of the legacy PM.
I have heard several users said they just use

   bugpoint -compile-custom -compile-command=./run a.ll
   (this usage is similar to llvm-reduce)

but not any other command listed on https://llvm.org/docs/Bugpoint.html .
(I am among them. 
https://llvm.org/docs/Bugpoint.html and
https://llvm.org/docs/CommandGuide/bugpoint.html
have no usage examples so I resorted to an external article
http://logan.tw/posts/2014/11/26/llvm-bugpoint/ )

The few things tied to the legacy pass manager are these "other
commands".
I hope the active users can

* help improve the documentation (several developers found bugpoint unintuitive
  to use and don't use features other than -compile-custom)
* Port legacy PM features to the new pass manager.
* If people think that new reducing strategies can be ported to llvm-reduce, and
keep
   bugpoint as the few misc features that'll also look good:)
>The C API can be worked through (we may need to introduce replacements to
>the legacy pass manager APIs), but bugpoint will be tricky since it has so
>many legacy PM-specific hacks and we may need to trim it down if we want it
>to work with the new PM. Anyway, I don't think any of the remaining
>blockers are large enough to block the switch (but comments welcome).
>
>I'd like to turn on the new PM by default soonish, after the 12.x
branch.
>Perhaps roughly a week from now barring any major newly discovered
>regressions?
>
>As for potential issues only uncovered after the switch, if there is a
>large issue I will roll it back, but for smaller issues I'd rather ask
>users to pin to the legacy PM while we fix the issues, either via the CMake
>flag -DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=OFF, or the corresponding
>compiler flags, like -flegacy-pass-manager for clang.
>
>Any concerns/comments?
>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> On Jan 26, 2021, at 22:32, Fangrui Song <maskray at google.com>
wrote:
> 
> Forked the thread and changed the subject to talk about bugpoint
specifically.
> CCed folks on https://bugs.llvm.org/show_bug.cgi?id=48813
Thanks!
> On 2021-01-26, Arthur Eubanks via llvm-dev wrote:
>> Hi all,
>> 
>> We've been fixing the various remaining issues in order to turn on
the new
>> pass manager for the optimization pipeline, and it's about time to
turn it
>> on. (Thanks to everyone who has helped with testing and fixing the new
pass
>> manager!)
>> 
>> https://reviews.llvm.org/D95380 is the change that would happen, which
sets
>> the CMake flag -DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=ON by default.
This
>> affects anything that uses the LLVM_ENABLE_NEW_PASS_MANAGER macro,
which
>> includes opt's handling of the `opt -instcombine` syntax, clang,
and
>> ThinLTO in lld drivers. This does not affect the backend
target-specific
>> codegen pipeline since that's mostly not been ported to use the new
PM
>> infrastructure yet.
>> 
>> Here <https://bugs.llvm.org/show_bug.cgi?id=46649> is the
umbrella bug for
>> turning on the new PM with blockers. The main one is loop unswitching
on
>> divergent loop conditions is unsafe
>> <https://bugs.llvm.org/show_bug.cgi?id=48819>, which is being
looked into.
>> There's also the LLVM C API and bugpoint that still use the legacy
PM,
>> which can be ported later on and only block the removal of the legacy
PM.
> 
> I have heard several users said they just use
> 
>  bugpoint -compile-custom -compile-command=./run a.ll
>  (this usage is similar to llvm-reduce)
> 
> but not any other command listed on https://llvm.org/docs/Bugpoint.html .
> (I am among them. https://llvm.org/docs/Bugpoint.html and
> https://llvm.org/docs/CommandGuide/bugpoint.html
> have no usage examples so I resorted to an external article
> http://logan.tw/posts/2014/11/26/llvm-bugpoint/ )
FWIW I often use bugpoint’s crashing pass reduction feature ( e.g. `bugpoint -O3
foo.bc` to reduce the sequence of passes exposing a crash) when investigating
crashes in LLVM optimizations. This is unfortunately one of the features
directly tied to the pass manager.

My main concern is that `opt` and `bugpoint` having different defaults seems to
have potential to be quite confusing to user  (e.g. `opt
-passes=default<O3>` crashes, but `bug point -passes=default<O3>`
does not, `bug point -O3 ` works, but does not reproduce the failure, e.g. due
to difference in the pipeline).

I don’t know how widely used bugpoint is, but it features prominently in the
documentation, e.g. https://llvm.org/docs/HowToSubmitABug.html , which is linked
from the getting started page and the documentation sidebar.

I think before switching, it would be good to at least update HowToSubmitABug.
We should probably also add a note to Bugpoint’s docs and refer the the
documentation for llvm-reduce. It is great if people take time to reduce the
test cases for their bug reports and we should try to make the experience as
smooth as possible

Cheers,
Florian