llvm dev - Jan 2021 - bugpoint and the new pass manager

> On Jan 26, 2021, at 22:32, Fangrui Song <maskray at google.com>
wrote:
> 
> Forked the thread and changed the subject to talk about bugpoint
specifically.
> CCed folks on https://bugs.llvm.org/show_bug.cgi?id=48813
Thanks!
> On 2021-01-26, Arthur Eubanks via llvm-dev wrote:
>> Hi all,
>> 
>> We've been fixing the various remaining issues in order to turn on
the new
>> pass manager for the optimization pipeline, and it's about time to
turn it
>> on. (Thanks to everyone who has helped with testing and fixing the new
pass
>> manager!)
>> 
>> https://reviews.llvm.org/D95380 is the change that would happen, which
sets
>> the CMake flag -DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=ON by default.
This
>> affects anything that uses the LLVM_ENABLE_NEW_PASS_MANAGER macro,
which
>> includes opt's handling of the `opt -instcombine` syntax, clang,
and
>> ThinLTO in lld drivers. This does not affect the backend
target-specific
>> codegen pipeline since that's mostly not been ported to use the new
PM
>> infrastructure yet.
>> 
>> Here <https://bugs.llvm.org/show_bug.cgi?id=46649> is the
umbrella bug for
>> turning on the new PM with blockers. The main one is loop unswitching
on
>> divergent loop conditions is unsafe
>> <https://bugs.llvm.org/show_bug.cgi?id=48819>, which is being
looked into.
>> There's also the LLVM C API and bugpoint that still use the legacy
PM,
>> which can be ported later on and only block the removal of the legacy
PM.
> 
> I have heard several users said they just use
> 
>  bugpoint -compile-custom -compile-command=./run a.ll
>  (this usage is similar to llvm-reduce)
> 
> but not any other command listed on https://llvm.org/docs/Bugpoint.html .
> (I am among them. https://llvm.org/docs/Bugpoint.html and
> https://llvm.org/docs/CommandGuide/bugpoint.html
> have no usage examples so I resorted to an external article
> http://logan.tw/posts/2014/11/26/llvm-bugpoint/ )
FWIW I often use bugpoint’s crashing pass reduction feature ( e.g. `bugpoint -O3
foo.bc` to reduce the sequence of passes exposing a crash) when investigating
crashes in LLVM optimizations. This is unfortunately one of the features
directly tied to the pass manager.

My main concern is that `opt` and `bugpoint` having different defaults seems to
have potential to be quite confusing to user  (e.g. `opt
-passes=default<O3>` crashes, but `bug point -passes=default<O3>`
does not, `bug point -O3 ` works, but does not reproduce the failure, e.g. due
to difference in the pipeline).

I don’t know how widely used bugpoint is, but it features prominently in the
documentation, e.g. https://llvm.org/docs/HowToSubmitABug.html , which is linked
from the getting started page and the documentation sidebar.

I think before switching, it would be good to at least update HowToSubmitABug.
We should probably also add a note to Bugpoint’s docs and refer the the
documentation for llvm-reduce. It is great if people take time to reduce the
test cases for their bug reports and we should try to make the experience as
smooth as possible

Cheers,
Florian

Florian Hahn via llvm-dev <llvm-dev at lists.llvm.org> writes:
> FWIW I often use bugpoint’s crashing pass reduction feature (
> e.g. `bugpoint -O3 foo.bc` to reduce the sequence of passes exposing a
> crash) when investigating crashes in LLVM optimizations. This is
> unfortunately one of the features directly tied to the pass manager.
Ditto.
> My main concern is that `opt` and `bugpoint` having different defaults
> seems to have potential to be quite confusing to user (e.g. `opt
> -passes=default<O3>` crashes, but `bug point
-passes=default<O3>` does
> not, `bug point -O3 ` works, but does not reproduce the failure,
> e.g. due to difference in the pipeline).
Agreed.
> I don’t know how widely used bugpoint is, but it features prominently
> in the documentation, e.g. https://llvm.org/docs/HowToSubmitABug.html
> , which is linked from the getting started page and the documentation
> sidebar.
I use bugpoint a ton.  That said, mostly I've used it for testcase
reduction and have done my own manual optimization pipeline reduction
mostly because codegen optimizations aren't as nicely layered and the
codegen pipeline can't be driven from bugpoint.

Getting codegen passes into the new pass manager has been raised
multiple times because and that would be a good thing to happen but it
doesn't begin to tackle the problem of automatic bug classification in
codegen.
> I think before switching, it would be good to at least update
> HowToSubmitABug. We should probably also add a note to Bugpoint’s docs
> and refer the the documentation for llvm-reduce. It is great if people
> take time to reduce the test cases for their bug reports and we should
> try to make the experience as smooth as possible
Yeah, we need a way to keep bug reports sane.  If llvm-reduce can do as
well or better than bugpoint, great, I'm all for it!

               -David

On 1/27/21 1:58 AM, Florian Hahn via llvm-dev wrote:
>
>> On Jan 26, 2021, at 22:32, Fangrui Song <maskray at google.com>
wrote:
>>
>> Forked the thread and changed the subject to talk about bugpoint
specifically.
>> CCed folks on https://bugs.llvm.org/show_bug.cgi?id=48813
> Thanks!
>
>> On 2021-01-26, Arthur Eubanks via llvm-dev wrote:
>>> Hi all,
>>>
>>> We've been fixing the various remaining issues in order to turn
on the new
>>> pass manager for the optimization pipeline, and it's about time
to turn it
>>> on. (Thanks to everyone who has helped with testing and fixing the
new pass
>>> manager!)
>>>
>>> https://reviews.llvm.org/D95380 is the change that would happen,
which sets
>>> the CMake flag -DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=ON by
default. This
>>> affects anything that uses the LLVM_ENABLE_NEW_PASS_MANAGER macro,
which
>>> includes opt's handling of the `opt -instcombine` syntax,
clang, and
>>> ThinLTO in lld drivers. This does not affect the backend
target-specific
>>> codegen pipeline since that's mostly not been ported to use the
new PM
>>> infrastructure yet.
>>>
>>> Here <https://bugs.llvm.org/show_bug.cgi?id=46649> is the
umbrella bug for
>>> turning on the new PM with blockers. The main one is loop
unswitching on
>>> divergent loop conditions is unsafe
>>> <https://bugs.llvm.org/show_bug.cgi?id=48819>, which is being
looked into.
>>> There's also the LLVM C API and bugpoint that still use the
legacy PM,
>>> which can be ported later on and only block the removal of the
legacy PM.
>> I have heard several users said they just use
>>
>>   bugpoint -compile-custom -compile-command=./run a.ll
>>   (this usage is similar to llvm-reduce)
>>
>> but not any other command listed on https://llvm.org/docs/Bugpoint.html
.
>> (I am among them. https://llvm.org/docs/Bugpoint.html and
>> https://llvm.org/docs/CommandGuide/bugpoint.html
>> have no usage examples so I resorted to an external article
>> http://logan.tw/posts/2014/11/26/llvm-bugpoint/ )
> FWIW I often use bugpoint’s crashing pass reduction feature ( e.g.
`bugpoint -O3 foo.bc` to reduce the sequence of passes exposing a crash) when
investigating crashes in LLVM optimizations. This is unfortunately one of the
features directly tied to the pass manager.
Same.
> My main concern is that `opt` and `bugpoint` having different defaults
seems to have potential to be quite confusing to user  (e.g. `opt
-passes=default<O3>` crashes, but `bug point -passes=default<O3>`
does not, `bug point -O3 ` works, but does not reproduce the failure, e.g. due
to difference in the pipeline).
>
> I don’t know how widely used bugpoint is, but it features prominently in
the documentation, e.g. https://llvm.org/docs/HowToSubmitABug.html , which is
linked from the getting started page and the documentation sidebar.
>
> I think before switching, it would be good to at least update
HowToSubmitABug. We should probably also add a note to Bugpoint’s docs and refer
the the documentation for llvm-reduce. It is great if people take time to reduce
the test cases for their bug reports and we should try to make the experience as
smooth as possible
Florian raises a great point here.  I'd be comfortable simply 
documenting the issue for the moment, but getting bugpoint properly 
ported to the new PM close to the point we switch seems very worthwhile 
for the reason Florian nicely describes.