llvm dev - Sep 2018 - OptBisect implementation for new pass manager

Greetings!

As the generic Pass Instrumentation framework for new pass manager is 
finally *in*,
I'm glad to start the discussion on implementation of -opt-bisect 
through that framework.

As it has already been discovered while porting other features (namely, 
-time-passes)
blindly copying the currently existing legacy implementation is most 
likely not a perfect
way forward. Now is a chance to take a fresh look at the overall 
approach and perhaps
do better, without the restrictions that legacy pass manager framework 
imposed on
the implementation.

Kind of a summary of what we have now:
   - There is a single OptBisect object, requested through LLVMContext
     (managed as ManagedStatic).

   - OptBisect is defined in lib/IR, but does use analyses,
     which is a known layering issue

   - Pass hierarchy provides skipModule etc helper functions

   - Individual passes opt-in to OptBisect activities by manually 
calling skip* helper functions
     whenever appropriate

With current state of new-pm PassInstrumentation potential OptBisect 
implementation
will have the following properties/issues:
   - OptBisect object that exists per compilation pipeline, managed 
similar to PassBuilder/PassManagers
     (which makes it more suitable for use in parallel compilations)

   - no more layering issues imposed by implementation since 
instrumentations by design
     can live anywhere - lib/Analysis, lib/Passes etc

   - since Codegen is still legacy-only we will have to make a joint 
implementation that
     provides a sequential passes numbering through both new-PM IR and 
legacy Codegen pipelines

   - as of right now there is no mechanism for opt-in/opt-out, so it 
needs to be designed/implemented
     Here I would like to ask:
         - what would be preferable - opt-in or opt-out?

         - with legacy implementation passes opt-in both for bisect and 
attribute-optnone support at once.
           Do we need to follow that in new-pm implementation?

Also, I would like to ask whether people see current user interface for 
opt-bisect limiting?
Do we need better controls for more sophisticated bisection?
Basically I'm looking for any ideas on improving opt-bisect user 
experience that might
affect design approaches we take on the initial implementation.

regards,
   Fedor.

Fedor Sergeev <fedor.sergeev at azul.com> writes:
>         - what would be preferable - opt-in or opt-out?
>
>         - with legacy implementation passes opt-in both for bisect and
> attribute-optnone support at once.
>           Do we need to follow that in new-pm implementation?
>
> Also, I would like to ask whether people see current user interface
> for opt-bisect limiting?
Ideally I would prefer an opt-out model.  Practiclaly speaking, I'm not
sure that's possible.  Presumably opt-bisect doesn't apply to analysis
passes.  What happens if a transformation pass depends on another
transformation pass earlier in the pipeline?  If the earlier
transformation pass is bisected out, then...???

It seems to me that optnone and opt-bisect are two different things and
therefore they should be handled orthogonally.

As for the user interface:

It might be nice to have opt-bisect skip/count capability similar to
DebugCounters.  But even that might not be enough.  Often a bug requires
some subset of transformations to run.  For example, if the pipeline is:

A->B->C->D->E

I might need B, D and E to run to see the bug (A and C don't matter).
If I understand things correctly, today opt-bisect would report that all
of A-E need to run to manifest the bug.  That's usually plenty good
enough as running and not running E may result in a small enough set of
diffs to be workable.  But it may be that running B->D->E and B->D
results in a smaller number of diffs.  Skip/count would get us to
B->C->D->E and even that might be an improvement.

I don't see this as a day-one requirement for the new opt-bisect, but
rather a longer-term goal to improve debugging precision.

How does all of this relate to bugpoint?  AFAIK bugpoint doesn't use
opt-bisect at all.  It can arrive at the B->D->E minimal pipeline.  It
seems worthwhile to put some effort into unifying capabilities and
sharing code.  bugpoint is a bit of a mess, unfortunately.  It would be
great to modernize it and make use of new internal debugging
capabilities that didn't exist when it was created.  Maybe that's a
project that could start after the new PM becomes default and the new
opt-bisect is in place.

                          -David

Hi Fedor,

can you make an example where a pass actually needs to opt-out? Because
IMO, bisect should quite literally to DebugCounter-style skip every step in
every ::run method's loop. Passes should not even be concerned with this.

Cheers,
Philip

On Wed, Sep 26, 2018 at 6:54 PM Fedor Sergeev via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Greetings!
>
> As the generic Pass Instrumentation framework for new pass manager is
> finally *in*,
> I'm glad to start the discussion on implementation of -opt-bisect
> through that framework.
>
> As it has already been discovered while porting other features (namely,
> -time-passes)
> blindly copying the currently existing legacy implementation is most
> likely not a perfect
> way forward. Now is a chance to take a fresh look at the overall
> approach and perhaps
> do better, without the restrictions that legacy pass manager framework
> imposed on
> the implementation.
>
> Kind of a summary of what we have now:
>    - There is a single OptBisect object, requested through LLVMContext
>      (managed as ManagedStatic).
>
>    - OptBisect is defined in lib/IR, but does use analyses,
>      which is a known layering issue
>
>    - Pass hierarchy provides skipModule etc helper functions
>
>    - Individual passes opt-in to OptBisect activities by manually
> calling skip* helper functions
>      whenever appropriate
>
> With current state of new-pm PassInstrumentation potential OptBisect
> implementation
> will have the following properties/issues:
>    - OptBisect object that exists per compilation pipeline, managed
> similar to PassBuilder/PassManagers
>      (which makes it more suitable for use in parallel compilations)
>
>    - no more layering issues imposed by implementation since
> instrumentations by design
>      can live anywhere - lib/Analysis, lib/Passes etc
>
>    - since Codegen is still legacy-only we will have to make a joint
> implementation that
>      provides a sequential passes numbering through both new-PM IR and
> legacy Codegen pipelines
>
>    - as of right now there is no mechanism for opt-in/opt-out, so it
> needs to be designed/implemented
>      Here I would like to ask:
>          - what would be preferable - opt-in or opt-out?
>
>          - with legacy implementation passes opt-in both for bisect and
> attribute-optnone support at once.
>            Do we need to follow that in new-pm implementation?
>
> Also, I would like to ask whether people see current user interface for
> opt-bisect limiting?
> Do we need better controls for more sophisticated bisection?
> Basically I'm looking for any ideas on improving opt-bisect user
> experience that might
> affect design approaches we take on the initial implementation.
>
> regards,
>    Fedor.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180926/7e5a4d41/attachment.html>

On 9/26/2018 10:47 AM, Philip Pfaffe via llvm-dev wrote:
> Hi Fedor,
>
> can you make an example where a pass actually needs to opt-out? 
> Because IMO, bisect should quite literally to DebugCounter-style skip 
> every step in every ::run method's loop. Passes should not even be 
> concerned with this.
This isn't so much an issue for the optimization pipeline, but code 
generation involves some passes which are mandatory (e.g. isel).

-Eli

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux
Foundation Collaborative Project

Philip,

It kinda depends on user expectations.
What do you really expect as a result of your compilation when you set 
-opt-bisect-limit=X?
Do you just get a resulting IR and scan for the bad pattern?
Then you dont care about pass sequences and do brute-force bisect.

Do you expect to get a runnable code at the end and check for buggy 
run-time behavior?
Then you need to keep the passes that are required to produce runnable code.
In our Java JIT compiler we have quite a number of passes where we lower 
the semantics
to C level and those lowerings are absolutely required for the code to 
be runnable.

regards,
   Fedor.

On 09/26/2018 08:47 PM, Philip Pfaffe wrote:
> Hi Fedor,
>
> can you make an example where a pass actually needs to opt-out? 
> Because IMO, bisect should quite literally to DebugCounter-style skip 
> every step in every ::run method's loop. Passes should not even be 
> concerned with this.
>
> Cheers,
> Philip
>
> On Wed, Sep 26, 2018 at 6:54 PM Fedor Sergeev via llvm-dev 
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
>     Greetings!
>
>     As the generic Pass Instrumentation framework for new pass manager is
>     finally *in*,
>     I'm glad to start the discussion on implementation of -opt-bisect
>     through that framework.
>
>     As it has already been discovered while porting other features
>     (namely,
>     -time-passes)
>     blindly copying the currently existing legacy implementation is most
>     likely not a perfect
>     way forward. Now is a chance to take a fresh look at the overall
>     approach and perhaps
>     do better, without the restrictions that legacy pass manager
>     framework
>     imposed on
>     the implementation.
>
>     Kind of a summary of what we have now:
>        - There is a single OptBisect object, requested through LLVMContext
>          (managed as ManagedStatic).
>
>        - OptBisect is defined in lib/IR, but does use analyses,
>          which is a known layering issue
>
>        - Pass hierarchy provides skipModule etc helper functions
>
>        - Individual passes opt-in to OptBisect activities by manually
>     calling skip* helper functions
>          whenever appropriate
>
>     With current state of new-pm PassInstrumentation potential OptBisect
>     implementation
>     will have the following properties/issues:
>        - OptBisect object that exists per compilation pipeline, managed
>     similar to PassBuilder/PassManagers
>          (which makes it more suitable for use in parallel compilations)
>
>        - no more layering issues imposed by implementation since
>     instrumentations by design
>          can live anywhere - lib/Analysis, lib/Passes etc
>
>        - since Codegen is still legacy-only we will have to make a joint
>     implementation that
>          provides a sequential passes numbering through both new-PM IR
>     and
>     legacy Codegen pipelines
>
>        - as of right now there is no mechanism for opt-in/opt-out, so it
>     needs to be designed/implemented
>          Here I would like to ask:
>              - what would be preferable - opt-in or opt-out?
>
>              - with legacy implementation passes opt-in both for
>     bisect and
>     attribute-optnone support at once.
>                Do we need to follow that in new-pm implementation?
>
>     Also, I would like to ask whether people see current user
>     interface for
>     opt-bisect limiting?
>     Do we need better controls for more sophisticated bisection?
>     Basically I'm looking for any ideas on improving opt-bisect user
>     experience that might
>     affect design approaches we take on the initial implementation.
>
>     regards,
>        Fedor.
>
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180926/296c49f0/attachment.html>

On 09/26/2018 08:13 PM, David Greene wrote:
 > Fedor Sergeev <fedor.sergeev at azul.com> writes:
 >
 >>         - what would be preferable - opt-in or opt-out?
 >>
 >>         - with legacy implementation passes opt-in both for bisect and
 >> attribute-optnone support at once.
 >>           Do we need to follow that in new-pm implementation?
 >>
 >> Also, I would like to ask whether people see current user interface
 >> for opt-bisect limiting?
 >
 > Ideally I would prefer an opt-out model.  Practiclaly speaking, I'm
not
 > sure that's possible.  Presumably opt-bisect doesn't apply to
analysis
 > passes.
Sure it doesnt.

 >   What happens if a transformation pass depends on another
 > transformation pass earlier in the pipeline?  If the earlier
 > transformation pass is bisected out, then...???
If we talk about current bisection scheme then the only passes that
are run after the bisection point are those required-to-run passes.
And that means required passes should not depend on anything else than
other required passes. Which seems to be a very reasonable restriction 
to me.

 > It seems to me that optnone and opt-bisect are two different things and
 > therefore they should be handled orthogonally.
I have heard this opinion more than once :)
Makes sense to me as well.

regards,
   Fedor.

We absolutely need to be able to generate executable programs using opt-bisect,
so some mechanism for not skipping required passes is needed. It might be nice
to have a mode where no passes are skipped and the IR/MIR is dumped when the
bisect limit is reached, but I don't see that as a requirement.

Regarding opt-in versus opt-out, I think we want to make this as easy and
transparent to pass developers as possible. It would be nice to have the
mechanism be opt-out so that passes that were added with no awareness of
opt-bisect would be automatically included. However, there is a small wrinkle to
this. I can't defend this as a reasonable design choice, but the
SelectionDAGISel pass has a sort of hybrid behavior. It can't actually be
skipped, but it OptBisect says it should be skipped it drops the optimization
level to OptNone. That's a machine function pass, so it doesn't matter
so much right now. It's just something to think about.

One of the reasons that we combined the optnone handling and the opt-bisect
handling is that we specifically wanted these two behaviors to be linked. The
exact rule we would like to use for opt bisect is that no pass which runs at O0
is skipped by opt-bisect. There's a test that verifies this. Conversely, if
a pass is able to respect the optnone attribute then it should also be skippable
by opt-bisect. Of course, I would be open to considering a use case where this
reasoning isn't desirable.


With regard to there being one OptBisect object per compilation pipeline, I have
some concerns. Specifically, the behavior of opt-bisect depends on the sequence
of passes run before the limit is reached being consistent and repeatable. My
inclination would be to not allow parallel compilation when opt-bisect is
enabled. I can imagine cases where you might specifically want to debug
something that only happens in a parallel build, but it's more difficult to
imagine something that only happens in a parallel build and doesn't depend
on interactions between threads. In such a case, would we be able to guarantee
that the sequence of passes and any interaction between pipelines was
repeatable. Basically, here I feel like I'm exploring a hypothetical idea
where other people have specific use cases. If so, please explain the use case
to me.

-Andy

-----Original Message-----
From: Fedor Sergeev [mailto:fedor.sergeev at azul.com] 
Sent: Wednesday, September 26, 2018 9:54 AM
To: llvm-dev <llvm-dev at lists.llvm.org>; Zhizhou Yang <zhizhouy at
google.com>; David Greene <dag at cray.com>; David Blaikie <dblaikie
at gmail.com>; Kaylor, Andrew <andrew.kaylor at intel.com>; Chandler
Carruth <chandlerc at gmail.com>
Subject: OptBisect implementation for new pass manager

Greetings!

As the generic Pass Instrumentation framework for new pass manager is finally
*in*, I'm glad to start the discussion on implementation of -opt-bisect
through that framework.

As it has already been discovered while porting other features (namely,
-time-passes)
blindly copying the currently existing legacy implementation is most likely not
a perfect way forward. Now is a chance to take a fresh look at the overall
approach and perhaps do better, without the restrictions that legacy pass
manager framework imposed on the implementation.

Kind of a summary of what we have now:
   - There is a single OptBisect object, requested through LLVMContext
     (managed as ManagedStatic).

   - OptBisect is defined in lib/IR, but does use analyses,
     which is a known layering issue

   - Pass hierarchy provides skipModule etc helper functions

   - Individual passes opt-in to OptBisect activities by manually calling skip*
helper functions
     whenever appropriate

With current state of new-pm PassInstrumentation potential OptBisect
implementation will have the following properties/issues:
   - OptBisect object that exists per compilation pipeline, managed similar to
PassBuilder/PassManagers
     (which makes it more suitable for use in parallel compilations)

   - no more layering issues imposed by implementation since instrumentations by
design
     can live anywhere - lib/Analysis, lib/Passes etc

   - since Codegen is still legacy-only we will have to make a joint
implementation that
     provides a sequential passes numbering through both new-PM IR and legacy
Codegen pipelines

   - as of right now there is no mechanism for opt-in/opt-out, so it needs to be
designed/implemented
     Here I would like to ask:
         - what would be preferable - opt-in or opt-out?

         - with legacy implementation passes opt-in both for bisect and
attribute-optnone support at once.
           Do we need to follow that in new-pm implementation?

Also, I would like to ask whether people see current user interface for
opt-bisect limiting?
Do we need better controls for more sophisticated bisection?
Basically I'm looking for any ideas on improving opt-bisect user experience
that might affect design approaches we take on the initial implementation.

regards,
   Fedor.

Hi Andrew,

We absolutely need to be able to generate executable programs
using
> opt-bisect, so some mechanism for not skipping required passes is needed.
> It might be nice to have a mode where no passes are skipped and the IR/MIR
> is dumped when the bisect limit is reached, but I don't see that as a
> requirement.
>
At this point it makes no sense to worry about the code generation
pipeline. As long as there is no new-PM design for that, this point is just
moot. Designing opt-bisect against code generation without actually
understanding what we're designing against is guaranteed to produce a bad
architecture. So lets figure out the optimizer bisect first, and
incrementally upgrade that once we've ironed out codegen.


Regarding opt-in versus opt-out, I think we want to make this as easy
and
> transparent to pass developers as possible. It would be nice to have the
> mechanism be opt-out so that passes that were added with no awareness of
> opt-bisect would be automatically included. However, there is a small
> wrinkle to this. I can't defend this as a reasonable design choice, but
the
> SelectionDAGISel pass has a sort of hybrid behavior. It can't actually
be
> skipped, but it OptBisect says it should be skipped it drops the
> optimization level to OptNone. That's a machine function pass, so it
> doesn't matter so much right now. It's just something to think
about.
>
> One of the reasons that we combined the optnone handling and the
> opt-bisect handling is that we specifically wanted these two behaviors to
> be linked. The exact rule we would like to use for opt bisect is that no
> pass which runs at O0 is skipped by opt-bisect. There's a test that
> verifies this. Conversely, if a pass is able to respect the optnone
> attribute then it should also be skippable by opt-bisect. Of course, I
> would be open to considering a use case where this reasoning isn't
> desirable.
>
Mixing OptNone and bisect is a software engineering bug: it's mixing
different layers of abstraction. Bisect is something that's at pass manager
scope: run passes until whatever. OptNone in turn doesn't belong in the
pass manager layer. It only concerns function passes, and crumbles quickly
considering other IRUnits or even codegen. I don't have a clear vision what
OptNone handling should look like, but at this point I consider it entirely
orthogonal to bisect handling.
>From a software architecture perspective I don't see a reason why passes
should even _know_ about something like bisect happening. That is simply
not their domain. If a pass shouldn't be skipped for whatever reason,
that's not something the pass should worry about, that's the bisect
driver's problem! My proposal here would be make it an opt-out design, but
let the driver control that. E.g., for skipping, let the user provide a
list of passes they don't want skipped.


> With regard to there being one OptBisect object per compilation pipeline,
> I have some concerns. Specifically, the behavior of opt-bisect depends on
> the sequence of passes run before the limit is reached being consistent and
> repeatable. My inclination would be to not allow parallel compilation when
> opt-bisect is enabled.
I don't have a strong opinion here, but just as a data point: my mental
model here is to expect bisect to expect a deterministic outcome _per
module_. That model isn't threatened by parallel execution.


Cheers,
Philip

> I can imagine cases where you might specifically want to debug something
> that only happens in a parallel build, but it's more difficult to
imagine
> something that only happens in a parallel build and doesn't depend on
> interactions between threads. In such a case, would we be able to guarantee
> that the sequence of passes and any interaction between pipelines was
> repeatable. Basically, here I feel like I'm exploring a hypothetical
idea
> where other people have specific use cases. If so, please explain the use
> case to me.
>
> -Andy
>
> -----Original Message-----
> From: Fedor Sergeev [mailto:fedor.sergeev at azul.com]
> Sent: Wednesday, September 26, 2018 9:54 AM
> To: llvm-dev <llvm-dev at lists.llvm.org>; Zhizhou Yang <zhizhouy
at google.com>;
> David Greene <dag at cray.com>; David Blaikie <dblaikie at
gmail.com>; Kaylor,
> Andrew <andrew.kaylor at intel.com>; Chandler Carruth <chandlerc
at gmail.com>
> Subject: OptBisect implementation for new pass manager
>
> Greetings!
>
> As the generic Pass Instrumentation framework for new pass manager is
> finally *in*, I'm glad to start the discussion on implementation of
> -opt-bisect through that framework.
>
> As it has already been discovered while porting other features (namely,
> -time-passes)
> blindly copying the currently existing legacy implementation is most
> likely not a perfect way forward. Now is a chance to take a fresh look at
> the overall approach and perhaps do better, without the restrictions that
> legacy pass manager framework imposed on the implementation.
>
> Kind of a summary of what we have now:
>    - There is a single OptBisect object, requested through LLVMContext
>      (managed as ManagedStatic).
>
>    - OptBisect is defined in lib/IR, but does use analyses,
>      which is a known layering issue
>
>    - Pass hierarchy provides skipModule etc helper functions
>
>    - Individual passes opt-in to OptBisect activities by manually calling
> skip* helper functions
>      whenever appropriate
>
> With current state of new-pm PassInstrumentation potential OptBisect
> implementation will have the following properties/issues:
>    - OptBisect object that exists per compilation pipeline, managed
> similar to PassBuilder/PassManagers
>      (which makes it more suitable for use in parallel compilations)
>
>    - no more layering issues imposed by implementation since
> instrumentations by design
>      can live anywhere - lib/Analysis, lib/Passes etc
>
>    - since Codegen is still legacy-only we will have to make a joint
> implementation that
>      provides a sequential passes numbering through both new-PM IR and
> legacy Codegen pipelines
>
>    - as of right now there is no mechanism for opt-in/opt-out, so it needs
> to be designed/implemented
>      Here I would like to ask:
>          - what would be preferable - opt-in or opt-out?
>
>          - with legacy implementation passes opt-in both for bisect and
> attribute-optnone support at once.
>            Do we need to follow that in new-pm implementation?
>
> Also, I would like to ask whether people see current user interface for
> opt-bisect limiting?
> Do we need better controls for more sophisticated bisection?
> Basically I'm looking for any ideas on improving opt-bisect user
> experience that might affect design approaches we take on the initial
> implementation.
>
> regards,
>    Fedor.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180927/20daa4aa/attachment.html>

On 09/27/2018 02:18 AM, Kaylor, Andrew wrote:
> With regard to there being one OptBisect object per compilation pipeline, I
have some concerns. Specifically, the behavior of opt-bisect depends on the
sequence of passes run before the limit is reached being consistent and
repeatable. My inclination would be to not allow parallel compilation when
opt-bisect is enabled. I can imagine cases where you might specifically want to
debug something that only happens in a parallel build, but it's more
difficult to imagine something that only happens in a parallel build and
doesn't depend on interactions between threads. In such a case, would we be
able to guarantee that the sequence of passes and any interaction between
pipelines was repeatable. Basically, here I feel like I'm exploring a
hypothetical idea where other people have specific use cases. If so, please
explain the use case to me.
Well, I did not really mean to emphasize parallel compilation topic here.
But as you asked - our JIT compiler which is LLVM library linked into 
java machine typically performs
multiple parallel compilations, each compiling a separate module from a 
separate context in a single thread.
It is not your typical fully-parallel compilation since every compile is 
independent on LLVM level.
However as it is a library in a single process, it does share all the 
globals, OptBisect object in particular.
Naturally, that makes -opt-bisect-limit usage for live in-VM compilation 
rather problematic.

Having OptBisect object handled separately for each individual 
compilation pipeline enables us
with ability to apply bisection to the live in-VM compilation.
(recently we did introduce OptPassGate interface that allows to install 
a different OptBisect for each context,
but it is much less clean than what we can get with new-pm).

regards,
   Fedor.

On 09/27/2018 02:18 AM, Kaylor, Andrew wrote:
> We absolutely need to be able to generate executable programs using
opt-bisect, so some mechanism for not skipping required passes is needed. It
might be nice to have a mode where no passes are skipped and the IR/MIR is
dumped when the bisect limit is reached, but I don't see that as a
requirement.
Do you expect this dumping functionality to be used directly by users, 
or is it going to be fed into some tool?
(implementing something like that should not present any difficulties, 
basically it is just a merge of two instrumentations -  
print-after=<optbisect-number>)
> Regarding opt-in versus opt-out, I think we want to make this as easy and
transparent to pass developers as possible. It would be nice to have the
mechanism be opt-out so that passes that were added with no awareness of
opt-bisect would be automatically included. However, there is a small wrinkle to
this. I can't defend this as a reasonable design choice, but the
SelectionDAGISel pass has a sort of hybrid behavior. It can't actually be
skipped, but it OptBisect says it should be skipped it drops the optimization
level to OptNone. That's a machine function pass, so it doesn't matter
so much right now. It's just something to think about.
It does sound like a hack :)
Perhaps it needs a direct integration with OptBisect?

regards,
   Fedor.
> One of the reasons that we combined the optnone handling and the opt-bisect
handling is that we specifically wanted these two behaviors to be linked. The
exact rule we would like to use for opt bisect is that no pass which runs at O0
is skipped by opt-bisect. There's a test that verifies this. Conversely, if
a pass is able to respect the optnone attribute then it should also be skippable
by opt-bisect. Of course, I would be open to considering a use case where this
reasoning isn't desirable.
>
>
> With regard to there being one OptBisect object per compilation pipeline, I
have some concerns. Specifically, the behavior of opt-bisect depends on the
sequence of passes run before the limit is reached being consistent and
repeatable. My inclination would be to not allow parallel compilation when
opt-bisect is enabled. I can imagine cases where you might specifically want to
debug something that only happens in a parallel build, but it's more
difficult to imagine something that only happens in a parallel build and
doesn't depend on interactions between threads. In such a case, would we be
able to guarantee that the sequence of passes and any interaction between
pipelines was repeatable. Basically, here I feel like I'm exploring a
hypothetical idea where other people have specific use cases. If so, please
explain the use case to me.
>
> -Andy
>
> -----Original Message-----
> From: Fedor Sergeev [mailto:fedor.sergeev at azul.com]
> Sent: Wednesday, September 26, 2018 9:54 AM
> To: llvm-dev <llvm-dev at lists.llvm.org>; Zhizhou Yang <zhizhouy
at google.com>; David Greene <dag at cray.com>; David Blaikie
<dblaikie at gmail.com>; Kaylor, Andrew <andrew.kaylor at
intel.com>; Chandler Carruth <chandlerc at gmail.com>
> Subject: OptBisect implementation for new pass manager
>
> Greetings!
>
> As the generic Pass Instrumentation framework for new pass manager is
finally *in*, I'm glad to start the discussion on implementation of
-opt-bisect through that framework.
>
> As it has already been discovered while porting other features (namely,
> -time-passes)
> blindly copying the currently existing legacy implementation is most likely
not a perfect way forward. Now is a chance to take a fresh look at the overall
approach and perhaps do better, without the restrictions that legacy pass
manager framework imposed on the implementation.
>
> Kind of a summary of what we have now:
>     - There is a single OptBisect object, requested through LLVMContext
>       (managed as ManagedStatic).
>
>     - OptBisect is defined in lib/IR, but does use analyses,
>       which is a known layering issue
>
>     - Pass hierarchy provides skipModule etc helper functions
>
>     - Individual passes opt-in to OptBisect activities by manually calling
skip* helper functions
>       whenever appropriate
>
> With current state of new-pm PassInstrumentation potential OptBisect
implementation will have the following properties/issues:
>     - OptBisect object that exists per compilation pipeline, managed
similar to PassBuilder/PassManagers
>       (which makes it more suitable for use in parallel compilations)
>
>     - no more layering issues imposed by implementation since
instrumentations by design
>       can live anywhere - lib/Analysis, lib/Passes etc
>
>     - since Codegen is still legacy-only we will have to make a joint
implementation that
>       provides a sequential passes numbering through both new-PM IR and
legacy Codegen pipelines
>
>     - as of right now there is no mechanism for opt-in/opt-out, so it needs
to be designed/implemented
>       Here I would like to ask:
>           - what would be preferable - opt-in or opt-out?
>
>           - with legacy implementation passes opt-in both for bisect and
attribute-optnone support at once.
>             Do we need to follow that in new-pm implementation?
>
> Also, I would like to ask whether people see current user interface for
opt-bisect limiting?
> Do we need better controls for more sophisticated bisection?
> Basically I'm looking for any ideas on improving opt-bisect user
experience that might affect design approaches we take on the initial
implementation.
>
> regards,
>     Fedor.
>

On Wed, 26 Sep 2018 at 17:54, Fedor Sergeev via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Greetings!
>
> As the generic Pass Instrumentation framework for new pass manager is
> finally *in*,
> I'm glad to start the discussion on implementation of -opt-bisect
> through that framework.
>
>
> I don't think I can help this discussion, but I would like to
understand
the context of it.
Do we have any requirements?
What are the use cases that "opt-bisect" is intended to help with?

Some use cases that I would like to understand are:
Q1) Take LLVM IR instruction X.
Which pass removed it (optimized it out) ?
Q2) Take LLVM IR instruction X.
Run pass Y once. Leaves instruction X in there.
Run pass Y a second time should not then remove instruction X.
My thinking that if pass Y was doing its job correctly, it should have
removed instruction X on the first pass Y. i.e. pass Y should not be lazy.
One problem with the "lazy" approach, is "How many times should
pass Y be
run?"
Q3) Do any LLVM IR passes do delayed instruction elimination?
I.e.
Pass 1 marks some instructions for removal.
Pass 2 marks some more instructions for removal.
Pass 3 actually removes the instructions. (the expensive bit where caches,
indexes etc. need rebuilding)
I only mention this Q3, because I think it could affect the way opt-bisect
should work. (see Q1 above) Should the answer be Pass 1 or Pass 3 ?

Is opt-bisect something that can help with these use cases?

Kind Regards

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180928/bee5960c/attachment.html>

> I don't think I can help this discussion, but I would like to
understand the context of it.
> Do we have any requirements?
> What are the use cases that "opt-bisect" is intended to help
with?
Basically, it’s meant to help isolate the pass that introduced problems that
occur in optimized code. I mainly use it to debug failures that occur when a
program is run, but it’s a pretty general facility and can be used in many ways.

It’s documented here: https://llvm.org/docs/OptBisect.html

> Q1) Take LLVM IR instruction X.
> Which pass removed it (optimized it out) ?
Opt-bisect, combined with the bisect script in utils, can help with this. You’d
need to provide some script that knows how to detect the IR change you’re
interested in. I’ve used opt-bisect for this. I think you can do this with
bugpoint too, but you wouldn’t enjoy it.

> Q2) Take LLVM IR instruction X.
> Run pass Y once. Leaves instruction X in there.
> Run pass Y a second time should not then remove instruction X.
> My thinking that if pass Y was doing its job correctly, it should have
removed instruction X on the first pass Y. i.e. pass Y should not be lazy.
> One problem with the "lazy" approach, is "How many times
should pass Y be run?"
I don’t think opt-bisect helps with this.

> Q3) Do any LLVM IR passes do delayed instruction elimination?
> I.e.
> Pass 1 marks some instructions for removal.
> Pass 2 marks some more instructions for removal.
> Pass 3 actually removes the instructions. (the expensive bit where caches,
indexes etc. need rebuilding)
> I only mention this Q3, because I think it could affect the way opt-bisect
should work. (see Q1 above) Should the answer be Pass 1 or Pass 3 ?
I don’t believe this kind of dependency between passes is supposed to happen. 
There are some gray areas, I guess, like one pass can add metadata that another
pass interprets in such a way that it decides to delete instructions. In any
event, the IR should be complete and legal after each pass, so if someone
decides to implement passes this way I think they’re on their own determined
what to make of the opt-bisect results.

-Andy



From: James Courtier-Dutton [mailto:james.dutton at gmail.com]
Sent: Friday, September 28, 2018 3:07 PM
To: fedor.sergeev at azul.com
Cc: llvm-dev <llvm-dev at lists.llvm.org>; zhizhouy at google.com; dag at
cray.com; David Blaikie <dblaikie at gmail.com>; Kaylor, Andrew
<andrew.kaylor at intel.com>; chandlerc at gmail.com
Subject: Re: [llvm-dev] OptBisect implementation for new pass manager

On Wed, 26 Sep 2018 at 17:54, Fedor Sergeev via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Greetings!

As the generic Pass Instrumentation framework for new pass manager is
finally *in*,
I'm glad to start the discussion on implementation of -opt-bisect
through that framework.

I don't think I can help this discussion, but I would like to understand the
context of it.
Do we have any requirements?
What are the use cases that "opt-bisect" is intended to help with?

Some use cases that I would like to understand are:
Q1) Take LLVM IR instruction X.
Which pass removed it (optimized it out) ?
Q2) Take LLVM IR instruction X.
Run pass Y once. Leaves instruction X in there.
Run pass Y a second time should not then remove instruction X.
My thinking that if pass Y was doing its job correctly, it should have removed
instruction X on the first pass Y. i.e. pass Y should not be lazy.
One problem with the "lazy" approach, is "How many times should
pass Y be run?"
Q3) Do any LLVM IR passes do delayed instruction elimination?
I.e.
Pass 1 marks some instructions for removal.
Pass 2 marks some more instructions for removal.
Pass 3 actually removes the instructions. (the expensive bit where caches,
indexes etc. need rebuilding)
I only mention this Q3, because I think it could affect the way opt-bisect
should work. (see Q1 above) Should the answer be Pass 1 or Pass 3 ?

Is opt-bisect something that can help with these use cases?

Kind Regards

James

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180928/7faa419c/attachment.html>

James Courtier-Dutton <james.dutton at gmail.com> writes:
> Q2) Take LLVM IR instruction X.
> Run pass Y once. Leaves instruction X in there.
> Run pass Y a second time should not then remove instruction X.
Do you mean multiple runs of Y back-to-back?  If so, this is a fine
requirement.  If there are intervening passes then running Y again most
certainly could make changes and remove instruction X.

I would expect if the thing constructing pass pipelines expects pass Y
to not be idempotent, it would construct the pipeline accordingly,
inserting multiple instances of pass Y itself.  Then OptBisect would
naturally eliminate the multiple instances, possibly uncovering a bug in
one of those instances.

In some ways, operation like that is desirable, as it would reduce the
number of diffs between a passing and failing run.  What is OptBisect's
role in the common way a pass is constructed?

  while (Change) {
    Change = runOnePassRound();
  }

Should OptBisect insert itself into this loop and shut off pass
operation after N iterations?  I would say no, as it seems like that
would complicate both OptBisect and the pass itself.  Once a pass is
identified as problematic, it's usually straightforward to insert things
like DebugCounters to further reduce the diffs.
> Q3) Do any LLVM IR passes do delayed instruction elimination?
> I.e.
> Pass 1 marks some instructions for removal.
> Pass 2 marks some more instructions for removal.
> Pass 3 actually removes the instructions. (the expensive bit where
> caches, indexes etc. need rebuilding)
> I only mention this Q3, because I think it could affect the way
> opt-bisect should work. (see Q1 above) Should the answer be Pass 1 or
> Pass 3 ?
>
> Is opt-bisect something that can help with these use cases?
Instruction selection works this way.  It consults metadata to decide
which instructions to pick.  It does this for non-temporal operations,
for example.

As to whether Pass 1 or Pass 3 should be disabled, it depends.  In the
isel case it must be Pass 1 that is disabled, because isel cannot be
disabled.

If Pass 1 and Pass 2 are considered transformation passes, which they
are because they are adding information to the IR, then they will be
candidates for OptBisect to disable.  So disabling any of Pass 1, Pass 2
or Pass 3 may cause the bug to go away.  This is an example where a
simple bisect is not enough.  Ideally the tool would report back that
all three passes are needed to expose the bug.  That would give a clue
to the developer where the real problem is.  But that is an enhancement
that can be added later.  It's a relatively uncommon case.

                         -David