David A. Greene via llvm-dev
2018-Jun-11 21:04 UTC
[llvm-dev] RFC: Pass Execution Instrumentation interface
I was going to write something up about fine-grained opt-bisect but
didn't get to it last week.
We've had a -pass-max option here for some time and have hand-added
instrumentation to various passes to honor it. It's saved us man-years
of debug time. I was planning on sending it upstream but saw this
effort with pass execution instrumentation and thought it might fit
there.
Initially I think some very simple APIs in PassInstrumentationAnalysis
would be fine, something like:
// PIA - PassInstrumentationAnalysis
if (PIA->skipTransformation()) {
return;
}
// Do it.
PIA->didTransformation();
This kind of interface also encourages good pass design like doing all
the analysis for a transformation before actually doing the
transformation. Some passes mix analysis with transformation and those
are much harder to instrument to support -pass-max operation.
In our implementation we can set a -pass-max per pass, like
-pass-max=instcombine=524. A global index might be even more useful.
If it interacted with opt-bisect, even better. It seems like APIs that
cover both the opt-bisect pass-level operation and the finer-grained
operation could be quite powerful. As passes opt-in to the
finer-grained control, the opt-bisect limit would become more powerfuly
automatically.
I've always wanted a bugpoint that could point not just to a pass but to
a specific transformation within a pass.
-David
Fedor Sergeev via llvm-dev <llvm-dev at lists.llvm.org> writes:
> On 06/08/2018 08:36 PM, Zhizhou Yang wrote:
>
> Thanks Craig, that's exactly what I mean, stopping at particular
> changes inside a pass.
>
> PassInstrumentation at its current base design only instruments Pass
> execution from a caller (PM) side.
> Stopping at a particular change inside a pass definitely requires pass
> commitment and is out of scope
> for this RFC.
>
> Yet it appears to be rather orthogonal and I dont see anything that
> precludes from enhancing PassInstrumentationAnalysis
> to also cover points other than those in-PMs. For sure,
> PassInstrumentationAnalysis should readily be available
> inside the pass through the AnalysisManager.
>
> regards,
> Fedor.
>
>
>
>
> Would you please refer me the discuss about combining opt-bisect
> with debug counters? Is it already under implementation?
>
>
>
> On Fri, Jun 8, 2018 at 12:19 AM Craig Topper
> <craig.topper at gmail.com> wrote:
>
>
> I think that "level" was referring to what level of
> granularity the opt-bisect should control it wasn't mean to be
> read as "optimization level". I think Zhizhou was saying
that
> it should be able to disable individual optimization steps
> within a pass. Like if a particular run of InstCombine made 20
> changes, opt-bisect should be able to skip each of those
> changes. I think this is the combining opt-bisect with debug
> counters idea that's been mentioned previously.
>
>
>
>
>
> ~Craig
>
>
>
>
>
> On Fri, Jun 8, 2018 at 12:10 AM Fedor Sergeev via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>
>
>
> Care to expand a bit on what you mean by per-optimization
> level? Preferably with a use case.
>
>
>
> To me optbisect is a low level developer tool and it
> doesn't cope well with a crude user level hammer of
> optimization level.
>
>
> F.
>
>
>
>
> On Fri, Jun 8, 2018 at 9:12 AM +0300, "Zhizhou Yang"
> <zhizhouy at google.com> wrote:
>
>
>
> Hi Fedor,
>
>
> Thanks for replying my questions about porting the
> OptBisecting to new PM.
>
>
> This thread looks like a great improvement on what we
> currently have.
> Though we are also trying to make opt-bisect more
> granular.
>
>
> In particular, we think it would be helpful if we
> could have opt-bisect work on a per-optimization level
> rather than per-pass level.
> I believe this will be a more invasive change and we
> would like to do this as a follow-up to this thread.
>
>
> How difficult do you think it would be to support this
> use-case with your design?
>
>
> Thank you!
> Zhizhou
>
>
>
>
> On Wed, Jun 6, 2018 at 5:00 PM Fedor Sergeev via
> llvm-dev <llvm-dev at lists.llvm.org> wrote:
>
> TL;DR
> ===>
> This RFC suggests an API to enable customizable
> instrumentation of pass
> execution.
> The intent is to have a common machinery to
> implement all the
> pass-execution-debugging
> features mentioned here.
>
> Prime target of the interface is the new pass
> manager.
> The overall approach and most of the
> implementation details should be
> equially applicable
> to the legacy one though.
>
>
> Background
> =========>
> There are quite a few important debugging
> facilities in LLVM that affect
> pass execution sequence:
>
> -print-after/-print-before[-all]
> execute IR-print pass before or after a
> particularly
> insteresting pass
> (or all passes)
>
> -verify-each
> execute verifier pass after each
>
> -opt-bisect-limit
> execute passes only up to a selected
> "pass-counter"
>
> -time-passes
> track execution time for each pass
>
> There are also quite a few similar ideas floating
> around, i.e:
> -git-commit-after-all
> -debugify-each
>
> All these facilities essentially require
> instrumentation of pass
> execution points
> in the pass manager, each being implemented in a
> legacy pass manager
> through their
> own custom way, e.g:
> * -time-passes has a bunch of dedicated code in
> each of the pass managers
>
> * -print-before/after/verify-each insert
> additional passes before/after
> the passes in the pipeline
>
> And there is no implementation of any of these
> features for the new pass
> manager,
> which obviously is a problem for new pass manager
> transition.
>
> Proposal
> =======>
> Main idea:
> - introduce an API that allows to instrument
> points of pass execution
> - access through LLVM Context (allows to control
> life-time and scope
> in multi-context execution)
> - wrap it into an analysis for easier access from
> pass managers
>
>
> Details:
> 1. introduce llvm::PassInstrumentation
>
> This is the main interface that handles the
> customization and
> provides instrumentation calls
>
> - resides in IR
> - is accessible through
> LLVMContext::getPassInstrumentation()
> (with context owning this object).
>
> 2. every single point of Pass execution in the
> (new) PassManager(s)
> will query
> this analysis and run instrumentation call
> specific to a
> particular point.
>
> Instrumentation points:
>
> bool BeforePass (PassID, PassExecutionCounter);
> void AfterPass (PassID, PassExecutionCounter);
>
> Run before/after a particular pass execution
> BeforePass instrumentation call returns true if
> this
> execution is allowed to run.
>
> 'PassID'
> certain unique identifier for a pass (pass name?).
>
> 'PassExecutionCounter'
> a number that uniquely identifies this particular
> pass
> execution
> in current pipeline, as tracked by Pass Manager.
>
> void StartPipeline()
> void EndPipeline()
>
> Run at the start/end of a pass pipeline execution.
> (useful for initialization/finalization purposes)
>
>
> 3. custom callbacks are registered with
> PassInstrumentation::register* interfaces
>
> A sequence of registered callbacks is called at
> each
> instrumentation point as appropriate.
>
> 4. introduce llvm::ExecutionCounter to track
> execution of passes
>
> (akin to DebugCounter, yet enabled in Release mode
> as well?)
>
> Note: it is somewhat nontrivial to uniquely track
> pass executions
> with counters in new pass
> manager as its pipeline schedule can be dynamic.
> Ideas are welcome
> on how to efficiently
> implement unique execution tracking that does not
> break in
> presence of fixed-point iteration
> passes like RepeatedPass/DevirtSCCRepeatedPass
>
> Also, the intent is for execution counters to be
> able provide
> thread-safety in multi-threaded
> pipeline execution (though no work planned for it
> yet).
>
> 5. introduce a new analysis
> llvm::PassInstrumentationAnalysis
>
> This is a convenience wrapper to provide an access
> to
> PassInstrumentation via analysis framework.
> If using analysis is not convenient (?legacy) then
> PassInstrumentation can be queried
> directly from LLVMContext.
>
>
> Additional goals
> ===============>
> - layering problem
> Currently OptBisect/OptPassGate has layering issue
> - interface
> dependencies on all the "IR units",
> even those that are analyses - Loop, CallGraphSCC.
>
> Generic PassInstrumentation facilitiy allows to
> inject arbitrary
> call-backs in run-time,
> removing any compile-time interface dependencies
> on internals of
> those callbacks,
> effectively solving this layering issue.
>
> - life-time/scope control for multi-context
> execution
>
> Currently there are issues with multi-context
> execution of, say,
> -time-passes which store
> their data in global maps.
>
> With LLVMContext owning PassInstrumentation there
> should be no
> problem with multi-context execution
> (callbacks can be made owning the instrumentation
> data).
>
> Open Questions
> =============>
> - whats the best way to handle ownership of
> PassInstrumentation
>
> Any problems with owning by LLVMContext?
> Something similar to TargetLibraryInfo (owned by
> TargetLibraryAnalysis/TargetLibraryInfoWrapperPass)
> ?
>
> - using PassInstrumentationAnalysis or directly
> querying LLVMContext
>
> PassInstrumentationAnalysis appeared to be a nice
> idea, only until
> I tried querying it
> in new pass manager framework, and amount of
> hooplas to jump over
> makes me shiver a bit...
>
> Querying LLVMContext is plain and straightforward,
> but we do not
> have a generic way to access LLVMContext
> from a PassManager template (need to introduce
> generic
> IRUnit::getContext?)
>
> Implementation
> =============>
> PassInstrumentationAnalysis proof-of-concept
> unfinished prototype
> implementation:
> (Heavily under construction, do not enter without
> wearing a hard hat...)
>
> https://reviews.llvm.org/D47858
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-
> dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Fedor Sergeev via llvm-dev
2018-Jun-13 12:38 UTC
[llvm-dev] RFC: Pass Execution Instrumentation interface
On 06/12/2018 12:04 AM, David A. Greene wrote:> I was going to write something up about fine-grained opt-bisect but > didn't get to it last week. > > We've had a -pass-max option here for some time and have hand-added > instrumentation to various passes to honor it. It's saved us man-years > of debug time. I was planning on sending it upstream but saw this > effort with pass execution instrumentation and thought it might fit > there. > > Initially I think some very simple APIs in PassInstrumentationAnalysis > would be fine, something like: > > // PIA - PassInstrumentationAnalysis > if (PIA->skipTransformation()) { > return; > } > // Do it. > PIA->didTransformation();That should be easily doable (though the interface would be part of PassInstrumentation rather than PassInstrumentationAnalysis).> > This kind of interface also encourages good pass design like doing all > the analysis for a transformation before actually doing the > transformation. Some passes mix analysis with transformation and those > are much harder to instrument to support -pass-max operation.I'm not sure everybody would agree on this definition of a good pass design :) Ability to mix analysis with transformation might appear to be rather useful when heavy analysis is only needed in a very special corner case of an overall transformation. regards, Fedor.> In our implementation we can set a -pass-max per pass, like > -pass-max=instcombine=524. A global index might be even more useful. > If it interacted with opt-bisect, even better. It seems like APIs that > cover both the opt-bisect pass-level operation and the finer-grained > operation could be quite powerful. As passes opt-in to the > finer-grained control, the opt-bisect limit would become more powerfuly > automatically. > > I've always wanted a bugpoint that could point not just to a pass but to > a specific transformation within a pass. > > -David > > Fedor Sergeev via llvm-dev <llvm-dev at lists.llvm.org> writes: > >> On 06/08/2018 08:36 PM, Zhizhou Yang wrote: >> >> Thanks Craig, that's exactly what I mean, stopping at particular >> changes inside a pass. >> >> PassInstrumentation at its current base design only instruments Pass >> execution from a caller (PM) side. >> Stopping at a particular change inside a pass definitely requires pass >> commitment and is out of scope >> for this RFC. >> >> Yet it appears to be rather orthogonal and I dont see anything that >> precludes from enhancing PassInstrumentationAnalysis >> to also cover points other than those in-PMs. For sure, >> PassInstrumentationAnalysis should readily be available >> inside the pass through the AnalysisManager. >> >> regards, >> Fedor. >> >> >> >> >> Would you please refer me the discuss about combining opt-bisect >> with debug counters? Is it already under implementation? >> >> >> >> On Fri, Jun 8, 2018 at 12:19 AM Craig Topper >> <craig.topper at gmail.com> wrote: >> >> >> I think that "level" was referring to what level of >> granularity the opt-bisect should control it wasn't mean to be >> read as "optimization level". I think Zhizhou was saying that >> it should be able to disable individual optimization steps >> within a pass. Like if a particular run of InstCombine made 20 >> changes, opt-bisect should be able to skip each of those >> changes. I think this is the combining opt-bisect with debug >> counters idea that's been mentioned previously. >> >> >> >> >> >> ~Craig >> >> >> >> >> >> On Fri, Jun 8, 2018 at 12:10 AM Fedor Sergeev via llvm-dev >> <llvm-dev at lists.llvm.org> wrote: >> >> >> >> Care to expand a bit on what you mean by per-optimization >> level? Preferably with a use case. >> >> >> >> To me optbisect is a low level developer tool and it >> doesn't cope well with a crude user level hammer of >> optimization level. >> >> >> F. >> >> >> >> >> On Fri, Jun 8, 2018 at 9:12 AM +0300, "Zhizhou Yang" >> <zhizhouy at google.com> wrote: >> >> >> >> Hi Fedor, >> >> >> Thanks for replying my questions about porting the >> OptBisecting to new PM. >> >> >> This thread looks like a great improvement on what we >> currently have. >> Though we are also trying to make opt-bisect more >> granular. >> >> >> In particular, we think it would be helpful if we >> could have opt-bisect work on a per-optimization level >> rather than per-pass level. >> I believe this will be a more invasive change and we >> would like to do this as a follow-up to this thread. >> >> >> How difficult do you think it would be to support this >> use-case with your design? >> >> >> Thank you! >> Zhizhou >> >> >> >> >> On Wed, Jun 6, 2018 at 5:00 PM Fedor Sergeev via >> llvm-dev <llvm-dev at lists.llvm.org> wrote: >> >> TL;DR >> ===>> >> This RFC suggests an API to enable customizable >> instrumentation of pass >> execution. >> The intent is to have a common machinery to >> implement all the >> pass-execution-debugging >> features mentioned here. >> >> Prime target of the interface is the new pass >> manager. >> The overall approach and most of the >> implementation details should be >> equially applicable >> to the legacy one though. >> >> >> Background >> =========>> >> There are quite a few important debugging >> facilities in LLVM that affect >> pass execution sequence: >> >> -print-after/-print-before[-all] >> execute IR-print pass before or after a >> particularly >> insteresting pass >> (or all passes) >> >> -verify-each >> execute verifier pass after each >> >> -opt-bisect-limit >> execute passes only up to a selected >> "pass-counter" >> >> -time-passes >> track execution time for each pass >> >> There are also quite a few similar ideas floating >> around, i.e: >> -git-commit-after-all >> -debugify-each >> >> All these facilities essentially require >> instrumentation of pass >> execution points >> in the pass manager, each being implemented in a >> legacy pass manager >> through their >> own custom way, e.g: >> * -time-passes has a bunch of dedicated code in >> each of the pass managers >> >> * -print-before/after/verify-each insert >> additional passes before/after >> the passes in the pipeline >> >> And there is no implementation of any of these >> features for the new pass >> manager, >> which obviously is a problem for new pass manager >> transition. >> >> Proposal >> =======>> >> Main idea: >> - introduce an API that allows to instrument >> points of pass execution >> - access through LLVM Context (allows to control >> life-time and scope >> in multi-context execution) >> - wrap it into an analysis for easier access from >> pass managers >> >> >> Details: >> 1. introduce llvm::PassInstrumentation >> >> This is the main interface that handles the >> customization and >> provides instrumentation calls >> >> - resides in IR >> - is accessible through >> LLVMContext::getPassInstrumentation() >> (with context owning this object). >> >> 2. every single point of Pass execution in the >> (new) PassManager(s) >> will query >> this analysis and run instrumentation call >> specific to a >> particular point. >> >> Instrumentation points: >> >> bool BeforePass (PassID, PassExecutionCounter); >> void AfterPass (PassID, PassExecutionCounter); >> >> Run before/after a particular pass execution >> BeforePass instrumentation call returns true if >> this >> execution is allowed to run. >> >> 'PassID' >> certain unique identifier for a pass (pass name?). >> >> 'PassExecutionCounter' >> a number that uniquely identifies this particular >> pass >> execution >> in current pipeline, as tracked by Pass Manager. >> >> void StartPipeline() >> void EndPipeline() >> >> Run at the start/end of a pass pipeline execution. >> (useful for initialization/finalization purposes) >> >> >> 3. custom callbacks are registered with >> PassInstrumentation::register* interfaces >> >> A sequence of registered callbacks is called at >> each >> instrumentation point as appropriate. >> >> 4. introduce llvm::ExecutionCounter to track >> execution of passes >> >> (akin to DebugCounter, yet enabled in Release mode >> as well?) >> >> Note: it is somewhat nontrivial to uniquely track >> pass executions >> with counters in new pass >> manager as its pipeline schedule can be dynamic. >> Ideas are welcome >> on how to efficiently >> implement unique execution tracking that does not >> break in >> presence of fixed-point iteration >> passes like RepeatedPass/DevirtSCCRepeatedPass >> >> Also, the intent is for execution counters to be >> able provide >> thread-safety in multi-threaded >> pipeline execution (though no work planned for it >> yet). >> >> 5. introduce a new analysis >> llvm::PassInstrumentationAnalysis >> >> This is a convenience wrapper to provide an access >> to >> PassInstrumentation via analysis framework. >> If using analysis is not convenient (?legacy) then >> PassInstrumentation can be queried >> directly from LLVMContext. >> >> >> Additional goals >> ===============>> >> - layering problem >> Currently OptBisect/OptPassGate has layering issue >> - interface >> dependencies on all the "IR units", >> even those that are analyses - Loop, CallGraphSCC. >> >> Generic PassInstrumentation facilitiy allows to >> inject arbitrary >> call-backs in run-time, >> removing any compile-time interface dependencies >> on internals of >> those callbacks, >> effectively solving this layering issue. >> >> - life-time/scope control for multi-context >> execution >> >> Currently there are issues with multi-context >> execution of, say, >> -time-passes which store >> their data in global maps. >> >> With LLVMContext owning PassInstrumentation there >> should be no >> problem with multi-context execution >> (callbacks can be made owning the instrumentation >> data). >> >> Open Questions >> =============>> >> - whats the best way to handle ownership of >> PassInstrumentation >> >> Any problems with owning by LLVMContext? >> Something similar to TargetLibraryInfo (owned by >> TargetLibraryAnalysis/TargetLibraryInfoWrapperPass) >> ? >> >> - using PassInstrumentationAnalysis or directly >> querying LLVMContext >> >> PassInstrumentationAnalysis appeared to be a nice >> idea, only until >> I tried querying it >> in new pass manager framework, and amount of >> hooplas to jump over >> makes me shiver a bit... >> >> Querying LLVMContext is plain and straightforward, >> but we do not >> have a generic way to access LLVMContext >> from a PassManager template (need to introduce >> generic >> IRUnit::getContext?) >> >> Implementation >> =============>> >> PassInstrumentationAnalysis proof-of-concept >> unfinished prototype >> implementation: >> (Heavily under construction, do not enter without >> wearing a hard hat...) >> >> https://reviews.llvm.org/D47858 >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm- >> dev >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
David A. Greene via llvm-dev
2018-Jun-13 16:46 UTC
[llvm-dev] RFC: Pass Execution Instrumentation interface
Fedor Sergeev <fedor.sergeev at azul.com> writes:> On 06/12/2018 12:04 AM, David A. Greene wrote: >> // PIA - PassInstrumentationAnalysis >> if (PIA->skipTransformation()) { >> return; >> } >> // Do it. >> PIA->didTransformation();> That should be easily doable (though the interface would be part of > PassInstrumentation > rather than PassInstrumentationAnalysis).Ok. The way I envision this working from a user standpoint is -opt-bisect-limit <n> would mean "n applications of code transformation." where "code transformation" could mean an entire pass run or individual transforms within a pass. Each pass would decide what it supports.>> This kind of interface also encourages good pass design like doing all >> the analysis for a transformation before actually doing the >> transformation. Some passes mix analysis with transformation and those >> are much harder to instrument to support -pass-max operation.> I'm not sure everybody would agree on this definition of a good pass > design :) > Ability to mix analysis with transformation might appear to be rather useful > when heavy analysis is only needed in a very special corner case of an > overall > transformation.Yes, I'm sure there are exceptions. I'm not referring to things like instcombine that have individual rules that guard transformations and the pass iterates applying transformations when rules are matched. That's straightforward to instrument. The harder cases are where the analysis phase itself does some transformation (possily to facilitate analysis) and then decides the larger-goal transformation is not viable. If the pass then tries to undo the first transformation, it's possible that -pass-max will result in code that never would have been generated, because it could do the first transformation but then not undo it because it hit the max number of transforms. Sometimes it's difficult to find where things are undone and update the transformation index (basically allow the undo and decrement the index to reflect the undo). In code: if (not hit max) do anlysis transform ++index return <some other function> if (transform legal) if (not hit max) do big transform ++index return <some third function> if (need to undo analysis transform) if (not hit max) undo it ++index Sometimes it is not obvious that these three places are logically connected. Ideally we wouldn't increment the index for the analysis transform or we would allow the undo and decrement the index, but it's not always clear from the code that that is what should happen. -David