llvm dev - May 2021 - Legacy PM deprecation for optimization pipeline timeline

On Mon, Apr 26, 2021 at 1:48 PM Mats Larsen <me at supergrecko.com> wrote:
> Hello everyone
>
> I wouldn't mind looking into hooking the new PM infrastructure into the
> LLVM C API. However, I would like to know a little more about the task.
> I've had a look at the new PM infrastructure, and from taking a quick
> glance at the available APIs and how it's used in the tests I assume
we'd
> look to do something similar to the existing legacy PM hooks.
>
> Things are a bit different as far as I've understood because we have
> Module, CGSCC, Function, and Loop passes, each of which may be adapted into
> "higher" kinds, eg FunctionPM -> ModulePM, all of which
differs a bit from
> the legacy PM. From what I've gathered, we'll need to set up the
PMs
> themselves as you mentioned and we'll have to hook a set (all?) of the
new
> PM passes into this. Would it be possible to get a more comprehensive list
> of what needs to be done?
>
> I'm pretty new around here so I'll probably require some guidance
further
> down the line.
>>
>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>
> Awesome, I'm happy to do reviews and answer any further questions.
I'm imagining a new PassBuilder.h in llvm/include/llvm-c where
PassManagerBuilder.h was.
Initially I was thinking we'd need wrappers around the various types of
pass managers as you've mentioned, as well as the adaptors
(e.g. createModuleToFunctionPassAdaptor()). And each of the LLVMAdd*Pass()
would need a corresponding new PM alternative that adds the pass to the
proper pass manager type.

But now that I think about it some more, we might be able to bypass all of
that and instead use the new PM's pass pipeline text parsing (the same
thing that `opt -passes=foo` uses). This seems exactly the sort of thing it
would be useful for. We'll bypass the need to create wrappers for the
different types of pass managers since we just get back a ModulePassManager
from the PassBuilder.
The code in NewPMDriver.cpp should be a good starting place to figure out
what's necessary to create a PassBuilder, then how to get a
ModulePassManager from it and run it on some IR.
I imagine first you'll need some way to setup the options to pass to the
PassBuilder, so the C wrapper would probably own various things like the
various
*AnalysisManagers, PassInstrumentationCallbacks, StandardInstrumentations,
and PipelineTuningOptions. Then the user can ask for some pipeline via a
string, whether that's an individual pass like "instcombine" or a
full
pipeline like "default<O2>", and you create a PassBuilder from
the
options, then tell it to create the ModulePassManager via
PassBuilder::parsePassPipeline().
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210427/d7c94154/attachment.html>

Thank you Arthur, that sounds great. Using opt's pass name parser sounds
like a good plan and a great way to save ourselves from complications with
the different PM types. We can then construct the ModulePassManager and run
that over an LLVM module the user passes to us.

I'll have a closer look at this over the next few days and I'll possibly
gather a tiny list of questions I may have before we get started.

Mats

On Tue, Apr 27, 2021 at 6:26 PM Arthur Eubanks <aeubanks at google.com>
wrote:
>
>
> On Mon, Apr 26, 2021 at 1:48 PM Mats Larsen <me at supergrecko.com>
wrote:
>
>> Hello everyone
>>
>> I wouldn't mind looking into hooking the new PM infrastructure into
the
>> LLVM C API. However, I would like to know a little more about the task.
>> I've had a look at the new PM infrastructure, and from taking a
quick
>> glance at the available APIs and how it's used in the tests I
assume we'd
>> look to do something similar to the existing legacy PM hooks.
>>
>> Things are a bit different as far as I've understood because we
have
>> Module, CGSCC, Function, and Loop passes, each of which may be adapted
into
>> "higher" kinds, eg FunctionPM -> ModulePM, all of which
differs a bit from
>> the legacy PM. From what I've gathered, we'll need to set up
the PMs
>> themselves as you mentioned and we'll have to hook a set (all?) of
the new
>> PM passes into this. Would it be possible to get a more comprehensive
list
>> of what needs to be done?
>>
>> I'm pretty new around here so I'll probably require some
guidance further
>> down the line.
>>>
>>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>>
>> Awesome, I'm happy to do reviews and answer any further questions.
>
> I'm imagining a new PassBuilder.h in llvm/include/llvm-c where
> PassManagerBuilder.h was.
> Initially I was thinking we'd need wrappers around the various types of
> pass managers as you've mentioned, as well as the adaptors
> (e.g. createModuleToFunctionPassAdaptor()). And each of the LLVMAdd*Pass()
> would need a corresponding new PM alternative that adds the pass to the
> proper pass manager type.
>
> But now that I think about it some more, we might be able to bypass all of
> that and instead use the new PM's pass pipeline text parsing (the same
> thing that `opt -passes=foo` uses). This seems exactly the sort of thing it
> would be useful for. We'll bypass the need to create wrappers for the
> different types of pass managers since we just get back a ModulePassManager
> from the PassBuilder.
> The code in NewPMDriver.cpp should be a good starting place to figure out
> what's necessary to create a PassBuilder, then how to get a
> ModulePassManager from it and run it on some IR.
> I imagine first you'll need some way to setup the options to pass to
the
> PassBuilder, so the C wrapper would probably own various things like the
> various
> *AnalysisManagers, PassInstrumentationCallbacks, StandardInstrumentations,
> and PipelineTuningOptions. Then the user can ask for some pipeline via a
> string, whether that's an individual pass like "instcombine"
or a full
> pipeline like "default<O2>", and you create a PassBuilder
from the
> options, then tell it to create the ModulePassManager via
> PassBuilder::parsePassPipeline().
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210428/32c2af38/attachment.html>

Greetings, I've had the time to take a better look at the code in the
NewPMDriver.cpp source and I believe I have a good overview of the task
now. Here are some things which I'm not too sure about so I'd like your
input on this:

- What are the PreservedAnalyses and how do we use them? Is it crucial for
the LLVM-C API?
- How do we want to handle the various callbacks for the
PassInstrumentationCallbacks? They're taking the templated IRUnit which
could be different types upon each invocation.
- Where do we want the binding code itself? Some places have files like
ExecutionEngineBindings.cpp, while others have it at the end of files
relevant to the code being hooked into.
- I see the C++ API accepts PGO Options, should that be hooked into as well?

With that being said, I think I have a decent plan for the interface
itself. We'll provide a PassBuilder which we can construct with the
PassInstrumentationCallbacks, PipelineTuningOptions, etc. The various
*AnalysisManagers may also be added to the PassBuilder. The
StandardInstrumentation can be added by passing a
PassInstrumentationCallbacks and optionally a FunctionAnalysisManager like
the C++ llvm::StandardInstrumentation::registerCallbacks method does.

The PassBuilder will then be able to accept a Module and an opt-style
string with the passes to be used. A ModulePassManager will be constructed
as you described and it will run the passes over the module.

Let me know what you think of an approach like this...

Best regards
Mats Larsen

On Wed, Apr 28, 2021 at 11:13 PM Mats Larsen <me at supergrecko.com>
wrote:
> Thank you Arthur, that sounds great. Using opt's pass name parser
sounds
> like a good plan and a great way to save ourselves from complications with
> the different PM types. We can then construct the ModulePassManager and run
> that over an LLVM module the user passes to us.
>
> I'll have a closer look at this over the next few days and I'll
possibly
> gather a tiny list of questions I may have before we get started.
>
> Mats
>
> On Tue, Apr 27, 2021 at 6:26 PM Arthur Eubanks <aeubanks at
google.com>
> wrote:
>
>>
>>
>> On Mon, Apr 26, 2021 at 1:48 PM Mats Larsen <me at
supergrecko.com> wrote:
>>
>>> Hello everyone
>>>
>>> I wouldn't mind looking into hooking the new PM infrastructure
into the
>>> LLVM C API. However, I would like to know a little more about the
task.
>>> I've had a look at the new PM infrastructure, and from taking a
quick
>>> glance at the available APIs and how it's used in the tests I
assume we'd
>>> look to do something similar to the existing legacy PM hooks.
>>>
>>> Things are a bit different as far as I've understood because we
have
>>> Module, CGSCC, Function, and Loop passes, each of which may be
adapted into
>>> "higher" kinds, eg FunctionPM -> ModulePM, all of
which differs a bit from
>>> the legacy PM. From what I've gathered, we'll need to set
up the PMs
>>> themselves as you mentioned and we'll have to hook a set (all?)
of the new
>>> PM passes into this. Would it be possible to get a more
comprehensive list
>>> of what needs to be done?
>>>
>>> I'm pretty new around here so I'll probably require some
guidance
>>> further down the line.
>>>>
>>>>
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>>>
>>> Awesome, I'm happy to do reviews and answer any further
questions.
>>
>> I'm imagining a new PassBuilder.h in llvm/include/llvm-c where
>> PassManagerBuilder.h was.
>> Initially I was thinking we'd need wrappers around the various
types of
>> pass managers as you've mentioned, as well as the adaptors
>> (e.g. createModuleToFunctionPassAdaptor()). And each of the
LLVMAdd*Pass()
>> would need a corresponding new PM alternative that adds the pass to the
>> proper pass manager type.
>>
>> But now that I think about it some more, we might be able to bypass all
>> of that and instead use the new PM's pass pipeline text parsing
(the same
>> thing that `opt -passes=foo` uses). This seems exactly the sort of
thing it
>> would be useful for. We'll bypass the need to create wrappers for
the
>> different types of pass managers since we just get back a
ModulePassManager
>> from the PassBuilder.
>> The code in NewPMDriver.cpp should be a good starting place to figure
out
>> what's necessary to create a PassBuilder, then how to get a
>> ModulePassManager from it and run it on some IR.
>> I imagine first you'll need some way to setup the options to pass
to the
>> PassBuilder, so the C wrapper would probably own various things like
the
>> various
>> *AnalysisManagers, PassInstrumentationCallbacks,
StandardInstrumentations,
>> and PipelineTuningOptions. Then the user can ask for some pipeline via
a
>> string, whether that's an individual pass like
"instcombine" or a full
>> pipeline like "default<O2>", and you create a
PassBuilder from the
>> options, then tell it to create the ModulePassManager via
>> PassBuilder::parsePassPipeline().
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210501/83880332/attachment.html>