llvm dev - May 2021 - A Few Questions Regarding The New Pass Manager, LTO, and LLD

LLVM Developers,
       Hi!
       There is a scenario where I want to do full program analysis, ideally
after link time optimization is done and before assembly is emitted, when all
symbols are in LLVM IR form and contained in a single module. But after some
time browsing stackoverflow, lld documentation, and the mailing lists, I still
have little clue how to do it… I am aware of the email sent by Y Liu and titled
“how to add my own passes to LTO pass”. But it doesn’t seem to work for the new
pass manager. A few questions:

1.  The legacy pass manager has an extension point,
`PassManagerBuilder::EP_FullLinkTimeOptimizationLast`. Unfortunately, I can’t
find a alternative in the new pass manager. The `Bye` example seems to suggest
using `PB.registerVectorizerStartEPCallback`. I am not sure whether doing so
will result in my module pass being ran at compile or link time?

2.  I wonder how does clang communicates passes that need to be run at lto to
lld. The command line that I attempt to get lto running is `clang
-fexperimental-new-pass-manager -fpass-plugin=libmymodulepass.so -flto
-fuse-ld=lld …`. I straced the command for execv calls, and expected to see
libmymodulepass.so to be passed to ld.lld as an argument, but found out it
wasn’t. Could  you shed some light on how clang and lld cope with each other
during lto?

3.  Greping `ld.lld --help` reveals an interesting option:
`--lto-newpm-passes=<value>`. Does the <value> here mean built-in
pass names like `dce`/`mem2reg`, or is it an option to insert our plugin like
libmymodulepass.so?
I’ve looking into this for a while but haven’t made much progress…Any examples /
command line on how to do this are greatly appreciated!

Best Wishes,
Chibin Zhang
2021.5.8
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210508/2ee465d9/attachment.html>

On Sat, May 8, 2021 at 8:15 AM 张驰斌 via llvm-dev <llvm-dev at
lists.llvm.org>
wrote:
> LLVM Developers,
>
>        Hi!
>
>        There is a scenario where I want to do full program analysis,
> ideally after link time optimization is done and before assembly is
> emitted, when all symbols are in LLVM IR form and contained in a single
> module. But after some time browsing stackoverflow, lld documentation, and
> the mailing lists, I still have little clue how to do it…
>
LLVM docs are _useful_ but IMAO they only provide really high level
guidelines. Therefore most of the time I just read the code. Most of the
LLVM code has high quality comments.

> I am aware of the email sent by Y Liu and titled “how to add my own
> passes to LTO pass”. But it doesn’t seem to work for the new pass
> manager. A few questions:
>
> 1.  The legacy pass manager has an extension point,
> `PassManagerBuilder::EP_FullLinkTimeOptimizationLast`. Unfortunately, I can
> ’t find a alternative in the new pass manager. The `Bye` example seems to
> suggest using `PB.registerVectorizerStartEPCallback`. I am not sure
> whether doing so will result in my module pass being ran at compile or link
> time?
>
PassBuilder (https://llvm.org/doxygen/classllvm_1_1PassBuilder.html) is the
right place to insert EP. If you're building a pipeline for normal
optimization (non-LTO) you can leverage registerOptimizerLastEPCallback.
But I think right now you can't insert any EP into LTO optimization (not
pre-link one) pipeline, except peephole EP (registerPeepholeEPCallback). I
don't know the rationale behind this decision, maybe folks familiar with
LTO can give you more details.
The callback registered with registerVectorizerStartEPC will only be
executed by normal optimization pipeline (including O0)
> 2.  I wonder how does clang communicates passes that need to be run at
> lto to lld.
>
Most of the time they're communicated via optimization level (i.e. O0 ~
O3...). Clang tells LLD what optimization level to be performed during LTO.
LLD then (indirectly) asks PassBuilder to build the corresponding LTO
optimization pipeline.

> The command line that I attempt to get lto running is `clang
> -fexperimental-new-pass-manager -fpass-plugin=libmymodulepass.so -flto
> -fuse-ld=lld …`. I straced the command for execv calls, and expected to
> see libmymodulepass.so to be passed to ld.lld as an argument, but found out
> it wasn’t. Could  you shed some light on how clang and lld cope with each
> other during lto?
>
I think neither LLD nor the gold plugin (llvm/tools/gold) can load (NewPM)
PassPlugin into their LTO optimization pipeline. Your PassPlugin will
eventually be loaded, but it's unable to register your custom Pass into the
LTO optimization pipeline.
The `llvm-lto` and `llvm-lto2` command line tools do support PassPlugin
though -- but these tools are only intended for debug and development.

> 3.  Greping `ld.lld --help` reveals an interesting option:
> `--lto-newpm-passes=<value>`. Does the <value> here mean
built-in pass
> names like `dce`/`mem2reg`,
>
Yes, for all available names please refer to this file:
https://github.com/llvm/llvm-project/blob/main/llvm/lib/Passes/PassRegistry.def


> or is it an option to insert our plugin like libmymodulepass.so?
>
No, I don't think LLD support PassPlugin now.

> I’ve looking into this for a while but haven’t made much progress…Any
> examples / command line on how to do this are greatly appreciated!
>
The rule of thumb here is that Clang doesn't explicitly tell LLD what
Passes to run during LTO -- it only tells LLD the optimization level (for
LTO) which is delegated to PassBuilder to construct a pre-defined
optimization pipeline accordingly.
And unfortunately I think there are little ways to customize LTO
optimization pipeline via EP. But maybe this is just a missing feature, if
that's the case feel free to send a patch.

-Min

>
>
> Best Wishes,
>
> Chibin Zhang
>
> 2021.5.8
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
Min-Yih Hsu
Ph.D Student in ICS Department, University of California, Irvine (UCI).
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210508/a8fef4c0/attachment.html>

On Sat, May 8, 2021 at 8:15 AM 张驰斌 via llvm-dev <llvm-dev at
lists.llvm.org>
wrote:
> LLVM Developers,
>
>        Hi!
>
>        There is a scenario where I want to do full program analysis,
> ideally after link time optimization is done and before assembly is
> emitted, when all symbols are in LLVM IR form and contained in a single
> module. But after some time browsing stackoverflow, lld documentation, and
> the mailing lists, I still have little clue how to do it… I am aware of
> the email sent by Y Liu and titled “how to add my own passes to LTO pass”.
> But it doesn’t seem to work for the new pass manager. A few questions:
>
> 1.  The legacy pass manager has an extension point,
> `PassManagerBuilder::EP_FullLinkTimeOptimizationLast`. Unfortunately, I can
> ’t find a alternative in the new pass manager. The `Bye` example seems to
> suggest using `PB.registerVectorizerStartEPCallback`. I am not sure
> whether doing so will result in my module pass being ran at compile or link
> time?
>
Feel free to create a patch to add a PassBuilder equivalent
of EP_FullLinkTimeOptimizationLast.
> 2.  I wonder how does clang communicates passes that need to be run at
> lto to lld. The command line that I attempt to get lto running is `clang
> -fexperimental-new-pass-manager -fpass-plugin=libmymodulepass.so -flto
> -fuse-ld=lld …`. I straced the command for execv calls, and expected to
> see libmymodulepass.so to be passed to ld.lld as an argument, but found out
> it wasn’t. Could  you shed some light on how clang and lld cope with each
> other during lto?
>
I don't know pass plugins that well, but maybe llvm/examples/Bye/Bye.cpp
and its surrounding infrastructure may help?
> 3.  Greping `ld.lld --help` reveals an interesting option:
> `--lto-newpm-passes=<value>`. Does the <value> here mean
built-in pass
> names like `dce`/`mem2reg`, or is it an option to insert our plugin like
> libmymodulepass.so?
>
> I’ve looking into this for a while but haven’t made much progress…Any
> examples / command line on how to do this are greatly appreciated!
>
>
>
> Best Wishes,
>
> Chibin Zhang
>
> 2021.5.8
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210510/92ac5ad6/attachment.html>