llvm dev - Oct 2019 - Exposing the New Pass Manager in the LLVM C API

Hi all,

At some time in the near future we would like to allow for Rust to use the
new PM. As I understand it, Rust uses LLVM through the stable LLVM C API,
which currently only provides an interface to the legacy PM. We wanted to
ask what people think about how we should go about exposing the new PM
through this API. We can think of 3 possibilities for now:

1) Just replace the API to expose the new PM instead of the legacy one.

I imagine we wouldn't want to do this since it would be a pretty sudden
change and there wouldn't be new PM equivalents for the
Initialize/FinalizeFunctionPassManager functions.

2) Add a new interface that exposes the new PM while still keeping the
current one.

The interface would probably be similar to the current one but just not
have the corresponding Initialize/FinalizeFunctionPassManager functions. I
think with this method we could also slowly roll out usage of the new PM
through the C API and eventually deprecate the legacy PM interface in favor
of the new one.

3) Add a build flag that switches between PMs used in the C API.

This could just be some flag in CMake that users can change at build time.

Any other suggestions/thoughts?

- Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191003/a83cdd8c/attachment.html>

On Fri, 4 Oct 2019 at 03:50, Leonard Chan via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi all,
>
> At some time in the near future we would like to allow for Rust to use the
> new PM. As I understand it, Rust uses LLVM through the stable LLVM C API,
> which currently only provides an interface to the legacy PM. We wanted to
> ask what people think about how we should go about exposing the new PM
> through this API. We can think of 3 possibilities for now:
>
Hi Leo,

I agree that increasing coverage of the stable C API (especially for such
essentials as the new PM) is a worthwhile goal that will benefit various
projects using LLVM.

About Rust specifically though, note that we're not restricted to the C
API. Whenever we need something that isn't part of the C API (which is
common), we just create our own bindings for it, i.e., write some C++ code
that wraps whatever we need in a C API (see the src/rustllvm directory). It
also seems unlikely Rust's use cases can be fully served by a C API in the
near term, we already have substantial amounts of custom C++ code
interacting with the legacy PM to do things not available in the C API
(e.g. driving ThinLTO) and this customization need won't go away with the
PM switch.

Regardless, more C API coverage is still nice for Rust, and will be a much
bigger enabler for other projects that don't have their own bindings to
fill the gaps.

> 1) Just replace the API to expose the new PM instead of the legacy one.
>
> I imagine we wouldn't want to do this since it would be a pretty sudden
> change and there wouldn't be new PM equivalents for the
> Initialize/FinalizeFunctionPassManager functions.
>
> 2) Add a new interface that exposes the new PM while still keeping the
> current one.
>
> The interface would probably be similar to the current one but just not
> have the corresponding Initialize/FinalizeFunctionPassManager functions. I
> think with this method we could also slowly roll out usage of the new PM
> through the C API and eventually deprecate the legacy PM interface in favor
> of the new one.
>
+1 for this option.

>
> 3) Add a build flag that switches between PMs used in the C API.
>
> This could just be some flag in CMake that users can change at build time.
>
I don't think this is a good option because many consumers of the C API
don't control the build process of the LLVM they use (e.g., they're
linked
to a distro-provided library). It's also not useful to programs that want
to leave the choice of pass manager to runtime/end users, like clang does
with -fexperimental-new-pass-manager.

Cheers,
Robin

>
> Any other suggestions/thoughts?
>
> - Leo
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191004/aa5407db/attachment.html>

On Fri, Oct 4, 2019 at 3:50 AM Leonard Chan via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> At some time in the near future we would like to allow for Rust to use the
new PM. As I understand it, Rust uses LLVM through the stable LLVM C API, which
currently only provides an interface to the legacy PM. We wanted to ask what
people think about how we should go about exposing the new PM through this API.
We can think of 3 possibilities for now:
+1 to allowing the NewPM to be used from the C API.

The stability expectations on the C interface are higher than those on
the C++ interface. You should treat the C API as stable as much as
possible. This means your option (2) -- adding a new parallel API for
the NewPM, while keeping the old PM API -- is really the only _real_
option that is available here.

Cheers,
Nicolai

>
> 1) Just replace the API to expose the new PM instead of the legacy one.
>
> I imagine we wouldn't want to do this since it would be a pretty sudden
change and there wouldn't be new PM equivalents for the
Initialize/FinalizeFunctionPassManager functions.
>
> 2) Add a new interface that exposes the new PM while still keeping the
current one.
>
> The interface would probably be similar to the current one but just not
have the corresponding Initialize/FinalizeFunctionPassManager functions. I think
with this method we could also slowly roll out usage of the new PM through the C
API and eventually deprecate the legacy PM interface in favor of the new one.
>
> 3) Add a build flag that switches between PMs used in the C API.
>
> This could just be some flag in CMake that users can change at build time.
>
> Any other suggestions/thoughts?
>
> - Leo
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


-- 
Lerne, wie die Welt wirklich ist,
aber vergiss niemals, wie sie sein sollte.

Am Di., 8. Okt. 2019 um 07:33 Uhr schrieb Nicolai Hähnle-Montoro via
llvm-dev <llvm-dev at lists.llvm.org>:
> The stability expectations on the C interface are higher than those on
> the C++ interface. You should treat the C API as stable as much as
> possible. This means your option (2) -- adding a new parallel API for
> the NewPM, while keeping the old PM API -- is really the only _real_
> option that is available here.
Does this mean that we will have to maintain two pass mangers forever?

Michael

Hi,

On 2019-10-04 08:58:14 +0200, Robin Kruppe via llvm-dev
wrote:
> I agree that increasing coverage of the stable C API (especially for such
> essentials as the new PM) is a worthwhile goal that will benefit various
> projects using LLVM.
Same. For projects that need to compile against various versions of
LLVM, and that don't have very devoted resources, it's otherwise quite
hard to keep up.

> It also seems unlikely Rust's use cases can be fully served by a C API
> in the near term, we already have substantial amounts of custom C++
> code interacting with the legacy PM to do things not available in the
> C API (e.g. driving ThinLTO) and this customization need won't go away
> with the PM switch.
It'd kinda be good to encapsule this logic better. There's quite a few
copies of thinlink like logic out there. I e.g. wrote my own crappy
version for postgres' JIT, Rust has a copy, etc.  And it's really hard
to understand from the outside what one even iss supposed to use.


> > 2) Add a new interface that exposes the new PM while still keeping the
> > current one.
> >
> > The interface would probably be similar to the current one but just
not
> > have the corresponding Initialize/FinalizeFunctionPassManager
functions. I
> > think with this method we could also slowly roll out usage of the new
PM
> > through the C API and eventually deprecate the legacy PM interface in
favor
> > of the new one.
> >
> 
> +1 for this option.
Same.

Greetings,

Andres Freund