llvm dev - Aug 2019 - Switching to the New Pass Manager by Default

Hello all,

As of now, all LLVM and Clang tests have been updated/addressed to run
under the new/experimental pass manager (at least the ones that failed when
using the new PM).

For those who aren't aware of what the new pass manager (PM) is, the tl;dr
is that this will serve as a replacement for the legacy PM, and promises
faster build times by restructuring how passes are run over IR units. For
some time, when someone wants to create a new pass, they may need to
implement it for both pass managers since the legacy one is what is enabled
by default. Passes that were initially made under the new PM have also
slowly been ported over time.

Now that LLVM 9.0.0 has branched, we have about 6 months before the next
release. We think it would be a good idea to take this opportunity and make
the new PM the default one which gives us enough time to work out any kinks
that might come out of this switch before LLVM 10.0.0 branch point. We
suspect that there may be other LLVM projects that will be affected by
this, probably from unported passes.

Does anyone have any opinions on this?

Off the top of my head, the next immediate work would be to update the docs
with instructions on how to write or port a new PM pass, and address any
breakages for other LLVM projects.

Thanks,
Leonard
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190802/34dec2d3/attachment.html>

On Fri, Aug 2, 2019 at 10:25 AM Leonard Chan via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello all,
>
> As of now, all LLVM and Clang tests have been updated/addressed to run
> under the new/experimental pass manager (at least the ones that failed when
> using the new PM).
>
> For those who aren't aware of what the new pass manager (PM) is, the
tl;dr
> is that this will serve as a replacement for the legacy PM, and promises
> faster build times by restructuring how passes are run over IR units. For
> some time, when someone wants to create a new pass, they may need to
> implement it for both pass managers since the legacy one is what is enabled
> by default. Passes that were initially made under the new PM have also
> slowly been ported over time.
>
> Now that LLVM 9.0.0 has branched, we have about 6 months before the next
> release. We think it would be a good idea to take this opportunity and make
> the new PM the default one which gives us enough time to work out any kinks
> that might come out of this switch before LLVM 10.0.0 branch point. We
> suspect that there may be other LLVM projects that will be affected by
> this, probably from unported passes.
>
> Does anyone have any opinions on this?
>
I just wanted to say: huge +1 from me, and a huge thanks to you and several
others who have been working tirelessly to track down the long tail of
issues and get them resolved so we could get to this point.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190802/5a8e9291/attachment.html>

On 8/2/19 12:25 PM, Leonard Chan via llvm-dev wrote:
Hello all,

As of now, all LLVM and Clang tests have been updated/addressed to run under the
new/experimental pass manager (at least the ones that failed when using the new
PM).

For those who aren't aware of what the new pass manager (PM) is, the tl;dr
is that this will serve as a replacement for the legacy PM, and promises faster
build times by restructuring how passes are run over IR units. For some time,
when someone wants to create a new pass, they may need to implement it for both
pass managers since the legacy one is what is enabled by default. Passes that
were initially made under the new PM have also slowly been ported over time.

Now that LLVM 9.0.0 has branched, we have about 6 months before the next
release. We think it would be a good idea to take this opportunity and make the
new PM the default one which gives us enough time to work out any kinks that
might come out of this switch before LLVM 10.0.0 branch point. We suspect that
there may be other LLVM projects that will be affected by this, probably from
unported passes.

Does anyone have any opinions on this?


A huge thank you to everyone who has worked on this! This is great news.
You're exactly right: we should change the default early in the release
cycle. +1 to doing this.

 -Hal


Off the top of my head, the next immediate work would be to update the docs with
instructions on how to write or port a new PM pass, and address any breakages
for other LLVM projects.

Thanks,
Leonard



_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


--
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190802/ac12e33b/attachment.html>

On Aug 2, 2019, at 10:25 AM, Leonard Chan via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hello all,
> 
> As of now, all LLVM and Clang tests have been updated/addressed to run
under the new/experimental pass manager (at least the ones that failed when
using the new PM).
> 
> For those who aren't aware of what the new pass manager (PM) is, the
tl;dr is that this will serve as a replacement for the legacy PM, and promises
faster build times by restructuring how passes are run over IR units. For some
time, when someone wants to create a new pass, they may need to implement it for
both pass managers since the legacy one is what is enabled by default. Passes
that were initially made under the new PM have also slowly been ported over
time.
> 
> Now that LLVM 9.0.0 has branched, we have about 6 months before the next
release. We think it would be a good idea to take this opportunity and make the
new PM the default one which gives us enough time to work out any kinks that
might come out of this switch before LLVM 10.0.0 branch point. We suspect that
there may be other LLVM projects that will be affected by this, probably from
unported passes.
Hooray!! You have no idea how happy this makes me.  Thank you for the great
work!

-Chris

Thanks for all the hard work that went into getting it here.

+1

Can you spell out what performance validation has been done? Informally, 
I know there's been quite a bit, but getting a summary in one place for 
later reference would be super helpful.

Philip

On 8/2/19 10:25 AM, Leonard Chan via llvm-dev wrote:
> Hello all,
>
> As of now, all LLVM and Clang tests have been updated/addressed to run 
> under the new/experimental pass manager (at least the ones that failed 
> when using the new PM).
>
> For those who aren't aware of what the new pass manager (PM) is, the 
> tl;dr is that this will serve as a replacement for the legacy PM, and 
> promises faster build times by restructuring how passes are run over 
> IR units. For some time, when someone wants to create a new pass, they 
> may need to implement it for both pass managers since the legacy one 
> is what is enabled by default. Passes that were initially made under 
> the new PM have also slowly been ported over time.
>
> Now that LLVM 9.0.0 has branched, we have about 6 months before the 
> next release. We think it would be a good idea to take this 
> opportunity and make the new PM the default one which gives us enough 
> time to work out any kinks that might come out of this switch before 
> LLVM 10.0.0 branch point. We suspect that there may be other LLVM 
> projects that will be affected by this, probably from unported passes.
>
> Does anyone have any opinions on this?
>
> Off the top of my head, the next immediate work would be to update the 
> docs with instructions on how to write or port a new PM pass, and 
> address any breakages for other LLVM projects.
>
> Thanks,
> Leonard
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190802/1fff7e30/attachment.html>

I think so far most individual projects have their own statistics on build
time/performance impact on switching to the new PM, but I agree that there
should at least be one place that people can reference on the impact.

On Fri, Aug 2, 2019 at 1:14 PM Philip Reames <listmail at
philipreames.com>
wrote:
> Thanks for all the hard work that went into getting it here.
>
> +1
>
> Can you spell out what performance validation has been done?  Informally,
> I know there's been quite a bit, but getting a summary in one place for
> later reference would be super helpful.
>
> Philip
> On 8/2/19 10:25 AM, Leonard Chan via llvm-dev wrote:
>
> Hello all,
>
> As of now, all LLVM and Clang tests have been updated/addressed to run
> under the new/experimental pass manager (at least the ones that failed when
> using the new PM).
>
> For those who aren't aware of what the new pass manager (PM) is, the
tl;dr
> is that this will serve as a replacement for the legacy PM, and promises
> faster build times by restructuring how passes are run over IR units. For
> some time, when someone wants to create a new pass, they may need to
> implement it for both pass managers since the legacy one is what is enabled
> by default. Passes that were initially made under the new PM have also
> slowly been ported over time.
>
> Now that LLVM 9.0.0 has branched, we have about 6 months before the next
> release. We think it would be a good idea to take this opportunity and make
> the new PM the default one which gives us enough time to work out any kinks
> that might come out of this switch before LLVM 10.0.0 branch point. We
> suspect that there may be other LLVM projects that will be affected by
> this, probably from unported passes.
>
> Does anyone have any opinions on this?
>
> Off the top of my head, the next immediate work would be to update the
> docs with instructions on how to write or port a new PM pass, and address
> any breakages for other LLVM projects.
>
> Thanks,
> Leonard
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190802/052493ea/attachment.html>

+1 from me, but I think we should keep some bots running on the old
pass manager, to make sure the failures aren't just because of the new
pass manager's structure.

At least for a while, until we have enough coverage on all buildbots
and feedback from enough downstream projects that all is well.

On Fri, 2 Aug 2019 at 18:25, Leonard Chan via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Hello all,
>
> As of now, all LLVM and Clang tests have been updated/addressed to run
under the new/experimental pass manager (at least the ones that failed when
using the new PM).
>
> For those who aren't aware of what the new pass manager (PM) is, the
tl;dr is that this will serve as a replacement for the legacy PM, and promises
faster build times by restructuring how passes are run over IR units. For some
time, when someone wants to create a new pass, they may need to implement it for
both pass managers since the legacy one is what is enabled by default. Passes
that were initially made under the new PM have also slowly been ported over
time.
>
> Now that LLVM 9.0.0 has branched, we have about 6 months before the next
release. We think it would be a good idea to take this opportunity and make the
new PM the default one which gives us enough time to work out any kinks that
might come out of this switch before LLVM 10.0.0 branch point. We suspect that
there may be other LLVM projects that will be affected by this, probably from
unported passes.
>
> Does anyone have any opinions on this?
>
> Off the top of my head, the next immediate work would be to update the docs
with instructions on how to write or port a new PM pass, and address any
breakages for other LLVM projects.
>
> Thanks,
> Leonard
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 08/02/2019 10:25 AM, Leonard Chan via llvm-dev wrote:
> Hello all,
> 
> As of now, all LLVM and Clang tests have been updated/addressed to run
under the new/experimental pass manager (at least the ones that failed when
using the new PM).
> 
> For those who aren't aware of what the new pass manager (PM) is, the
tl;dr is that this will serve as a replacement for the legacy PM, and promises
faster build times by restructuring how passes are run over IR units. For some
time, when someone wants to create a new pass, they may need to implement it for
both pass managers since the legacy one is what is enabled by default. Passes
that were initially made under the new PM have also slowly been ported over
time.
> 
> Now that LLVM 9.0.0 has branched, we have about 6 months before the next
release. We think it would be a good idea to take this opportunity and make the
new PM the default one which gives us enough time to work out any kinks that
might come out of this switch before LLVM 10.0.0 branch point. We suspect that
there may be other LLVM projects that will be affected by this, probably from
unported passes.
> 
> Does anyone have any opinions on this?
> 
What is involved in making the new pass manager the default?
Does it just mean updating clang, or will the C API functions,
like LLVMCreatePassManager(), etc. be updated too?

-Tom
> Off the top of my head, the next immediate work would be to update the docs
with instructions on how to write or port a new PM pass, and address any
breakages for other LLVM projects.
> 
> Thanks,
> Leonard
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

One option is just to change the default
for ENABLE_EXPERIMENTAL_NEW_PASS_MANAGER in Clang's CMakeLists.txt:
https://github.com/llvm/llvm-project/blob/0930643ff6f1684bf234a97f6a3b76bf8e30b0a1/clang/CMakeLists.txt#L236.
That approach allows developers and bots to easily switch back to the
legacy pass manager by simply
passing -DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=OFF to their CMake build.

On Tue, Aug 6, 2019 at 2:45 PM Tom Stellard via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 08/02/2019 10:25 AM, Leonard Chan via llvm-dev wrote:
> > Hello all,
> >
> > As of now, all LLVM and Clang tests have been updated/addressed to run
> under the new/experimental pass manager (at least the ones that failed when
> using the new PM).
> >
> > For those who aren't aware of what the new pass manager (PM) is,
the
> tl;dr is that this will serve as a replacement for the legacy PM, and
> promises faster build times by restructuring how passes are run over IR
> units. For some time, when someone wants to create a new pass, they may
> need to implement it for both pass managers since the legacy one is what is
> enabled by default. Passes that were initially made under the new PM have
> also slowly been ported over time.
> >
> > Now that LLVM 9.0.0 has branched, we have about 6 months before the
next
> release. We think it would be a good idea to take this opportunity and make
> the new PM the default one which gives us enough time to work out any kinks
> that might come out of this switch before LLVM 10.0.0 branch point. We
> suspect that there may be other LLVM projects that will be affected by
> this, probably from unported passes.
> >
> > Does anyone have any opinions on this?
> >
>
> What is involved in making the new pass manager the default?
> Does it just mean updating clang, or will the C API functions,
> like LLVMCreatePassManager(), etc. be updated too?
>
> -Tom
>
> > Off the top of my head, the next immediate work would be to update the
> docs with instructions on how to write or port a new PM pass, and address
> any breakages for other LLVM projects.
> >
> > Thanks,
> > Leonard
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190806/0d711aad/attachment.html>