llvm dev - Oct 2017 - RFC: Switching to the new pass manager by default

Sorry, by debug build I actually meant asserts enabled.  Thus, this 
issue can show up in either a debug or release build, if asserts are 
enabled.


On 10/26/2017 4:05 PM, Chad Rosier via llvm-dev wrote:
>
> Chandler/All,
>
> We've just started testing the new pass manager this week and we ran 
> into a 548x slowdown (i.e., 6.28s to 3443.83s) for one of the files 
> from SPEC2017/blender.  The issue arises only in debug builds due to 
> the numerous calls to RefSCC::verify() and SCC::verify() in the 
> LazyCallGraph implementation.  Would it make sense to start 
> predicating these calls with the EXPENSIVE_CHECKS macro, rather than 
> NDEBUG?
>
>  Chad
>
>
> On 10/18/2017 2:50 AM, Chandler Carruth via llvm-dev wrote:
>> Greetings everyone!
>>
>> The new pass manager is getting extremely close to the point where 
>> I'm not aware of any significant outstanding work needed, and
I'd
>> like to see what else would be needed to enable it by default. Here 
>> are the current functionality I'm aware of outstanding:
>>
>> 1) Does not do non-trivial loop unswitching. Majority of this is in 
>> https://reviews.llvm.org/D34200 but will need one or two small 
>> follow-ups.
>>
>> 2) Currently, sanitizers don't work correctly with it. Thanks to
the
>> work of others, the missing infrastructure has been added and I'll 
>> send a patch to wire this up this week.
>>
>> 3) Missing support for 'optnone'. I've been working on
this, but the
>> existing testing wasn't as thorough as I wanted, so it is going 
>> slowly. I've got about 1/4 of this implemented and should have 
>> patches this week or next.
>>
>> 4) Missing opt-bisect (or similar) facility. This looks pretty 
>> trivial to add, but I've not even started. If anyone is interested
in
>> it, go for it. We might even be able to do something simpler using 
>> the generic debug counters and get equivalent functionality.
>>
>> ... that's it?
>>
>> Optimization quality / run-time performance:
>> - We've been using it at Google extensively and are very happy with
>> the optimization quality. Benchmarks look *very* good here.
>> - More data from other users would be important.
>> - You can try it out with `-fexperimental-new-pass-manager` to Clang
>>
>> Compile-time performance:
>> - Sometimes *much* better due to cached analyses.
>> - Sometimes worse, typically due to more / different inlining in turn 
>> running main pipeline (GVN + InstCombine) more times or over more code.
>> - Overall somewhat a wash, but the increased compile times typically 
>> due to the optimizer "trying" harder, so not too concerning
on our end.
>> - Again, more feedback from other users good: 
>> `-fexperimental-new-pass-manager` to Clang
>>
>> Once the four missing things land, I'll also happily work on 
>> collecting some of the basics on the test-suite and CTMark. But I 
>> suspect more "in the wild" data would really be useful here
given the
>> significance of the change.
>>
>> Thoughts? What else (beyond the four items above and feedback on 
>> run-time and compile-time) would folks like to see?
>>
>> Once this happens, I'll also be preparing some batch, mechanical 
>> updates to the test suite to primarily use the new pass manager. Also 
>> there is lots of documentation updates that will be needed here.
>>
>> -Chandler
>>
>> PS: I'll be sending a note to cfe-dev as a "heads up"
about this
>> discussion as in some ways, the default flip is mostly a Clang 
>> default flip. But hopefully our doc updates will trigger this being 
>> "perceived" as the default for other frontends, and I'll
try to reach
>> out to other major frontends as well (Swift and Rust are on my radar, 
>> and I've already started talking with Philip Reames about their 
>> Falcon JIT).
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171026/43f2ffe2/attachment.html>

I thought most of the extremely superlinear asserts were already behind
EXPENSIVE_CHECKS, but we can add this one if you have a test case. Could
you file a bug w/ a test case? I'd also be happy to try and just make the
verify a bit less egregious. But I don't have that version of SPEC....

On Thu, Oct 26, 2017 at 1:16 PM Chad Rosier <mcrosier at codeaurora.org>
wrote:
> Sorry, by debug build I actually meant asserts enabled.  Thus, this issue
> can show up in either a debug or release build, if asserts are enabled.
>
> On 10/26/2017 4:05 PM, Chad Rosier via llvm-dev wrote:
>
> Chandler/All,
>
> We've just started testing the new pass manager this week and we ran
into
> a 548x slowdown (i.e., 6.28s to 3443.83s) for one of the files from
> SPEC2017/blender.  The issue arises only in debug builds due to the
> numerous calls to RefSCC::verify() and SCC::verify() in the LazyCallGraph
> implementation.  Would it make sense to start predicating these calls with
> the EXPENSIVE_CHECKS macro, rather than NDEBUG?
>
>  Chad
>
> On 10/18/2017 2:50 AM, Chandler Carruth via llvm-dev wrote:
>
> Greetings everyone!
>
> The new pass manager is getting extremely close to the point where I'm
not
> aware of any significant outstanding work needed, and I'd like to see
what
> else would be needed to enable it by default. Here are the current
> functionality I'm aware of outstanding:
>
> 1) Does not do non-trivial loop unswitching. Majority of this is in
> https://reviews.llvm.org/D34200 but will need one or two small follow-ups.
>
> 2) Currently, sanitizers don't work correctly with it. Thanks to the
work
> of others, the missing infrastructure has been added and I'll send a
patch
> to wire this up this week.
>
> 3) Missing support for 'optnone'. I've been working on this,
but the
> existing testing wasn't as thorough as I wanted, so it is going slowly.
> I've got about 1/4 of this implemented and should have patches this
week or
> next.
>
> 4) Missing opt-bisect (or similar) facility. This looks pretty trivial to
> add, but I've not even started. If anyone is interested in it, go for
it.
> We might even be able to do something simpler using the generic debug
> counters and get equivalent functionality.
>
> ... that's it?
>
> Optimization quality / run-time performance:
> - We've been using it at Google extensively and are very happy with the
> optimization quality. Benchmarks look *very* good here.
> - More data from other users would be important.
> - You can try it out with `-fexperimental-new-pass-manager` to Clang
>
> Compile-time performance:
> - Sometimes *much* better due to cached analyses.
> - Sometimes worse, typically due to more / different inlining in turn
> running main pipeline (GVN + InstCombine) more times or over more code.
> - Overall somewhat a wash, but the increased compile times typically due
> to the optimizer "trying" harder, so not too concerning on our
end.
> - Again, more feedback from other users good:
> `-fexperimental-new-pass-manager` to Clang
>
> Once the four missing things land, I'll also happily work on collecting
> some of the basics on the test-suite and CTMark. But I suspect more
"in the
> wild" data would really be useful here given the significance of the
change.
>
> Thoughts? What else (beyond the four items above and feedback on run-time
> and compile-time) would folks like to see?
>
> Once this happens, I'll also be preparing some batch, mechanical
updates
> to the test suite to primarily use the new pass manager. Also there is lots
> of documentation updates that will be needed here.
>
> -Chandler
>
> PS: I'll be sending a note to cfe-dev as a "heads up" about
this
> discussion as in some ways, the default flip is mostly a Clang default
> flip. But hopefully our doc updates will trigger this being
"perceived" as
> the default for other frontends, and I'll try to reach out to other
major
> frontends as well (Swift and Rust are on my radar, and I've already
started
> talking with Philip Reames about their Falcon JIT).
>
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171026/bf625e69/attachment-0001.html>

Sure, I'll file a bug and investigate a bit in the morning.  I'll also 
see if I can't find a similar regression with one of the llvm test suite 
tests, so it will be easier to reproduce on your end.


On 10/26/2017 4:52 PM, Chandler Carruth wrote:
> I thought most of the extremely superlinear asserts were already 
> behind EXPENSIVE_CHECKS, but we can add this one if you have a test 
> case. Could you file a bug w/ a test case? I'd also be happy to try 
> and just make the verify a bit less egregious. But I don't have that 
> version of SPEC....
>
> On Thu, Oct 26, 2017 at 1:16 PM Chad Rosier <mcrosier at codeaurora.org 
> <mailto:mcrosier at codeaurora.org>> wrote:
>
>     Sorry, by debug build I actually meant asserts enabled. Thus, this
>     issue can show up in either a debug or release build, if asserts
>     are enabled.
>
>
>     On 10/26/2017 4:05 PM, Chad Rosier via llvm-dev wrote:
>>
>>     Chandler/All,
>>
>>     We've just started testing the new pass manager this week and
we
>>     ran into a 548x slowdown (i.e., 6.28s to 3443.83s) for one of the
>>     files from SPEC2017/blender. The issue arises only in debug
>>     builds due to the numerous calls to RefSCC::verify() and
>>     SCC::verify() in the LazyCallGraph implementation.  Would it make
>>     sense to start predicating these calls with the EXPENSIVE_CHECKS
>>     macro, rather than NDEBUG?
>>
>>      Chad
>>
>>
>>     On 10/18/2017 2:50 AM, Chandler Carruth via llvm-dev wrote:
>>>     Greetings everyone!
>>>
>>>     The new pass manager is getting extremely close to the point
>>>     where I'm not aware of any significant outstanding work
needed,
>>>     and I'd like to see what else would be needed to enable it
by
>>>     default. Here are the current functionality I'm aware of
>>>     outstanding:
>>>
>>>     1) Does not do non-trivial loop unswitching. Majority of this
is
>>>     in https://reviews.llvm.org/D34200 but will need one or two
>>>     small follow-ups.
>>>
>>>     2) Currently, sanitizers don't work correctly with it.
Thanks to
>>>     the work of others, the missing infrastructure has been added
>>>     and I'll send a patch to wire this up this week.
>>>
>>>     3) Missing support for 'optnone'. I've been working
on this, but
>>>     the existing testing wasn't as thorough as I wanted, so it
is
>>>     going slowly. I've got about 1/4 of this implemented and
should
>>>     have patches this week or next.
>>>
>>>     4) Missing opt-bisect (or similar) facility. This looks pretty
>>>     trivial to add, but I've not even started. If anyone is
>>>     interested in it, go for it. We might even be able to do
>>>     something simpler using the generic debug counters and get
>>>     equivalent functionality.
>>>
>>>     ... that's it?
>>>
>>>     Optimization quality / run-time performance:
>>>     - We've been using it at Google extensively and are very
happy
>>>     with the optimization quality. Benchmarks look *very* good
here.
>>>     - More data from other users would be important.
>>>     - You can try it out with `-fexperimental-new-pass-manager` to
Clang
>>>
>>>     Compile-time performance:
>>>     - Sometimes *much* better due to cached analyses.
>>>     - Sometimes worse, typically due to more / different inlining
in
>>>     turn running main pipeline (GVN + InstCombine) more times or
>>>     over more code.
>>>     - Overall somewhat a wash, but the increased compile times
>>>     typically due to the optimizer "trying" harder, so
not too
>>>     concerning on our end.
>>>     - Again, more feedback from other users good:
>>>     `-fexperimental-new-pass-manager` to Clang
>>>
>>>     Once the four missing things land, I'll also happily work
on
>>>     collecting some of the basics on the test-suite and CTMark. But
>>>     I suspect more "in the wild" data would really be
useful here
>>>     given the significance of the change.
>>>
>>>     Thoughts? What else (beyond the four items above and feedback
on
>>>     run-time and compile-time) would folks like to see?
>>>
>>>     Once this happens, I'll also be preparing some batch,
mechanical
>>>     updates to the test suite to primarily use the new pass
manager.
>>>     Also there is lots of documentation updates that will be needed
>>>     here.
>>>
>>>     -Chandler
>>>
>>>     PS: I'll be sending a note to cfe-dev as a "heads
up" about this
>>>     discussion as in some ways, the default flip is mostly a Clang
>>>     default flip. But hopefully our doc updates will trigger this
>>>     being "perceived" as the default for other frontends,
and I'll
>>>     try to reach out to other major frontends as well (Swift and
>>>     Rust are on my radar, and I've already started talking with
>>>     Philip Reames about their Falcon JIT).
>>>
>>>
>>>     _______________________________________________
>>>     LLVM Developers mailing list
>>>     llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>>     _______________________________________________
>>     LLVM Developers mailing list
>>     llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171026/fd7f13c4/attachment.html>

Hi Chandler,

See: https://bugs.llvm.org/show_bug.cgi?id=35110

Let me know if I can be of further assistance.

  Chad

On 2017-10-26 16:52, Chandler Carruth wrote:
> I thought most of the extremely superlinear asserts were already
> behind EXPENSIVE_CHECKS, but we can add this one if you have a test
> case. Could you file a bug w/ a test case? I'd also be happy to try
> and just make the verify a bit less egregious. But I don't have that
> version of SPEC....
> 
> On Thu, Oct 26, 2017 at 1:16 PM Chad Rosier <mcrosier at
codeaurora.org>
> wrote:
> 
>> Sorry, by debug build I actually meant asserts enabled.  Thus, this
>> issue can show up in either a debug or release build, if asserts are
>> enabled.
>> 
>> On 10/26/2017 4:05 PM, Chad Rosier via llvm-dev wrote:
>> 
>> Chandler/All,
>> 
>> We've just started testing the new pass manager this week and we
ran
>> into a 548x slowdown (i.e., 6.28s to 3443.83s) for one of the files
>> from SPEC2017/blender.  The issue arises only in debug builds due to
>> the numerous calls to RefSCC::verify() and SCC::verify() in the
>> LazyCallGraph implementation.  Would it make sense to start
>> predicating these calls with the EXPENSIVE_CHECKS macro, rather than
>> NDEBUG?
>> 
>> Chad
>> 
>> On 10/18/2017 2:50 AM, Chandler Carruth via llvm-dev wrote:
>> 
>> Greetings everyone!
>> 
>> The new pass manager is getting extremely close to the point where
>> I'm not aware of any significant outstanding work needed, and
I'd
>> like to see what else would be needed to enable it by default. Here
>> are the current functionality I'm aware of outstanding:
>> 
>> 1) Does not do non-trivial loop unswitching. Majority of this is in
>> https://reviews.llvm.org/D34200 but will need one or two small
>> follow-ups.
>> 
>> 2) Currently, sanitizers don't work correctly with it. Thanks to
the
>> work of others, the missing infrastructure has been added and I'll
>> send a patch to wire this up this week.
>> 
>> 3) Missing support for 'optnone'. I've been working on
this, but the
>> existing testing wasn't as thorough as I wanted, so it is going
>> slowly. I've got about 1/4 of this implemented and should have
>> patches this week or next.
>> 
>> 4) Missing opt-bisect (or similar) facility. This looks pretty
>> trivial to add, but I've not even started. If anyone is interested
>> in it, go for it. We might even be able to do something simpler
>> using the generic debug counters and get equivalent functionality.
>> 
>> ... that's it?
>> 
>> Optimization quality / run-time performance:
>> - We've been using it at Google extensively and are very happy with
>> the optimization quality. Benchmarks look *very* good here.
>> - More data from other users would be important.
>> - You can try it out with `-fexperimental-new-pass-manager` to Clang
>> 
>> 
>> Compile-time performance:
>> - Sometimes *much* better due to cached analyses.
>> - Sometimes worse, typically due to more / different inlining in
>> turn running main pipeline (GVN + InstCombine) more times or over
>> more code.
>> - Overall somewhat a wash, but the increased compile times typically
>> due to the optimizer "trying" harder, so not too concerning
on our
>> end.
>> - Again, more feedback from other users good:
>> `-fexperimental-new-pass-manager` to Clang
>> 
>> Once the four missing things land, I'll also happily work on
>> collecting some of the basics on the test-suite and CTMark. But I
>> suspect more "in the wild" data would really be useful here
given
>> the significance of the change.
>> 
>> Thoughts? What else (beyond the four items above and feedback on
>> run-time and compile-time) would folks like to see?
>> 
>> Once this happens, I'll also be preparing some batch, mechanical
>> updates to the test suite to primarily use the new pass manager.
>> Also there is lots of documentation updates that will be needed
>> here.
>> 
>> -Chandler
>> 
>> PS: I'll be sending a note to cfe-dev as a "heads up"
about this
>> discussion as in some ways, the default flip is mostly a Clang
>> default flip. But hopefully our doc updates will trigger this being
>> "perceived" as the default for other frontends, and I'll
try to
>> reach out to other major frontends as well (Swift and Rust are on my
>> radar, and I've already started talking with Philip Reames about
>> their Falcon JIT).
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev