llvm dev - Nov 2018 - llvm.org pre-built clang significantly slower than apple/xcode clang

Hello LLVM/Clang developers,

We recently switched to use the same clang version on all our platforms.
This included switching from apple-clang from xcode to a pre-built binary
we downloaded from llvm.org. We noticed that this actually came with a
pretty big performance regression in compile times.

If we do the simplest test program like this:

#include <string>
#include <iostream>
int main()
{
    std::cout << "Hello world" << std::endl;
}

and compile that with Xcode Clang (Xcode 10.1 apple-clang
clang-1000.11.45.5):
clang++ test.cpp -o test  0.31s user 0.06s system 97% cpu 0.380 total

with clang 7 binaries found on llvm.org 7.0.0:
~/Downloads/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++ -o test
test.cpp  0.53s user 0.11s system 62% cpu 1.032 total

If we now run that on our whole project:
with xcode clang:
368.17s user 32.00s system 663% cpu 1:00.30 total

with clang 7:
423.31s user 31.65s system 662% cpu 1:08.69 total

That's a pretty hefty difference. Any ideas what can account for this
discrepancy? Does apple-clang contain any special patches or build flags
that differ a lot from the binaries on llvm.org?

I know about PGO - and I guess the best we could do is to get profile data
out of compiling my whole tree and use that when building clang - but this
process seems not very well documented and unsure if this would even help.

Thankful for any ideas or feedback.
Tobias
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181120/41454f81/attachment.html>

I don't know about what sort of secret sauce apple may have (& they
won't
necessarily be able to talk about it) - though mostly I believe they have
fun tional rather than performance patches.

Profile guided optimisation is likely important - Google uses it internally
for compiler releases too.

On Tue, Nov 20, 2018, 6:56 AM Tobias Hieta via cfe-dev <
cfe-dev at lists.llvm.org wrote:
> Hello LLVM/Clang developers,
>
> We recently switched to use the same clang version on all our platforms.
> This included switching from apple-clang from xcode to a pre-built binary
> we downloaded from llvm.org. We noticed that this actually came with a
> pretty big performance regression in compile times.
>
> If we do the simplest test program like this:
>
> #include <string>
> #include <iostream>
> int main()
> {
>     std::cout << "Hello world" << std::endl;
> }
>
> and compile that with Xcode Clang (Xcode 10.1 apple-clang
> clang-1000.11.45.5):
> clang++ test.cpp -o test  0.31s user 0.06s system 97% cpu 0.380 total
>
> with clang 7 binaries found on llvm.org 7.0.0:
> ~/Downloads/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++ -o test
> test.cpp  0.53s user 0.11s system 62% cpu 1.032 total
>
> If we now run that on our whole project:
> with xcode clang:
> 368.17s user 32.00s system 663% cpu 1:00.30 total
>
> with clang 7:
> 423.31s user 31.65s system 662% cpu 1:08.69 total
>
> That's a pretty hefty difference. Any ideas what can account for this
> discrepancy? Does apple-clang contain any special patches or build flags
> that differ a lot from the binaries on llvm.org?
>
> I know about PGO - and I guess the best we could do is to get profile data
> out of compiling my whole tree and use that when building clang - but this
> process seems not very well documented and unsure if this would even help.
>
> Thankful for any ideas or feedback.
> Tobias
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181120/dae09055/attachment.html>

The obvious question is whether the llvm.org builds are using
-DLLVM_ENABLE_ASSERTIONS:OFF
-DCMAKE_BUILD_TYPE:STRING=Release -DLLVM_LINK_LLVM_DYLIB:BOOL=ON which
would improve the load time of the compiler by combining all of the llvm
libs into a single dylib and would eliminate the speed decrease from using
the default use of assertions in the built compiler.
Jack

On Tue, Nov 20, 2018 at 6:56 AM Tobias Hieta via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Hello LLVM/Clang developers,
>
> We recently switched to use the same clang version on all our platforms.
> This included switching from apple-clang from xcode to a pre-built binary
> we downloaded from llvm.org. We noticed that this actually came with a
> pretty big performance regression in compile times.
>
> If we do the simplest test program like this:
>
> #include <string>
> #include <iostream>
> int main()
> {
>     std::cout << "Hello world" << std::endl;
> }
>
> and compile that with Xcode Clang (Xcode 10.1 apple-clang
> clang-1000.11.45.5):
> clang++ test.cpp -o test  0.31s user 0.06s system 97% cpu 0.380 total
>
> with clang 7 binaries found on llvm.org 7.0.0:
> ~/Downloads/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++ -o test
> test.cpp  0.53s user 0.11s system 62% cpu 1.032 total
>
> If we now run that on our whole project:
> with xcode clang:
> 368.17s user 32.00s system 663% cpu 1:00.30 total
>
> with clang 7:
> 423.31s user 31.65s system 662% cpu 1:08.69 total
>
> That's a pretty hefty difference. Any ideas what can account for this
> discrepancy? Does apple-clang contain any special patches or build flags
> that differ a lot from the binaries on llvm.org?
>
> I know about PGO - and I guess the best we could do is to get profile data
> out of compiling my whole tree and use that when building clang - but this
> process seems not very well documented and unsure if this would even help.
>
> Thankful for any ideas or feedback.
> Tobias
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181120/c85bda0b/attachment.html>

Hi Tobias,

If I recall correctly the release builds are built with fdo and
thinlto/lto. The cmake scripts should be in tree somewhere (I'm on my phone
which makes looking hard). Those could be worth 20% for build time that
you're seeing.

Thanks!

-eric

On Tue, Nov 20, 2018, 3:56 AM Tobias Hieta via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Hello LLVM/Clang developers,
>
> We recently switched to use the same clang version on all our platforms.
> This included switching from apple-clang from xcode to a pre-built binary
> we downloaded from llvm.org. We noticed that this actually came with a
> pretty big performance regression in compile times.
>
> If we do the simplest test program like this:
>
> #include <string>
> #include <iostream>
> int main()
> {
>     std::cout << "Hello world" << std::endl;
> }
>
> and compile that with Xcode Clang (Xcode 10.1 apple-clang
> clang-1000.11.45.5):
> clang++ test.cpp -o test  0.31s user 0.06s system 97% cpu 0.380 total
>
> with clang 7 binaries found on llvm.org 7.0.0:
> ~/Downloads/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++ -o test
> test.cpp  0.53s user 0.11s system 62% cpu 1.032 total
>
> If we now run that on our whole project:
> with xcode clang:
> 368.17s user 32.00s system 663% cpu 1:00.30 total
>
> with clang 7:
> 423.31s user 31.65s system 662% cpu 1:08.69 total
>
> That's a pretty hefty difference. Any ideas what can account for this
> discrepancy? Does apple-clang contain any special patches or build flags
> that differ a lot from the binaries on llvm.org?
>
> I know about PGO - and I guess the best we could do is to get profile data
> out of compiling my whole tree and use that when building clang - but this
> process seems not very well documented and unsure if this would even help.
>
> Thankful for any ideas or feedback.
> Tobias
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181120/642a689f/attachment.html>

I don’t think Apple disable assertion on the release build. I remember having
clang and llvm crash because of assertion failure regularly at some point in the
past.
Nowadays, it is far more unusual to get a clang crash, so I can’t tell, but I
doubt they change the configuration.
> Le 20 nov. 2018 à 16:32, Jack Howarth via cfe-dev <cfe-dev at
lists.llvm.org> a écrit :
> 
> The obvious question is whether the llvm.org <http://llvm.org/>
builds are using -DLLVM_ENABLE_ASSERTIONS:OFF -DCMAKE_BUILD_TYPE:STRING=Release
-DLLVM_LINK_LLVM_DYLIB:BOOL=ON which would improve the load time of the compiler
by combining all of the llvm libs into a single dylib and would eliminate the
speed decrease from using the default use of assertions in the built compiler.
>        Jack
> 
> On Tue, Nov 20, 2018 at 6:56 AM Tobias Hieta via cfe-dev <cfe-dev at
lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
> Hello LLVM/Clang developers,
> 
> We recently switched to use the same clang version on all our platforms.
This included switching from apple-clang from xcode to a pre-built binary we
downloaded from llvm.org <http://llvm.org/>. We noticed that this actually
came with a pretty big performance regression in compile times.
> 
> If we do the simplest test program like this:
> 
> #include <string>
> #include <iostream>
> int main()
> {
>     std::cout << "Hello world" << std::endl;
> }
> 
> and compile that with Xcode Clang (Xcode 10.1 apple-clang
clang-1000.11.45.5):
> clang++ test.cpp -o test  0.31s user 0.06s system 97% cpu 0.380 total
> 
> with clang 7 binaries found on llvm.org <http://llvm.org/> 7.0.0:
> ~/Downloads/clang+llvm-7.0.0-x86_64-apple-darwin/bin/clang++ -o test
test.cpp  0.53s user 0.11s system 62% cpu 1.032 total
> 
> If we now run that on our whole project:
> with xcode clang:
> 368.17s user 32.00s system 663% cpu 1:00.30 total
> 
> with clang 7:
> 423.31s user 31.65s system 662% cpu 1:08.69 total
> 
> That's a pretty hefty difference. Any ideas what can account for this
discrepancy? Does apple-clang contain any special patches or build flags that
differ a lot from the binaries on llvm.org <http://llvm.org/>?
> 
> I know about PGO - and I guess the best we could do is to get profile data
out of compiling my whole tree and use that when building clang - but this
process seems not very well documented and unsure if this would even help.
> 
> Thankful for any ideas or feedback.
> Tobias
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181120/43149a93/attachment.html>