llvm dev - Feb 2015 - [LLVMdev] Current state-of-the-art in whole program optimization

As I understand it, clang currently has the -flto flag which activates some
degree of whole program optimization. However, this requires clang to be
presented with all the source files at the same time. This is easy enough
to arrange in a greenfield project, but may be trickier with an existing
program.

On Unix, C and C++ programs are usually compiled with makefiles (whether
handwritten or auto generated), and these typically call the compiler on
one source file at a time and then call the system linker only after
compilation has proceeded to machine code object files. Thus, even with
clang as a drop-in replacement for GCC, there is the challenge of obtaining
exact compilation instructions from the makefile and then modifying these
to allow clang to see all the files at once, no later than bitcode stage.

On Windows, it's common to have a Visual Studio project file that msbuild
can use. This sometimes does call the compiler on multiple files at the
same time; then there's the question of how to get clang-cl invoked with
suitable options.

At least that's my thinking on this aspect of the problem so far; there are
doubtless things I'm missing. What's currently being done in this area?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150202/31449423/attachment.html>

On Mon, Feb 2, 2015 at 2:01 PM, Russell Wallace <russell.wallace at
gmail.com>
wrote:
> As I understand it, clang currently has the -flto flag which activates
> some degree of whole program optimization. However, this requires clang to
> be presented with all the source files at the same time. This is easy
> enough to arrange in a greenfield project, but may be trickier with an
> existing program.
>
> On Unix, C and C++ programs are usually compiled with makefiles (whether
> handwritten or auto generated), and these typically call the compiler on
> one source file at a time and then call the system linker only after
> compilation has proceeded to machine code object files. Thus, even with
> clang as a drop-in replacement for GCC, there is the challenge of obtaining
> exact compilation instructions from the makefile and then modifying these
> to allow clang to see all the files at once, no later than bitcode stage.
>
This is generally already addressed by just making 'object' files that
are
actually LLVM bitcode. I forget which flags are needed, exactly, but you'll
also need a linker that's aware of this trick (there's an LLVM plugin
for
the gold linker, for example).

I forget which flags you have to pass to clang to make it do this (maybe it
is just -flto) but once you pass that flag and have an LLVM-aware linker,
it should 'just work' at least as far as the build system issues are
concerned.

- David

>
> On Windows, it's common to have a Visual Studio project file that
msbuild
> can use. This sometimes does call the compiler on multiple files at the
> same time; then there's the question of how to get clang-cl invoked
with
> suitable options.
>
> At least that's my thinking on this aspect of the problem so far; there
> are doubtless things I'm missing. What's currently being done in
this area?
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150202/81487164/attachment.html>

On 2 February 2015 at 14:01, Russell Wallace <russell.wallace at
gmail.com>
wrote:
> As I understand it, clang currently has the -flto flag which activates
> some degree of whole program optimization. However, this requires clang to
> be presented with all the source files at the same time.
>
Not at all. The -flto flag causes clang to emit a half-compiled .o file
that is not at all a real object file. It contains no object code and is
not elf, mach-o nor pe/coff. It's an llvm bitcode file.

The second half of this story is that you're going to ask the linker to
compile a program with these non-object file .o files, and the linker needs
to handle it. On darwin "ld" is linked against libLTO [
http://llvm.org/docs/LinkTimeOptimization.html ]. On linux, you can use
"gold" (the newer ld in binutils) with a plugin [
http://llvm.org/docs/GoldPlugin.html ]. I'm told the plan is to make lld
support llvm lto when building windows/pecoff files, but this may not be
ready yet.

This is easy enough to arrange in a greenfield project, but may be
trickier
> with an existing program.
>
> On Unix, C and C++ programs are usually compiled with makefiles (whether
> handwritten or auto generated), and these typically call the compiler on
> one source file at a time and then call the system linker only after
> compilation has proceeded to machine code object files. Thus, even with
> clang as a drop-in replacement for GCC, there is the challenge of obtaining
> exact compilation instructions from the makefile and then modifying these
> to allow clang to see all the files at once, no later than bitcode stage.
>
The gold plugin webpage covers exactly this case, including a section on
how to make llvm LTO work with autotools (configure+make) projects.

Nick

On Windows, it's common to have a Visual Studio project file that
msbuild
> can use. This sometimes does call the compiler on multiple files at the
> same time; then there's the question of how to get clang-cl invoked
with
> suitable options.
>
> At least that's my thinking on this aspect of the problem so far; there
> are doubtless things I'm missing. What's currently being done in
this area?
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150202/d07b3944/attachment.html>