llvm dev - Aug 2016 - [Progress Update] LLVM Runtimes Subdirectory

Hi LLVM-Dev,

Over the past week or so I’ve been working on build system improvements around
the LLVM runtimes directory. You may remember this from the LLVM-Dev discussion
(http://lists.llvm.org/pipermail/llvm-dev/2016-June/100859.html), or the patch
review discussion (https://reviews.llvm.org/D20992) back in June.

Recently, I’ve made a handful of new changes that actually make it useful, and I
wanted to blast out this update and encourage people to give it a try.

Let me start with an update on the current state.

First and foremost, this is not currently supported with multi-configuration
generators (Visual Studio or Xcode). I’ve done my testing with Ninja, but I
expect it will work the same with Make.

Currently the runtimes directory is supported for building compiler-rt and
libcxx. You can enable the new build paths either by putting your compiler-rt
and libcxx checkouts into the runtimes subdirectory or by symlinking the
directories. I do plan to support variables for overriding paths to the runtimes
similar to how we support the LLVM projects directory today, but I haven’t
gotten there yet (let me know if this prevents you from using the feature and
I’ll prioritize it).

With support for libcxx I made the runtimes build generate top-level targets for
each runtime project. The runtime project targets are named for the project with
leading ‘lib’ prefixes removed (i.e. compiler-rt, cxx). Also note, for the
runtimes subdirectory compiler-rt is treated as a separate project from the
builtins. One of the current limitations of this system is that I still don’t
have a solution for building and running the builtins tests.

I’ve also added support to the runtimes for generating a target that runs all
runtime tests from a single lit invocation. This target is exposed in the build
as “check-runtimes”, and it is connected as a dependency of LLVM’s “check-all”
target. This makes it so “check-all” includes the runtime test, but not under
the same lit invocation. I have ideas for how to fix that which I’ll be working
on.

The last new feature I’ve added is the ability to expose a set of targets from
runtime projects into the top-level generated build files. I’ve connected this
new feature into compiler-rt, which allows exposing targets like asan and
check-asan through to the top-level build. This last feature has some slight
awkwardness and I’m curious to hear opinions on it.

To avoid having to hard-code a list of targets to expose I made the runtime
configuration step generate a list of targets to map back. This means that
before you can use targets from the runtimes build you need to build enough of
the toolchain to configure the runtime project build. What that means is you
can’t do the following in a new build tree:
> cmake ...
> ninja asan
Instead you would do:
> cmake ...
> ninja runtimes-configure
> ninja asan
I can’t think of any good ways around this. I’m curious if anyone has ideas, or
opinions on whether or not this is an acceptable workflow.

The big items on my to-do list are:
* Running the builtins tests
* Porting libcxxabi and libunwind
* Getting check-all with a single lit invocation
* IDE support
* Supporting specifying the path of runtime libraries

In general I think this is getting to a point where people can give it a test
drive and see what they think, and I’m eagerly looking for feedback.

Thanks,
-Chris

On 26 August 2016 at 22:45, Chris Bieneman via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Recently, I’ve made a handful of new changes that actually make it useful,
and I wanted to blast out this update and encourage people to give it a try.
Woot! Compiling RT as we speak... :)

Interesting, "ninja runtimes-configure" wants to build the whole
thing... Is that because you use the just-built clang to build the
multiple targets for each runtime?

What's the CMake variable to enable RT for ARM, even though I'm
running on x86_64?

>> cmake ...
>> ninja runtimes-configure
>> ninja asan
It does seem awkward. I don't know CMake well enough to know what's
the problem, much less how to fix it, sorry. :(

In pure Make, I'd just make the asan rule depend on the other, maybe
the ninja file could be changed in that way?

> The big items on my to-do list are:
> * Running the builtins tests
> * Porting libcxxabi and libunwind
> * Getting check-all with a single lit invocation
> * IDE support
> * Supporting specifying the path of runtime libraries
Sounds like a solid plan.

cheers,
--renato

On 26 August 2016 at 23:22, Renato Golin <renato.golin at linaro.org>
wrote:
> Woot! Compiling RT as we speak... :)
Right, all went well until I got this:

[0/1] Re-running CMake...
-- Builtin supported architectures: i386;x86_64
-- Configuring done
-- Generating done
-- Build files have been written to:
/home/rengolin/devel/llvm/workdir/libs/build/runtimes/builtins-bins
ninja: no work to do.
[677/681] No install step for 'builtins'
[680/681] Performing configure step for 'runtimes'
-- Compiler-RT supported architectures: x86_64;i386
CMake Error at CMakeLists.txt:93 (list):
  list sub-command REMOVE_DUPLICATES requires list to be present.

CMake logs attached.

--renato
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CMakeLogs.zip
Type: application/zip
Size: 3496 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160826/85893be8/attachment.zip>

> On Aug 26, 2016, at 3:22 PM, Renato Golin <renato.golin at
linaro.org> wrote:
> 
> On 26 August 2016 at 22:45, Chris Bieneman via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> Recently, I’ve made a handful of new changes that actually make it
useful, and I wanted to blast out this update and encourage people to give it a
try.
> 
> Woot! Compiling RT as we speak... :)
> 
> Interesting, "ninja runtimes-configure" wants to build the whole
> thing... Is that because you use the just-built clang to build the
> multiple targets for each runtime?
Yes, because it uses the just-built tools, clang is a dependency of the
configure step.
> 
> What's the CMake variable to enable RT for ARM, even though I'm
> running on x86_64?
This isn’t explicitly supported yet, but the standard COMPILER_RT_* prefixed
options will be passed through.
> 
> 
>>> cmake ...
>>> ninja runtimes-configure
>>> ninja asan
> 
> It does seem awkward. I don't know CMake well enough to know what's
> the problem, much less how to fix it, sorry. :(
> 
> In pure Make, I'd just make the asan rule depend on the other, maybe
> the ninja file could be changed in that way?
The problem is that the top-level CMake doesn’t actually know what targets to
generate until after you configure compiler-rt because the capabilities of the
compiler you build will impact what targets get generated.
> 
> 
>> The big items on my to-do list are:
>> * Running the builtins tests
>> * Porting libcxxabi and libunwind
>> * Getting check-all with a single lit invocation
>> * IDE support
>> * Supporting specifying the path of runtime libraries
> 
> Sounds like a solid plan.
-Chris
> 
> cheers,
> --renato