llvm dev - Jun 2015 - [LLVMdev] Long-Term Support for LLVM Projects Extension to Build System?

On 20 June 2015 at 02:46, David Chisnall <David.Chisnall at cl.cam.ac.uk>
wrote:
> On 19 Jun 2015, at 23:25, Dan Liew <dan at su-root.co.uk> wrote:
>>
>> This is what I effectively do. Although because I build the projects I
>> work on I use the CMake config file that LLVM exports to make it
>> trivially easy to use the LLVM libraries from within CMake (see
>> http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project).
>
> The last time that I tried this, I hit all sorts of errors when LLVM was
built as shared libraries and eventually just had my cmake files invoke
llvm-config and manually populate the relevant CXX / LD flags.
AFAIK the exported CMake targets only work with the static LLVM
libraries right now. The intention is that user's of LLVM should use
the llvm_map_components_to_libnames() function in CMake. Right now
that returns the names of the static library targets. In principle I
suppose it could be modified to return the shared library target
instead if a shared build was done. But there are a few issues to
consider here:

* When LLVM is built it possible to pick what components go into the
shared library which means in some cases
``llvm_map_components_to_libnames()`` would not work as intended if
just returns the LLVM shared library target name.
* A user of LLVM may want the ability to pick the static libraries
even though the shared libraries were built.
* In the shared library the C++ symbols hidden so we would need to add
an option to disable this behaviour.

Here's what I'd like to suggest:

- Modify ``llvm_map_components_to_libnames()`` to take an optional
keyword which is either STATIC or SHARED. To avoid ambiguity this
means we cannot have a component called "STATIC" or
"SHARED", I don't
imagine this would ever happen in practice. So the signature would
look like this

llvm_map_components_to_libnames(OUTPUTVAR [STATIC|SHARED] component0 ...)


- If set to static the static library names will be returned if the
components exists, otherwise emit an error message.
- If SHARED is specified CMake will check that the requested
components were added to the shared library when it was built. For
this to work the names of the components will need to be written into
LLVMConfig.cmake. If all the components requested are supposed to be
in the shared library then return the shared library target, otherwise
emit an error message (this includes the case where the shared library
was not built).
- If STATIC or SHARED is not specified use STATIC so as to not break
the interface for existing users of llvm_map_components_to_libnames().
- Switch C++ symbol hiding in the shared library off by default.

Thoughts?
> I don’t know if there’s something that we can do with buildbots to make
sure that this is better tested.
I was thinking it would be nice to make a very simple toy project that
uses CMake to build and uses ``llvm_map_components_to_libnames()``.
The CMake build bots could be taught to build this from scratch
against a completed build of LLVM. I thinking testing this is a very
good idea because I recently had to fix this feature which was broken
under Windows.

Thanks,
Dan.

On 22 Jun 2015, at 02:42, Dan Liew <dan at su-root.co.uk>
wrote:
> 
> AFAIK the exported CMake targets only work with the static LLVM
> libraries right now. The intention is that user's of LLVM should use
> the llvm_map_components_to_libnames() function in CMake. Right now
> that returns the names of the static library targets. In principle I
> suppose it could be modified to return the shared library target
> instead if a shared build was done. But there are a few issues to
> consider here:
> 
> * When LLVM is built it possible to pick what components go into the
> shared library which means in some cases
> ``llvm_map_components_to_libnames()`` would not work as intended if
> just returns the LLVM shared library target name.
> * A user of LLVM may want the ability to pick the static libraries
> even though the shared libraries were built.
> * In the shared library the C++ symbols hidden so we would need to add
> an option to disable this behaviour.
> 
> Here's what I'd like to suggest:
> 
> - Modify ``llvm_map_components_to_libnames()`` to take an optional
> keyword which is either STATIC or SHARED. To avoid ambiguity this
> means we cannot have a component called "STATIC" or
"SHARED", I don't
> imagine this would ever happen in practice. So the signature would
> look like this
> 
> llvm_map_components_to_libnames(OUTPUTVAR [STATIC|SHARED] component0 ...)
> 
> 
> - If set to static the static library names will be returned if the
> components exists, otherwise emit an error message.
> - If SHARED is specified CMake will check that the requested
> components were added to the shared library when it was built. For
> this to work the names of the components will need to be written into
> LLVMConfig.cmake. If all the components requested are supposed to be
> in the shared library then return the shared library target, otherwise
> emit an error message (this includes the case where the shared library
> was not built).
> - If STATIC or SHARED is not specified use STATIC so as to not break
> the interface for existing users of llvm_map_components_to_libnames().
> - Switch C++ symbol hiding in the shared library off by default.
> 
> Thoughts?
This all sounds great to me.  That said, my use case is quite unusual: I want a
single debug build of LLVM and a single release build on a shared file server so
that students can link things against a debug build without ending up with a few
hundred MBs of object code in their home directories for each exercise and
hitting quota issues.  I suspect that this isn’t a problem that doesn’t affect
most other people.

David

> On Jun 21, 2015, at 6:42 PM, Dan Liew <dan at su-root.co.uk> wrote:
> 
> On 20 June 2015 at 02:46, David Chisnall <David.Chisnall at
cl.cam.ac.uk> wrote:
>> On 19 Jun 2015, at 23:25, Dan Liew <dan at su-root.co.uk> wrote:
>>> 
>>> This is what I effectively do. Although because I build the
projects I
>>> work on I use the CMake config file that LLVM exports to make it
>>> trivially easy to use the LLVM libraries from within CMake (see
>>> http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project).
>> 
>> The last time that I tried this, I hit all sorts of errors when LLVM
was built as shared libraries and eventually just had my cmake files invoke
llvm-config and manually populate the relevant CXX / LD flags.
> 
> AFAIK the exported CMake targets only work with the static LLVM
> libraries right now.
That depends. It should work with shared libraries if the LLVM you are importing
was built with BUILD_SHARED_LIBS=On. It will also work with llvm-shlib if it was
built with LLVM_BUILD_LLVM_DYLIB=On.
> The intention is that user's of LLVM should use
> the llvm_map_components_to_libnames() function in CMake. Right now
> that returns the names of the static library targets. In principle I
> suppose it could be modified to return the shared library target
> instead if a shared build was done. But there are a few issues to
> consider here:
> 
> * When LLVM is built it possible to pick what components go into the
> shared library which means in some cases
> ``llvm_map_components_to_libnames()`` would not work as intended if
> just returns the LLVM shared library target name.
> * A user of LLVM may want the ability to pick the static libraries
> even though the shared libraries were built.
If static libraries are built, but the user wants shared libraries, there is no
way to handle this other than re-configuring and rebuilding LLVM.
> * In the shared library the C++ symbols hidden so we would need to add
> an option to disable this behaviour.
LLVM_DYLIB_EXPORT_ALL=On will export all symbols from the dylib.
> 
> Here's what I'd like to suggest:
> 
> - Modify ``llvm_map_components_to_libnames()`` to take an optional
> keyword which is either STATIC or SHARED. To avoid ambiguity this
> means we cannot have a component called "STATIC" or
"SHARED", I don't
> imagine this would ever happen in practice. So the signature would
> look like this
> 
> llvm_map_components_to_libnames(OUTPUTVAR [STATIC|SHARED] component0 ...)
> 
> 
> - If set to static the static library names will be returned if the
> components exists, otherwise emit an error message.
> - If SHARED is specified CMake will check that the requested
> components were added to the shared library when it was built. For
> this to work the names of the components will need to be written into
> LLVMConfig.cmake. If all the components requested are supposed to be
> in the shared library then return the shared library target, otherwise
> emit an error message (this includes the case where the shared library
> was not built).
I have thought about doing this, and it wouldn’t be terribly complicated. There
is already some goop in tools/llvm-c-test/CMakeLists.txt to check against the
list of libraries in the dylib before linking the libraries separately.

While I don’t disagree with what you’re proposing, I actually think we need a
larger reworking of how llvm_map_components_to_libnames works. The file that
function comes from is basically a re-implementation of llvm-config in CMake,
and it is really a bit of a dirty hack, and it unfortunately results in us
having 2 different systems for expressing libraries dependencies at varying
levels of robustness. I would like to find a way to get rid of all of the
duplication, and to have a project-agnostic way to do this so that clang and
other sub-projects could benefit from it as well.

That’s all a bit of a pipe dream though, so we shouldn’t let it prevent us from
making the existing hacks work better.
> - If STATIC or SHARED is not specified use STATIC so as to not break
> the interface for existing users of llvm_map_components_to_libnames().
> - Switch C++ symbol hiding in the shared library off by default.
I disagree with this. When I added llvm-shlib to the CMake build system I did it
this way intentionally with specific clients in mind (the key one being WebKit).
I’ve added lots of options to configure it in any way you want, but my
inclination is that the default configuration should be what it is now. The
library that llvm-shlib generates was not intended to be used by llvm in-tree
tools, nor was it intended to be used by projects that import our build system.
The whole point of having a single shared library is that your link command line
is trivial (-lLLVM). While I’m completely in favor of extending it to work
better for other use cases (and in fact we have a few planned uses ourselves), I
would prefer if the default behavior didn’t change.

-Chris
> 
> Thoughts?
> 
>> I don’t know if there’s something that we can do with buildbots to make
sure that this is better tested.
> 
> I was thinking it would be nice to make a very simple toy project that
> uses CMake to build and uses ``llvm_map_components_to_libnames()``.
> The CMake build bots could be taught to build this from scratch
> against a completed build of LLVM. I thinking testing this is a very
> good idea because I recently had to fix this feature which was broken
> under Windows.
> 
> Thanks,
> Dan.

> On Jun 22, 2015, at 12:28 AM, David Chisnall <David.Chisnall at
cl.cam.ac.uk> wrote:
> 
> On 22 Jun 2015, at 02:42, Dan Liew <dan at su-root.co.uk> wrote:
>> 
>> AFAIK the exported CMake targets only work with the static LLVM
>> libraries right now. The intention is that user's of LLVM should
use
>> the llvm_map_components_to_libnames() function in CMake. Right now
>> that returns the names of the static library targets. In principle I
>> suppose it could be modified to return the shared library target
>> instead if a shared build was done. But there are a few issues to
>> consider here:
>> 
>> * When LLVM is built it possible to pick what components go into the
>> shared library which means in some cases
>> ``llvm_map_components_to_libnames()`` would not work as intended if
>> just returns the LLVM shared library target name.
>> * A user of LLVM may want the ability to pick the static libraries
>> even though the shared libraries were built.
>> * In the shared library the C++ symbols hidden so we would need to add
>> an option to disable this behaviour.
>> 
>> Here's what I'd like to suggest:
>> 
>> - Modify ``llvm_map_components_to_libnames()`` to take an optional
>> keyword which is either STATIC or SHARED. To avoid ambiguity this
>> means we cannot have a component called "STATIC" or
"SHARED", I don't
>> imagine this would ever happen in practice. So the signature would
>> look like this
>> 
>> llvm_map_components_to_libnames(OUTPUTVAR [STATIC|SHARED] component0
...)
>> 
>> 
>> - If set to static the static library names will be returned if the
>> components exists, otherwise emit an error message.
>> - If SHARED is specified CMake will check that the requested
>> components were added to the shared library when it was built. For
>> this to work the names of the components will need to be written into
>> LLVMConfig.cmake. If all the components requested are supposed to be
>> in the shared library then return the shared library target, otherwise
>> emit an error message (this includes the case where the shared library
>> was not built).
>> - If STATIC or SHARED is not specified use STATIC so as to not break
>> the interface for existing users of llvm_map_components_to_libnames().
>> - Switch C++ symbol hiding in the shared library off by default.
>> 
>> Thoughts?
> 
> This all sounds great to me.  That said, my use case is quite unusual: I
want a single debug build of LLVM and a single release build on a shared file
server so that students can link things against a debug build without ending up
with a few hundred MBs of object code in their home directories for each
exercise and hitting quota issues.  I suspect that this isn’t a problem that
doesn’t affect most other people.
If you’re building this for students there is a much simpler way to do this,
with the existing build system.

Configure LLVM with a command like this:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DLLVM_OPTIMIZED_TABLEGEN=On
-DLLVM_TARGETS_TO_BUILD="X86;ARM;AArch64" -DLLVM_BUILD_LLVM_DYLIB=On
-DLLVM_DYLIB_EXPORT_ALL=On -DCMAKE_INSTALL_PREFIX=<install path> <path
to llvm>

Setting the install prefix will result in you having a directory you can tar up
and send to your students.

Build just the bits you need:
ninja LLVM

Install only the dylib and the headers:
ninja install-LLVM installhdrs

This will give you a working libLLVM that your students can link against, and
the headers they need in the most reduced capacity. No need to use CMake, no
need to rely on our build system. Much simpler.

You may need to check tools/llvm-shlib/CMakeLists.txt to make sure it has all
the LLVM components you need in the default configuration. If it doesn’t you can
change the list of components to include in the dylib by setting
LLVM_DYLIB_COMPONENTS.

-Chris
> 
> David
>