llvm dev - Jan 2016 - Building SVN head with CMake - shared libraries?

Hi,

On Sun, Jan 17, 2016 at 1:04 PM, Dan Liew <dan at su-root.co.uk>
wrote:
> On 16 January 2016 at 20:21, Ismail Donmez <ismail at i10z.com>
wrote:
>> On Sat, Jan 16, 2016 at 9:33 PM, Dan Liew <dan at su-root.co.uk>
wrote:
>>>> I am trying to enable this on openSUSE but it seems to break
>>>> standalone lldb (note that we don't ship static libs):
>>>>
>>>> cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
>>>> -DCMAKE_CXX_FLAGS=-stdlib=libc++ -DLLVM_BUILD_LLVM_DYLI
>>>> B=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_INSTALL_PREFIX=/usr
>>>> -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_LIBDIR_SUFFIX=64
-DLLDB_PA
>>>> TH_TO_LLVM_BUILD=/usr -DLLDB_PATH_TO_CLANG_BUILD=/usr
>>>>
-DLLVM_RUNTIME_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/buil
>>>> d/bin
-DLLVM_LIBRARY_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/build/lib64
>>>> -DPYTHON_VERSION_MAJOR=2 -DPYTHON_VERSIO
>>>> N_MINOR=7 -G Ninja ..
>>>>
>>>> [...]
>>>>
>>>> CMake Error at /usr/share/llvm/cmake/LLVMExports.cmake:1011
(message):
>>>>   The imported target "LLVMSupport" references the
file
>>>>
>>>>       "/usr/lib64/libLLVMSupport.a"
>>>
>>> Could you try appending ``-DLLVM_LINK_LLVM_DYLIB=ON`` to your CMake
>>> invocation? It looks like lldb is using the ``llvm_add_library()``
>>> function in the ``cmake/modules/AddLLVM.cmake`` (LLVM source tree)
>>> which uses ``LLVM_LINK_LLVM_DYLIB`` (and a few other variables) to
>>> decide whether to link against LLVM's static component
libraries or
>>> the monolithic shared library. I could be wrong on this though but
it
>>> should
>>> be pretty quick to check.
>>
>> I already use  -DLLVM_BUILD_LLVM_DYLIB=ON  -DLLVM_LINK_LLVM_DYLIB=ON
>
> That'll teach me to suggest something without actually trying it. Then
> I might have actually noticed you're already using that flag!
>
> Let me see if I can reproduce this. Are you building on LLVM trunk on
> the 3.8 release branch?
Trying the 3.8 branch atm.
>>> Out of curiosity though if you're building packages on openSUSE
why do
>>> you need a standalone build? I thought the normal way to do this
was
>>> to build everything and then generate split packages?
>>
>> Nice question. Since Mesa depends on llvm and Mesa is very crucial to
>> a distro bootstrap, adding any new dependency to llvm creates a
>> circular dependency which we can't have. And lldb itself brings
python
>> dependency which complicates the issue (python has a Mesa dependency).
>> I may try to pre-generate files with swig to remove the dependency but
>> it would be nice if this worked as expected.
>
> It's not entirely clear to me why adding a package (build or runtime?)
> dependency to LLVM creates a circular dependency.
> If you build llvm, clang and lldb and make split packages from the
> result you have a runtime dependency graph that is a tree with LLVM at
> the root.
> Maybe it's best not to digress. I agree that the standalone build of
> lldb should work as expected.
The cycle is

Mesa -> llvm -> python -> Mesa

@Brad: CC'ing you because I know you use
``llvm_map_components_to_libnames()`` so you will likely have
something to say about it breaking.

Okay Ismail, I took a much deeper look at this.

In LLDB the problem occurs when ``cmake/modules/LLDBStandalone.cmake``
tries to do

```
  # Import CMake library targets from LLVM and Clang.
 
include("${LLDB_PATH_TO_LLVM_BUILD}/share/llvm/cmake/LLVMConfig.cmake")
```

``LLVMConfig.cmake`` will include ``LLVMExports.cmake`` which checks
that the exported targets actually exist. The problem here is that you
have removed those installed libraries from the install tree and the
build system has know way of knowing that you were going to do this.

The most obvious solution to me is to add an option to LLVM which
informs CMake that you do not want to install the individual component
libraries. If you don't need to install the component libraries that
means you don't need to export the targets to ``LLVMExports.cmake`` in
the install tree (note the ``LLVMExport.cmake`` in the build tree
should still contain the targets).

I tried implementing this by adding an extra option
(LLVM_INSTALL_COMPONENT_LIBRARIES) which can be set to OFF in the
attached patch (0001-Add-LLVM_INSTALL_COMPONENT_LIBRARIES-option-which-is.patch
, apply against LLVM). It is a hack and I do not recommend using this
in production. Instead it's more of a starting point for discussion. I
ran into lots of problems because if the component libraries are not
in the install tree then components that link against them and are
installed can't be so I had hack some of the cmake macros/functions to
link against libLLVM instead.

With this patch I built LLVM as so

```
cd /home/dsl11/dev/llvm-upstream/test_build/
mkdir install
cmake -DCMAKE_INSTALL_PREFIX=`realpath install`
-DLLVM_INSTALL_COMPONENT_LIBRARIES=OFF -DLLVM_BUILD_LLVM_DYLIB=ON
-DLLVM_LINK_LLVM_DYLIB=ON  -G Ninja ../src
ninja install
```

You will notice there are no ``install/lib/libLLVM*.a`` files
installed. At least I got that right :)

Then I tried configuring LLD like so

```
cmake -DLLVM_LINK_LLVM_DYLIB=ON
-DLLDB_PATH_TO_LLVM_BUILD=/home/dsl11/dev/llvm-upstream/test_build/install
-DLLDB_PATH_TO_CLANG_BUILD=/home/dsl11/dev/llvm-upstream/test_build/install
../lldb-src
```

This didn't work out the box for me because it tries to pass
"${LLVM_MAIN_INCLUDE_DIR}" and "${CLANG_MAIN_INCLUDE_DIR}"
to
``include_directories()`` which doesn't like being passed empty
strings. The attached patch for lldb ( lldb_no_need_src_tree.patch )
fixes that.

The configure succeed but with loads of warnings like this

```
CMake Warning (dev) in source/Plugins/ScriptInterpreter/Python/CMakeLists.txt:
  Policy CMP0022 is not set: INTERFACE_LINK_LIBRARIES defines the link
  interface.  Run "cmake --help-policy CMP0022" for policy details. 
Use the
  cmake_policy command to set the policy and suppress this warning.

  Target "lldbPluginScriptInterpreterPython" has an
INTERFACE_LINK_LIBRARIES
  property.  This should be preferred as the source of the link interface for
  this library but because CMP0022 is not set CMake is ignoring the property
  and using the link implementation as the link interface instead.

  INTERFACE_LINK_LIBRARIES:

    LLVM;LLVM

  Link implementation:

    LLVM
```

I think the reason for LLVM appearing twice in
INTERFACE_LINK_LIBRARIES is that ``add_lldb_library()`` uses
``llvm_add_library()`` which will add a dependency on the LLVM target.
A few lines later it then calls ``llvm_config()`` on it later which
will add the dependency again!

There's also warnings about mixing the old style
``target_link_libraries()`` with ``target_link_libraries(target
(PUBLIC|PRIVATE|INTERFACE))``.

The build doesn't succeed. For some reason when building liblldb.so
there are missing references to pthreads.

```
[ 81%] Linking CXX shared library ../../lib/liblldb.so
../../lib/liblldbHost.a(Host.cpp.o): In function
`lldb_private::Host::ThreadLocalStorageCreate(void (*)(void*))':
Host.cpp:(.text._ZN12lldb_private4Host24ThreadLocalStorageCreateEPFvPvE+0x1b):
undefined reference to `pthread_key_create'
...

This surprises me because ``libLLVM-3.9svn.so`` which is part of the
link line for ``liblldb.so`` (after all the static libraries) which is
linked against ``libpthread.so.0`` when I examine it with the ``ldd``
tool.

Anyway... I guess this is a reasonable starting point for discussing
how to do this properly. After looking at this I have some concerns:

* The way that LLVM components are linked against seems really messy
to me. We have this ``llvm_config()`` macro for linking against
components but also ``llvm_add_library()`` does this too. We don't
need two ways of doing the same thing. I think there should be one
clean way of doing this.

* If the static libraries are not installed then external projects
that use LLVM via [1] won't build because
``llvm_map_components_to_libnames()`` only works with the static
libraries. We need to provide something better that does the right
thing automatically but allows forcing to use the static components or
the dynamic library.



[1] http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project

HTH,
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-LLVM_INSTALL_COMPONENT_LIBRARIES-option-which-is.patch
Type: text/x-patch
Size: 5394 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160117/c880908c/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lldb_no_need_src_tree.patch
Type: text/x-patch
Size: 1030 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160117/c880908c/attachment-0001.bin>

Hi,

On Sun, Jan 17, 2016 at 9:53 PM, Dan Liew <dan at su-root.co.uk>
wrote:
> @Brad: CC'ing you because I know you use
> ``llvm_map_components_to_libnames()`` so you will likely have
> something to say about it breaking.
>
> Okay Ismail, I took a much deeper look at this.
Thanks for the great analysis!
> In LLDB the problem occurs when ``cmake/modules/LLDBStandalone.cmake``
> tries to do
>
> ```
>   # Import CMake library targets from LLVM and Clang.
>  
include("${LLDB_PATH_TO_LLVM_BUILD}/share/llvm/cmake/LLVMConfig.cmake")
> ```
>
> ``LLVMConfig.cmake`` will include ``LLVMExports.cmake`` which checks
> that the exported targets actually exist. The problem here is that you
> have removed those installed libraries from the install tree and the
> build system has know way of knowing that you were going to do this.
>
> The most obvious solution to me is to add an option to LLVM which
> informs CMake that you do not want to install the individual component
> libraries. If you don't need to install the component libraries that
> means you don't need to export the targets to ``LLVMExports.cmake`` in
> the install tree (note the ``LLVMExport.cmake`` in the build tree
> should still contain the targets).
>
> I tried implementing this by adding an extra option
> (LLVM_INSTALL_COMPONENT_LIBRARIES) which can be set to OFF in the
> attached patch
(0001-Add-LLVM_INSTALL_COMPONENT_LIBRARIES-option-which-is.patch
> , apply against LLVM). It is a hack and I do not recommend using this
> in production. Instead it's more of a starting point for discussion. I
> ran into lots of problems because if the component libraries are not
> in the install tree then components that link against them and are
> installed can't be so I had hack some of the cmake macros/functions to
> link against libLLVM instead.
Indeed the correct solution would be such an option where we can skip
installing the components. The problem with installing the components
is that they are static libraries and in openSUSE we can't just ship
static libs to the end user. It would make updating libraries more
complicated and has security implications (even if it's less so for
llvm).

Regards,
ismail

On 01/17/2016 02:53 PM, Dan Liew wrote:
> @Brad: CC'ing you because I know you use
> ``llvm_map_components_to_libnames()`` so you will likely have
> something to say about it breaking.
I'm using it in CastXML but that could easily be adapted to something
different if needed.  The larger concern is that that API is the
documented way to use LLVM in an application with CMake so there could
be many such clients.
> * The way that LLVM components are linked against seems really messy
> to me. We have this ``llvm_config()`` macro for linking against
> components but also ``llvm_add_library()`` does this too. We don't
> need two ways of doing the same thing. I think there should be one
> clean way of doing this.
> 
> * If the static libraries are not installed then external projects
> that use LLVM via [1] won't build because
> ``llvm_map_components_to_libnames()`` only works with the static
> libraries. We need to provide something better that does the right
> thing automatically but allows forcing to use the static components or
> the dynamic library.
IIUC the purpose of asking applications to map component names
instead of directly naming libraries is so that the set of actual
libraries can be changed out underneath while retaining the same
components.  This abstraction means that if only the shared library
is installed then ``llvm_map_components_to_libnames()`` can simply
return it for all components built into it.  Its implementation just
needs to be made conditional on the option for static libraries.

-Brad