llvm dev - Jun 2015 - [LLVMdev] [RFC] Improving the testing of exported LLVM CMake targets

Hi,

Following on from another thread (Long-Term Support for LLVM Projects
Extension to Build System) I'd like to discuss the testing of the
exported LLVM CMake targets which can be used by consumers of LLVM as
documented in [1].

Right now we don't test this feature **at all** and as a result it has
been or is

* broken in trunk
* broken by those packaging LLVM
* broken in the official release binaries

I'd like to propose a few ideas I have to improve this situation.

I've created a small toy project [2] that uses the exported LLVM CMake
targets. What I'd like to happen is for this project to become part of
the LLVM source tree (or as a sub project) so that it can be tested in
various scenarios. I do not want this toy project to be a maintenance
burden so it makes very minimal use of LLVM's C++ API.

## Testing trunk

We could teach some of the existing build bots to try building this
toy project after building LLVM. The scenarios we should test are
building the toy project

- Against the LLVM build tree when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with the
Autoconf/Makefile build system

## Testing packaged LLVM

For those producing LLVM packages for Linux/*BSD, OSX and Windows it
would be good if we could encourage them to test that the toy project
actually builds against their package. Building the toy projects only
takes a few seconds so this shouldn't be much of a burden but can
greatly increase a packagers confidence that LLVM has been properly
packaged.

## Testing official LLVM release binaries

This situation is currently broken. I've observed that

- The LLVM 3.6.* binary tarballs have the LLVM CMake files completely
missing, how did this happen?
- The LLVM3.5 binary tarballs have the CMake files but they're
completely broken.

When making official releases I think it would be a good idea to test
that building the toy project against an extracted binary tarball
actually works.

There are two problems currently AFAICT:

First, the current test-release.sh script does not package an install
properly it uses --prefix to set a temporary directory for the tarball
install when instead it should be using the DESTDIR makefile variable.
I have sent a patch to llvm-commits to try to fix this [3]. The result
of this mistake is that the CMake files end up containing absolute
paths to the temporary tarball directory which means they cannot be
imported by another project.

Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake)
are not relocatable (they contain absolute paths which are based on
the install prefix). So even if the first issue is fixed we still have
the problem that a user cannot simply extract the binary tarball to an
arbitrary location their system and build the toy project against it.

I think the LLVMExport.cmake file and LLVM-Config.cmake files need to
be loaded from the same directory containing the found
LLVMConfig.cmake file. The automatically generated LLVMExports.cmake
file would also need to contain relative paths to the libraries (it
currently has absolute paths). I'm not sure how to do this for both of
LLVM's build systems. Any ideas?

Alternatively we could explicitly state in our docs that the LLVM
release binaries can only be used from CMake when it is installed into
the right prefix. This simpler but not very useful because it means
those who want to use the official LLVM binary release builds on their
machine can't unless they have root access (so they can write to
/usr/local/).

Thoughts?

[1] http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project
[2] https://github.com/delcypher/llvmCMakeImportDemo
[3]
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150622/284436.html

Thanks,
Dan.

> Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake) are
not relocatable (they contain absolute paths which are based on the install
prefix). So even if the first issue is fixed we still have the problem that a
user cannot simply extract the binary tarball to an arbitrary location their
system and build the toy project against it.
Really? My latest LLVMConfig (beginning of this month) is relative to
"LLVM_INSTALL_PREFIX" which is set by

get_filename_component(LLVM_INSTALL_PREFIX
"${CMAKE_CURRENT_LIST_FILE}" PATH)
get_filename_component(LLVM_INSTALL_PREFIX "${LLVM_INSTALL_PREFIX}"
PATH)
get_filename_component(LLVM_INSTALL_PREFIX "${LLVM_INSTALL_PREFIX}"
PATH)
get_filename_component(LLVM_INSTALL_PREFIX "${LLVM_INSTALL_PREFIX}"
PATH)

and analogously in LLVMExports:

get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_FILE}"
PATH)
get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)

Did someone change this recently?

--
Johannes S. Mueller-Roemer, MSc
Wiss. Mitarbeiter - Interactive Engineering Technologies (IET)

Fraunhofer-Institut für Graphische Datenverarbeitung IGD
Fraunhoferstr. 5  |  64283 Darmstadt  |  Germany
Tel +49 6151 155-606  |  Fax +49 6151 155-139
johannes.mueller-roemer at igd.fraunhofer.de  |  www.igd.fraunhofer.de


-----Original Message-----
From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On
Behalf Of Dan Liew
Sent: Saturday, June 27, 2015 10:17
To: LLVM Developers Mailing List
Subject: [LLVMdev] [RFC] Improving the testing of exported LLVM CMake targets

Hi,

Following on from another thread (Long-Term Support for LLVM Projects Extension
to Build System) I'd like to discuss the testing of the exported LLVM CMake
targets which can be used by consumers of LLVM as documented in [1].

Right now we don't test this feature **at all** and as a result it has been
or is

* broken in trunk
* broken by those packaging LLVM
* broken in the official release binaries

I'd like to propose a few ideas I have to improve this situation.

I've created a small toy project [2] that uses the exported LLVM CMake
targets. What I'd like to happen is for this project to become part of the
LLVM source tree (or as a sub project) so that it can be tested in various
scenarios. I do not want this toy project to be a maintenance burden so it makes
very minimal use of LLVM's C++ API.

## Testing trunk

We could teach some of the existing build bots to try building this toy project
after building LLVM. The scenarios we should test are building the toy project

- Against the LLVM build tree when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with the Autoconf/Makefile build
system

## Testing packaged LLVM

For those producing LLVM packages for Linux/*BSD, OSX and Windows it would be
good if we could encourage them to test that the toy project actually builds
against their package. Building the toy projects only takes a few seconds so
this shouldn't be much of a burden but can greatly increase a packagers
confidence that LLVM has been properly packaged.

## Testing official LLVM release binaries

This situation is currently broken. I've observed that

- The LLVM 3.6.* binary tarballs have the LLVM CMake files completely missing,
how did this happen?
- The LLVM3.5 binary tarballs have the CMake files but they're completely
broken.

When making official releases I think it would be a good idea to test that
building the toy project against an extracted binary tarball actually works.

There are two problems currently AFAICT:

First, the current test-release.sh script does not package an install properly
it uses --prefix to set a temporary directory for the tarball install when
instead it should be using the DESTDIR makefile variable.
I have sent a patch to llvm-commits to try to fix this [3]. The result of this
mistake is that the CMake files end up containing absolute paths to the
temporary tarball directory which means they cannot be imported by another
project.

Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake) are not
relocatable (they contain absolute paths which are based on the install prefix).
So even if the first issue is fixed we still have the problem that a user cannot
simply extract the binary tarball to an arbitrary location their system and
build the toy project against it.

I think the LLVMExport.cmake file and LLVM-Config.cmake files need to be loaded
from the same directory containing the found LLVMConfig.cmake file. The
automatically generated LLVMExports.cmake file would also need to contain
relative paths to the libraries (it currently has absolute paths). I'm not
sure how to do this for both of LLVM's build systems. Any ideas?

Alternatively we could explicitly state in our docs that the LLVM release
binaries can only be used from CMake when it is installed into the right prefix.
This simpler but not very useful because it means those who want to use the
official LLVM binary release builds on their machine can't unless they have
root access (so they can write to /usr/local/).

Thoughts?

[1] http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project
[2] https://github.com/delcypher/llvmCMakeImportDemo
[3]
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150622/284436.html

Thanks,
Dan.
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On 06/27/2015 04:17 AM, Dan Liew wrote:
> Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake)
> are not relocatable (they contain absolute paths which are based on
> the install prefix). So even if the first issue is fixed we still have
> the problem that a user cannot simply extract the binary tarball to an
> arbitrary location their system and build the toy project against it.
The files generated by the CMake build are completely relocatable
because they compute everything relative to the LLVMConfig.cmake
file location.  The Autoconf/Makefile build system would have to
be taught to do the same thing in the files it generates.  Take a
look at the CMake-generated files to see how they achieve it.

Thanks,
-Brad

Hi,


On 29 June 2015 at 01:30, Mueller-Roemer, Johannes Sebastian
<Johannes.Sebastian.Mueller-Roemer at igd.fraunhofer.de>
wrote:
>> Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake)
are not relocatable (they contain absolute paths which are based on the install
prefix). So even if the first issue is fixed we still have the problem that a
user cannot simply extract the binary tarball to an arbitrary location their
system and build the toy project against it.
>
> Really? My latest LLVMConfig (beginning of this month) is relative to
"LLVM_INSTALL_PREFIX" which is set by
>
> get_filename_component(LLVM_INSTALL_PREFIX
"${CMAKE_CURRENT_LIST_FILE}" PATH)
> get_filename_component(LLVM_INSTALL_PREFIX
"${LLVM_INSTALL_PREFIX}" PATH)
> get_filename_component(LLVM_INSTALL_PREFIX
"${LLVM_INSTALL_PREFIX}" PATH)
> get_filename_component(LLVM_INSTALL_PREFIX
"${LLVM_INSTALL_PREFIX}" PATH)
>
> and analogously in LLVMExports:
>
> get_filename_component(_IMPORT_PREFIX
"${CMAKE_CURRENT_LIST_FILE}" PATH)
> get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
> get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
> get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
>
> Did someone change this recently?
No. It turns out I gave my install directories stupid names and I
confused the install directory of the Autoconf/Makefile build system
with the CMake one. It is the CMake export files generated by the
Autoconf/Makefile build system which are not relocatable. Seeing as
the the non relocatable install behaviour is an issue with the
Autoconf/Makefile build system it should be fairly easy to fix.

On 29 June 2015 at 07:16, Brad King <brad.king at kitware.com>
wrote:
> On 06/27/2015 04:17 AM, Dan Liew wrote:
>> Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake)
>> are not relocatable (they contain absolute paths which are based on
>> the install prefix). So even if the first issue is fixed we still have
>> the problem that a user cannot simply extract the binary tarball to an
>> arbitrary location their system and build the toy project against it.
>
> The files generated by the CMake build are completely relocatable
> because they compute everything relative to the LLVMConfig.cmake
> file location.  The Autoconf/Makefile build system would have to
> be taught to do the same thing in the files it generates.  Take a
> look at the CMake-generated files to see how they achieve it.
Yes, sorry I got my install directories confused (I should give them
better names!). I'll try to write a patch for the autoconf/Makefile
build system. The Debian/Ubuntu packages modify the install location
of the LLVM CMake files so (assuming my patch gets accepted) the
maintainers will need to do some more patching in the next release in
order to avoid breaking the installed CMake files. The toy project
that I mentioned in the original e-mail would be useful here because
it would let the packagers test their package with minimal effort.

The other points that I mention in the original e-mail (broken binary
builds, the need for more testing, etc.) still stand though.

Thanks,
Dan.

Dan Liew wrote:
> This situation is currently broken. I've observed that
> 
> - The LLVM 3.6.* binary tarballs have the LLVM CMake files completely
> missing, how did this happen?
> - The LLVM3.5 binary tarballs have the CMake files but they're
> completely broken.
You could also consider changing the names of the llvm imported targets at 
this point to llvm::foo (and similar for the clang imported targets). This 
will give better diagnostics with current cmake releases, and may gain extra 
benefit in the future.

See 

 http://www.cmake.org/cmake/help/v3.2/policy/CMP0028.html

As of the resolution of 

 https://llvm.org/bugs/show_bug.cgi?id=19462

clang installs appropriate files when built with CMake, but I don't know 
about other llvm projects. 

I introduced a policy to disallow doing what clang was doing there, so you 
can use CMake 3.3-rc2 and CMP0062 to find any other possible instances of 
that problem.

Thanks,

Steve.

Hi,

On 29 June 2015 at 15:53, Stephen Kelly <steveire at gmail.com>
wrote:
> Dan Liew wrote:
>
>> This situation is currently broken. I've observed that
>>
>> - The LLVM 3.6.* binary tarballs have the LLVM CMake files completely
>> missing, how did this happen?
>> - The LLVM3.5 binary tarballs have the CMake files but they're
>> completely broken.
>
> You could also consider changing the names of the llvm imported targets at
> this point to llvm::foo (and similar for the clang imported targets). This
> will give better diagnostics with current cmake releases, and may gain
extra
> benefit in the future.
>
I'd like to do that but I worry about downstream projects that rely
directly on target names instead of using the
llvm_map_components_to_libnames() function. Would using aliases like
``add_library(LLVMSuport ALIAS llvm::Support)`` be sufficient to not
break downstream projects?

Thanks,
Dan.

Hi,

I'd like to try and revive this dead thread. Another breakage of the
CMake exported targets has occurred so it's about time something was
done about this.

> ## Testing trunk
>
> We could teach some of the existing build bots to try building this
> toy project after building LLVM. The scenarios we should test are
> building the toy project
>
> - Against the LLVM build tree when LLVM is built with CMake
> - Against the installed LLVM when LLVM is built with CMake
> - Against the installed LLVM when LLVM is built with the
> Autoconf/Makefile build system
Unless there are any objections I'd like to go ahead with this. I need
to know where the toy project should live SVN tree. Could someone tell
me where this should be and I'll add it? I should also probably give
the files the usual LLVM copyright notice.
> ## Testing packaged LLVM
>
> For those producing LLVM packages for Linux/*BSD, OSX and Windows it
> would be good if we could encourage them to test that the toy project
> actually builds against their package. Building the toy projects only
> takes a few seconds so this shouldn't be much of a burden but can
> greatly increase a packagers confidence that LLVM has been properly
> packaged.
Our documentation needs to be updated to ask packagers to check that
the toy project builds.
> ## Testing official LLVM release binaries
The test-release.sh script probably needs to be taught to try to build
the toy project as part of its testing (probably make it optional when
building with autoconf because the user might not have CMake).

Thanks,
Dan.