llvm dev - Feb 2014 - [LLVMdev] compiler-rt CMake build

The trouble is that the sanitizer tests are not really compiler-rt
tests.  The original compiler-rt tests are, but not the sanitizer
tests.  The sanitizer tests are "compiler-rt tests via clang".  The
sanitizer runtime is a C++ library requiring only a C++ compiler to
build it (and to *unit* test it).  The ASan lit tests, however,
require clang, because they are testing those C++ functions in the
context of a particular C/C++ compiler's instrumentation.  I believe
these tests should not be called "compiler-rt tests".  They are clang
tests.  In fact, they are just-built-clang tests.  They don't, for
example, verify the C++ library is useful to another compiler, another
version of clang, or simply as a C++ library.  They are integration
tests (tests clang+asan), as opposed to unit tests (tests only the
asan runtime).

I understand that the integration tests are more relevant to the
sanitizers than they are clang, and so it make sense to me that they
reside in the compiler-rt repo alongside the sanitizers.  But rather
than build them in the compiler-rt build, I think the directory should
be exported so that clang can optionally import it.  The clang build
can then add those tests to the clang tests suite, and install the
runtime in whatever location is most appropriate for clang.

Why is this important now?  The CMake developers are trying to make
the LLVM toolchain easy to build for package managers.  Compiler-rt's
dependency on the just-built-clang makes that difficult.  Without that
dependency, Compiler-rt could be built and tested against only an LLVM
package.  And likewise, a Clang package would depend on Compiler-rt
(and do further testing on compiler-rt with those lit tests).  But
with the dependency on the just-built-clang, there needs to be an
additional top-level package, let's call it ClangAndFriends.
ClangAndFriends would depend on Clang and Compiler-rt.  Compiler-rt
would depend on the Clang package as well.  Once both are built,
ClangAndFriends would copy the Clang install directory and then copy
the Compiler-rt install directory into it.  This extra package and
extra copy would be unnecessary if the Compiler-rt build dropped its
dependency on the just-built-clang.  After testing the sanitizers, the
Clang package would copy the runtime into its own install directory.

So what next?  Wait for Brad King's LLVM patches to be integrated.
Then we tweak the compiler-rt build to optionally build against the
LLVM install directory (instead of in the context of the LLVM source
directory).  Next, we use CMake to export the Sanitizer lit tests.
Lastly, Clang optionally imports the test suite and adds the sanitizer
runtime to its install directory.  What do you think?  Squeaky clean?

Thanks,
Greg



On Wed, Feb 5, 2014 at 12:55 AM, Alexey Samsonov <samsonov at google.com>
wrote:
>
> On Tue, Feb 4, 2014 at 10:17 PM, Greg Fitzgerald <garious at
gmail.com> wrote:
>>
>> Regarding Brad King's CMake patches for LLVM:
>>
>>   http://thread.gmane.org/gmane.comp.compilers.llvm.cvs/173517
>>
>> If compiler-rt could follow suit, the packaging for the toolchain
>> becomes a nice, clean package-manager-friendly DAG.  Currently, the
>> compiler-rt CMake build depends on the just-built-clang to run its
>> test suite.  It's more intuitive to me that either:
>>
>>  1) compiler-rt's sanitizer tests would live inside the clang
project
>>  2) compiler-rt would export a test suite that clang could optionally
>> import
>>
>> The former is simpler, but the latter allows the compiler-rt
>> developers to retain control over those tests, which are typically
>> updated alongside changes to compiler-rt's sanitizers.  Thoughts?
>
>
> I'm pretty sure the compiler-rt tests should live in the compiler-rt
> project. You're right:
> these tests should generally be updated in sync with the runtimes. I'm
fine
> with creating
> a large test suite comprising all individual tool-specific test suites we
> have now. I'm not
> sure how we can "export" this to clang, though, and why the
current
> situation is bad.
>
>
>>
>>
>> Thanks,
>> Greg
>
>
>
>
> --
> Alexey Samsonov, MSK

Hi Greg,

On Thu, Feb 6, 2014 at 12:55 AM, Greg Fitzgerald <garious at gmail.com>
wrote:
> The trouble is that the sanitizer tests are not really compiler-rt
> tests.  The original compiler-rt tests are, but not the sanitizer
> tests.  The sanitizer tests are "compiler-rt tests via clang". 
The
> sanitizer runtime is a C++ library requiring only a C++ compiler to
> build it (and to *unit* test it).  The ASan lit tests, however,
> require clang, because they are testing those C++ functions in the
> context of a particular C/C++ compiler's instrumentation.  I believe
> these tests should not be called "compiler-rt tests".  They are
clang
> tests.

Well, not really. I agree that there is a significant difference between
plain compiler-rt and
sanitizers:
1) compiler-rt is more or less a regular C++ library, with the exception
that a compiler
sometimes creates a calls into compiler-rt. compiler-rt can be built with
any C compiler,
and can be fairly easy unit-tested.
2) sanitizer libraries are special in a way that they more heavily rely on
compiler instrumentation and
code transformations. sanitizers can be built with any C++ compiler, but
"unit-testing" is far
less important for them, and although we have regular unit tests (that
don't expect
instrumentation) for some sanitizers, we are really more concentrated on
testing
compiler+run-time together.

But our tests (call them "integration tests" or "toolchain
tests") are not
Clang-specific - they
should work (and we'd love to make them work) with GCC version of ASan as
well. So, in
a perfect world, we would have a "sanitizer-tests" test suite, which
we can
run with
any compiler we specify at configuration time, be it just-built Clang, or
GCC [1].

>  In fact, they are just-built-clang tests.  They don't, for
> example, verify the C++ library is useful to another compiler, another
> version of clang, or simply as a C++ library.  They are integration
> tests (tests clang+asan), as opposed to unit tests (tests only the
> asan runtime).
>
> I understand that the integration tests are more relevant to the
> sanitizers than they are clang, and so it make sense to me that they
> reside in the compiler-rt repo alongside the sanitizers.  But rather
> than build them in the compiler-rt build, I think the directory should
> be exported so that clang can optionally import it.  The clang build
> can then add those tests to the clang tests suite, and install the
> runtime in whatever location is most appropriate for clang.
>
> Why is this important now?  The CMake developers are trying to make
> the LLVM toolchain easy to build for package managers.  Compiler-rt's
> dependency on the just-built-clang makes that difficult.  Without that
> dependency, Compiler-rt could be built and tested against only an LLVM
> package.

Why does compiler-rt needs to be tested against LLVM package? All
compiler-rt runtimes can be built and unit-tested with any C/C++ host
compiler,
if they don't depend on just-built-Clang. Or do you mean LLVM testing
utilities (llvm-lit, FileCheck) here? If yes, maybe it makes sense to move
*them*
to another package as well?

>  And likewise, a Clang package would depend on Compiler-rt
> (and do further testing on compiler-rt with those lit tests).

That's a bit weird - what if compiler-rt is not supported on a platform,
where Clang is supported?

>  But
> with the dependency on the just-built-clang, there needs to be an
> additional top-level package, let's call it ClangAndFriends.
> ClangAndFriends would depend on Clang and Compiler-rt.  Compiler-rt
> would depend on the Clang package as well.  Once both are built,
> ClangAndFriends would copy the Clang install directory and then copy
> the Compiler-rt install directory into it.

Can we instead make a package SanitizerTests (or whatever)? It won't even
need to have any dependencies - instead we should make it configurable -
SanitizerTests should take as a parameter a path to the compiler, and use
it to run the lit-tests and GTest-style unit tests that require
instrumentation.

>  This extra package and
> extra copy would be unnecessary if the Compiler-rt build dropped its
> dependency on the just-built-clang.  After testing the sanitizers, the
> Clang package would copy the runtime into its own install directory.
>
> So what next?  Wait for Brad King's LLVM patches to be integrated.
> Then we tweak the compiler-rt build to optionally build against the
> LLVM install directory (instead of in the context of the LLVM source
> directory).

Please note that it makes a lot of sense to built compiler-rt (and
sanitizers) with just-built
Clang. In fact, even though we should support building it with another
compilers (gcc, MSVC),
using just-built-Clang should be a default scenario, like it is in
configure+make build.
In this way we'll be able to build compiler-rt libs for platforms that
just-build
Clang can target, and host compiler probably can't...

>  Next, we use CMake to export the Sanitizer lit tests.
> Lastly, Clang optionally imports the test suite and adds the sanitizer
> runtime to its install directory.  What do you think?  Squeaky clean?
>
> Thanks,
> Greg
>
[1] Except that we frequently add new features to sanitizers and modify
compiler<->runtime interface, so the test-suite is kind of tied to the ToT
Clang.
But, at the moment we merge Sanitizer runtime into GCC tree at revison X,
we can cut off Sanitizer testsuite at revision X as well, and expect it to
work.

>
>
> On Wed, Feb 5, 2014 at 12:55 AM, Alexey Samsonov <samsonov at
google.com>
> wrote:
> >
> > On Tue, Feb 4, 2014 at 10:17 PM, Greg Fitzgerald <garious at
gmail.com>
> wrote:
> >>
> >> Regarding Brad King's CMake patches for LLVM:
> >>
> >>   http://thread.gmane.org/gmane.comp.compilers.llvm.cvs/173517
> >>
> >> If compiler-rt could follow suit, the packaging for the toolchain
> >> becomes a nice, clean package-manager-friendly DAG.  Currently,
the
> >> compiler-rt CMake build depends on the just-built-clang to run its
> >> test suite.  It's more intuitive to me that either:
> >>
> >>  1) compiler-rt's sanitizer tests would live inside the clang
project
> >>  2) compiler-rt would export a test suite that clang could
optionally
> >> import
> >>
> >> The former is simpler, but the latter allows the compiler-rt
> >> developers to retain control over those tests, which are typically
> >> updated alongside changes to compiler-rt's sanitizers. 
Thoughts?
> >
> >
> > I'm pretty sure the compiler-rt tests should live in the
compiler-rt
> > project. You're right:
> > these tests should generally be updated in sync with the runtimes.
I'm
> fine
> > with creating
> > a large test suite comprising all individual tool-specific test suites
we
> > have now. I'm not
> > sure how we can "export" this to clang, though, and why the
current
> > situation is bad.
> >
> >
> >>
> >>
> >> Thanks,
> >> Greg
> >
> >
> >
> >
> > --
> > Alexey Samsonov, MSK
>


-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140206/4efb680b/attachment.html>

On 02/06/2014 08:12 AM, Alexey Samsonov wrote:
> Please note that it makes a lot of sense to built compiler-rt (and
sanitizers) with just-built
> Clang. In fact, even though we should support building it with another
compilers (gcc, MSVC),
> using just-built-Clang should be a default scenario, like it is in
configure+make build.
This is possible with CMake using the ExternalProject module:

 http://www.cmake.org/cmake/help/v2.8.12/cmake.html#module:ExternalProject

The add_llvm_external_project macro already in LLVM could be taught
to call ExternalProject_Add instead of add_subdirectory.  This could
create a custom target with dependencies appropriately configured to
build after Clang.  The custom target would run CMake to configure the
project like an outside build and then launch the build.

IIUC there is a desire for Clang to be able to be built externally to
LLVM rather than subsumed into its build process.  The top-level
CMakeLists.txt file of Clang already has code to do that, though it
can be much cleaner after my patches to LLVM to provide LLVMConfig.cmake
are integrated.  Then it will even be possible to build Clang using
CMake against a LLVM that was built and installed using configure+make.

If one would like to drive compiler-rt as part of testing a Clang built
outside of LLVM then the appropriate place for the ExternalProject_Add
call would be inside the build of Clang.

Either way, from a quick glance at the top-level CMakeLists.txt file
in compiler-rt it appears to want to know a bunch of information about
Clang rather than LLVM.  In this case having Clang export enough info
for applications to write find_package(Clang) would make sense.  This
is possible to do an can be implemented using techniques similar to
that in my proposed LLVMConfig.cmake patch series.

-Brad