On Thu, Apr 3, 2014 at 5:04 AM, Greg Fitzgerald <garious at gmail.com>
wrote:
> > we would still want to use compiler-rt test-suite in a standalone
mode,
> to test fully built/installed toolchains,
> and even GCC.
>
> Sounds good.
>
>
> > Clang driver links the static xsan runtimes from a hardcoded
> > paths in Clang resource directory, and doesn't add flags like
> > "-lasan -L/path/to/clang/resource/dir". I find this behavior
reasonable.
>
> Can we change the flags to "-lasan
-L/path/to/clang/resource/dir"? If
> we make that change,
I don't think it's a good idea to let user hijack the driver and stuff
in
custom version of
ASan runtime instead the one installed/built with compiler :)
> I can stop suggesting that the clang build should
> invoke the compiler-rt test suite directly. :-)
>
I haven't yet fully understood your complaints. Note that if you
build/configure compiler-rt
in a standalone mode, then "make check-compiler-rt" in that tree
wouldn't
rebuild runtimes,
and instead assume that they are provided by toolchain (that is, in
standalone mode runtimes and
test suites are effectively independent). What do you mean by "invoking the
test-suite directly"?
>
>
> > Do you want to give it a try?
>
> I'll take a crack it, sure.
>
>
> By the way, locally, I now have just over half the ASan test suite
> passing ARM-Linux via QEMU. It's been really convenient to build
> compiler-rt in standalone mode. The edit-compile-run cycle is much
> shorter this way. Thanks for your help in making this work!
>
> -Greg
>
>
>
> On Wed, Apr 2, 2014 at 7:56 AM, Alexey Samsonov <samsonov at
google.com>
> wrote:
> > Hi Greg,
> >
> > On Wed, Apr 2, 2014 at 2:50 AM, Greg Fitzgerald <garious at
gmail.com>
> wrote:
> >>
> >> Alexey, Evgeniy,
> >>
> >> I propose the following steps to unify multi-arch support in
> compiler-rt:
> >>
> >> 1) The compiler-rt test suite adds
"-L${CMAKE_CURRENT_BINARY_DIR}/lib"
> >> to its 'clang' variables. This way we can test the
sanitizers without
> >> installing any libs to the just-built-clang install directory.
> >
> >
> > This is possible, but please keep in mind that:
> > 1) we would still want to use compiler-rt test-suite in a standalone
> mode,
> > to test fully built/installed toolchains,
> > and even GCC.
> > 2) Adding -L... wouldn't work: Clang driver links the static xsan
> runtimes
> > from a hardcoded paths in Clang resource directory,
> > and doesn't add flags like "-lasan
-L/path/to/clang/resource/dir". I find
> > this behavior reasonable.
> >
> > I don't see a problem with the current approach - we can make
"run
> sanitizer
> > test suite" command in the
> > top-level build tree depend on "build/install compiler-rt
> ExternalProject"
> > steps (see how it's done currently).
> >
> >>
> >>
> >> 2) The "clang/runtime" build calls ExternalProject once
for each arch
> >> it needs of compiler-rt. So once to create x86_64 libs and once
for
> >> i386 libs.
> >>
> >> 3) The compiler-rt build drops the ${arch} suffix from its libs,
and
> >> the "clang/runtime" build uses CMAKE_INSTALL_PREFIX to
control where
> >> the compiler-rt libs go.
> >>
> >>
>
"CMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}/lib/clang/${CLANG_VERSION}/x86_64-linux"
> >>
> >>
> >> 4) Remove multi-arch support from the compiler-rt build. Instead,
> >> declare compiler-rt as an "any-arch" build, configured
with:
> >> CMAKE_CXX_COMPILER (defaults to just-built-clang)
> >> CMAKE_CXX_FLAGS (defaults to llvm's default target)
> >> COMPILER_RT_TEST_COMPILER (defaults to CMAKE_CXX_COMPILER)
> >> COMPILER_RT_TEST_FLAGS (defaults to CMAKE_CXX_FLAGS)
> >
> >
> > Unfortunately, this could be problematic for Mac - we use
"universal
> > binaries" there - a single static/dynamic runtime
> > which contains runtimes for all the possible architectures. We might
work
> > around this, however, by adding a custom
> > Mac-specific step that would merge all single-arch binaries into a
> > multi-arch one - I think this is fine as long as we
> > greatly simplify the rest of the ugly build system.
> >
> > Do you want to give it a try?
> >
> >>
> >>
> >> Thoughts?
> >>
> >> Thanks,
> >> Greg
> >>
> >> On Tue, Apr 1, 2014 at 12:58 AM, Evgeniy Stepanov
> >> <eugeni.stepanov at gmail.com> wrote:
> >> > It does sound like Android is better suited for
"honest"
> >> > cross-compilation, rather than "build compiler-rt for
all targets we
> >> > can find" model.
> >> >
> >> > I'm still not convinced that we must require the
"ninja install" step.
> >> > Could we just "ninja clang" and then build the
second stage against
> >> > the first stage build directory? Will this
"find_package" thing not
> >> > work that way, or do you mean it as a way to copy target asan
runtime
> >> > where the first stage compiler will find it (or probably
both)?
> >> >
> >> > On Mon, Mar 31, 2014 at 6:46 PM, Alexey Samsonov <samsonov
at google.com
> >
> >> > wrote:
> >> >>
> >> >> On Fri, Mar 28, 2014 at 10:56 PM, Greg Fitzgerald
<garious at gmail.com
> >
> >> >> wrote:
> >> >>>
> >> >>> > Note that ASan tests on Android require
llvm-symbolizer binary.
> >> >>>
> >> >>> That's a really good point. And I see that
llvm-symbolizer can't
> just
> >> >>> be pulled into compiler-rt because it has
dependencies on DebugInfo,
> >> >>> Object, and Support libraries.
> >> >>>
> >> >>>
> >> >>> This throws a big wrench in Alexey's plan to have
the native
> >> >>> compiler-rt build generate the cross-compiled
binaries for all
> >> >>> supported targets. We would need to do the same in
the llvm repo.
> >> >>> The alternative is to run a separate cross-compiled
build for each
> >> >>> architecture. :-/
> >> >>
> >> >>
> >> >> Yes, this llvm-symbolizer part is frustrating... I though
we can get
> >> >> away
> >> >> with it at first, as llvm-symbolizer is optional, and we
generally
> >> >> won't be
> >> >> able
> >> >> to produce working *binaries* for architectures different
from the
> host
> >> >> one
> >> >> (we might not have a linker, for instance).
> >> >> Android with its NDK looks like an exception, though.
> >> >>
> >> >>>
> >> >>>
> >> >>>
> >> >>> > Yes, so "ninja install" is optional.
> >> >>>
> >> >>> But the 'install' depends on the
'all' target.
> >> >>>
> >> >>> -Greg
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Mar 28, 2014 at 12:39 AM, Evgeniy Stepanov
> >> >>> <eugeni.stepanov at gmail.com> wrote:
> >> >>> > On Thu, Mar 27, 2014 at 10:18 PM, Greg
Fitzgerald
> >> >>> > <garious at gmail.com>
> >> >>> > wrote:
> >> >>> >>> Alexey's approach with CMake
sub-projects.
> >> >>> >>
> >> >>> >> I prefer that direction as well, but what
I've proposed is a
> >> >>> >> solution
> >> >>> >> that works today. To support
cross-compilation, we'll need to
> loop
> >> >>> >> over each supported arch (llvm-config
--targets-built), then loop
> >> >>> >> over
> >> >>> >> each supported triple for each arch
(hard-coded map?), and then
> >> >>> >> pair
> >> >>> >> up each triple with a sysroot (system paths
provided by the
> user).
> >> >>> >>
> >> >>> >>
> >> >>> >>> I think it already copies
> >> >>> >>> compiler-rt build products to the parent
build directory, right?
> >> >>> >>
> >> >>> >> The Android compiler-rt build does this, but
in an unnecessarily
> >> >>> >> complex way. My proposal is a
simplification of that process.
> >> >>> >> What
> >> >>> >> is described in
"llvm/cmake/platforms/Android.cmake" is to build
> >> >>> >> llvm+clang for the host architecture and
then run a second build
> to
> >> >>> >> cross-compile llvm+compiler-rt to get a the
ASan shared object
> and
> >> >>> >> test suite. Instead, I propose building
only llvm for the host
> >> >>> >> architecture (for the purpose of building
llvm-config) and then
> >> >>> >> only
> >> >>> >> compile compiler-rt for Android. No need to
build clang or to
> >> >>> >> cross-compile llvm.
> >> >>> >>
> >> >>> >>
> >> >>> >>> Requiring that llvm is installed
complicates things.
> >> >>> >>
> >> >>> >> Sorry for the confusion. By "llvm
install directory", I don't
> mean
> >> >>> >> to
> >> >>> >> install llvm to the root directory. I'm
referring to the
> >> >>> >> intermediary
> >> >>> >> directory pointed to by
CMAKE_INSTALL_PREFIX. When you run
> "ninja
> >> >>> >> install", the build populates this
directory with only the libs,
> >> >>> >> headers, docs, and executables that the LLVM
build intends
> >> >>> >> customers
> >> >>> >> to use directly. In the case of
compiler-rt, we need LLVM's
> >> >>> >> "bin/llvm-config" to determine the
build mode, flags, and targets
> >> >>> >> available.
> >> >>> >
> >> >>> > Yes, so "ninja install" is optional.
This should work with LLVM
> >> >>> > build
> >> >>> > directory just as well.
> >> >>> >
> >> >>> >>
> >> >>> >>
> >> >>> >>> This should not be too hard to fix.
> >> >>> >>
> >> >>> >> Porting the LLVM build to Android? That
seems like a more
> >> >>> >> challenging
> >> >>> >> solution than to cut out the unused
cross-compiled llvm build.
> >> >>> >
> >> >>> > Note that ASan tests on Android require
llvm-symbolizer binary.
> >> >>> > Building just compiler-rt for the target would
not be enough.
> >> >>> >
> >> >>> >>
> >> >>> >> -Greg
> >> >>> >>
> >> >>> >> On Thu, Mar 27, 2014 at 1:14 AM, Evgeniy
Stepanov
> >> >>> >> <eugeni.stepanov at gmail.com> wrote:
> >> >>> >>> I'd prefer something based on
Alexey's approach with CMake
> >> >>> >>> sub-projects. I think it already copies
compiler-rt build
> products
> >> >>> >>> to
> >> >>> >>> the parent build directory, right?
> >> >>> >>>
> >> >>> >>> Requiring that llvm is installed
complicates things.
> >> >>> >>>
> >> >>> >>> On Thu, Mar 27, 2014 at 5:00 AM, Greg
Fitzgerald
> >> >>> >>> <garious at gmail.com>
> >> >>> >>> wrote:
> >> >>> >>>> The build for the Android sanitizers
is unique in that it needs
> >> >>> >>>> to
> >> >>> >>>> link against the Android system
libraries to create a shared
> >> >>> >>>> object
> >> >>> >>>> and its test suites. The current
solution to build ASan is to
> >> >>> >>>> drop
> >> >>> >>>> the compiler-rt repo into the llvm
source tree and
> cross-compile
> >> >>> >>>> the
> >> >>> >>>> llvm build for Android. This is a
bit awkward for few reasons:
> >> >>> >>>>
> >> >>> >>>> 1) Not all of llvm can be
cross-compiled for Android. "ninja
> >> >>> >>>> all"
> >> >>> >>>> causes expected build errors.
> >> >>> >>>
> >> >>> >>> This should not be too hard to fix.
> >> >>> >>>
> >> >>> >>>> 2) The sanitizers don't depend
on any LLVM libraries, only
> >> >>> >>>> llvm-config
> >> >>> >>>> for its build configuration.
> >> >>> >>>> 3) No "install" rule.
Instead, you cherry-pick files from the
> >> >>> >>>> build
> >> >>> >>>> directory.
> >> >>> >>>>
> >> >>> >>>> Building against the LLVM install
directory cleans this up
> nicely
> >> >>> >>>> and
> >> >>> >>>> as it turns out, this mostly works
today (see CMake
> configuration
> >> >>> >>>> below). The only missing pieces are
that the shared object is
> >> >>> >>>> not
> >> >>> >>>> added to the install directory and
the test suites are not
> built.
> >> >>> >>>> Is
> >> >>> >>>> this a build configuration you'd
consider using?
> >> >>> >>>>
> >> >>> >>>> $ cmake -G Ninja .. \
> >> >>> >>>> -DCMAKE_INSTALL_PREFIX=ship \
> >> >>> >>>>
-DCMAKE_C_COMPILER=arm-linux-androideabi-gcc \
> >> >>> >>>>
-DCMAKE_CXX_COMPILER=arm-linux-androideabi-g++ \
> >> >>> >>>>
-DCMAKE_PREFIX_PATH=`pwd`/../../llvm/out/ship \
> >> >>> >>>> -DANDROID=1 \
> >> >>> >>>> -DCMAKE_SYSTEM_NAME=Linux \
> >> >>> >>>>
> >> >>> >>>>
> -DCMAKE_C_FLAGS=--sysroot=$(ndkDir)/platforms/android-19/arch-arm
> >> >>> >>>> \
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> -DCMAKE_CXX_FLAGS=--sysroot=$(ndkDir)/platforms/android-19/arch-arm
> >> >>> >>>> ...
> >> >>> >>>> $ ninja install
> >> >>> >>>> ...
> >> >>> >>>> [68/68]
> >> >>> >>>> -- Install configuration:
"Release"
> >> >>> >>>> -- Installing:
ship/include/sanitizer/asan_interface.h
> >> >>> >>>> -- Installing:
ship/include/sanitizer/common_interface_defs.h
> >> >>> >>>> -- Installing:
ship/include/sanitizer/dfsan_interface.h
> >> >>> >>>> -- Installing:
ship/include/sanitizer/linux_syscall_hooks.h
> >> >>> >>>> -- Installing:
ship/include/sanitizer/lsan_interface.h
> >> >>> >>>> -- Installing:
ship/include/sanitizer/msan_interface.h
> >> >>> >>>> -- Installing:
ship/include/sanitizer/tsan_interface_atomic.h
> >> >>> >>>> -- Installing:
ship/asan_blacklist.txt
> >> >>> >>>> -- Installing:
ship/bin/asan_device_setup
> >> >>> >>>>
> >> >>> >>>> Thanks,
> >> >>> >>>> Greg
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Alexey Samsonov, MSK
> >
> >
> >
> >
> > --
> > Alexey Samsonov, MSK
>
--
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140403/cb1d86e9/attachment.html>