llvm dev - Sep 2015 - Link statically against libc++ on Cray XC 40

Hi folks,

In our scientific simulation tool, we set up our build system to link against
LLVMSupport to print stack traces for debugging purposes, also from other
compilers. This works very well on most hosts, but we fail to make it work with
gcc on a Cray XC 40 cluster. They have a very restrictive toolchain setup
(everything is controlled by the “module” command; you never get to see any real
paths or commands but just use “CC” as the compiler and everything is
included/linked to automagically).

I already could solve some of the problems but now I am stuck at the linking
stage. Specifically, their CC tool tries to link everything statically, but
since we built LLVM using libc++, we also need to link against libc++, which
fails with the error

/usr/bin/ld: attempted static link of dynamic object
`/zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib/libc++.so’

Does anyone have a suggestion what I can do to make this work (link)?

Regards,

Michael

P.S.: This is the full linker command minus some (many) object files and
-Wl,-rpath,library/paths:

/opt/cray/craype/2.4.1/bin/CC   -O0 -g3 -fno-inline -std=c++11 -pedantic 
-fdiagnostics-color=auto   
-L/zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib
<many_object_files_go_here> CMakeFiles/zfs.dir/main.cpp.o  -o zfs
-rdynamic /zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib/libLLVMSupport.a
-ldl -lpthread /zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib/libc++.so
/zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib/libc++abi.a


--
Michael Schlottke-Lakemper

Chair of Fluid Mechanics and Institute of Aerodynamics
RWTH Aachen University
Wüllnerstraße 5a
52062 Aachen
Germany

Phone: +49 (241) 80 95188
Fax: +49 (241) 80 92257
Mail: m.schlottke-lakemper at aia.rwth-aachen.de<mailto:m.schlottke-lakemper
at aia.rwth-aachen.de>
Web: http://www.aia.rwth-aachen.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150930/ad927679/attachment.html>

On Wed, Sep 30, 2015 at 1:12 PM, Schlottke-Lakemper, Michael via
llvm-dev <llvm-dev at lists.llvm.org> wrote:
> Hi folks,
>
> In our scientific simulation tool, we set up our build system to link
> against LLVMSupport to print stack traces for debugging purposes, also from
> other compilers. This works very well on most hosts, but we fail to make it
> work with gcc on a Cray XC 40 cluster. They have a very restrictive
> toolchain setup (everything is controlled by the “module” command; you
never
> get to see any real paths or commands but just use “CC” as the compiler and
> everything is included/linked to automagically).
>
> I already could solve some of the problems but now I am stuck at the
linking
> stage. Specifically, their CC tool tries to link everything statically, but
> since we built LLVM using libc++, we also need to link against libc++,
which
> fails with the error
>
> /usr/bin/ld: attempted static link of dynamic object
> `/zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib/libc++.so’
>
> Does anyone have a suggestion what I can do to make this work (link)?
>
> Regards,
>
> Michael
>
> P.S.: This is the full linker command minus some (many) object files and
> -Wl,-rpath,library/paths:
>
> /opt/cray/craype/2.4.1/bin/CC   -O0 -g3 -fno-inline -std=c++11 -pedantic
> -fdiagnostics-color=auto
> -L/zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib
> <many_object_files_go_here> CMakeFiles/zfs.dir/main.cpp.o  -o zfs
-rdynamic
> /zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib/libLLVMSupport.a -ldl
> -lpthread /zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib/libc++.so
> /zhome/academic/HLRS/xac/xacmicha/.pool/llvm/cur/lib/libc++abi.a
I don't recommend to use shared libs, but if you add CC -shared it
will do shared linking. Also if you add -v to compilation you can see
the collect2 line (which is basically the linker line). That info
would give you enough to manually figure out your -L and -rpath. From
there you can do a simple clang wrapper script to work-around modules.

The problem isn't going to be resolving static compilation, but making
mpiCC work. The modified mpich which Cray uses links *everything*
against GNU C++ STL and runtime. That will out-of-the-box conflict
with libc++. So you can get clang to work using the gnu
stl/runtime/libunwind, but libc++ is no-go.

If you *really* want to test libc++ and clang there's 2 options
1) Get Cray to build mpi with clang and libc++ (I suspect they will
pushback and not do this)
2) Email me off list and I'll share more of my work (which partially
overlaps with this)

side note: Why are you trying to pull in libLLVMSupport.a (I suspect
this may not be correct unless you're trying to build clang itself)

I don't recommend to use shared libs, but if you add CC -shared it
will do shared linking.
No, you are right, I would prefer not to do that, since that means I also have
to change other things in the compilation (like adding -fPIC everywhere).

Also if you add -v to compilation you can see
the collect2 line (which is basically the linker line). That info
would give you enough to manually figure out your -L and -rpath. From
there you can do a simple clang wrapper script to work-around modules.
My goal is not to use clang to compile & link our program, but rather just
to link against libLLVMSupport, so I already know all paths I need. Using clang
as the main compiler on the XC 40 is probably a little more challenging, so
unless someone from the computer center where the XC40 is located, or someone
from Cray directly, is willing to help us with that, we are not trying to go
there alone (we have asked but no reply yet).

The problem isn't going to be resolving static compilation, but making
mpiCC work. The modified mpich which Cray uses links *everything*
against GNU C++ STL and runtime. That will out-of-the-box conflict
with libc++. So you can get clang to work using the gnu
stl/runtime/libunwind, but libc++ is no-go.
Hm, maybe I don’t understand. We already do that (successfully) on our institute
cluster, where we compile with GCC as usual and just link against
libLLVMSupport, libdl, libtinfo, libpthread, libc++, and libc++abi, all in
addition to the normal libraries (including libstdc++). But I am not an expert
on such things, maybe I just did not grasp your intent.

side note: Why are you trying to pull in libLLVMSupport.a (I suspect
this may not be correct unless you're trying to build clang itself)
We use its "llvm::sys::PrintStackTrace(llvm::errs())" functionality as
a portable and easy-to-use (at least on hosts where clang is preinstalled) way
to print stacktraces in case of self-detected runtime errors. Before we used a
hack using GCC extensions and calls to “addr2line”, but that works only with GCC
(and not reliably).

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150930/5063e497/attachment.html>