David Blaikie via llvm-dev
2017-Mar-11 00:25 UTC
[llvm-dev] Use of host/target compiler when building compiler-rt
On Thu, Mar 9, 2017 at 3:00 PM Chris Bieneman <beanz at apple.com> wrote:> I'll try and reproduce later today. Is this Linux? Can you give me your > CMake command line? >Excuse the delay, been busy setting up a new machine - also an opportunity to try clean cmake setups rather than my aging configurations that have a bunch of old stuff baked in and manual variables changed, etc. (& I do so wish that CMakeCache.txt had a comment at teh top describing the command that produced it like the config.log - that way I'd probably be more likely to copy/paste and modify the command line than hand editing the cache file) Anyway, so I don't have the compounding issue (local libstdc++ install with ABI incompatibility to my system libstdc++) so I can build/test compiler-rt when it's in projects at the moment (though I will soon locally install GCC 6.3 and have the problems with xray again). But when I place compiler-rt in runtimes I do have problems. Here's my cmake command: cmake -G Ninja -DCMAKE_INSTALL_PREFIX=$HOME/dev/llvm/install -DCMAKE_BUILD_TYPE=Debug ../../src/ -DCMAKE_C_COMPILER=$HOME/install/bin/clang -DCMAKE_CXX_COMPILER=$HOME/install/bin/clang++ -DLLVM_USE_SPLIT_DWARF=ON -DLLVM_ENABLE_WERROR=ON -DLLVM_BUILD_EXAMPLES=ON -DLLVM_INCLUDE_GO_TESTS=OFF & here's the result of "ninja check-all": [3671/3683] cd /usr/local/google/home/blaikie/dev/llvm/build/default...fault/runtimes/runtimes-bins/ --target check-runtimes --config Debug [318/409] Generating tsan_mman_test.cc.x86_64.o FAILED: compiler-rt/lib/tsan/tests/unit/tsan_mman_test.cc.x86_64.o cd /usr/local/google/home/blaikie/dev/llvm/build/default/runtimes/runtimes-bins/compiler-rt/lib/tsan/tests/unit && /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/clang -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -std=c++11 -fcolor-diagnostics -Wall -Werror -std=c++11 -Wno-unused-parameter -Wno-unknown-warning-option -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fvisibility-inlines-hidden -fno-lto -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-non-virtual-dtor -fPIE -fno-rtti -Wno-covered-switch-default -DGTEST_NO_LLVM_RAW_OSTREAM=1 -DGTEST_HAS_RTTI=0 -I/usr/local/google/home/blaikie/dev/llvm/src/utils/unittest/googletest/include -I/usr/local/google/home/blaikie/dev/llvm/src/utils/unittest/googletest -I/usr/local/google/home/blaikie/dev/llvm/src/runtimes/compiler-rt/lib -I/usr/local/google/home/blaikie/dev/llvm/src/runtimes/compiler-rt/lib/tsan/rtl -DGTEST_HAS_RTTI=0 -m64 -c -o tsan_mman_test.cc.x86_64.o /usr/local/google/home/blaikie/dev/llvm/src/runtimes/compiler-rt/lib/tsan/tests/unit/tsan_mman_test.cc /usr/local/google/home/blaikie/dev/llvm/src/runtimes/compiler-rt/lib/tsan/tests/unit/tsan_mman_test.cc:14:10: fatal error: 'sanitizer/allocator_interface.h' file not found #include <sanitizer/allocator_interface.h> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 error generated. [370/409] Generating ASAN_INST_TEST_OBJECTS.asan_test.cc.x86_64-inline.o> -Chris > > On Mar 9, 2017, at 11:45 AM, David Blaikie <dblaikie at gmail.com> wrote: > > > On Thu, Mar 9, 2017 at 11:25 AM Chris Bieneman <beanz at apple.com> wrote: > > On Mar 8, 2017, at 4:42 PM, David Blaikie <dblaikie at gmail.com> wrote: > > > > On Wed, Mar 8, 2017 at 3:23 PM Chris Bieneman <beanz at apple.com> wrote: > > On Mar 8, 2017, at 3:16 PM, David Blaikie <dblaikie at gmail.com> wrote: > > > > On Wed, Mar 8, 2017 at 2:55 PM Chris Bieneman <beanz at apple.com> wrote: > > David, > > This is an area that has had a lot of development over the last two years. > > There are two supported ways in the LLVM build system to build compiler-rt > with the just-built compiler. > > 1) The legacy way is for if compiler-rt is under LLVM/projects. You can > specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure > compiler-rt using the just-built clang after clang is built. > > > I thought the BUILD_EXTERNAL variables were for use when projects were not > embedded within the llvm source tree (mostly in use by Takumi's flat > buildbots that checout the top-level project without embedding, say, clang > or compiler-rt within the llvm source tree)? > > > You are confusing this with the similarly > named LLVM_EXTERNAL_${nameUPPER}_SOURCE_DIR variables. > > > Ah, right - indeed. > > > 2) The new way, is to place compiler-rt under LLVM/runtimes. In this path > the build system will automatically build with the just-built compiler. > This path also splits compiler-rt into two separate build steps, one that > configures and builds the builtins with the just-built compiler, and a > second that configures and builds the sanitizer libraries. > > > Huh, OK - could someone remove the legacy format, then? If it's a trap. > > > I'm not sure the new path is fully supported in every workflow, so > removing it seems like a not great idea at the moment. > > > That said, I tried putting compiler-rt in runtimes instead of projects and > I got a bunch of cmake errors starting with: > > CMake Error at > /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 > (add_dependencies): > The dependency target "GotsanRuntimeCheck" of target "check-runtimes" > does > not exist. > Call Stack (most recent call first): > CMakeLists.txt:110 (add_lit_target) > > Any ideas? > > > I have never encountered that issue. It looks like the tsan test code is > out of sync. If you go into tsan/test/CMakeLists.txt and on Line 2 add this > to the if statement "AND TARGET GotsanRuntimeCheck" that should fix the > issue. > > > Hrm - not sure which CMakeLists.txt you're referring to? In my > runtimes/compiler-rt/lib/tsan/tests/CMakeLists.txt the first few lines are: > > > Sorry, I meant <compiler-rt>/test/tsan/CMakeLists.txt, not tsan/test. > > > Think that got me past that error, but (sorry I wasn't clear) it was only > one of many errors. Here are the next... 5, say... > > CMake Error at > /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 > (add_dependencies): > The dependency target "TsanUnitTests" of target "check-runtimes" does not > exist. > Call Stack (most recent call first): > CMakeLists.txt:110 (add_lit_target) > > > CMake Error at > /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 > (add_dependencies): > The dependency target "cfi" of target "check-runtimes" does not exist. > Call Stack (most recent call first): > CMakeLists.txt:110 (add_lit_target) > > > CMake Error at > /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 > (add_dependencies): > The dependency target "TsanUnitTests" of target "check-compiler-rt" does > not exist. > Call Stack (most recent call first): > compiler-rt/test/CMakeLists.txt:94 (add_lit_target) > > > CMake Error at > /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 > (add_dependencies): > The dependency target "cfi" of target "check-compiler-rt" does not exist. > Call Stack (most recent call first): > compiler-rt/test/CMakeLists.txt:94 (add_lit_target) > > > CMake Error at > /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 > (add_dependencies): > The dependency target "TsanUnitTests" of target "check-tsan" does not > exist. > Call Stack (most recent call first): > > /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1195 > (add_lit_target) > compiler-rt/test/tsan/CMakeLists.txt:46 (add_lit_testsuite) > > > I'm assuming there's some systemic problem with the way I'm holding this > (if it's working for other people)? > > - Dave > > > > -Chris > > > include_directories(../rtl) > > add_custom_target(TsanUnitTests) > set_target_properties(TsanUnitTests PROPERTIES > FOLDER "TSan unittests") > > no if condition I could modify? > > > > -Chris > > > > > The second path also works for many (but not all) of our other runtime > library projects. I know it works for libcxx, libcxxabi, and > libunwind. Petr Hosek (CC'd) has also been working on support for > multi-arch builtin and runtime library builds so that you can generate full > cross-compilers from a single cmake invocation. > > -Chris > > > On Mar 8, 2017, at 2:35 PM, David Blaikie via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > > > On Wed, Mar 8, 2017 at 2:03 PM Sterling Augustine <saugustine at google.com> > wrote: > > Yes, this is a aspect of the larger problem that clang bootstrap doesn't > work for a cross-compiler. The build (mostly?) assumes that host==target > during the build of clang itself, and then if you want another architecture > also, you run a second build of the target libraries, and manually merge > the trees. > > > I kind of roughly follow that, but not too well. > > > If you think about compiler-rt as being compiled for the target rather > than the host, the problem you describe here is exactly the same one, and > we have been getting lucky. > > > Sure - if a PPC clang is being built from an x86 host, how would > compiler-rt be built (OK, it could be built with the just-built clang, > which it isn't at the moment) and tested (can't really be tested because > the host can't run PPC binaries). > > > At the moment, the blaze builds of clang do exactly the procedure > described above, so this hasn't been a terrible problem for Google, but I > do think it is something that should be fixed (I'm working on another > aspect of compiler-rt bringup at the moment, so won't solve this in the > immediate future.) > > > Rightio > > > > gnu systems have a make variable, "CC_FOR_TARGET" that addresses this > problem. I imagine llvm should adopt a similar mechanism inside cmake. > > > Not sure I follow on the need/use of CC_FOR_TARGET compared to using the > just-built clang as the CC_FOR_TARGET (which it seems we have some plumbing > for already - the just-built clang is used for building the compiler-rt > tests, but not for building the library. I /think/ it should be used for > both) > > - Dave > > > > On Wed, Mar 8, 2017 at 1:54 PM, David Blaikie <dblaikie at gmail.com> wrote: > > I stumbled across what seems to be a bug (to me) in the compiler-rt build: > > The compiler-rt libraries themselves are built with the host compiler > while the tests are built and then linked with the just-built clang. > > It was my understanding that the goal/intent/need was to have the > compiler-rt library build with the just-built clang? Did I misunderstand > that?* > > Sterling: Chandler seemed to think you might be interested in this issue & > possibly addressing it given you're working on compiler-rt bring-up? It'd > probably be useful to have compiler-rt built with the just-built clang for > performance reasons. > > Evgeniy - not sure if you're interested in this or have much context? Know > if this is right/wrong/neutral, etc? > > * reasons include performance, ABI compatibility, etc (I thought this was > necessary for correctness in some way) - also, otherwise it seems excessive > to hold up the whole build on waiting for just-built clang to finish, then > use that to compile some tests. (well, I realize some of the tests are > end-to-end, so they do need the just-built compiler) > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170311/f5b01fb8/attachment.html>
Matthias Braun via llvm-dev
2017-Mar-11 01:57 UTC
[llvm-dev] Use of host/target compiler when building compiler-rt
> On Mar 10, 2017, at 4:25 PM, David Blaikie via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > > > On Thu, Mar 9, 2017 at 3:00 PM Chris Bieneman <beanz at apple.com <mailto:beanz at apple.com>> wrote: > I'll try and reproduce later today. Is this Linux? Can you give me your CMake command line? > > Excuse the delay, been busy setting up a new machine - also an opportunity to try clean cmake setups rather than my aging configurations that have a bunch of old stuff baked in and manual variables changed, etc. (& I do so wish that CMakeCache.txt had a comment at teh top describing the command that produced it like the config.log - that way I'd probably be more likely to copy/paste and modify the command line than hand editing the cache file)At least newer cmakes have something like: LLVM_TARGETS_TO_BUILD-MODIFIED:INTERNAL=ON in the cache file to indicate that the user changed LLVM_TARGET_TOBUILD. So at least the information is there and you can manually look around / reproduce it. Unfortunately I don't know a command that automatically cleans the cache except for the XXX-MODIFIED variables... - Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170310/854153fd/attachment.html>
Matthias Braun via llvm-dev
2017-Mar-11 02:07 UTC
[llvm-dev] Use of host/target compiler when building compiler-rt
> On Mar 10, 2017, at 5:57 PM, Matthias Braun via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > >> On Mar 10, 2017, at 4:25 PM, David Blaikie via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> >> >> On Thu, Mar 9, 2017 at 3:00 PM Chris Bieneman <beanz at apple.com <mailto:beanz at apple.com>> wrote: >> I'll try and reproduce later today. Is this Linux? Can you give me your CMake command line? >> >> Excuse the delay, been busy setting up a new machine - also an opportunity to try clean cmake setups rather than my aging configurations that have a bunch of old stuff baked in and manual variables changed, etc. (& I do so wish that CMakeCache.txt had a comment at teh top describing the command that produced it like the config.log - that way I'd probably be more likely to copy/paste and modify the command line than hand editing the cache file) > At least newer cmakes have something like: > LLVM_TARGETS_TO_BUILD-MODIFIED:INTERNAL=ON > in the cache file to indicate that the user changed LLVM_TARGET_TOBUILD. > > So at least the information is there and you can manually look around / reproduce it. Unfortunately I don't know a command that automatically cleans the cache except for the XXX-MODIFIED variables...Just checked this again, it seems the XXX-MODIFIED bits are only set if you change the variable through ccmake, but not when you changed it with -D on the commandline. That doesn't really help then for most people. - Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170310/3b1a3847/attachment.html>