llvm dev - Feb 2019 - How do I run llvm's asan tests?

Hi llvm-dev,

I'm trying to figure out how to contribute to LLVM, in particular a followup
to
kcc's commit 6bde702a in sanitzer_suppressions.cc. However I can't find
a way
to get the tests to run before I even change anything.

The relevant unit test is
compiler-rt/lib/sanitizer_common/tests/sanitizer_suppressions_test.cc, so I
expect I want `ninja check-asan` (right?). Here's what I tried, on my Debian
machine, following (and adjusting) the outdated llvm documentation about
setting up a GCC toolchain:

    # Install the GCC toolchain.
    # See
http://llvm.org/docs/GettingStarted.html#getting-a-modern-host-c-toolchain
    # Adjusted for GCC 7.4.0, which doesn't ship in a bzip2 archive.
    gcc_version=7.4.0
    wget
https://ftp.gnu.org/gnu/gcc/gcc-${gcc_version}/gcc-${gcc_version}.tar.gz
    wget
https://ftp.gnu.org/gnu/gcc/gcc-${gcc_version}/gcc-${gcc_version}.tar.gz.sig
    wget https://ftp.gnu.org/gnu/gnu-keyring.gpg
    signature_invalid=`gpg --verify --no-default-keyring --keyring
./gnu-keyring.gpg gcc-${gcc_version}.tar.gz.sig`
    if [ $signature_invalid ]; then echo "Invalid signature" ; exit 1
; fi
    tar -xvzf gcc-${gcc_version}.tar.gz
    cd gcc-${gcc_version}
    ./contrib/download_prerequisites
    cd ..
    mkdir gcc-${gcc_version}-build
    cd gcc-${gcc_version}-build
    mkdir -p $HOME/toolchains
    $PWD/../gcc-${gcc_version}/configure --prefix=$HOME/toolchains
--enable-languages=c,c++
    make -j$(nproc)
    make install

    # Clone llvm and try to test it.
    mkdir -p ~/clients
    cd ~/clients
    git clone https://github.com/llvm/llvm-project.git
    # Get a coffee
    cd llvm-project
    mkdir build
    cd build
    CC=$HOME/toolchains/bin/gcc CXX=$HOME/toolchains/bin/g++ \
      cmake -G Ninja
-DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$HOME/toolchains/lib64
-L$HOME/toolchains/lib64"
-DLLVM_ENABLE_PROJECTS='clang;compiler-rt;libcxx;libcxxabi' ../llvm
    ninja check-asan
    # Get another coffee

After about 30 minutes, the ninja command fails with an error about how the
string header isn't found:

    FAILED:
projects/compiler-rt/lib/asan/tests/ASAN_NOINST_TEST_OBJECTS.asan_noinst_test.cc.x86_64-inline.o
    cd
/usr/local/google/home/jacobsa/clients/llvm-project/build/projects/compiler-rt/lib/asan/tests
&& /usr/local/google/home/jacobsa/clients/llvm-project/build/./bin/clang
-fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall
-Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual
-Wno-missing-field-initializers -pedantic -Wno-long-long
-Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type
-Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -Wall
-std=c++11 -Wno-unused-parameter -Wno-unknown-warning-option
-DGTEST_NO_LLVM_RAW_OSTREAM=1 -DGTEST_HAS_RTTI=0
-I/usr/local/google/home/jacobsa/clients/llvm-project/llvm/utils/unittest/googletest/include
-I/usr/local/google/home/jacobsa/clients/llvm-project/llvm/utils/unittest/googletest
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/include
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/sanitizer_common/tests
-fno-rtti -O2 -Wno-format -Werror=sign-compare -Wno-non-virtual-dtor
-Wno-variadic-macros -gline-tables-only -DASAN_HAS_BLACKLIST=1
-DASAN_HAS_EXCEPTIONS=1 -DASAN_UAR=0 -m64 -c -o
ASAN_NOINST_TEST_OBJECTS.asan_noinst_test.cc.x86_64-inline.o
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_noinst_test.cc
    In file included from
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_noinst_test.cc:17:
    In file included from
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_test_utils.h:18:
   
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_test_config.h:19:10:
fatal error: 'string' file not found
    #include <string>
             ^~~~~~~~
    1 error generated.

This doesn't happen for `ninja check-all` (but despite its name that target
doesn't seem to run the test), so I wonder if it's a hermeticism issue
in the
asan tests? It's also very possible I've just done it wrong. Help?

Thanks,
Aaron

Your steps seem to be correct....

I've just done this, and it worked fine:

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build
cd build
cmake -G Ninja -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;' ../llvm
ninja check-asan

My gcc is "gcc (Debian 7.3.0-5) 7.3.0"

No good ideas what's broken on your side...

--kcc


On Tue, Feb 19, 2019 at 5:58 PM Aaron Jacobs via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Hi llvm-dev,
>
> I'm trying to figure out how to contribute to LLVM, in particular a
followup to
> kcc's commit 6bde702a in sanitzer_suppressions.cc. However I can't
find a way
> to get the tests to run before I even change anything.
>
> The relevant unit test is
> compiler-rt/lib/sanitizer_common/tests/sanitizer_suppressions_test.cc, so I
> expect I want `ninja check-asan` (right?). Here's what I tried, on my
Debian
> machine, following (and adjusting) the outdated llvm documentation about
> setting up a GCC toolchain:
>
>     # Install the GCC toolchain.
>     # See
http://llvm.org/docs/GettingStarted.html#getting-a-modern-host-c-toolchain
>     # Adjusted for GCC 7.4.0, which doesn't ship in a bzip2 archive.
>     gcc_version=7.4.0
>     wget
https://ftp.gnu.org/gnu/gcc/gcc-${gcc_version}/gcc-${gcc_version}.tar.gz
>     wget
https://ftp.gnu.org/gnu/gcc/gcc-${gcc_version}/gcc-${gcc_version}.tar.gz.sig
>     wget https://ftp.gnu.org/gnu/gnu-keyring.gpg
>     signature_invalid=`gpg --verify --no-default-keyring --keyring
> ./gnu-keyring.gpg gcc-${gcc_version}.tar.gz.sig`
>     if [ $signature_invalid ]; then echo "Invalid signature" ;
exit 1 ; fi
>     tar -xvzf gcc-${gcc_version}.tar.gz
>     cd gcc-${gcc_version}
>     ./contrib/download_prerequisites
>     cd ..
>     mkdir gcc-${gcc_version}-build
>     cd gcc-${gcc_version}-build
>     mkdir -p $HOME/toolchains
>     $PWD/../gcc-${gcc_version}/configure --prefix=$HOME/toolchains
> --enable-languages=c,c++
>     make -j$(nproc)
>     make install
>
>     # Clone llvm and try to test it.
>     mkdir -p ~/clients
>     cd ~/clients
>     git clone https://github.com/llvm/llvm-project.git
>     # Get a coffee
>     cd llvm-project
>     mkdir build
>     cd build
>     CC=$HOME/toolchains/bin/gcc CXX=$HOME/toolchains/bin/g++ \
>       cmake -G Ninja
> -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$HOME/toolchains/lib64
> -L$HOME/toolchains/lib64"
> -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;libcxx;libcxxabi' ../llvm
>     ninja check-asan
>     # Get another coffee
>
> After about 30 minutes, the ninja command fails with an error about how the
> string header isn't found:
>
>     FAILED:
projects/compiler-rt/lib/asan/tests/ASAN_NOINST_TEST_OBJECTS.asan_noinst_test.cc.x86_64-inline.o
>     cd
/usr/local/google/home/jacobsa/clients/llvm-project/build/projects/compiler-rt/lib/asan/tests
> &&
/usr/local/google/home/jacobsa/clients/llvm-project/build/./bin/clang
> -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall
> -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual
> -Wno-missing-field-initializers -pedantic -Wno-long-long
> -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type
> -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -Wall
> -std=c++11 -Wno-unused-parameter -Wno-unknown-warning-option
> -DGTEST_NO_LLVM_RAW_OSTREAM=1 -DGTEST_HAS_RTTI=0
>
-I/usr/local/google/home/jacobsa/clients/llvm-project/llvm/utils/unittest/googletest/include
>
-I/usr/local/google/home/jacobsa/clients/llvm-project/llvm/utils/unittest/googletest
> -I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/include
> -I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib
> -I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan
>
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/sanitizer_common/tests
> -fno-rtti -O2 -Wno-format -Werror=sign-compare -Wno-non-virtual-dtor
> -Wno-variadic-macros -gline-tables-only -DASAN_HAS_BLACKLIST=1
> -DASAN_HAS_EXCEPTIONS=1 -DASAN_UAR=0 -m64 -c -o
> ASAN_NOINST_TEST_OBJECTS.asan_noinst_test.cc.x86_64-inline.o
>
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_noinst_test.cc
>     In file included from
>
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_noinst_test.cc:17:
>     In file included from
>
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_test_utils.h:18:
>    
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_test_config.h:19:10:
> fatal error: 'string' file not found
>     #include <string>
>              ^~~~~~~~
>     1 error generated.
>
> This doesn't happen for `ninja check-all` (but despite its name that
target
> doesn't seem to run the test), so I wonder if it's a hermeticism
issue in the
> asan tests? It's also very possible I've just done it wrong. Help?
>
> Thanks,
> Aaron
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

clang can't find your STL headers, I guess. You probably have some GCC
libraries installed in /usr/lib for some recent version of GCC (8.N), but
no headers for that same version. Clang's GCC installation detection logic
will find the libraries, and use the corresponding version to calculate
standard library include paths.

The easiest fix is probably to run `sudo apt-get install libstdc++-8-dev`,
assuming you have some libraries for libstdc++8 but no headers.

On Tue, Feb 19, 2019 at 5:58 PM Aaron Jacobs via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi llvm-dev,
>
> I'm trying to figure out how to contribute to LLVM, in particular a
> followup to
> kcc's commit 6bde702a in sanitzer_suppressions.cc. However I can't
find a
> way
> to get the tests to run before I even change anything.
>
> The relevant unit test is
> compiler-rt/lib/sanitizer_common/tests/sanitizer_suppressions_test.cc, so I
> expect I want `ninja check-asan` (right?). Here's what I tried, on my
> Debian
> machine, following (and adjusting) the outdated llvm documentation about
> setting up a GCC toolchain:
>
>     # Install the GCC toolchain.
>     # See
> http://llvm.org/docs/GettingStarted.html#getting-a-modern-host-c-toolchain
>     # Adjusted for GCC 7.4.0, which doesn't ship in a bzip2 archive.
>     gcc_version=7.4.0
>     wget
> https://ftp.gnu.org/gnu/gcc/gcc-${gcc_version}/gcc-${gcc_version}.tar.gz
>     wget
>
https://ftp.gnu.org/gnu/gcc/gcc-${gcc_version}/gcc-${gcc_version}.tar.gz.sig
>     wget https://ftp.gnu.org/gnu/gnu-keyring.gpg
>     signature_invalid=`gpg --verify --no-default-keyring --keyring
> ./gnu-keyring.gpg gcc-${gcc_version}.tar.gz.sig`
>     if [ $signature_invalid ]; then echo "Invalid signature" ;
exit 1 ; fi
>     tar -xvzf gcc-${gcc_version}.tar.gz
>     cd gcc-${gcc_version}
>     ./contrib/download_prerequisites
>     cd ..
>     mkdir gcc-${gcc_version}-build
>     cd gcc-${gcc_version}-build
>     mkdir -p $HOME/toolchains
>     $PWD/../gcc-${gcc_version}/configure --prefix=$HOME/toolchains
> --enable-languages=c,c++
>     make -j$(nproc)
>     make install
>
>     # Clone llvm and try to test it.
>     mkdir -p ~/clients
>     cd ~/clients
>     git clone https://github.com/llvm/llvm-project.git
>     # Get a coffee
>     cd llvm-project
>     mkdir build
>     cd build
>     CC=$HOME/toolchains/bin/gcc CXX=$HOME/toolchains/bin/g++ \
>       cmake -G Ninja
> -DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$HOME/toolchains/lib64
> -L$HOME/toolchains/lib64"
> -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;libcxx;libcxxabi' ../llvm
>     ninja check-asan
>     # Get another coffee
>
> After about 30 minutes, the ninja command fails with an error about how the
> string header isn't found:
>
>     FAILED: projects/compiler-rt/lib/asan/tests/ASAN_NOINST_TEST_
> OBJECTS.asan_noinst_test.cc.x86_64-inline.o
>     cd
>
/usr/local/google/home/jacobsa/clients/llvm-project/build/projects/compiler-rt/lib/asan/tests
> &&
/usr/local/google/home/jacobsa/clients/llvm-project/build/./bin/clang
> -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall
> -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual
> -Wno-missing-field-initializers -pedantic -Wno-long-long
> -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type
> -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -Wall
> -std=c++11 -Wno-unused-parameter -Wno-unknown-warning-option
> -DGTEST_NO_LLVM_RAW_OSTREAM=1 -DGTEST_HAS_RTTI=0
>
>
-I/usr/local/google/home/jacobsa/clients/llvm-project/llvm/utils/unittest/googletest/include
>
>
-I/usr/local/google/home/jacobsa/clients/llvm-project/llvm/utils/unittest/googletest
> -I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/include
> -I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib
> -I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan
>
>
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/sanitizer_common/tests
> -fno-rtti -O2 -Wno-format -Werror=sign-compare -Wno-non-virtual-dtor
> -Wno-variadic-macros -gline-tables-only -DASAN_HAS_BLACKLIST=1
> -DASAN_HAS_EXCEPTIONS=1 -DASAN_UAR=0 -m64 -c -o
> ASAN_NOINST_TEST_OBJECTS.asan_noinst_test.cc.x86_64-inline.o
>
>
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_noinst_test.cc
>     In file included from
>
>
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_noinst_test.cc:17:
>     In file included from
>
>
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_test_utils.h:18:
>
>
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_test_config.h:19:10:
> fatal error: 'string' file not found
>     #include <string>
>              ^~~~~~~~
>     1 error generated.
>
> This doesn't happen for `ninja check-all` (but despite its name that
target
> doesn't seem to run the test), so I wonder if it's a hermeticism
issue in
> the
> asan tests? It's also very possible I've just done it wrong. Help?
>
> Thanks,
> Aaron
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190220/caed33f3/attachment.html>

Thanks Kostya and Reid for confirming the usual process. After a tremendous
amount of faffing around, I did manage to make this work on a Google Cloud
Engine instance that I created from scratch, as follows:

    gcloud compute instances create debian --image-project
debian-cloud --image-family debian-9 --machine-type n1-standard-16
--project <redacted> --zone australia-southeast1-a --boot-disk-size
1TB
    gcloud compute ssh debian

    sudo apt-get install gcc g++ git cmake ninja-build
    gcc --version # gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
    g++ --version # ++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
    cmake --version # cmake version 3.7.2
    ninja --version # 1.7.2

    git clone https://github.com/llvm/llvm-project.git
    cd llvm-project
    mkdir build
    cd build
    cmake -G Ninja -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;' ../llvm
    ninja check-asan

    # But that doesn't actually run the test I want, so:
    ninja check-sanitizer

So I guess this was a problem with my local setup (a relatively stock
Google-internal linux image) in the end. Sorry for the noise.

Aaron