llvm dev - Jun 2012 - [LLVMdev] AddressSanitizer+CMake unittest question

Context: I'm trying to implement support for ASan's unittest suite in
CMake. This is ... quite challenging.

I think I can get it to work with one significant caveat: it will require
manual dependency management. None of the automatic header tracking. I
think this is fine in some cases, and not so fine in other cases. Let me
explain.

It feels like these tests are really comprised of two distinct collections
of tests:

1) Those that rather directly test the ASan runtime. These do not rely upon
the compiler instrumenting the code, and simply exercise the runtime
library directly.
2) Those that expect to be instrumented by the compiler, and exercise the
runtime through GoogleTest's death tests on seemingly innocuous code.

For the first bucket, there is no problem. We should be able to handle
these easily.

For the second bucket, this can be a bit tricky. It requires compiling the
tests with a custom compiler and flags. Let's talk about the options for
supporting this case.

A) We could require the host compiler to have support for
-faddress-sanitizer, but ensure that the just-built runtime library is used
rather than the host compiler's runtime library.
B) We can depend upon the Clang built in the same LLVM/Clang/CompilerRT
checkout, and provide a custom compilation strategy to use it to instrument
the unittest code.


Option A has fairly obvious problems: it introduces version skew into the
equation, and would require a full bootstrap to test new instrumentation.
However, it plays very nicely with the build system, requiring no special
magic. It also would "Just Work" in the cross-compilation scenario, as
much
as any unittest would.

Option B avoids any version skew issues, but at the cost of requiring us to
implement a "complete" custom compilation strategy for these source
files.
At the very least, this will not be portable and thus will only be enabled
on a few platforms, and it will not get automatic header dependency
tracking.


I can pursue either of these options, but I'd like to know which the ASan
folks would be most interested in working with.

Second, it would be extremely helpful to have a very firm separation
between bucket #1 and bucket #2. Currently, there is one collection of
tests clearly in bucket #1 (asan_noinst_test.cc), and plenty of tests
clearly in bucket #2, but there seems to be quite a bit of overlap as well.
For example, I would naively expect asan_interface_test.cc to not require
instrumentation -- it almost exclusively directly calls the runtime. But it
is currently being compiled with instrumentation by the Makefile.old, so
I'm at a loss. Separating these two scenarios into at least separate files,
and ideally linking them into two separate binaries would make everything
much simpler. For example, bucket #2 shouldn't need to include any of
ASan's header files, making it easier to manage manual dependencies if
necessary.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120625/9a2fdd96/attachment.html>

Hi CMake'ers,

See the email below from llvmdev. I was wondering whether there's a better
way to solve this. Ideas?

Thanks!
/Manuel

On Mon, Jun 25, 2012 at 2:00 PM, Chandler Carruth <chandlerc at
google.com>wrote:
> Context: I'm trying to implement support for ASan's unittest suite
in
> CMake. This is ... quite challenging.
>
> I think I can get it to work with one significant caveat: it will require
> manual dependency management. None of the automatic header tracking. I
> think this is fine in some cases, and not so fine in other cases. Let me
> explain.
>
> It feels like these tests are really comprised of two distinct collections
> of tests:
>
> 1) Those that rather directly test the ASan runtime. These do not rely
> upon the compiler instrumenting the code, and simply exercise the runtime
> library directly.
> 2) Those that expect to be instrumented by the compiler, and exercise the
> runtime through GoogleTest's death tests on seemingly innocuous code.
>
> For the first bucket, there is no problem. We should be able to handle
> these easily.
>
> For the second bucket, this can be a bit tricky. It requires compiling the
> tests with a custom compiler and flags. Let's talk about the options
for
> supporting this case.
>
> A) We could require the host compiler to have support for
> -faddress-sanitizer, but ensure that the just-built runtime library is used
> rather than the host compiler's runtime library.
> B) We can depend upon the Clang built in the same LLVM/Clang/CompilerRT
> checkout, and provide a custom compilation strategy to use it to instrument
> the unittest code.
>
>
> Option A has fairly obvious problems: it introduces version skew into the
> equation, and would require a full bootstrap to test new instrumentation.
> However, it plays very nicely with the build system, requiring no special
> magic. It also would "Just Work" in the cross-compilation
scenario, as much
> as any unittest would.
>
> Option B avoids any version skew issues, but at the cost of requiring us
> to implement a "complete" custom compilation strategy for these
source
> files. At the very least, this will not be portable and thus will only be
> enabled on a few platforms, and it will not get automatic header dependency
> tracking.
>
>
> I can pursue either of these options, but I'd like to know which the
ASan
> folks would be most interested in working with.
>
> Second, it would be extremely helpful to have a very firm separation
> between bucket #1 and bucket #2. Currently, there is one collection of
> tests clearly in bucket #1 (asan_noinst_test.cc), and plenty of tests
> clearly in bucket #2, but there seems to be quite a bit of overlap as well.
> For example, I would naively expect asan_interface_test.cc to not require
> instrumentation -- it almost exclusively directly calls the runtime. But it
> is currently being compiled with instrumentation by the Makefile.old, so
> I'm at a loss. Separating these two scenarios into at least separate
files,
> and ideally linking them into two separate binaries would make everything
> much simpler. For example, bucket #2 shouldn't need to include any of
> ASan's header files, making it easier to manage manual dependencies if
> necessary.
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120625/69553950/attachment.html>

On Mon, Jun 25, 2012 at 5:00 AM, Chandler Carruth <chandlerc at
google.com>wrote:
>
> A) We could require the host compiler to have support for
> -faddress-sanitizer, but ensure that the just-built runtime library is used
> rather than the host compiler's runtime library.
>
FWIW, I've started implementing this and folks can try it out as of r159134.

The way to test out the current CMake build is to comment out the exclusion
of 'compiler-rt' from the LLVM build in
'projects/CMakeLists.txt'. Then you
can build, for example 'make AsanTests' and it should compile both the
instrumented and non-instrumented unit tests.

Currently, you have to run them by hand -- I need to refactor a bit of the
LLVM cmake build before i can wire in nice convenience targets for running
these -- but I've tried and they seem to work for me...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120625/b694c265/attachment.html>

On Mon, Jun 25, 2012 at 4:00 PM, Chandler Carruth <chandlerc at
google.com>wrote:
> Context: I'm trying to implement support for ASan's unittest suite
in
> CMake. This is ... quite challenging.
>
> I think I can get it to work with one significant caveat: it will require
> manual dependency management. None of the automatic header tracking. I
> think this is fine in some cases, and not so fine in other cases. Let me
> explain.
>
> It feels like these tests are really comprised of two distinct collections
> of tests:
>
> 1) Those that rather directly test the ASan runtime. These do not rely
> upon the compiler instrumenting the code, and simply exercise the runtime
> library directly.
> 2) Those that expect to be instrumented by the compiler, and exercise the
> runtime through GoogleTest's death tests on seemingly innocuous code.
>
> For the first bucket, there is no problem. We should be able to handle
> these easily.
>
> For the second bucket, this can be a bit tricky. It requires compiling the
> tests with a custom compiler and flags. Let's talk about the options
for
> supporting this case.
>
> A) We could require the host compiler to have support for
> -faddress-sanitizer, but ensure that the just-built runtime library is used
> rather than the host compiler's runtime library.
> B) We can depend upon the Clang built in the same LLVM/Clang/CompilerRT
> checkout, and provide a custom compilation strategy to use it to instrument
> the unittest code.
>
>
> Option A has fairly obvious problems: it introduces version skew into the
> equation, and would require a full bootstrap to test new instrumentation.
> However, it plays very nicely with the build system, requiring no special
> magic. It also would "Just Work" in the cross-compilation
scenario, as much
> as any unittest would.
>
> Option B avoids any version skew issues, but at the cost of requiring us
> to implement a "complete" custom compilation strategy for these
source
> files. At the very least, this will not be portable and thus will only be
> enabled on a few platforms, and it will not get automatic header dependency
> tracking.
>
>
> I can pursue either of these options, but I'd like to know which the
ASan
> folks would be most interested in working with.
>
> Second, it would be extremely helpful to have a very firm separation
> between bucket #1 and bucket #2.
>
(answering parts at a time)
Agree. I am going to fix that now.

> Currently, there is one collection of tests clearly in bucket #1
> (asan_noinst_test.cc), and plenty of tests clearly in bucket #2, but there
> seems to be quite a bit of overlap as well. For example, I would naively
> expect asan_interface_test.cc to not require instrumentation -- it almost
> exclusively directly calls the runtime. But it is currently being compiled
> with instrumentation by the Makefile.old, so I'm at a loss. Separating
> these two scenarios into at least separate files, and ideally linking them
> into two separate binaries would make everything much simpler. For example,
> bucket #2 shouldn't need to include any of ASan's header files,
making it
> easier to manage manual dependencies if necessary.
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120625/60917df1/attachment.html>

On Mon, Jun 25, 2012 at 5:43 PM, Kostya Serebryany <kcc at google.com>
wrote:
>
>
> On Mon, Jun 25, 2012 at 4:00 PM, Chandler Carruth <chandlerc at
google.com>wrote:
>
>> Context: I'm trying to implement support for ASan's unittest
suite in
>> CMake. This is ... quite challenging.
>>
>> I think I can get it to work with one significant caveat: it will
require
>> manual dependency management. None of the automatic header tracking. I
>> think this is fine in some cases, and not so fine in other cases. Let
me
>> explain.
>>
>> It feels like these tests are really comprised of two distinct
>> collections of tests:
>>
>> 1) Those that rather directly test the ASan runtime. These do not rely
>> upon the compiler instrumenting the code, and simply exercise the
runtime
>> library directly.
>> 2) Those that expect to be instrumented by the compiler, and exercise
the
>> runtime through GoogleTest's death tests on seemingly innocuous
code.
>>
>> For the first bucket, there is no problem. We should be able to handle
>> these easily.
>>
>> For the second bucket, this can be a bit tricky. It requires compiling
>> the tests with a custom compiler and flags. Let's talk about the
options
>> for supporting this case.
>>
>> A) We could require the host compiler to have support for
>> -faddress-sanitizer, but ensure that the just-built runtime library is
used
>> rather than the host compiler's runtime library.
>> B) We can depend upon the Clang built in the same LLVM/Clang/CompilerRT
>> checkout, and provide a custom compilation strategy to use it to
instrument
>> the unittest code.
>>
>>
>> Option A has fairly obvious problems: it introduces version skew into
the
>> equation, and would require a full bootstrap to test new
instrumentation.
>> However, it plays very nicely with the build system, requiring no
special
>> magic. It also would "Just Work" in the cross-compilation
scenario, as much
>> as any unittest would.
>>
>> Option B avoids any version skew issues, but at the cost of requiring
us
>> to implement a "complete" custom compilation strategy for
these source
>> files. At the very least, this will not be portable and thus will only
be
>> enabled on a few platforms, and it will not get automatic header
dependency
>> tracking.
>>
>
B is highly preferable.
I.e.:
  1. Build clang
  2. Build asan-rt (doesn't not matter with which compiler)
  3. build asan-rt-tests using clang from #1
This is what we use now anyway.
There could of course be dependencies between asan-rt and asan-rt-tests,
but even worth, there could be dependencies between the instrumentation
module in LLVM and asan-rt-tests.


>
>>
>> I can pursue either of these options, but I'd like to know which
the ASan
>> folks would be most interested in working with.
>>
>> Second, it would be extremely helpful to have a very firm separation
>> between bucket #1 and bucket #2.
>>
>
> (answering parts at a time)
> Agree. I am going to fix that now.
>
This part is fixed by r159135.

thanks!!

--kcc

>
>
>>  Currently, there is one collection of tests clearly in bucket #1
>> (asan_noinst_test.cc), and plenty of tests clearly in bucket #2, but
there
>> seems to be quite a bit of overlap as well. For example, I would
naively
>> expect asan_interface_test.cc to not require instrumentation -- it
almost
>> exclusively directly calls the runtime. But it is currently being
compiled
>> with instrumentation by the Makefile.old, so I'm at a loss.
Separating
>> these two scenarios into at least separate files, and ideally linking
them
>> into two separate binaries would make everything much simpler. For
example,
>> bucket #2 shouldn't need to include any of ASan's header files,
making it
>> easier to manage manual dependencies if necessary.
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120625/0eac447a/attachment.html>