llvm dev - Jul 2017 - moving libfuzzer to compiler-rt?

On Wed, Jul 12, 2017 at 11:30 AM, George Karpenkov via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
> I already forgot why we decided not to move the code to compiler-rt.
>
> This would solve at least this problem.
> Since we now have -fsanitize=fuzzer it will actually be pretty natural.
>
>
> Licensing concerns, compiler-rt has a different license.
>
> BTW libFuzzer CMake has a crazy amount of hacks to work under Windows,
> where logic in many parts
> is entirely different, so any help on testing and fixing arising issues
> would be much appreciated.
> All I can do is to make a commit and then try to understand the response
> from a bots,
> which is not very efficient.
>
I got volunteered to help with that, I'll try your patch.

There's more stuff to unpack in the rest of your email, though. I'd
really
rather go to compiler-rt instead of runtimes. runtimes seems like it's
going the opposite direction from the monorepo efforts.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170712/b419a75e/attachment.html>

Reid Kleckner via llvm-dev <llvm-dev at lists.llvm.org>
writes:
> On Wed, Jul 12, 2017 at 11:30 AM, George Karpenkov via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>>
>> I already forgot why we decided not to move the code to compiler-rt.
>>
>> This would solve at least this problem.
>> Since we now have -fsanitize=fuzzer it will actually be pretty natural.
>>
>>
>> Licensing concerns, compiler-rt has a different license.
>>
>> BTW libFuzzer CMake has a crazy amount of hacks to work under Windows,
>> where logic in many parts
>> is entirely different, so any help on testing and fixing arising issues
>> would be much appreciated.
>> All I can do is to make a commit and then try to understand the
response
>> from a bots,
>> which is not very efficient.
>>
>
> I got volunteered to help with that, I'll try your patch.
>
> There's more stuff to unpack in the rest of your email, though. I'd
really
> rather go to compiler-rt instead of runtimes. runtimes seems like it's
> going the opposite direction from the monorepo efforts.
I don't see how runtimes is going either for- or against- the monorepo
stuff. We really need a model like runtimes/ rather than a model like
projects/ going forward regardless of where you find the code. This is
necessary if we want to build cross-compiled compiler-rt libraries in
any sort of sane way.

> I really like the property of libFuzzer living in its own place so that
> it's easy to use without building the world
But it’s not: the implementation of the coverage instrumentation
is done in one of the sanitizers, so it’s impossible to just use libFuzzer
without them.
Furthermore, I would think that almost all libFuzzer users would use a sanitizer
while fuzzing.

I can see a few more benefits of moving to compiler-rt:

1) It would enforce the property
that either sanitizers and libfuzzer are both in “runtimes”, or they are both
aren’t.
Then we can naturally add a dependency from libFuzzer test to sanitizers using
CMake.
Otherwise, adding a dependency from libfuzzer tests to sanitizers requires
hacky if/elses in CMake configuration (and we already have too many of those).

2) Orthogonally to the moving-of-source-code discussion, I would like to move
the generated library
to the folder where other clang libraries and sanitizers are.
That would considerably simplify toolchain installation problem (as we know that
sanitizers are already installed
correctly in the toolchain) as well as simplifying driver logic for both Clang
and Swift.
[Diverting even further, I think it would be beneficial to build a dynamic
library as well as an archive,
and use that at least on Mac, where it can solve the linking issues discussed in
https://github.com/google/sanitizers/issues/832.
It would be also be much easier if the resulting dylib would be in the same
folder where other sanitizers are, due to the fact
that we already have the code for setting the expected rpath on the produced
binary].

This move can of course be done from any source directory, as it’s just a matter
of setting a target property.
Yet the logic for selecting the folder is a 20-30 lines chunk of CMake, which I
would like to avoid copying,
and which we could otherwise just reuse from compiler-rt.

3) As Chris Bieneman has previously mentioned in the beginning of this thread,
adding libFuzzer to the toolchain installation might be problematic under the
current license,
which requires binary attribution.
Performing the move and resolving the licensing concern would solve that.

> On Jul 12, 2017, at 1:27 PM, Justin Bogner <mail at justinbogner.com>
wrote:
> 
> Reid Kleckner via llvm-dev <llvm-dev at lists.llvm.org> writes:
>> On Wed, Jul 12, 2017 at 11:30 AM, George Karpenkov via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>> 
>>> I already forgot why we decided not to move the code to
compiler-rt.
>>> 
>>> This would solve at least this problem.
>>> Since we now have -fsanitize=fuzzer it will actually be pretty
natural.
>>> 
>>> 
>>> Licensing concerns, compiler-rt has a different license.
>>> 
>>> BTW libFuzzer CMake has a crazy amount of hacks to work under
Windows,
>>> where logic in many parts
>>> is entirely different, so any help on testing and fixing arising
issues
>>> would be much appreciated.
>>> All I can do is to make a commit and then try to understand the
response
>>> from a bots,
>>> which is not very efficient.
>>> 
>> 
>> I got volunteered to help with that, I'll try your patch.
>> 
>> There's more stuff to unpack in the rest of your email, though.
I'd really
>> rather go to compiler-rt instead of runtimes. runtimes seems like
it's
>> going the opposite direction from the monorepo efforts.
> 
> I don't see how runtimes is going either for- or against- the monorepo
> stuff. We really need a model like runtimes/ rather than a model like
> projects/ going forward regardless of where you find the code. This is
> necessary if we want to build cross-compiled compiler-rt libraries in
> any sort of sane way.