llvm dev - May 2017 - moving libfuzzer to compiler-rt?

On Tue, May 2, 2017 at 4:28 PM, Kostya Serebryany via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Tue, May 2, 2017 at 12:26 PM, George Karpenkov <ekarpenkov at
apple.com>
> wrote:
>
>> From my understanding, all these problems can be solved entirely
>>
> by moving libfuzzer to compiler-rt, where (other) sanitizers already
>> reside.
>>
>
> Yes, that might be a reasonable thing to do.
> I am trying to remember the reasons why we've put libFuzzer into llvm
and
> not into compiler-rt in the first place, and failing to do so.
>
IIRC you thought libFuzzer would be more tightly coupled to LLVM
instrumentation, so the goal was to reap monorepo-like productivity
developments without waiting for it to happen?

Maybe it was a licensing concern, UIUC vs dual UIUC/MIT?

> Any thoughts on the suggestion?
>>
>> It would be still possible to compile just libfuzzer with no
>> dependencies, by simply making a partial checkout from SVN,
>> and only the repo path would change.
>>
>
> This would cause some annoyance for me and maybe some users, but nothing
> we can't tolerate.
> Still what are our other options?
>
> * wait for the mono repo to happen, then we'll be able to make
libFuzzer
> depend on clang. (Or no?)
> * move libFuzzer to a separate repo, parallel to compiler-rt? (not a large
> win, just listing as a choice)
> * anything else?
>
> Does anyone see good reasons why libFuzzer should remain in llvm repo (as
> opposed to moving it to compiler-rt)?
>
I'd like to move libFuzzer out of llvm/lib/, since it's definitely a
runtime library, and not compiler library infrastructure. compiler-rt seems
like the best place to put it for now, unless there are licensing concerns.

Mechanically, we can do this in one history-preserving commit by checking
out the entire SVN repo the way that the git-llvm script does. SVN users
and users of https://github.com/llvm-project/llvm-project will still see
the right history. The final monorepo will presumably also have accurate
history.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170502/a2709776/attachment.html>

Reid Kleckner via llvm-dev <llvm-dev at lists.llvm.org>
writes:
> On Tue, May 2, 2017 at 4:28 PM, Kostya Serebryany via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> On Tue, May 2, 2017 at 12:26 PM, George Karpenkov <ekarpenkov at
apple.com>
>> wrote:
>>
>>> From my understanding, all these problems can be solved entirely
>>>
>> by moving libfuzzer to compiler-rt, where (other) sanitizers already
>>> reside.
>>>
>>
>> Yes, that might be a reasonable thing to do.
>> I am trying to remember the reasons why we've put libFuzzer into
llvm and
>> not into compiler-rt in the first place, and failing to do so.
>>
>
> IIRC you thought libFuzzer would be more tightly coupled to LLVM
> instrumentation, so the goal was to reap monorepo-like productivity
> developments without waiting for it to happen?
>
> Maybe it was a licensing concern, UIUC vs dual UIUC/MIT?
>
>
>> Any thoughts on the suggestion?
>>>
>>> It would be still possible to compile just libfuzzer with no
>>> dependencies, by simply making a partial checkout from SVN,
>>> and only the repo path would change.
>>>
>>
>> This would cause some annoyance for me and maybe some users, but
nothing
>> we can't tolerate.
>> Still what are our other options?
>>
>> * wait for the mono repo to happen, then we'll be able to make
libFuzzer
>> depend on clang. (Or no?)
>> * move libFuzzer to a separate repo, parallel to compiler-rt? (not a
large
>> win, just listing as a choice)
>> * anything else?
>>
>> Does anyone see good reasons why libFuzzer should remain in llvm repo
(as
>> opposed to moving it to compiler-rt)?
>>
>
> I'd like to move libFuzzer out of llvm/lib/, since it's definitely
a
> runtime library, and not compiler library infrastructure. compiler-rt seems
> like the best place to put it for now, unless there are licensing concerns.
Maybe we should just move it under llvm/runtimes and set it up with a
fairly simple cmake config? I think this makes it pretty trivial to
build it with the just-built clang. It doesn't have terribly much to do
with compiler-rt IMO - this isn't something that enables a compiler
feature per se, it's more of its own thing.
> Mechanically, we can do this in one history-preserving commit by checking
> out the entire SVN repo the way that the git-llvm script does. SVN users
> and users of https://github.com/llvm-project/llvm-project will still see
> the right history. The final monorepo will presumably also have accurate
> history.

On Tue, May 2, 2017 at 9:34 PM, Justin Bogner <mail at justinbogner.com>
wrote:
> Reid Kleckner via llvm-dev <llvm-dev at lists.llvm.org> writes:
> > On Tue, May 2, 2017 at 4:28 PM, Kostya Serebryany via llvm-dev <
> > llvm-dev at lists.llvm.org> wrote:
> >
> >> On Tue, May 2, 2017 at 12:26 PM, George Karpenkov <ekarpenkov
at apple.com
> >
> >> wrote:
> >>
> >>> From my understanding, all these problems can be solved
entirely
> >>>
> >> by moving libfuzzer to compiler-rt, where (other) sanitizers
already
> >>> reside.
> >>>
> >>
> >> Yes, that might be a reasonable thing to do.
> >> I am trying to remember the reasons why we've put libFuzzer
into llvm
> and
> >> not into compiler-rt in the first place, and failing to do so.
> >>
> >
> > IIRC you thought libFuzzer would be more tightly coupled to LLVM
> > instrumentation, so the goal was to reap monorepo-like productivity
> > developments without waiting for it to happen?
> >
> > Maybe it was a licensing concern, UIUC vs dual UIUC/MIT?
> >
> >
> >> Any thoughts on the suggestion?
> >>>
> >>> It would be still possible to compile just libfuzzer with no
> >>> dependencies, by simply making a partial checkout from SVN,
> >>> and only the repo path would change.
> >>>
> >>
> >> This would cause some annoyance for me and maybe some users, but
nothing
> >> we can't tolerate.
> >> Still what are our other options?
> >>
> >> * wait for the mono repo to happen, then we'll be able to make
libFuzzer
> >> depend on clang. (Or no?)
> >> * move libFuzzer to a separate repo, parallel to compiler-rt? (not
a
> large
> >> win, just listing as a choice)
> >> * anything else?
> >>
> >> Does anyone see good reasons why libFuzzer should remain in llvm
repo
> (as
> >> opposed to moving it to compiler-rt)?
> >>
> >
> > I'd like to move libFuzzer out of llvm/lib/, since it's
definitely a
> > runtime library, and not compiler library infrastructure. compiler-rt
> seems
> > like the best place to put it for now, unless there are licensing
> concerns.
>
> Maybe we should just move it under llvm/runtimes and set it up with a
> fairly simple cmake config?

can we?
If we can, then we probably can also leave the code where it is, and tweak
the cmake to use just-built-clang. No?

> I think this makes it pretty trivial to
> build it with the just-built clang. It doesn't have terribly much to do
> with compiler-rt IMO - this isn't something that enables a compiler
> feature per se, it's more of its own thing.
>
> > Mechanically, we can do this in one history-preserving commit by
checking
> > out the entire SVN repo the way that the git-llvm script does. SVN
users
> > and users of https://github.com/llvm-project/llvm-project will still
see
> > the right history. The final monorepo will presumably also have
accurate
> > history.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170503/19740198/attachment.html>

>From my understanding, the entire file
`compiler-rt/cmake/modules/CompileRTCompile.cmake` is dedicated to using the
trick
of using the just-built-compiler for compiling the sources.
Copying it seems suboptimal.
I can move it to `llvm/cmake/modules`, but that would be one step away from the
ability to built `compiler-rt` without the parent repository.
Should that matter?
> not build at all
Why not? libfuzzer itself does not require instrumentation (at least from my
understanding),
the actual problem arises with tests.
> On May 2, 2017, at 9:34 PM, Justin Bogner via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Reid Kleckner via llvm-dev <llvm-dev at lists.llvm.org> writes:
>> On Tue, May 2, 2017 at 4:28 PM, Kostya Serebryany via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>> 
>>> On Tue, May 2, 2017 at 12:26 PM, George Karpenkov <ekarpenkov at
apple.com>
>>> wrote:
>>> 
>>>> From my understanding, all these problems can be solved
entirely
>>>> 
>>> by moving libfuzzer to compiler-rt, where (other) sanitizers
already
>>>> reside.
>>>> 
>>> 
>>> Yes, that might be a reasonable thing to do.
>>> I am trying to remember the reasons why we've put libFuzzer
into llvm and
>>> not into compiler-rt in the first place, and failing to do so.
>>> 
>> 
>> IIRC you thought libFuzzer would be more tightly coupled to LLVM
>> instrumentation, so the goal was to reap monorepo-like productivity
>> developments without waiting for it to happen?
>> 
>> Maybe it was a licensing concern, UIUC vs dual UIUC/MIT?
>> 
>> 
>>> Any thoughts on the suggestion?
>>>> 
>>>> It would be still possible to compile just libfuzzer with no
>>>> dependencies, by simply making a partial checkout from SVN,
>>>> and only the repo path would change.
>>>> 
>>> 
>>> This would cause some annoyance for me and maybe some users, but
nothing
>>> we can't tolerate.
>>> Still what are our other options?
>>> 
>>> * wait for the mono repo to happen, then we'll be able to make
libFuzzer
>>> depend on clang. (Or no?)
>>> * move libFuzzer to a separate repo, parallel to compiler-rt? (not
a large
>>> win, just listing as a choice)
>>> * anything else?
>>> 
>>> Does anyone see good reasons why libFuzzer should remain in llvm
repo (as
>>> opposed to moving it to compiler-rt)?
>>> 
>> 
>> I'd like to move libFuzzer out of llvm/lib/, since it's
definitely a
>> runtime library, and not compiler library infrastructure. compiler-rt
seems
>> like the best place to put it for now, unless there are licensing
concerns.
> 
> Maybe we should just move it under llvm/runtimes and set it up with a
> fairly simple cmake config? I think this makes it pretty trivial to
> build it with the just-built clang. It doesn't have terribly much to do
> with compiler-rt IMO - this isn't something that enables a compiler
> feature per se, it's more of its own thing.
> 
>> Mechanically, we can do this in one history-preserving commit by
checking
>> out the entire SVN repo the way that the git-llvm script does. SVN
users
>> and users of https://github.com/llvm-project/llvm-project will still
see
>> the right history. The final monorepo will presumably also have
accurate
>> history.
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev