llvm dev - May 2017 - moving libfuzzer to compiler-rt?

> On May 9, 2017, at 3:00 PM, Kostya Serebryany <kcc at google.com>
wrote:
> 
> Thanks for the explanations! (it was worth asking) 
> 
> I do want to build libFuzzer itself (and its tests) using the just-built
clang. So, llvm/runtimes then.
> I'd name the directory llvm/runtimes/libFuzzer, if possible (the old
path was lib/Fuzzer which is how the tool got it's name, actually)
> George, would you like to send the change for review? 
OK
> 
> --kcc
> 
> 
> 
> On Tue, May 9, 2017 at 2:37 PM, Chris Bieneman <cbieneman at apple.com
<mailto:cbieneman at apple.com>> wrote:
> 
>> On May 9, 2017, at 2:19 PM, George Karpenkov <ekarpenkov at
apple.com <mailto:ekarpenkov at apple.com>> wrote:
>> 
>> +Chris.
>> 
>> My understanding was that it is technically impossible for things in
“lib”, as they are built first, and there’s no way to tell them to do that
before “clang”.
>> I’m not a CMake expert, and I might be wrong.
> 
> It is not impossible, it would just involve excessive hacks. Since it seems
like this isn't a short-term solution we're talking about I am very
opposed to throwing hacks into the build system. I'd rather we actually fix
the problem(s). More below.
> 
>> 
>>> On May 9, 2017, at 2:15 PM, Kostya Serebryany <kcc at google.com
<mailto:kcc at google.com>> wrote:
>>> 
>>> 
>>> 
>>> On Tue, May 9, 2017 at 1:56 PM, George Karpenkov <ekarpenkov at
apple.com <mailto:ekarpenkov at apple.com>> wrote:
>>> Again, after offline conversation with Chris Bieneman:
>>> 
>>>  - move to compiler-rt would be too complicated due to change in
licenses
>>>  - it would make much more sense to move to “tools” folder instead,
for the following reasons:
>>>     * conceptually, it’s a tool, not a library
>>>     * all other projects in “lib” depend on LLVM and can not build
without LLVM, libFuzzer does not
>>>     * practically speaking, CMake has no way of knowing whether
Clang is being built when
>>>       “lib” is compiled, yet it does know for projects in tools.
>>> 
>>> Using a freshly built clang for projects in “tools” is
embarrassingly easy and only requires a couple of lines
>>> of configuration change.
>>> 
>>> Kostya, what about moving to “tools” then?
>>> 
>>> Well, ok, this sounds cool. 
>>> But can we make one more step and try to preserve the code where it
is, for the sake of compatibility?
> 
> Please no. This code doesn't actually belong in lib, it has never fit
the model of an LLVM library, we really need to pull it out of there.
> 
>>> E.g. can we have the CMake in tools while still keeping the code in
lib?
> 
> Could we contrive a hack in the build system to do it? Yes, but I will
fight violently against allowing that change into the build system because the
right answer here is to move the code.
> 
>>> Or a link of some kind?
> 
> Links are incredibly fragile on Windows, and they trip up a lot of SCM
tools. We have one in LLDB's repo that causes me nothing but problems, so I
am also strongly opposed to that.
> 
>>> 
>>> My worry is that there are already quite a few places that know
where libFuzzer code is,
>>> and I don't control all of them. 
> 
> Downstream clients will have to update. That is kinda how these things
work, I can't imagine re-pointing an SCM checkout being a huge burden.
> 
>>> 
>>> And, finally, I really don't get why we can do something in
tools and can't do the same in lib.
>>> Or we simply don't want to do it to keep things simple? 
> 
> Not all functionality in CMake is order-independent. Specifically the
detection of targets is not. In order to support what you're trying to do
you are going to change behavior based on the presence of the clang target.
Which means the clang target must be added before your CMake is processed.
> 
> To support this our build system has strict ordering requirements such that
things in lib cannot depend on things in tools. If you need to depend on clang,
you need to not be in lib.
> 
> Also, generally speaking Fuzzer is a library under lib that also has nested
tests, which is *not* how the lib directory is supposed to be structured. It
never should have been allowed to be structured like that. If you want the tests
next to the library, it is a tool or a runtime, but not a lib.
> 
> As I see there are two options to move forward with, and it really depends
on how you intend to use the just-built clang.
> 
> (1) If you want to use just-built clang to build libFuzzer and its tests,
it should be a runtime.
> (2) If you want to use just-built clang to only build libFuzzer's
tests, it should be a tool.
> 
> I think that since it is a runtime library, it should be a runtime, and I
expect it would mostly work to just copy the Fuzzer directory into
llvm/runtimes.
> 
> -Chris
> 
>>> 
>>> --kcc 
>>> 
>>>  
>>> 
>>>> On May 9, 2017, at 11:07 AM, Dan Liew <dan at su-root.co.uk
<mailto:dan at su-root.co.uk>> wrote:
>>>> 
>>>> On 9 May 2017 at 18:55, Kostya Serebryany <kcc at google.com
<mailto:kcc at google.com>> wrote:
>>>>> 
>>>>> 
>>>>> On Tue, May 9, 2017 at 10:23 AM, Dan Liew <dan at
su-root.co.uk <mailto:dan at su-root.co.uk>> wrote:
>>>>>> 
>>>>>>> Does anyone see good reasons why libFuzzer should
remain in llvm repo
>>>>>>> (as
>>>>>>> opposed to moving it to compiler-rt)?
>>>>>> 
>>>>>> Does moving LibFuzzer to compiler-rt imply that it is
compiled as part
>>>>>> of compiler-rt and shipped with it?
>>>>>> 
>>>>>> How does that fit with LibFuzzer's model of
allowing the user to
>>>>>> provide their own `main()`.
>>>>> 
>>>>> 
>>>>> libFuzzer doesn't allow users to use their own main
(not any more).
>>>>> Although I am not sure how that's related to moving
libFuzzer somewhere.
>>>> 
>>>> Oops. That shows how long it's been since I looked at the
source code.
>>>> 
>>>> It was related in that if LibFuzzer was shipped as part of
compiler-rt
>>>> I presumed we would need to supply both libraries to end users.
>>>> Given that this feature was removed it is a non-issue.
>> 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170509/a7422a89/attachment.html>

Actually, there’s another problem we have missed: libraries under `build/lib`
are not installed into toolchain
on mac os (and neither on linux, I would suppose).
Thus installations of Clang would not contain libLLVMFuzzer, but we would like
them to, so that users would not have
to compile anything, and could just call `clang -fsanitize=fuzzer`.

That could probably be done with another CMake change, but I have no idea how to
do that.
> On May 9, 2017, at 4:04 PM, George Karpenkov via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>> 
>> On May 9, 2017, at 3:00 PM, Kostya Serebryany <kcc at google.com
<mailto:kcc at google.com>> wrote:
>> 
>> Thanks for the explanations! (it was worth asking) 
>> 
>> I do want to build libFuzzer itself (and its tests) using the
just-built clang. So, llvm/runtimes then.
>> I'd name the directory llvm/runtimes/libFuzzer, if possible (the
old path was lib/Fuzzer which is how the tool got it's name, actually)
>> George, would you like to send the change for review? 
> OK
>> 
>> --kcc
>> 
>> 
>> 
>> On Tue, May 9, 2017 at 2:37 PM, Chris Bieneman <cbieneman at
apple.com <mailto:cbieneman at apple.com>> wrote:
>> 
>>> On May 9, 2017, at 2:19 PM, George Karpenkov <ekarpenkov at
apple.com <mailto:ekarpenkov at apple.com>> wrote:
>>> 
>>> +Chris.
>>> 
>>> My understanding was that it is technically impossible for things
in “lib”, as they are built first, and there’s no way to tell them to do that
before “clang”.
>>> I’m not a CMake expert, and I might be wrong.
>> 
>> It is not impossible, it would just involve excessive hacks. Since it
seems like this isn't a short-term solution we're talking about I am
very opposed to throwing hacks into the build system. I'd rather we actually
fix the problem(s). More below.
>> 
>>> 
>>>> On May 9, 2017, at 2:15 PM, Kostya Serebryany <kcc at
google.com <mailto:kcc at google.com>> wrote:
>>>> 
>>>> 
>>>> 
>>>> On Tue, May 9, 2017 at 1:56 PM, George Karpenkov <ekarpenkov
at apple.com <mailto:ekarpenkov at apple.com>> wrote:
>>>> Again, after offline conversation with Chris Bieneman:
>>>> 
>>>>  - move to compiler-rt would be too complicated due to change
in licenses
>>>>  - it would make much more sense to move to “tools” folder
instead, for the following reasons:
>>>>     * conceptually, it’s a tool, not a library
>>>>     * all other projects in “lib” depend on LLVM and can not
build without LLVM, libFuzzer does not
>>>>     * practically speaking, CMake has no way of knowing whether
Clang is being built when
>>>>       “lib” is compiled, yet it does know for projects in
tools.
>>>> 
>>>> Using a freshly built clang for projects in “tools” is
embarrassingly easy and only requires a couple of lines
>>>> of configuration change.
>>>> 
>>>> Kostya, what about moving to “tools” then?
>>>> 
>>>> Well, ok, this sounds cool. 
>>>> But can we make one more step and try to preserve the code
where it is, for the sake of compatibility?
>> 
>> Please no. This code doesn't actually belong in lib, it has never
fit the model of an LLVM library, we really need to pull it out of there.
>> 
>>>> E.g. can we have the CMake in tools while still keeping the
code in lib?
>> 
>> Could we contrive a hack in the build system to do it? Yes, but I will
fight violently against allowing that change into the build system because the
right answer here is to move the code.
>> 
>>>> Or a link of some kind?
>> 
>> Links are incredibly fragile on Windows, and they trip up a lot of SCM
tools. We have one in LLDB's repo that causes me nothing but problems, so I
am also strongly opposed to that.
>> 
>>>> 
>>>> My worry is that there are already quite a few places that know
where libFuzzer code is,
>>>> and I don't control all of them. 
>> 
>> Downstream clients will have to update. That is kinda how these things
work, I can't imagine re-pointing an SCM checkout being a huge burden.
>> 
>>>> 
>>>> And, finally, I really don't get why we can do something in
tools and can't do the same in lib.
>>>> Or we simply don't want to do it to keep things simple? 
>> 
>> Not all functionality in CMake is order-independent. Specifically the
detection of targets is not. In order to support what you're trying to do
you are going to change behavior based on the presence of the clang target.
Which means the clang target must be added before your CMake is processed.
>> 
>> To support this our build system has strict ordering requirements such
that things in lib cannot depend on things in tools. If you need to depend on
clang, you need to not be in lib.
>> 
>> Also, generally speaking Fuzzer is a library under lib that also has
nested tests, which is *not* how the lib directory is supposed to be structured.
It never should have been allowed to be structured like that. If you want the
tests next to the library, it is a tool or a runtime, but not a lib.
>> 
>> As I see there are two options to move forward with, and it really
depends on how you intend to use the just-built clang.
>> 
>> (1) If you want to use just-built clang to build libFuzzer and its
tests, it should be a runtime.
>> (2) If you want to use just-built clang to only build libFuzzer's
tests, it should be a tool.
>> 
>> I think that since it is a runtime library, it should be a runtime, and
I expect it would mostly work to just copy the Fuzzer directory into
llvm/runtimes.
>> 
>> -Chris
>> 
>>>> 
>>>> --kcc 
>>>> 
>>>>  
>>>> 
>>>>> On May 9, 2017, at 11:07 AM, Dan Liew <dan at
su-root.co.uk <mailto:dan at su-root.co.uk>> wrote:
>>>>> 
>>>>> On 9 May 2017 at 18:55, Kostya Serebryany <kcc at
google.com <mailto:kcc at google.com>> wrote:
>>>>>> 
>>>>>> 
>>>>>> On Tue, May 9, 2017 at 10:23 AM, Dan Liew <dan at
su-root.co.uk <mailto:dan at su-root.co.uk>> wrote:
>>>>>>> 
>>>>>>>> Does anyone see good reasons why libFuzzer
should remain in llvm repo
>>>>>>>> (as
>>>>>>>> opposed to moving it to compiler-rt)?
>>>>>>> 
>>>>>>> Does moving LibFuzzer to compiler-rt imply that it
is compiled as part
>>>>>>> of compiler-rt and shipped with it?
>>>>>>> 
>>>>>>> How does that fit with LibFuzzer's model of
allowing the user to
>>>>>>> provide their own `main()`.
>>>>>> 
>>>>>> 
>>>>>> libFuzzer doesn't allow users to use their own main
(not any more).
>>>>>> Although I am not sure how that's related to moving
libFuzzer somewhere.
>>>>> 
>>>>> Oops. That shows how long it's been since I looked at
the source code.
>>>>> 
>>>>> It was related in that if LibFuzzer was shipped as part of
compiler-rt
>>>>> I presumed we would need to supply both libraries to end
users.
>>>>> Given that this feature was removed it is a non-issue.
>>> 
>> 
>> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170510/be1c7926/attachment-0001.html>

> On May 10, 2017, at 4:43 PM, George Karpenkov via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Actually, there’s another problem we have missed: libraries under
`build/lib` are not installed into toolchain
> on mac os (and neither on linux, I would suppose).
Actually that isn't accurate. By default we don't install the LLVM
libraries, but that is completely configurable in the build system. It
doesn't work for libFuzzer because the CMake build for libFuzzer is not
built using any of the LLVM CMake modules or following any of LLVM's
conventions.
> Thus installations of Clang would not contain libLLVMFuzzer, but we would
like them to, so that users would not have
> to compile anything, and could just call `clang -fsanitize=fuzzer`.
> 
> That could probably be done with another CMake change, but I have no idea
how to do that.
Yea, libFuzzer's CMake really needs a big overhaul, and probably an almost
complete rewrite.

-Chris
> 
>> On May 9, 2017, at 4:04 PM, George Karpenkov via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>>> 
>>> On May 9, 2017, at 3:00 PM, Kostya Serebryany <kcc at google.com
<mailto:kcc at google.com>> wrote:
>>> 
>>> Thanks for the explanations! (it was worth asking) 
>>> 
>>> I do want to build libFuzzer itself (and its tests) using the
just-built clang. So, llvm/runtimes then.
>>> I'd name the directory llvm/runtimes/libFuzzer, if possible
(the old path was lib/Fuzzer which is how the tool got it's name, actually)
>>> George, would you like to send the change for review? 
>> OK
>>> 
>>> --kcc
>>> 
>>> 
>>> 
>>> On Tue, May 9, 2017 at 2:37 PM, Chris Bieneman <cbieneman at
apple.com <mailto:cbieneman at apple.com>> wrote:
>>> 
>>>> On May 9, 2017, at 2:19 PM, George Karpenkov <ekarpenkov at
apple.com <mailto:ekarpenkov at apple.com>> wrote:
>>>> 
>>>> +Chris.
>>>> 
>>>> My understanding was that it is technically impossible for
things in “lib”, as they are built first, and there’s no way to tell them to do
that before “clang”.
>>>> I’m not a CMake expert, and I might be wrong.
>>> 
>>> It is not impossible, it would just involve excessive hacks. Since
it seems like this isn't a short-term solution we're talking about I am
very opposed to throwing hacks into the build system. I'd rather we actually
fix the problem(s). More below.
>>> 
>>>> 
>>>>> On May 9, 2017, at 2:15 PM, Kostya Serebryany <kcc at
google.com <mailto:kcc at google.com>> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, May 9, 2017 at 1:56 PM, George Karpenkov
<ekarpenkov at apple.com <mailto:ekarpenkov at apple.com>> wrote:
>>>>> Again, after offline conversation with Chris Bieneman:
>>>>> 
>>>>>  - move to compiler-rt would be too complicated due to
change in licenses
>>>>>  - it would make much more sense to move to “tools” folder
instead, for the following reasons:
>>>>>     * conceptually, it’s a tool, not a library
>>>>>     * all other projects in “lib” depend on LLVM and can
not build without LLVM, libFuzzer does not
>>>>>     * practically speaking, CMake has no way of knowing
whether Clang is being built when
>>>>>       “lib” is compiled, yet it does know for projects in
tools.
>>>>> 
>>>>> Using a freshly built clang for projects in “tools” is
embarrassingly easy and only requires a couple of lines
>>>>> of configuration change.
>>>>> 
>>>>> Kostya, what about moving to “tools” then?
>>>>> 
>>>>> Well, ok, this sounds cool. 
>>>>> But can we make one more step and try to preserve the code
where it is, for the sake of compatibility?
>>> 
>>> Please no. This code doesn't actually belong in lib, it has
never fit the model of an LLVM library, we really need to pull it out of there.
>>> 
>>>>> E.g. can we have the CMake in tools while still keeping the
code in lib?
>>> 
>>> Could we contrive a hack in the build system to do it? Yes, but I
will fight violently against allowing that change into the build system because
the right answer here is to move the code.
>>> 
>>>>> Or a link of some kind?
>>> 
>>> Links are incredibly fragile on Windows, and they trip up a lot of
SCM tools. We have one in LLDB's repo that causes me nothing but problems,
so I am also strongly opposed to that.
>>> 
>>>>> 
>>>>> My worry is that there are already quite a few places that
know where libFuzzer code is,
>>>>> and I don't control all of them. 
>>> 
>>> Downstream clients will have to update. That is kinda how these
things work, I can't imagine re-pointing an SCM checkout being a huge
burden.
>>> 
>>>>> 
>>>>> And, finally, I really don't get why we can do
something in tools and can't do the same in lib.
>>>>> Or we simply don't want to do it to keep things simple?
>>> 
>>> Not all functionality in CMake is order-independent. Specifically
the detection of targets is not. In order to support what you're trying to
do you are going to change behavior based on the presence of the clang target.
Which means the clang target must be added before your CMake is processed.
>>> 
>>> To support this our build system has strict ordering requirements
such that things in lib cannot depend on things in tools. If you need to depend
on clang, you need to not be in lib.
>>> 
>>> Also, generally speaking Fuzzer is a library under lib that also
has nested tests, which is *not* how the lib directory is supposed to be
structured. It never should have been allowed to be structured like that. If you
want the tests next to the library, it is a tool or a runtime, but not a lib.
>>> 
>>> As I see there are two options to move forward with, and it really
depends on how you intend to use the just-built clang.
>>> 
>>> (1) If you want to use just-built clang to build libFuzzer and its
tests, it should be a runtime.
>>> (2) If you want to use just-built clang to only build
libFuzzer's tests, it should be a tool.
>>> 
>>> I think that since it is a runtime library, it should be a runtime,
and I expect it would mostly work to just copy the Fuzzer directory into
llvm/runtimes.
>>> 
>>> -Chris
>>> 
>>>>> 
>>>>> --kcc 
>>>>> 
>>>>>  
>>>>> 
>>>>>> On May 9, 2017, at 11:07 AM, Dan Liew <dan at
su-root.co.uk <mailto:dan at su-root.co.uk>> wrote:
>>>>>> 
>>>>>> On 9 May 2017 at 18:55, Kostya Serebryany <kcc at
google.com <mailto:kcc at google.com>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, May 9, 2017 at 10:23 AM, Dan Liew <dan
at su-root.co.uk <mailto:dan at su-root.co.uk>> wrote:
>>>>>>>> 
>>>>>>>>> Does anyone see good reasons why libFuzzer
should remain in llvm repo
>>>>>>>>> (as
>>>>>>>>> opposed to moving it to compiler-rt)?
>>>>>>>> 
>>>>>>>> Does moving LibFuzzer to compiler-rt imply that
it is compiled as part
>>>>>>>> of compiler-rt and shipped with it?
>>>>>>>> 
>>>>>>>> How does that fit with LibFuzzer's model of
allowing the user to
>>>>>>>> provide their own `main()`.
>>>>>>> 
>>>>>>> 
>>>>>>> libFuzzer doesn't allow users to use their own
main (not any more).
>>>>>>> Although I am not sure how that's related to
moving libFuzzer somewhere.
>>>>>> 
>>>>>> Oops. That shows how long it's been since I looked
at the source code.
>>>>>> 
>>>>>> It was related in that if LibFuzzer was shipped as part
of compiler-rt
>>>>>> I presumed we would need to supply both libraries to
end users.
>>>>>> Given that this feature was removed it is a non-issue.
>>>> 
>>> 
>>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170510/894aa84e/attachment.html>