George Karpenkov via llvm-dev
2017-May-02 19:26 UTC
[llvm-dev] moving libfuzzer to compiler-rt?
Hi All, Currently libfuzzer depends on (often freshly built) clang, yet the dependency is not explicitly specified in cmake. That leads to various issues: for instance, it’s not possible to check out LLVM repo and run libfuzzer tests: one would often need to compile fresh clang first, and then create a separate build directory, where libfuzzer could be tested. For the buildbot this problem is approached by grabbing a freshly built binary from another buildbot and using that for testing. Needless to say, that could be quite annoying. Additionally, my recent changes start using libfuzzer from Clang driver: and finding the actual archive file requires some hardcoding of directory paths, as one would need to go up the tree from the Clang binary (in swift, for example, the situation is even worse, as the path to Clang is a symlink, and getting an archive file from the LLVM tree would require going quite a few levels up). From my understanding, all these problems can be solved entirely by moving libfuzzer to compiler-rt, where (other) sanitizers already reside. Any thoughts on the suggestion? It would be still possible to compile just libfuzzer with no dependencies, by simply making a partial checkout from SVN, and only the repo path would change. George
Kostya Serebryany via llvm-dev
2017-May-02 23:28 UTC
[llvm-dev] moving libfuzzer to compiler-rt?
On Tue, May 2, 2017 at 12:26 PM, George Karpenkov <ekarpenkov at apple.com> wrote:> Hi All, > > Currently libfuzzer depends on (often freshly built) clang, yet the > dependency is not explicitly specified > in cmake. >Correct.> That leads to various issues: for instance, it’s not possible to check out > LLVM repo and run libfuzzer > tests: one would often need to compile fresh clang first, and then create > a separate build directory, > where libfuzzer could be tested. > For the buildbot this problem is approached by grabbing a freshly built > binary from another buildbot > and using that for testing. >And correct again.> > Needless to say, that could be quite annoying. > Additionally, my recent changes start using libfuzzer from Clang driver: > and finding the actual archive file > requires some hardcoding of directory paths, as one would need to go up > the tree from the Clang binary > (in swift, for example, the situation is even worse, as the path to Clang > is a symlink, and getting an archive file > from the LLVM tree would require going quite a few levels up). > > From my understanding, all these problems can be solved entirely > by moving libfuzzer to compiler-rt, where (other) sanitizers already > reside. >Yes, that might be a reasonable thing to do. I am trying to remember the reasons why we've put libFuzzer into llvm and not into compiler-rt in the first place, and failing to do so.> > Any thoughts on the suggestion? > > It would be still possible to compile just libfuzzer with no dependencies, > by simply making a partial checkout from SVN, > and only the repo path would change. >This would cause some annoyance for me and maybe some users, but nothing we can't tolerate. Still what are our other options? * wait for the mono repo to happen, then we'll be able to make libFuzzer depend on clang. (Or no?) * move libFuzzer to a separate repo, parallel to compiler-rt? (not a large win, just listing as a choice) * anything else? Does anyone see good reasons why libFuzzer should remain in llvm repo (as opposed to moving it to compiler-rt)? --kcc> > George-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170502/b96b9336/attachment.html>
Reid Kleckner via llvm-dev
2017-May-03 01:18 UTC
[llvm-dev] moving libfuzzer to compiler-rt?
On Tue, May 2, 2017 at 4:28 PM, Kostya Serebryany via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On Tue, May 2, 2017 at 12:26 PM, George Karpenkov <ekarpenkov at apple.com> > wrote: > >> From my understanding, all these problems can be solved entirely >> > by moving libfuzzer to compiler-rt, where (other) sanitizers already >> reside. >> > > Yes, that might be a reasonable thing to do. > I am trying to remember the reasons why we've put libFuzzer into llvm and > not into compiler-rt in the first place, and failing to do so. >IIRC you thought libFuzzer would be more tightly coupled to LLVM instrumentation, so the goal was to reap monorepo-like productivity developments without waiting for it to happen? Maybe it was a licensing concern, UIUC vs dual UIUC/MIT?> Any thoughts on the suggestion? >> >> It would be still possible to compile just libfuzzer with no >> dependencies, by simply making a partial checkout from SVN, >> and only the repo path would change. >> > > This would cause some annoyance for me and maybe some users, but nothing > we can't tolerate. > Still what are our other options? > > * wait for the mono repo to happen, then we'll be able to make libFuzzer > depend on clang. (Or no?) > * move libFuzzer to a separate repo, parallel to compiler-rt? (not a large > win, just listing as a choice) > * anything else? > > Does anyone see good reasons why libFuzzer should remain in llvm repo (as > opposed to moving it to compiler-rt)? >I'd like to move libFuzzer out of llvm/lib/, since it's definitely a runtime library, and not compiler library infrastructure. compiler-rt seems like the best place to put it for now, unless there are licensing concerns. Mechanically, we can do this in one history-preserving commit by checking out the entire SVN repo the way that the git-llvm script does. SVN users and users of https://github.com/llvm-project/llvm-project will still see the right history. The final monorepo will presumably also have accurate history. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170502/a2709776/attachment.html>
> Does anyone see good reasons why libFuzzer should remain in llvm repo (as > opposed to moving it to compiler-rt)?Does moving LibFuzzer to compiler-rt imply that it is compiled as part of compiler-rt and shipped with it? How does that fit with LibFuzzer's model of allowing the user to provide their own `main()`. Would you just build two different libraries. One with a `main()` and one without? Thanks, Dan.
Johan Engelen via llvm-dev
2017-Jul-12 22:07 UTC
[llvm-dev] moving libfuzzer to compiler-rt?
I'd like to add another perspective: LDC (LLVM D Compiler) can use libFuzzer and AddressSanitizer from compiler-rt, and it'd be great if building them would not require a fresh-built clang. I believe the only reason libFuzzer needs a fresh-built clang is because of the dependence on the sanitizer interface? compiler-rt can already be built without using clang, libFuzzer is a little more tricky: libFuzzer depends on <sanitizer/coverage_interface.h>, note the angle brackets, meaning that you have to install compiler-rt's include files to have libFuzzer be in version-sync with the sanitizers. My hope is that moving libFuzzer to compiler-rt will make it possible to build it together with the sanitizers, so that it is a little easier to build it for non-clang use cases. Kind regards, Johan On Tue, May 2, 2017 at 9:26 PM, George Karpenkov via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi All, > > Currently libfuzzer depends on (often freshly built) clang, yet the > dependency is not explicitly specified > in cmake. > That leads to various issues: for instance, it’s not possible to check out > LLVM repo and run libfuzzer > tests: one would often need to compile fresh clang first, and then create > a separate build directory, > where libfuzzer could be tested. > For the buildbot this problem is approached by grabbing a freshly built > binary from another buildbot > and using that for testing. > > Needless to say, that could be quite annoying. > Additionally, my recent changes start using libfuzzer from Clang driver: > and finding the actual archive file > requires some hardcoding of directory paths, as one would need to go up > the tree from the Clang binary > (in swift, for example, the situation is even worse, as the path to Clang > is a symlink, and getting an archive file > from the LLVM tree would require going quite a few levels up). > > From my understanding, all these problems can be solved entirely > by moving libfuzzer to compiler-rt, where (other) sanitizers already > reside. > > Any thoughts on the suggestion? > > It would be still possible to compile just libfuzzer with no dependencies, > by simply making a partial checkout from SVN, > and only the repo path would change. > > George > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170713/f78b0d49/attachment.html>
Kostya Serebryany via llvm-dev
2017-Jul-12 22:26 UTC
[llvm-dev] moving libfuzzer to compiler-rt?
On Wed, Jul 12, 2017 at 3:07 PM, Johan Engelen via llvm-dev < llvm-dev at lists.llvm.org> wrote:> I'd like to add another perspective: LDC (LLVM D Compiler) can use > libFuzzer and AddressSanitizer from compiler-rt, and it'd be great if > building them would not require a fresh-built clang. I believe the only > reason libFuzzer needs a fresh-built clang is because of the dependence on > the sanitizer interface? >Not really. libFuzzer depends on the coverage instrumentation applied to the target code. I.e we can build libFuzzer with any compiler (even some old GCC) but we have to use a recent clang to build the target code (including libFuzzer's own tests).> compiler-rt can already be built without using clang, libFuzzer is a > little more tricky: libFuzzer depends on <sanitizer/coverage_interface.h>, > >Not any more, removed in r307858. It loosely depends on sanitizer/lsan_interface.h though> note the angle brackets, meaning that you have to install compiler-rt's > include files to have libFuzzer be in version-sync with the sanitizers. > My hope is that moving libFuzzer to compiler-rt will make it possible to > build it together with the sanitizers, so that it is a little easier to > build it for non-clang use cases. > > Kind regards, > Johan > > > On Tue, May 2, 2017 at 9:26 PM, George Karpenkov via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> Hi All, >> >> Currently libfuzzer depends on (often freshly built) clang, yet the >> dependency is not explicitly specified >> in cmake. >> That leads to various issues: for instance, it’s not possible to check >> out LLVM repo and run libfuzzer >> tests: one would often need to compile fresh clang first, and then create >> a separate build directory, >> where libfuzzer could be tested. >> For the buildbot this problem is approached by grabbing a freshly built >> binary from another buildbot >> and using that for testing. >> >> Needless to say, that could be quite annoying. >> Additionally, my recent changes start using libfuzzer from Clang driver: >> and finding the actual archive file >> requires some hardcoding of directory paths, as one would need to go up >> the tree from the Clang binary >> (in swift, for example, the situation is even worse, as the path to Clang >> is a symlink, and getting an archive file >> from the LLVM tree would require going quite a few levels up). >> >> From my understanding, all these problems can be solved entirely >> by moving libfuzzer to compiler-rt, where (other) sanitizers already >> reside. >> >> Any thoughts on the suggestion? >> >> It would be still possible to compile just libfuzzer with no >> dependencies, by simply making a partial checkout from SVN, >> and only the repo path would change. >> >> George >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170712/074ef402/attachment.html>