llvm dev - Feb 2015 - [LLVMdev] RFC: Dropping support for building sanitizers with autotools

Unfortunately, we do rely on the Autotools to build for darwin. 

The lack of proper testing is very unfortunate. In the future, we will either
drop the dependency on Makefile/autoconf build or add the support for sanitizer
testing.

Anna.
> On Feb 17, 2015, at 10:54 AM, Kostya Serebryany <kcc at google.com>
wrote:
> 
> +1 (in case you still need mine)
> 
> On Thu, Feb 12, 2015 at 6:31 PM, Alexey Samsonov <vonosmas at gmail.com
<mailto:vonosmas at gmail.com>> wrote:
> Hi everyone,
> 
> TL;DR I plan to do subj unless anyone objects.
> 
> Autotools support for building sanitizers was never complete, is a proper
subset of CMake capabilities, and the gap between the features of CMake build
and autotools is quickly increasing:
> 1) OS: autotools support Linux, Mac OS X and Android; CMake also supports
FreeBSD and Windows.
> 2) Architectures: autotools support i386 and x86_64, CMake also supports
MIPS, ARM/AArch64, PowerPC).
> 3) Library variants: autotools don't support building shared ASan
runtime on Linux.
> 4 [!!!]) Autotools build doesn't have *any* support for building and
running tests.
> 5) The sets of compiler flags we use to build runtimes are different in two
builds, and are hard to kept in sync.
> 
> Because of (4), sanitizer runtimes built with autotools are severely
undertested, and maintaining two different build systems is a burden I would
like to get rid of. Now seems to be a good time for that: we've cut off 3.6
release branch, and Hans assures me that 3.7 release process will use CMake.
> 
> Are there existing users of configure/make that want sanitizers to be
available in their build? Are there other arguments for keeping the autotools
build "working" I'm missing?
> 
> -- 
> Alexey Samsonov
> vonosmas at gmail.com <mailto:vonosmas at gmail.com>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150217/36102db7/attachment.html>

Do you have an ETA on this effort?

-eric

On Tue Feb 17 2015 at 3:52:49 PM Anna Zaks <ganna at apple.com> wrote:
> Unfortunately, we do rely on the Autotools to build for darwin.
>
> The lack of proper testing is very unfortunate. In the future, we will
> either drop the dependency on Makefile/autoconf build or add the support
> for sanitizer testing.
>
> Anna.
>
>
> On Feb 17, 2015, at 10:54 AM, Kostya Serebryany <kcc at google.com>
wrote:
>
> +1 (in case you still need mine)
>
> On Thu, Feb 12, 2015 at 6:31 PM, Alexey Samsonov <vonosmas at
gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> TL;DR I plan to do subj unless anyone objects.
>>
>> Autotools support for building sanitizers was never complete, is a
proper
>> subset of CMake capabilities, and the gap between the features of CMake
>> build and autotools is quickly increasing:
>> 1) OS: autotools support Linux, Mac OS X and Android; CMake also
supports
>> FreeBSD and Windows.
>> 2) Architectures: autotools support i386 and x86_64, CMake also
supports
>> MIPS, ARM/AArch64, PowerPC).
>> 3) Library variants: autotools don't support building shared ASan
runtime
>> on Linux.
>> 4 [!!!]) Autotools build doesn't have *any* support for building
and
>> running tests.
>> 5) The sets of compiler flags we use to build runtimes are different in
>> two builds, and are hard to kept in sync.
>>
>> Because of (4), sanitizer runtimes built with autotools are severely
>> undertested, and maintaining two different build systems is a burden I
>> would like to get rid of. Now seems to be a good time for that:
we've cut
>> off 3.6 release branch, and Hans assures me that 3.7 release process
will
>> use CMake.
>>
>> Are there existing users of configure/make that want sanitizers to be
>> available in their build? Are there other arguments for keeping the
>> autotools build "working" I'm missing?
>>
>> --
>> Alexey Samsonov
>> vonosmas at gmail.com
>>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150217/7bcfa1f5/attachment.html>

On Tue, Feb 17, 2015 at 3:37 PM, Anna Zaks <ganna at apple.com> wrote:
> Unfortunately, we do rely on the Autotools to build for darwin.
>
> The lack of proper testing is very unfortunate. In the future, we will
> either drop the dependency on Makefile/autoconf build or add the support
> for sanitizer testing.
>
Sorry, for some reason I had the impression that you're also more
interested in CMake buiild :-/. Only now I noticed that recent Kuba's
patches about -isysroot fix both makefile and CMake
build system (and he's probably not very happy about it).

I believe the effort required to port sanitizer unit tests to autotools is
not worth it, and it's better to spend that time migrating the release
tools to CMake. Keeping two build systems "working"
was and is a manageable, but unpleasant burden. Providing a good test
coverage for two runtime versions is hard, and keeping two runtime versions
"equivalent" (built with the same compiler flags)
is close to impossible.

I understand that removing autotools support right now breaks your
integration. What if we just delete autotools support from all Linux builds?

Speaking of autotools support, I should lay out the reason why I raised the
question of removing it last week. We plan to make intrusive changes to
ASan and UBSan library layout:
1) embed UBSan runtime into ASan to solve various initialization and
integration problems of using ASan+UBSan (fixing
http://llvm.org/bugs/show_bug.cgi?id=21112)
2) make UBSan runtime standalone, and forbid to use it with another
sanitizers except for ASan.
Unfortunately, it means changing a bunch of build rules in *both* CMake and
makefile build.

Ideally, I would be happy to transfer ownership of current autotools build
on Darwin to you, and ask for help in implementing/testing necessary
changes on Mac OS X side :)
At the moment we don't even have a live buildbot running autotools-built
ASan on Mac, so you're in much better shape to catch and notice regressions.

> Anna.
>
> On Feb 17, 2015, at 10:54 AM, Kostya Serebryany <kcc at google.com>
wrote:
>
> +1 (in case you still need mine)
>
> On Thu, Feb 12, 2015 at 6:31 PM, Alexey Samsonov <vonosmas at
gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> TL;DR I plan to do subj unless anyone objects.
>>
>> Autotools support for building sanitizers was never complete, is a
proper
>> subset of CMake capabilities, and the gap between the features of CMake
>> build and autotools is quickly increasing:
>> 1) OS: autotools support Linux, Mac OS X and Android; CMake also
supports
>> FreeBSD and Windows.
>> 2) Architectures: autotools support i386 and x86_64, CMake also
supports
>> MIPS, ARM/AArch64, PowerPC).
>> 3) Library variants: autotools don't support building shared ASan
runtime
>> on Linux.
>> 4 [!!!]) Autotools build doesn't have *any* support for building
and
>> running tests.
>> 5) The sets of compiler flags we use to build runtimes are different in
>> two builds, and are hard to kept in sync.
>>
>> Because of (4), sanitizer runtimes built with autotools are severely
>> undertested, and maintaining two different build systems is a burden I
>> would like to get rid of. Now seems to be a good time for that:
we've cut
>> off 3.6 release branch, and Hans assures me that 3.7 release process
will
>> use CMake.
>>
>> Are there existing users of configure/make that want sanitizers to be
>> available in their build? Are there other arguments for keeping the
>> autotools build "working" I'm missing?
>>
>> --
>> Alexey Samsonov
>> vonosmas at gmail.com
>>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>

-- 
Alexey Samsonov
vonosmas at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150217/3cf4d30a/attachment.html>

On Tue, Feb 17, 2015 at 3:37 PM, Anna Zaks <ganna at apple.com> wrote:
> Unfortunately, we do rely on the Autotools to build for darwin.
>
> The lack of proper testing is very unfortunate.
>
This isn't just "proper testing". This is any testing at all. I
have to
guess that you're not using it for *development*, just using it as part of
some release process build system?

If that's the case, I think it would be better for you to maintain whatever
you need to support that in the interim out-of-tree. If the tests don't
run, I don't think it's really reasonable to keep a build system in tree
and expect open source contributors to update it and keep it working. How
will they even know if it is working? We don't even have build bots that
cover it as far as I know? (If we do, that somewhat lessens this pain...)


In the future, we will either drop the dependency on
Makefile/autoconf
> build or add the support for sanitizer testing.
>
Given the push to move away from autotools altogether, it would seem really
weird to add support for testing to it rather than migrating the dependency
away. You can even use the cmake build *from* an autoconf/Makefile setup by
running cmake directly. It's a bit annoying, but it can be done.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150217/26c4c97a/attachment.html>

> On Feb 17, 2015, at 4:09 PM, Chandler Carruth <chandlerc at
google.com> wrote:
> 
> 
> On Tue, Feb 17, 2015 at 3:37 PM, Anna Zaks <ganna at apple.com
<mailto:ganna at apple.com>> wrote:
> Unfortunately, we do rely on the Autotools to build for darwin. 
> 
> The lack of proper testing is very unfortunate.
> 
> This isn't just "proper testing". This is any testing at all.
I have to guess that you're not using it for *development*, just using it as
part of some release process build system?
> 
> If that's the case, I think it would be better for you to maintain
whatever you need to support that in the interim out-of-tree. If the tests
don't run, I don't think it's really reasonable to keep a build
system in tree and expect open source contributors to update it and keep it
working. How will they even know if it is working? We don't even have build
bots that cover it as far as I know? (If we do, that somewhat lessens this
pain…)
We do have buildbots, e.g.:
http://lab.llvm.org:8080/green/view/Clang/job/clang-stage1-configure-RA_build/
<http://lab.llvm.org:8080/green/view/Clang/job/clang-stage1-configure-RA_build/>
> 
> 
> In the future, we will either drop the dependency on Makefile/autoconf
build or add the support for sanitizer testing.
> 
> Given the push to move away from autotools altogether, it would seem really
weird to add support for testing to it rather than migrating the dependency
away. You can even use the cmake build *from* an autoconf/Makefile setup by
running cmake directly. It's a bit annoying, but it can be done.
There are still a lot of issues blocking full CMake equivalence with
autoconf/Makefile (PR15732). Removing the ASan makefile support is really
premature.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150217/e5c45bd3/attachment.html>

> On Feb 17, 2015, at 4:00 PM, Alexey Samsonov <vonosmas at gmail.com>
wrote:
> 
> 
> On Tue, Feb 17, 2015 at 3:37 PM, Anna Zaks <ganna at apple.com
<mailto:ganna at apple.com>> wrote:
> Unfortunately, we do rely on the Autotools to build for darwin. 
> 
> The lack of proper testing is very unfortunate. In the future, we will
either drop the dependency on Makefile/autoconf build or add the support for
sanitizer testing.
> 
> Sorry, for some reason I had the impression that you're also more
interested in CMake buiild :-/. Only now I noticed that recent Kuba's
patches about -isysroot fix both makefile and CMake
> build system (and he's probably not very happy about it).
> 
> I believe the effort required to port sanitizer unit tests to autotools is
not worth it, and it's better to spend that time migrating the release tools
to CMake. Keeping two build systems "working"
> was and is a manageable, but unpleasant burden. Providing a good test
coverage for two runtime versions is hard, and keeping two runtime versions
"equivalent" (built with the same compiler flags)
> is close to impossible.
I agree. The solution where we only rely on cmake build would be the best
option.
>  
> I understand that removing autotools support right now breaks your
integration. What if we just delete autotools support from all Linux builds?
> 
Yes. That is fine by me.
> Speaking of autotools support, I should lay out the reason why I raised the
question of removing it last week. We plan to make intrusive changes to ASan and
UBSan library layout:
> 1) embed UBSan runtime into ASan to solve various initialization and
integration problems of using ASan+UBSan (fixing
http://llvm.org/bugs/show_bug.cgi?id=21112
<http://llvm.org/bugs/show_bug.cgi?id=21112>)
> 2) make UBSan runtime standalone, and forbid to use it with another
sanitizers except for ASan.
> Unfortunately, it means changing a bunch of build rules in *both* CMake and
makefile build.
> 
> Ideally, I would be happy to transfer ownership of current autotools build
on Darwin to you, and ask for help in implementing/testing necessary changes on
Mac OS X side :)
> At the moment we don't even have a live buildbot running
autotools-built ASan on Mac, so you're in much better shape to catch and
notice regressions.
> 
We do have public buildbots set up for OS X:

Here is one that is testing a makefile build. There are probably others in
different configurations.
http://lab.llvm.org:8080/green/view/Clang/job/clang-stage1-configure-RA_build/
<http://lab.llvm.org:8080/green/view/Clang/job/clang-stage1-configure-RA_build/>

We also have a buildbot running cmake tests on OS X:
http://lab.llvm.org:8080/green/view/All/job/clang-stage1-cmake-RA_check/
<http://lab.llvm.org:8080/green/view/All/job/clang-stage1-cmake-RA_check/>
> 
> Anna.
> 
>> On Feb 17, 2015, at 10:54 AM, Kostya Serebryany <kcc at google.com
<mailto:kcc at google.com>> wrote:
>> 
>> +1 (in case you still need mine)
>> 
>> On Thu, Feb 12, 2015 at 6:31 PM, Alexey Samsonov <vonosmas at
gmail.com <mailto:vonosmas at gmail.com>> wrote:
>> Hi everyone,
>> 
>> TL;DR I plan to do subj unless anyone objects.
>> 
>> Autotools support for building sanitizers was never complete, is a
proper subset of CMake capabilities, and the gap between the features of CMake
build and autotools is quickly increasing:
>> 1) OS: autotools support Linux, Mac OS X and Android; CMake also
supports FreeBSD and Windows.
>> 2) Architectures: autotools support i386 and x86_64, CMake also
supports MIPS, ARM/AArch64, PowerPC).
>> 3) Library variants: autotools don't support building shared ASan
runtime on Linux.
>> 4 [!!!]) Autotools build doesn't have *any* support for building
and running tests.
>> 5) The sets of compiler flags we use to build runtimes are different in
two builds, and are hard to kept in sync.
>> 
>> Because of (4), sanitizer runtimes built with autotools are severely
undertested, and maintaining two different build systems is a burden I would
like to get rid of. Now seems to be a good time for that: we've cut off 3.6
release branch, and Hans assures me that 3.7 release process will use CMake.
>> 
>> Are there existing users of configure/make that want sanitizers to be
available in their build? Are there other arguments for keeping the autotools
build "working" I'm missing?
>> 
>> -- 
>> Alexey Samsonov
>> vonosmas at gmail.com <mailto:vonosmas at gmail.com>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
> 
> 
> 
> -- 
> Alexey Samsonov
> vonosmas at gmail.com <mailto:vonosmas at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150217/cc4e8694/attachment.html>

Hi Alexey,

I'd like to be pinged when you start doing that refactoring, just in case I
miss it.

Thanks

  Filipe

On Tue, Feb 17, 2015 at 4:00 PM, Alexey Samsonov <vonosmas at gmail.com>
wrote:
>
> On Tue, Feb 17, 2015 at 3:37 PM, Anna Zaks <ganna at apple.com>
wrote:
>
>> Unfortunately, we do rely on the Autotools to build for darwin.
>>
>
>> The lack of proper testing is very unfortunate. In the future, we will
>> either drop the dependency on Makefile/autoconf build or add the
support
>> for sanitizer testing.
>>
>
> Sorry, for some reason I had the impression that you're also more
> interested in CMake buiild :-/. Only now I noticed that recent Kuba's
> patches about -isysroot fix both makefile and CMake
> build system (and he's probably not very happy about it).
>
> I believe the effort required to port sanitizer unit tests to autotools is
> not worth it, and it's better to spend that time migrating the release
> tools to CMake. Keeping two build systems "working"
> was and is a manageable, but unpleasant burden. Providing a good test
> coverage for two runtime versions is hard, and keeping two runtime versions
> "equivalent" (built with the same compiler flags)
> is close to impossible.
>
> I understand that removing autotools support right now breaks your
> integration. What if we just delete autotools support from all Linux
builds?
>
> Speaking of autotools support, I should lay out the reason why I raised
> the question of removing it last week. We plan to make intrusive changes to
> ASan and UBSan library layout:
> 1) embed UBSan runtime into ASan to solve various initialization and
> integration problems of using ASan+UBSan (fixing
> http://llvm.org/bugs/show_bug.cgi?id=21112)
> 2) make UBSan runtime standalone, and forbid to use it with another
> sanitizers except for ASan.
> Unfortunately, it means changing a bunch of build rules in *both* CMake
> and makefile build.
>
> Ideally, I would be happy to transfer ownership of current autotools build
> on Darwin to you, and ask for help in implementing/testing necessary
> changes on Mac OS X side :)
> At the moment we don't even have a live buildbot running
autotools-built
> ASan on Mac, so you're in much better shape to catch and notice
regressions.
>
>
>> Anna.
>>
>> On Feb 17, 2015, at 10:54 AM, Kostya Serebryany <kcc at
google.com> wrote:
>>
>> +1 (in case you still need mine)
>>
>> On Thu, Feb 12, 2015 at 6:31 PM, Alexey Samsonov <vonosmas at
gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> TL;DR I plan to do subj unless anyone objects.
>>>
>>> Autotools support for building sanitizers was never complete, is a
>>> proper subset of CMake capabilities, and the gap between the
features of
>>> CMake build and autotools is quickly increasing:
>>> 1) OS: autotools support Linux, Mac OS X and Android; CMake also
>>> supports FreeBSD and Windows.
>>> 2) Architectures: autotools support i386 and x86_64, CMake also
supports
>>> MIPS, ARM/AArch64, PowerPC).
>>> 3) Library variants: autotools don't support building shared
ASan
>>> runtime on Linux.
>>> 4 [!!!]) Autotools build doesn't have *any* support for
building and
>>> running tests.
>>> 5) The sets of compiler flags we use to build runtimes are
different in
>>> two builds, and are hard to kept in sync.
>>>
>>> Because of (4), sanitizer runtimes built with autotools are
severely
>>> undertested, and maintaining two different build systems is a
burden I
>>> would like to get rid of. Now seems to be a good time for that:
we've cut
>>> off 3.6 release branch, and Hans assures me that 3.7 release
process will
>>> use CMake.
>>>
>>> Are there existing users of configure/make that want sanitizers to
be
>>> available in their build? Are there other arguments for keeping the
>>> autotools build "working" I'm missing?
>>>
>>> --
>>> Alexey Samsonov
>>> vonosmas at gmail.com
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>
>>
>
>
> --
> Alexey Samsonov
> vonosmas at gmail.com
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150218/3998694e/attachment.html>