Hi LLVMDev, I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana). The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take. (1) Reconcile out-of-tree functionality Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that. D11083 [CMake] Add experimental support for building compiler-rt for iOS D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module D11073 Architectures for darwin need to be conditionalized based on the operating system. One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking. (2) Build the new Darwin build behavior One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin. The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths. There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers. (3) Make compiler-rt an external project when built in-tree with Clang Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users. This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory. Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together. (4) Celebrate! That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things. So... what's next? With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly. Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds. Questions, comments, concerns, condolences? Thanks, -Chris
On Wed, Jul 29, 2015 at 1:39 PM Chris Bieneman <beanz at apple.com> wrote:> Hi LLVMDev, > > I wanted to take some time to write up and roll out a proposed road map > for CMake over the next few months. Apologies in advance for the > substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp > mana). > >Excellent email...> Questions, comments, concerns, condolences? > >Condolences for sure. I appreciate all the work you've been doing here and thanks! -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150729/7dd14bb4/attachment.html>
On Wed, Jul 29, 2015 at 01:36:43PM -0700, Chris Bieneman wrote:> Hi LLVMDev, > > I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana). > > The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take. > > (1) Reconcile out-of-tree functionality > > Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that. > > D11083 [CMake] Add experimental support for building compiler-rt for iOS > D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module > D11073 Architectures for darwin need to be conditionalized based on the operating system. > > One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking. > > (2) Build the new Darwin build behavior > > One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin. > > The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths. > > There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers. > > (3) Make compiler-rt an external project when built in-tree with Clang > > Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users. > > This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory. > > Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together. > > (4) Celebrate! > > That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things. > > So... what's next? > > With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly. > > Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds. > > Questions, comments, concerns, condolences? >Are we close enough to fixing all the CMake bugs that we can mark autoconf as deprecated now and plan to remove it after the 3.8 release? -Tom> Thanks, > -Chris > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> On Jul 29, 2015, at 7:03 PM, Tom Stellard <tom at stellard.net> wrote: > > On Wed, Jul 29, 2015 at 01:36:43PM -0700, Chris Bieneman wrote: >> Hi LLVMDev, >> >> I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana). >> >> The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take. >> >> (1) Reconcile out-of-tree functionality >> >> Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that. >> >> D11083 [CMake] Add experimental support for building compiler-rt for iOS >> D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module >> D11073 Architectures for darwin need to be conditionalized based on the operating system. >> >> One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking. >> >> (2) Build the new Darwin build behavior >> >> One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin. >> >> The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths. >> >> There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers. >> >> (3) Make compiler-rt an external project when built in-tree with Clang >> >> Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users. >> >> This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory. >> >> Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together. >> >> (4) Celebrate! >> >> That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things. >> >> So... what's next? >> >> With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly. >> >> Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds. >> >> Questions, comments, concerns, condolences? >> > > Are we close enough to fixing all the CMake bugs that we can mark autoconf > as deprecated now and plan to remove it after the 3.8 release? > > -TomThat is premature. The compiler-rt build is not even close to parity with the makefiles (at least for Apple platforms). Chris is working on a road map to fix that, but we’re still at the beginning of that road, not the end.> >> Thanks, >> -Chris >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev> > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150729/3194d78c/attachment.html>
> > Questions, comments, concerns, condolences?While in the process of moving away from autotools and fully supporting cmake be should look at the underlying generator to build with. I'm sure many of the llvm clang devs already use cmake with the ninja build tool provided here. http://martine.github.io/ninja/ We have an interesting issue however when we want to install a single components of llvm with this tool. https://github.com/martine/ninja/issues/932 This seems like something that would have to be done in the CMakeLists with new targets like install-clang similarly to how we already have check-clang. Of course I man this to be done for every component not just clang Thoughts? On Wed, Jul 29, 2015 at 9:36 PM, Chris Bieneman <beanz at apple.com> wrote:> Hi LLVMDev, > > I wanted to take some time to write up and roll out a proposed road map > for CMake over the next few months. Apologies in advance for the > substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp > mana). > > The main thing I want to talk about is PR21562. For Apple PR21562 is the > biggest reason we can't fully abandon autoconf. I've spent some time over > the last month hacking on compiler-rt's build system and I've come up with > a basic outline of the approach I'd like to take. > > (1) Reconcile out-of-tree functionality > > Apple has a little bit of out-of-tree code that adds functionality to > compiler-rt's CMake build system. Specifically the ability to build for > iOS. Before I can do any substantial refactoring of the open source build > system I need to bring it to feature parity with our internal system. I > have three remaining patches for that. > > D11083 [CMake] Add experimental support for building compiler-rt for iOS > D11082 [CMake] Adding some utility functions for Darwin builds into a new > CompilerRTDarwinUtils.cmake module > D11073 Architectures for darwin need to be conditionalized based on the > operating system. > > One thing worth noting here. I need to land some variant of this > functionality, but the way this is done is hacky and should go away. Once > these are landed I can begin the more substantial reworking. > > (2) Build the new Darwin build behavior > > One of the big problems with building compiler-rt for Darwin is that the > CMake has been hacked to build multiple platforms at once (yes I know my > patches above only make this worse). The first thing I want to do is add a > new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to > toggle into a new mode for building on Darwin. > > The new Darwin build mode will build a single triple at a time in much the > same way the Android and Linux builds work. In fact, my plan is to have the > new Darwin functionality almost completely follow the Linux and Android > code paths. > > There is a healthy bit of hand waving here because this task is larger > than I'm making it sound. Compiler-rt is comprised of two very different > sets of components, and they have some significantly different > requirements. A big part of this work is going to be the root of PR21562, > replacing the specific functionality in make/platform/clang_darwin.mk. > That functionality drives building the runtime pieces of compiler-rt. There > will need to be similar, but slightly different work done for the > sanitizers. > > (3) Make compiler-rt an external project when built in-tree with Clang > > Once compiler-rt builds for one and only one target at a time, we will > need to hook that into the LLVM build system so that you can do in-tree > builds of compiler-rt for all the targets you can target with your new > clang. I will gate this functionality on a flag just like the work above so > it won't break existing users. > > This re-working will also provide a more robust solution for PR14109. > PR14109 can't be properly fixed without making compiler-rt an external > project because CMake doesn't really support changing the compiler after > you've already configured a directory. > > Some of this functionality already exists in clang/runtime, but we'll want > to transition to that being the only supported way to build compiler-rt, > and (for Darwin) we'll need to add some support for lipo-ing thin binaries > together. > > (4) Celebrate! > > That's really the hard bits. Once we get the new behavior working through > the stack we'll need to test it like nobody's business and then we can talk > about making it default and cleaning up all the code that supported the old > way of doing things. > > So... what's next? > > With PR14109 and PR21562 done Apple will be able to migrate off autoconf, > and there will only be a few currently identified issues preventing CMake > from replacing autoconf. Tackling the remaining issues should be fairly > straight forward as none of them are too gnarly. > > Moving beyond this all I have some ideas for improving our CMake scripts > to make developers more productive, and I have some ideas for improvements > that we could get by working with the CMake community. For example, I want > our CMake build system to have convenience targets for things that we > currently drive with out-of-tree tooling. This includes clang bootstrap > builds, and generating PGO data and linker order files. I also would like > to work with the CMake community to get a way to wire up external projects > so that they could all be mapped into a single Ninja build file for better > parallelism and faster incremental builds. > > Questions, comments, concerns, condolences? > > Thanks, > -Chris > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150730/bdd63402/attachment.html>
> On Jul 30, 2015, at 7:20 AM, Martell Malone <martellmalone at gmail.com> wrote: > > Questions, comments, concerns, condolences? > > While in the process of moving away from autotools and fully supporting cmake be should look at the underlying generator to build with. > I'm sure many of the llvm clang devs already use cmake with the ninja build tool provided here. > http://martine.github.io/ninja/ <http://martine.github.io/ninja/> > > We have an interesting issue however when we want to install a single components of llvm with this tool. > https://github.com/martine/ninja/issues/932 <https://github.com/martine/ninja/issues/932> > > This seems like something that would have to be done in the CMakeLists with new targets like install-clang similarly to how we already have check-clang. > > Of course I man this to be done for every component not just clang > > Thoughts?Funny enough, I landed a patch back in February that added install-${name} targets to all the LLVM tools (r229727). Unfortunately that patch doesn’t apply to the clang tools because they don’t use the AddLLVM infrastructure. I’ve been thinking about adding a similar hook for clang too. I will probably do that at some point. -Chris> > On Wed, Jul 29, 2015 at 9:36 PM, Chris Bieneman <beanz at apple.com <mailto:beanz at apple.com>> wrote: > Hi LLVMDev, > > I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana). > > The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take. > > (1) Reconcile out-of-tree functionality > > Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that. > > D11083 [CMake] Add experimental support for building compiler-rt for iOS > D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module > D11073 Architectures for darwin need to be conditionalized based on the operating system. > > One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking. > > (2) Build the new Darwin build behavior > > One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin. > > The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths. > > There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk <http://clang_darwin.mk/>. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers. > > (3) Make compiler-rt an external project when built in-tree with Clang > > Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users. > > This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory. > > Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together. > > (4) Celebrate! > > That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things. > > So... what's next? > > With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly. > > Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds. > > Questions, comments, concerns, condolences? > > Thanks, > -Chris > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150730/ac12d565/attachment.html>
On Wed, Jul 29, 2015 at 1:36 PM, Chris Bieneman <beanz at apple.com> wrote:> I wanted to take some time to write up and roll out a proposed road map > for CMake over the next few months. Apologies in advance for the > substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp > mana). >Mono black devotion ftw! Thanks for working on this. :) The main thing I'm trying to understand is, at each point along the way, will CMake be able to generate a combined build.ninja for all compilation jobs? You're doing the work, so it's fine if that's a nongoal. However, I'd like it if the simple case where compiler-rt is being compiled for the host, the generated build files don't shell out to another build. The main thing I want to talk about is PR21562. For Apple PR21562 is the> biggest reason we can't fully abandon autoconf. I've spent some time over > the last month hacking on compiler-rt's build system and I've come up with > a basic outline of the approach I'd like to take. > > (1) Reconcile out-of-tree functionality > > Apple has a little bit of out-of-tree code that adds functionality to > compiler-rt's CMake build system. Specifically the ability to build for > iOS. Before I can do any substantial refactoring of the open source build > system I need to bring it to feature parity with our internal system. I > have three remaining patches for that. > > D11083 [CMake] Add experimental support for building compiler-rt for iOS > D11082 [CMake] Adding some utility functions for Darwin builds into a new > CompilerRTDarwinUtils.cmake module > D11073 Architectures for darwin need to be conditionalized based on the > operating system. > > One thing worth noting here. I need to land some variant of this > functionality, but the way this is done is hacky and should go away. Once > these are landed I can begin the more substantial reworking. > > (2) Build the new Darwin build behavior > > One of the big problems with building compiler-rt for Darwin is that the > CMake has been hacked to build multiple platforms at once (yes I know my > patches above only make this worse). The first thing I want to do is add a > new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to > toggle into a new mode for building on Darwin. > > The new Darwin build mode will build a single triple at a time in much the > same way the Android and Linux builds work. In fact, my plan is to have the > new Darwin functionality almost completely follow the Linux and Android > code paths. > > There is a healthy bit of hand waving here because this task is larger > than I'm making it sound. Compiler-rt is comprised of two very different > sets of components, and they have some significantly different > requirements. A big part of this work is going to be the root of PR21562, > replacing the specific functionality in make/platform/clang_darwin.mk. > That functionality drives building the runtime pieces of compiler-rt. There > will need to be similar, but slightly different work done for the > sanitizers. > > (3) Make compiler-rt an external project when built in-tree with Clang > > Once compiler-rt builds for one and only one target at a time, we will > need to hook that into the LLVM build system so that you can do in-tree > builds of compiler-rt for all the targets you can target with your new > clang. I will gate this functionality on a flag just like the work above so > it won't break existing users. > > This re-working will also provide a more robust solution for PR14109. > PR14109 can't be properly fixed without making compiler-rt an external > project because CMake doesn't really support changing the compiler after > you've already configured a directory. > > Some of this functionality already exists in clang/runtime, but we'll want > to transition to that being the only supported way to build compiler-rt, > and (for Darwin) we'll need to add some support for lipo-ing thin binaries > together. > > (4) Celebrate! > > That's really the hard bits. Once we get the new behavior working through > the stack we'll need to test it like nobody's business and then we can talk > about making it default and cleaning up all the code that supported the old > way of doing things. > > So... what's next? > > With PR14109 and PR21562 done Apple will be able to migrate off autoconf, > and there will only be a few currently identified issues preventing CMake > from replacing autoconf. Tackling the remaining issues should be fairly > straight forward as none of them are too gnarly. > > Moving beyond this all I have some ideas for improving our CMake scripts > to make developers more productive, and I have some ideas for improvements > that we could get by working with the CMake community. For example, I want > our CMake build system to have convenience targets for things that we > currently drive with out-of-tree tooling. This includes clang bootstrap > builds, and generating PGO data and linker order files. I also would like > to work with the CMake community to get a way to wire up external projects > so that they could all be mapped into a single Ninja build file for better > parallelism and faster incremental builds. > > Questions, comments, concerns, condolences? > > Thanks, > -Chris > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150730/f512046b/attachment.html>
> On Jul 30, 2015, at 10:30 AM, Reid Kleckner <rnk at google.com> wrote: > > On Wed, Jul 29, 2015 at 1:36 PM, Chris Bieneman <beanz at apple.com <mailto:beanz at apple.com>> wrote: > I wanted to take some time to write up and roll out a proposed road map for CMake over the next few months. Apologies in advance for the substantial 0/7 Wall of Text I've summoned here (it only cost me 3 swamp mana). > > Mono black devotion ftw! Thanks for working on this. :) > > The main thing I'm trying to understand is, at each point along the way, will CMake be able to generate a combined build.ninja for all compilation jobs?In order to generate a single combined build.ninja file we’ll actually need some changes to CMake. My intention is to have a high-level build file (generator agnostic) that will invoke the build tool for sub-projects for compiler-rt. For the Ninja generator we can use a custom job pool for the sub-projects to prevent over saturation of the build machine, but there will be sub-projects.> > You're doing the work, so it's fine if that's a nongoal. However, I'd like it if the simple case where compiler-rt is being compiled for the host, the generated build files don't shell out to another build.I agree, and that is how I would prefer this to work. I had a discussion with some of the guys from Sony a few weeks back about what it would take to be able to generate a single combined build.ninja file. There were three features we would need from CMake to make this work properly. (1) Prefixing rules so that the sub-project rules wouldn’t conflict with each other or the master project’s rules (2) Importing the sub-projects into the parent ninja (3) Mark the CMake invocations for the external projects as “generator” rules so that Ninja would re-exec itself if CMake changed the build files -Chris> > The main thing I want to talk about is PR21562. For Apple PR21562 is the biggest reason we can't fully abandon autoconf. I've spent some time over the last month hacking on compiler-rt's build system and I've come up with a basic outline of the approach I'd like to take. > > (1) Reconcile out-of-tree functionality > > Apple has a little bit of out-of-tree code that adds functionality to compiler-rt's CMake build system. Specifically the ability to build for iOS. Before I can do any substantial refactoring of the open source build system I need to bring it to feature parity with our internal system. I have three remaining patches for that. > > D11083 [CMake] Add experimental support for building compiler-rt for iOS > D11082 [CMake] Adding some utility functions for Darwin builds into a new CompilerRTDarwinUtils.cmake module > D11073 Architectures for darwin need to be conditionalized based on the operating system. > > One thing worth noting here. I need to land some variant of this functionality, but the way this is done is hacky and should go away. Once these are landed I can begin the more substantial reworking. > > (2) Build the new Darwin build behavior > > One of the big problems with building compiler-rt for Darwin is that the CMake has been hacked to build multiple platforms at once (yes I know my patches above only make this worse). The first thing I want to do is add a new variable (something like COMPILER_RT_EXPERIMENTAL_DARWIN_BUILD) to toggle into a new mode for building on Darwin. > > The new Darwin build mode will build a single triple at a time in much the same way the Android and Linux builds work. In fact, my plan is to have the new Darwin functionality almost completely follow the Linux and Android code paths. > > There is a healthy bit of hand waving here because this task is larger than I'm making it sound. Compiler-rt is comprised of two very different sets of components, and they have some significantly different requirements. A big part of this work is going to be the root of PR21562, replacing the specific functionality in make/platform/clang_darwin.mk <http://clang_darwin.mk/>. That functionality drives building the runtime pieces of compiler-rt. There will need to be similar, but slightly different work done for the sanitizers. > > (3) Make compiler-rt an external project when built in-tree with Clang > > Once compiler-rt builds for one and only one target at a time, we will need to hook that into the LLVM build system so that you can do in-tree builds of compiler-rt for all the targets you can target with your new clang. I will gate this functionality on a flag just like the work above so it won't break existing users. > > This re-working will also provide a more robust solution for PR14109. PR14109 can't be properly fixed without making compiler-rt an external project because CMake doesn't really support changing the compiler after you've already configured a directory. > > Some of this functionality already exists in clang/runtime, but we'll want to transition to that being the only supported way to build compiler-rt, and (for Darwin) we'll need to add some support for lipo-ing thin binaries together. > > (4) Celebrate! > > That's really the hard bits. Once we get the new behavior working through the stack we'll need to test it like nobody's business and then we can talk about making it default and cleaning up all the code that supported the old way of doing things. > > So... what's next? > > With PR14109 and PR21562 done Apple will be able to migrate off autoconf, and there will only be a few currently identified issues preventing CMake from replacing autoconf. Tackling the remaining issues should be fairly straight forward as none of them are too gnarly. > > Moving beyond this all I have some ideas for improving our CMake scripts to make developers more productive, and I have some ideas for improvements that we could get by working with the CMake community. For example, I want our CMake build system to have convenience targets for things that we currently drive with out-of-tree tooling. This includes clang bootstrap builds, and generating PGO data and linker order files. I also would like to work with the CMake community to get a way to wire up external projects so that they could all be mapped into a single Ninja build file for better parallelism and faster incremental builds. > > Questions, comments, concerns, condolences? > > Thanks, > -Chris > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150730/8a81f997/attachment.html>