Mehdi Amini via llvm-dev
2016-Mar-01 18:01 UTC
[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS
> On Mar 1, 2016, at 9:57 AM, Chris Bieneman <cbieneman at apple.com> wrote: > > There are a few notes I'd like to add to this thread. > > (1) we have a number of places throughout out CMake build where we use features from newer CMakes gated by version checks. Most of these features are performance or usability related. None of them are correctness. Using the latest CMake release will often result in faster builds, so I encourage it. > > (2) CMake's "install" target will pretty much always be slower from clean than the old autoconf/make "install" target. This is because in CMake "install" depends on "all", and our CMake builds more stuff in "all" than autoconf did. To help with this or CMake system has lots of convenient "install-${name}" targets that support component-based installation. Not every component has one of these rules, but if one you need is missing let me know. I also recently (r261681) added a new option (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of components that have custom install targets. It then creates a new "install-distribution" target that wraps just the components you want. For Apple this is almost a 40% speed up over "ninja install".That sounds great, I want to use it! It would even be more awesome with an description/example in docs/CMake.rst :) -- Mehdi> > On Feb 25, 2016, at 11:08 AM, Sedat Dilek via llvm-dev <llvm-dev at lists.llvm.org> wrote: > >>> Which combination of cmake/ninja versions are you using (latest are >>> v3.4.3 and v1.6.0)? >> >> With this combination I could reduce build-time down from approx. 3h >> down to 01h20m. >> >> $ egrep -i 'jobs|ninja' llvm-build/CMakeCache.txt >> //Program used to build from build.ninja files. >> CMAKE_MAKE_PROGRAM:FILEPATH=/opt/cmake/bin/ninja >> //Define the maximum number of concurrent compilation jobs. >> LLVM_PARALLEL_COMPILE_JOBS:STRING=3 >> //Define the maximum number of concurrent link jobs. >> LLVM_PARALLEL_LINK_JOBS:STRING=1 >> CMAKE_GENERATOR:INTERNAL=Ninja >> >> $ LC_ALL=C ls -alt logs/3.8.0rc3_clang-3-8-0-rc3_cmake-3-4-3_ninja-1-6-0/ >> total 360 >> drwxr-xr-x 2 wearefam wearefam 4096 Feb 25 19:58 . >> drwxr-xr-x 6 wearefam wearefam 4096 Feb 25 19:58 .. >> -rw-r--r-- 1 wearefam wearefam 130196 Feb 25 19:54 >> install-log_llvm-toolchain-3.8.0rc3.txt >> -rw-r--r-- 1 wearefam wearefam 205762 Feb 25 19:51 >> build-log_llvm-toolchain-3.8.0rc3.txt >> -rw-r--r-- 1 wearefam wearefam 14331 Feb 25 18:30 >> configure-log_llvm-toolchain-3.8.0rc3.txt >> >> $ LC_ALL=C du -s -m llvm* /opt/llvm-toolchain-3.8.0rc3 >> 315 llvm >> 941 llvm-build >> 609 /opt/llvm-toolchain-3.8.0rc3 >> >> - Sedat - >> >> [1] https://cmake.org/files/v3.5/cmake-3.5.0-rc3-Linux-x86_64.tar.gz >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
ChrisBieneman via llvm-dev
2016-Mar-01 20:17 UTC
[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS
> On Mar 1, 2016, at 10:01 AM, Mehdi Amini <mehdi.amini at apple.com> wrote: > > >> On Mar 1, 2016, at 9:57 AM, Chris Bieneman <cbieneman at apple.com> wrote: >> >> There are a few notes I'd like to add to this thread. >> >> (1) we have a number of places throughout out CMake build where we use features from newer CMakes gated by version checks. Most of these features are performance or usability related. None of them are correctness. Using the latest CMake release will often result in faster builds, so I encourage it. >> >> (2) CMake's "install" target will pretty much always be slower from clean than the old autoconf/make "install" target. This is because in CMake "install" depends on "all", and our CMake builds more stuff in "all" than autoconf did. To help with this or CMake system has lots of convenient "install-${name}" targets that support component-based installation. Not every component has one of these rules, but if one you need is missing let me know. I also recently (r261681) added a new option (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of components that have custom install targets. It then creates a new "install-distribution" target that wraps just the components you want. For Apple this is almost a 40% speed up over "ninja install". > > That sounds great, I want to use it! > It would even be more awesome with an description/example in docs/CMake.rst :)Once I get the last of the kinks worked out for our internal adoption I'm going to open source our config files that use it. I've also made a note to remind myself to document it in docs/CMake.rst. I need to do a pass updating that with a bunch of the cool new things we're doing with CMake. Thanks for the reminder. -Chris> > -- > Mehdi >> >> On Feb 25, 2016, at 11:08 AM, Sedat Dilek via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> >>>> Which combination of cmake/ninja versions are you using (latest are >>>> v3.4.3 and v1.6.0)? >>> >>> With this combination I could reduce build-time down from approx. 3h >>> down to 01h20m. >>> >>> $ egrep -i 'jobs|ninja' llvm-build/CMakeCache.txt >>> //Program used to build from build.ninja files. >>> CMAKE_MAKE_PROGRAM:FILEPATH=/opt/cmake/bin/ninja >>> //Define the maximum number of concurrent compilation jobs. >>> LLVM_PARALLEL_COMPILE_JOBS:STRING=3 >>> //Define the maximum number of concurrent link jobs. >>> LLVM_PARALLEL_LINK_JOBS:STRING=1 >>> CMAKE_GENERATOR:INTERNAL=Ninja >>> >>> $ LC_ALL=C ls -alt logs/3.8.0rc3_clang-3-8-0-rc3_cmake-3-4-3_ninja-1-6-0/ >>> total 360 >>> drwxr-xr-x 2 wearefam wearefam 4096 Feb 25 19:58 . >>> drwxr-xr-x 6 wearefam wearefam 4096 Feb 25 19:58 .. >>> -rw-r--r-- 1 wearefam wearefam 130196 Feb 25 19:54 >>> install-log_llvm-toolchain-3.8.0rc3.txt >>> -rw-r--r-- 1 wearefam wearefam 205762 Feb 25 19:51 >>> build-log_llvm-toolchain-3.8.0rc3.txt >>> -rw-r--r-- 1 wearefam wearefam 14331 Feb 25 18:30 >>> configure-log_llvm-toolchain-3.8.0rc3.txt >>> >>> $ LC_ALL=C du -s -m llvm* /opt/llvm-toolchain-3.8.0rc3 >>> 315 llvm >>> 941 llvm-build >>> 609 /opt/llvm-toolchain-3.8.0rc3 >>> >>> - Sedat - >>> >>> [1] https://cmake.org/files/v3.5/cmake-3.5.0-rc3-Linux-x86_64.tar.gz >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >
Fabio Pagani via llvm-dev
2016-Mar-01 21:12 UTC
[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS
For faster builds and rebuilds you should definitely read: https://blogs.s-osg.org/an-introduction-to-accelerating-your-build-with-clang/ https://blogs.s-osg.org/a-conclusion-to-accelerating-your-build-with-clang/ Hope this helps! On Tue, Mar 1, 2016 at 9:17 PM, ChrisBieneman via llvm-dev < llvm-dev at lists.llvm.org> wrote:> > > > On Mar 1, 2016, at 10:01 AM, Mehdi Amini <mehdi.amini at apple.com> wrote: > > > > > >> On Mar 1, 2016, at 9:57 AM, Chris Bieneman <cbieneman at apple.com> wrote: > >> > >> There are a few notes I'd like to add to this thread. > >> > >> (1) we have a number of places throughout out CMake build where we use > features from newer CMakes gated by version checks. Most of these features > are performance or usability related. None of them are correctness. Using > the latest CMake release will often result in faster builds, so I encourage > it. > >> > >> (2) CMake's "install" target will pretty much always be slower from > clean than the old autoconf/make "install" target. This is because in CMake > "install" depends on "all", and our CMake builds more stuff in "all" than > autoconf did. To help with this or CMake system has lots of convenient > "install-${name}" targets that support component-based installation. Not > every component has one of these rules, but if one you need is missing let > me know. I also recently (r261681) added a new option > (LLVM_DISTRIBUTION_COMPONENTS) that allows you to specify a list of > components that have custom install targets. It then creates a new > "install-distribution" target that wraps just the components you want. For > Apple this is almost a 40% speed up over "ninja install". > > > > That sounds great, I want to use it! > > It would even be more awesome with an description/example in > docs/CMake.rst :) > > Once I get the last of the kinks worked out for our internal adoption I'm > going to open source our config files that use it. > > I've also made a note to remind myself to document it in docs/CMake.rst. I > need to do a pass updating that with a bunch of the cool new things we're > doing with CMake. Thanks for the reminder. > > -Chris > > > > > -- > > Mehdi > >> > >> On Feb 25, 2016, at 11:08 AM, Sedat Dilek via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> > >>>> Which combination of cmake/ninja versions are you using (latest are > >>>> v3.4.3 and v1.6.0)? > >>> > >>> With this combination I could reduce build-time down from approx. 3h > >>> down to 01h20m. > >>> > >>> $ egrep -i 'jobs|ninja' llvm-build/CMakeCache.txt > >>> //Program used to build from build.ninja files. > >>> CMAKE_MAKE_PROGRAM:FILEPATH=/opt/cmake/bin/ninja > >>> //Define the maximum number of concurrent compilation jobs. > >>> LLVM_PARALLEL_COMPILE_JOBS:STRING=3 > >>> //Define the maximum number of concurrent link jobs. > >>> LLVM_PARALLEL_LINK_JOBS:STRING=1 > >>> CMAKE_GENERATOR:INTERNAL=Ninja > >>> > >>> $ LC_ALL=C ls -alt > logs/3.8.0rc3_clang-3-8-0-rc3_cmake-3-4-3_ninja-1-6-0/ > >>> total 360 > >>> drwxr-xr-x 2 wearefam wearefam 4096 Feb 25 19:58 . > >>> drwxr-xr-x 6 wearefam wearefam 4096 Feb 25 19:58 .. > >>> -rw-r--r-- 1 wearefam wearefam 130196 Feb 25 19:54 > >>> install-log_llvm-toolchain-3.8.0rc3.txt > >>> -rw-r--r-- 1 wearefam wearefam 205762 Feb 25 19:51 > >>> build-log_llvm-toolchain-3.8.0rc3.txt > >>> -rw-r--r-- 1 wearefam wearefam 14331 Feb 25 18:30 > >>> configure-log_llvm-toolchain-3.8.0rc3.txt > >>> > >>> $ LC_ALL=C du -s -m llvm* /opt/llvm-toolchain-3.8.0rc3 > >>> 315 llvm > >>> 941 llvm-build > >>> 609 /opt/llvm-toolchain-3.8.0rc3 > >>> > >>> - Sedat - > >>> > >>> [1] https://cmake.org/files/v3.5/cmake-3.5.0-rc3-Linux-x86_64.tar.gz > >>> _______________________________________________ > >>> LLVM Developers mailing list > >>> llvm-dev at lists.llvm.org > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160301/93394809/attachment.html>