Simon Pilgrim via llvm-dev
2019-Jul-02 08:31 UTC
[llvm-dev] Tablegen ridiculously slow when compiling for Debug
Much of this has been discussed (over many, many years) on https://bugs.llvm.org/show_bug.cgi?id=28222 Some of the issues that were identified included: 1 - Poor tablegen dependency handling leading to unexpected rebuilds. 2 - Debug STL iterator checks taking an insane amount of time (might be MSVC specific). 3 - Lack of parallelization of custom commands (XCode and VS builds) - VS at least has a recent (VS2017+?) 'build custom tools in parallel' option that can be enabled per project file - we should investigate setting that automatically. 4 - A lot of O(N^2), or worse, code that has built up over the years. 5 - Poor STL type selection resulting it excessive iteration/access times. Running a profiler every so often helps find some quick improvements but its not really fixing these core problems. Simon. On 01/07/2019 21:05, Chris Bieneman via llvm-dev wrote:> In CMake 3.7 and later the Ninja generator can handle depfiles which > gives us correct and accurate dependencies for tablegen, and we do use > that support if it is available. > > I'm surprised CMake has never extended that support to the Makefile > generator, but unsurprised it isn't supported in the IDE generators. > I'm reasonably confident that you can't add that support to Xcode > without treating tablegen as an extra compiler, which (I believe) > requires an Xcode plugin. Even if that isn't the case the Xcode build > system's extensibility is largely undocumented and I'm sure it would > be very challenging to extend it in this way. > > I think it would be possible to add that support to CMake's MSBuild > generator for Visual Studio, but I'm not sure why you would. It seems > like Microsoft's preferred approach to using CMake with Visual Studio > is with ninja as the build tool via the CMake Server integration. > > -Chris > >> On Jul 1, 2019, at 2:57 PM, David Blaikie <dblaikie at gmail.com >> <mailto:dblaikie at gmail.com>> wrote: >> >> If someone can manage it, it wouldn't be a bad thing - obviously open >> up more parallelism (I don't know how much of LLVM can be built >> before you hit everything that needs tblgen run - I guess libSupport >> and some other bits) >> >> On Mon, Jul 1, 2019 at 12:54 PM Zakharin, Vyacheslav P via llvm-dev >> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> [resending to the whole list] >> >> I wonder if we can stop rebuilding TD files unconditionally, i.e. >> generate dependencies for TD files based on include directives >> and just allow the build system do its job? Would that solve >> most of the build time issues? >> >> Thanks, >> >> Slava >> >> *From:*llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org >> <mailto:llvm-dev-bounces at lists.llvm.org>] *On Behalf Of *Chris >> Bieneman via llvm-dev >> *Sent:* Monday, July 1, 2019 10:35 AM >> *To:* Joan Lluch <joan.lluch at icloud.com >> <mailto:joan.lluch at icloud.com>> >> *Cc:* llvm-dev <llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org>> >> *Subject:* Re: [llvm-dev] Tablegen ridiculously slow when >> compiling for Debug >> >> Hey Joan, >> >> When looking for build support it is really useful to include a >> bunch of information about your build up front. Knowing that you >> are on macOS, and using the Xcode generator are really useful. >> >> On macOS, BUILD_SHARED_LIBS won’t really help much because the >> default linker (ld64) is pretty good. Using an IDE generator and >> setting LLVM_USE_OPTIMIZED_TABLEGEN will kill your release builds. >> >> In general Xcode takes 2x-3x longer than Ninja for incremental >> builds, and 1.5x-2x as long for clean builds. Lots of people use >> the Xcode generator to create a project file for navigation and >> editing, but most people I know doing LLVM development on macOS >> use Ninja for their builds. >> >> -Chris >> >> >> On Jun 30, 2019, at 12:18 PM, Joan Lluch via llvm-dev >> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> Hi Greg, >> >> I tried to setup Ninja before on my mac but I mush have done >> something wrong and I didn’t manage to get it work. I’m not >> familiarised at all with the procedures involved. I may try >> that again to see If I have some luck though. It’s a pity >> that LLVM is not particularly friendly with familiar IDEs >> such as xCode on macs and Visual Studio on windows. >> >> John >> >> On 30 Jun 2019, at 17:43, Greg Bedwell >> <gregbedwell at gmail.com <mailto:gregbedwell at gmail.com>> wrote: >> >> This is also the case with the Visual Studio generators. >> Custom commands in a single cmake file essentially get >> written out line by line into a single batch file that >> gets processed as a custom build step. In the VS case >> this means that it can, for example, run X86 and Aarch64 >> tablegen steps in parallel with each other but all of the >> individual X86 invocations get processed serially. I can >> well imagine that the Xcode situation is similar although >> I've no experience with it myself to know for sure. >> >> As previously mentioned, the best solution is probably to >> try to adjust your workflow to use the Ninja ( >> >> https://ninja-build.org <https://ninja-build.org/> >> >> ) CMake generator if at all possible. It's a bit of an >> adjustment but it does work very nicely with the LLVM >> build system. >> >> -Greg >> >> On Sun, 30 Jun 2019 at 12:08, Nicolai Hähnle via llvm-dev >> <llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> Are you saying that the TableGen execution isn't >> parallelized? That >> seems like an obvious Xcode-specific problem... >> >> The TableGen executions are parallelized with >> cmake/ninja just fine. >> >> Cheers, >> Nicolai >> >> On 30.06.19 11:28, Joan Lluch via llvm-dev wrote: >> > Hi Praveen, >> > >> > Thanks for the tip, but Xcode seems to spend all >> the time running >> > tablegen "custom shell scripts", one by one at a >> time, not linking. >> > Linking is actually very fast, possibly less than a >> second. The >> > “scripts” that take longer are >> “AArch64CommonTableGen" and >> > “AMDGPUCommonTableGen”. As said this is on LLVM 9.0. >> > >> > However, on LLVM 7.0.1, the same process takes just >> 5-6 seconds in >> > total, with individual “scripts” taking >> significantly less than 1 second >> > each. There must be some difference between LLVM >> 9.0 and LLVM 7.0 that >> > might cause this (?) >> > >> > John >> > >> > >> >> On 30 Jun 2019, at 11:17, Praveen Velliengiri >> >> <praveenvelliengiri at gmail.com >> <mailto:praveenvelliengiri at gmail.com> >> <mailto:praveenvelliengiri at gmail.com >> <mailto:praveenvelliengiri at gmail.com>>> >> >> wrote: >> >> >> >> * >> >> * >> >> cmake *BUILD_SHARED_LIBS* option, it builds llvm >> as .so not as .a. It >> >> will use less memory during linking so you can >> increase the link >> >> threads and your build time will be lesser. >> >> Check this in : https://llvm.org/docs/CMake.html >> >> >> >> ** >> >> ** >> >> >> >> On Sun, 30 Jun 2019 at 14:42, Joan Lluch >> <joan.lluch at icloud.com <mailto:joan.lluch at icloud.com> >> >> <mailto:joan.lluch at icloud.com >> <mailto:joan.lluch at icloud.com>>> wrote: >> >> >> >> Hi Praveen, >> >> >> >> Please, can you elaborate on this?. What do do >> mean by “building >> >> as shared objects”. >> >> >> >> Thanks, >> >> >> >> John >> >> >> >> >> >> >> >>> On 30 Jun 2019, at 07:32, Praveen Velliengiri >> >>> <praveenvelliengiri at gmail.com >> <mailto:praveenvelliengiri at gmail.com> >> >>> <mailto:praveenvelliengiri at gmail.com >> <mailto:praveenvelliengiri at gmail.com>>> wrote: >> >>> >> >>> Maybe try building llvm as a shared objects.. >> >>> >> >>> On Jun 30, 2019 1:30 AM, "Joan Lluch via >> llvm-dev" >> >>> <llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org> >> <mailto:llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org>>> wrote: >> >>> >> >>> Hi Florian, >> >>> >> >>> Ok, I ran this: >> >>> >> >>> cmake -S LLVM -DCMAKE_INSTALL_PREFIX=INSTALL >> >>> -DLLVM_OPTIMIZED_TABLEGEN=On -G Xcode >> >>> >> >>> Compiled it again from clean, and the >> situation is worse than >> >>> before. Incremental builds take an >> incredible amount of time >> >>> stuck in running Tablegen scripts for all >> targets. Now this >> >>> happens both in Release and Debug >> configurations. Just before >> >>> this, at least Release compiled fine, but >> that’s no longer >> >>> the case. >> >>> >> >>> Any other suggestions? What could >> actually cause this? >> >>> >> >>> Thanks >> >>> John >> >>> >> >>> >> >>> >> >>>> On 29 Jun 2019, at 19:37, Florian Hahn >> >>>> <florian_hahn at apple.com >> <mailto:florian_hahn at apple.com> >> <mailto:florian_hahn at apple.com >> <mailto:florian_hahn at apple.com>>> wrote: >> >>>> >> >>>> Hi, >> >>>> >> >>>>> On Jun 29, 2019, at 18:26, Joan Lluch via llvm-dev >> >>>>> <llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org> >> <mailto:llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org>>> >> >>>>> wrote: >> >>>>> >> >>>>> Hi all, >> >>>>> >> >>>>> On LLVM version 7.0.1, incremental builds are >> very fast for >> >>>>> both Release and Debug. I’m compiling with Xcode >> >>>>> >> >>>>> I recently downloaded LLVM 9.0 from the >> LLVM-mirror Github >> >>>>> repository and found that Incremental "Debug” >> builds take a >> >>>>> ridiculously long time due to Tablegen taking ages >> >>>>> (literally more than 10 minutes) to generate >> files. This >> >>>>> makes it totally unusable for debug purposes. >> However, >> >>>>> incremental ‘Release’ builds only take a few >> seconds. >> >>>>> >> >>>>> Why is that?. Any suggestions?. >> >>>> >> >>>> >> >>>> >> >>>> You could give setting >> LLVM_OPTIMIZED_TABLEGEN a try >> >>>> (https://llvm.org/docs/CMake.html). >> >>>> >> >>>> Cheers, >> >>>> Florian >> >>> >> >>> >> >>> _______________________________________________ >> >>> LLVM Developers mailing list >> >>> llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org> >> <mailto:llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org>> >> >>> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >>> >> >> >> > >> > >> > _______________________________________________ >> > LLVM Developers mailing list >> > llvm-dev at lists.llvm.org >> <mailto:llvm-dev at lists.llvm.org> >> > >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > >> >> -- >> Lerne, wie die Welt wirklich ist, >> Aber vergiss niemals, wie sie sein sollte. >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190702/4a09cab8/attachment.html>
Zachary Turner via llvm-dev
2019-Jul-02 15:36 UTC
[llvm-dev] Tablegen ridiculously slow when compiling for Debug
I think the O(N^2) behavior is probably the worst culprit, but I don’t really know anything about Tablegen to fix it. And I think that speaks to another problem: not many people do. It would be great if someone knowledgeable about Tablegen could look at it with the specific aim of improving the algorithmic complexity On Tue, Jul 2, 2019 at 1:31 AM Simon Pilgrim via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Much of this has been discussed (over many, many years) on > https://bugs.llvm.org/show_bug.cgi?id=28222 > > Some of the issues that were identified included: > > 1 - Poor tablegen dependency handling leading to unexpected rebuilds. > > 2 - Debug STL iterator checks taking an insane amount of time (might be > MSVC specific). > > 3 - Lack of parallelization of custom commands (XCode and VS builds) - VS > at least has a recent (VS2017+?) 'build custom tools in parallel' option > that can be enabled per project file - we should investigate setting that > automatically. > > 4 - A lot of O(N^2), or worse, code that has built up over the years. > > 5 - Poor STL type selection resulting it excessive iteration/access times. > > Running a profiler every so often helps find some quick improvements but > its not really fixing these core problems. > > > Simon. > > On 01/07/2019 21:05, Chris Bieneman via llvm-dev wrote: > > In CMake 3.7 and later the Ninja generator can handle depfiles which gives > us correct and accurate dependencies for tablegen, and we do use that > support if it is available. > > I'm surprised CMake has never extended that support to the Makefile > generator, but unsurprised it isn't supported in the IDE generators. I'm > reasonably confident that you can't add that support to Xcode without > treating tablegen as an extra compiler, which (I believe) requires an Xcode > plugin. Even if that isn't the case the Xcode build system's extensibility > is largely undocumented and I'm sure it would be very challenging to extend > it in this way. > > I think it would be possible to add that support to CMake's MSBuild > generator for Visual Studio, but I'm not sure why you would. It seems like > Microsoft's preferred approach to using CMake with Visual Studio is with > ninja as the build tool via the CMake Server integration. > > -Chris > > On Jul 1, 2019, at 2:57 PM, David Blaikie <dblaikie at gmail.com> wrote: > > If someone can manage it, it wouldn't be a bad thing - obviously open up > more parallelism (I don't know how much of LLVM can be built before you hit > everything that needs tblgen run - I guess libSupport and some other bits) > > On Mon, Jul 1, 2019 at 12:54 PM Zakharin, Vyacheslav P via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> [resending to the whole list] >> >> >> >> I wonder if we can stop rebuilding TD files unconditionally, i.e. >> generate dependencies for TD files based on include directives and just >> allow the build system do its job? Would that solve most of the build time >> issues? >> >> >> >> Thanks, >> >> Slava >> >> >> >> >> >> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of *Chris >> Bieneman via llvm-dev >> *Sent:* Monday, July 1, 2019 10:35 AM >> *To:* Joan Lluch <joan.lluch at icloud.com> >> *Cc:* llvm-dev <llvm-dev at lists.llvm.org> >> *Subject:* Re: [llvm-dev] Tablegen ridiculously slow when compiling for >> Debug >> >> >> >> Hey Joan, >> >> >> >> When looking for build support it is really useful to include a bunch of >> information about your build up front. Knowing that you are on macOS, and >> using the Xcode generator are really useful. >> >> >> >> On macOS, BUILD_SHARED_LIBS won’t really help much because the default >> linker (ld64) is pretty good. Using an IDE generator and setting >> LLVM_USE_OPTIMIZED_TABLEGEN will kill your release builds. >> >> >> >> In general Xcode takes 2x-3x longer than Ninja for incremental builds, >> and 1.5x-2x as long for clean builds. Lots of people use the Xcode >> generator to create a project file for navigation and editing, but most >> people I know doing LLVM development on macOS use Ninja for their builds. >> >> >> >> -Chris >> >> >> On Jun 30, 2019, at 12:18 PM, Joan Lluch via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >> Hi Greg, >> >> >> >> I tried to setup Ninja before on my mac but I mush have done something >> wrong and I didn’t manage to get it work. I’m not familiarised at all with >> the procedures involved. I may try that again to see If I have some luck >> though. It’s a pity that LLVM is not particularly friendly with familiar >> IDEs such as xCode on macs and Visual Studio on windows. >> >> >> >> John >> >> >> >> On 30 Jun 2019, at 17:43, Greg Bedwell <gregbedwell at gmail.com> wrote: >> >> >> >> This is also the case with the Visual Studio generators. Custom commands >> in a single cmake file essentially get written out line by line into a >> single batch file that gets processed as a custom build step. In the VS >> case this means that it can, for example, run X86 and Aarch64 tablegen >> steps in parallel with each other but all of the individual X86 invocations >> get processed serially. I can well imagine that the Xcode situation is >> similar although I've no experience with it myself to know for sure. >> >> >> >> As previously mentioned, the best solution is probably to try to adjust >> your workflow to use the Ninja ( >> >> https://ninja-build.org >> >> ) CMake generator if at all possible. It's a bit of an adjustment but >> it does work very nicely with the LLVM build system. >> >> >> >> -Greg >> >> >> >> >> >> On Sun, 30 Jun 2019 at 12:08, Nicolai Hähnle via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >> Are you saying that the TableGen execution isn't parallelized? That >> seems like an obvious Xcode-specific problem... >> >> The TableGen executions are parallelized with cmake/ninja just fine. >> >> Cheers, >> Nicolai >> >> On 30.06.19 11:28, Joan Lluch via llvm-dev wrote: >> > Hi Praveen, >> > >> > Thanks for the tip, but Xcode seems to spend all the time running >> > tablegen "custom shell scripts", one by one at a time, not linking. >> > Linking is actually very fast, possibly less than a second. The >> > “scripts” that take longer are “AArch64CommonTableGen" and >> > “AMDGPUCommonTableGen”. As said this is on LLVM 9.0. >> > >> > However, on LLVM 7.0.1, the same process takes just 5-6 seconds in >> > total, with individual “scripts” taking significantly less than 1 >> second >> > each. There must be some difference between LLVM 9.0 and LLVM 7.0 that >> > might cause this (?) >> > >> > John >> > >> > >> >> On 30 Jun 2019, at 11:17, Praveen Velliengiri >> >> <praveenvelliengiri at gmail.com <mailto:praveenvelliengiri at gmail.com>> >> >> wrote: >> >> >> >> * >> >> * >> >> cmake *BUILD_SHARED_LIBS* option, it builds llvm as .so not as .a. It >> >> will use less memory during linking so you can increase the link >> >> threads and your build time will be lesser. >> >> Check this in : https://llvm.org/docs/CMake.html >> >> >> >> ** >> >> ** >> >> >> >> On Sun, 30 Jun 2019 at 14:42, Joan Lluch <joan.lluch at icloud.com >> >> <mailto:joan.lluch at icloud.com>> wrote: >> >> >> >> Hi Praveen, >> >> >> >> Please, can you elaborate on this?. What do do mean by “building >> >> as shared objects”. >> >> >> >> Thanks, >> >> >> >> John >> >> >> >> >> >> >> >>> On 30 Jun 2019, at 07:32, Praveen Velliengiri >> >>> <praveenvelliengiri at gmail.com >> >>> <mailto:praveenvelliengiri at gmail.com>> wrote: >> >>> >> >>> Maybe try building llvm as a shared objects.. >> >>> >> >>> On Jun 30, 2019 1:30 AM, "Joan Lluch via llvm-dev" >> >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >>> >> >>> Hi Florian, >> >>> >> >>> Ok, I ran this: >> >>> >> >>> cmake -S LLVM -DCMAKE_INSTALL_PREFIX=INSTALL >> >>> -DLLVM_OPTIMIZED_TABLEGEN=On -G Xcode >> >>> >> >>> Compiled it again from clean, and the situation is worse than >> >>> before. Incremental builds take an incredible amount of time >> >>> stuck in running Tablegen scripts for all targets. Now this >> >>> happens both in Release and Debug configurations. Just before >> >>> this, at least Release compiled fine, but that’s no longer >> >>> the case. >> >>> >> >>> Any other suggestions? What could actually cause this? >> >>> >> >>> Thanks >> >>> John >> >>> >> >>> >> >>> >> >>>> On 29 Jun 2019, at 19:37, Florian Hahn >> >>>> <florian_hahn at apple.com <mailto:florian_hahn at apple.com>> >> wrote: >> >>>> >> >>>> Hi, >> >>>> >> >>>>> On Jun 29, 2019, at 18:26, Joan Lluch via llvm-dev >> >>>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> >> >>>>> wrote: >> >>>>> >> >>>>> Hi all, >> >>>>> >> >>>>> On LLVM version 7.0.1, incremental builds are very fast for >> >>>>> both Release and Debug. I’m compiling with Xcode >> >>>>> >> >>>>> I recently downloaded LLVM 9.0 from the LLVM-mirror Github >> >>>>> repository and found that Incremental "Debug” builds take a >> >>>>> ridiculously long time due to Tablegen taking ages >> >>>>> (literally more than 10 minutes) to generate files. This >> >>>>> makes it totally unusable for debug purposes. However, >> >>>>> incremental ‘Release’ builds only take a few seconds. >> >>>>> >> >>>>> Why is that?. Any suggestions?. >> >>>> >> >>>> >> >>>> >> >>>> You could give setting LLVM_OPTIMIZED_TABLEGEN a try >> >>>> (https://llvm.org/docs/CMake.html). >> >>>> >> >>>> Cheers, >> >>>> Florian >> >>> >> >>> >> >>> _______________________________________________ >> >>> LLVM Developers mailing list >> >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >>> >> >> >> > >> > >> > _______________________________________________ >> > LLVM Developers mailing list >> > llvm-dev at lists.llvm.org >> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > >> >> -- >> Lerne, wie die Welt wirklich ist, >> Aber vergiss niemals, wie sie sein sollte. >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > > > _______________________________________________ > LLVM Developers mailing listllvm-dev at lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190702/b81d4658/attachment.html>
Chris Bieneman via llvm-dev
2019-Jul-02 15:46 UTC
[llvm-dev] Tablegen ridiculously slow when compiling for Debug
There are other issues that come in WRT CMake + Xcode. We don't see a lot of people using that configuration, so we don't often discuss them. It isn't that Xcode doesn't parallelize custom commands, it doesn't parallelize targets unless you set a setting on the scheme. CMake has limited support for generating schemes which was only added in CMake 3.9. I had briefly looked at adding scheme support to our build, but I didn't because schemes don't really match super well to LLVM development workflows, so it would really need to be added by an Xcode user wanting to define a development flow for Xcode. Additionally the Xcode project format doesn't support adding custom tools in a declarative format. There are two routes that work either you embed them as shell scripts, or you wrap them in Makefiles and create Makefile build targets (CMake does the later). In either case Xcode is very limited in its ability to track dependencies in and out of shell and Makefile tasks, which frequently results in "throw hands in the air and rerun everything". CMake generating Makefile targets and not having historical support for Xcode schemes combined to make the Xcode build of LLVM much slower than it should be, and there isn't really a good solution to the problem that the LLVM or CMake communities can drive. -Chris> On Jul 2, 2019, at 3:31 AM, Simon Pilgrim via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Much of this has been discussed (over many, many years) on https://bugs.llvm.org/show_bug.cgi?id=28222 <https://bugs.llvm.org/show_bug.cgi?id=28222> > Some of the issues that were identified included: > > 1 - Poor tablegen dependency handling leading to unexpected rebuilds. > > 2 - Debug STL iterator checks taking an insane amount of time (might be MSVC specific). > > 3 - Lack of parallelization of custom commands (XCode and VS builds) - VS at least has a recent (VS2017+?) 'build custom tools in parallel' option that can be enabled per project file - we should investigate setting that automatically. > > 4 - A lot of O(N^2), or worse, code that has built up over the years. > > 5 - Poor STL type selection resulting it excessive iteration/access times. > > Running a profiler every so often helps find some quick improvements but its not really fixing these core problems. > > Simon. > > On 01/07/2019 21:05, Chris Bieneman via llvm-dev wrote: > >> In CMake 3.7 and later the Ninja generator can handle depfiles which gives us correct and accurate dependencies for tablegen, and we do use that support if it is available. >> >> I'm surprised CMake has never extended that support to the Makefile generator, but unsurprised it isn't supported in the IDE generators. I'm reasonably confident that you can't add that support to Xcode without treating tablegen as an extra compiler, which (I believe) requires an Xcode plugin. Even if that isn't the case the Xcode build system's extensibility is largely undocumented and I'm sure it would be very challenging to extend it in this way. >> >> I think it would be possible to add that support to CMake's MSBuild generator for Visual Studio, but I'm not sure why you would. It seems like Microsoft's preferred approach to using CMake with Visual Studio is with ninja as the build tool via the CMake Server integration. >> >> -Chris >> >>> On Jul 1, 2019, at 2:57 PM, David Blaikie <dblaikie at gmail.com <mailto:dblaikie at gmail.com>> wrote: >>> >>> If someone can manage it, it wouldn't be a bad thing - obviously open up more parallelism (I don't know how much of LLVM can be built before you hit everything that needs tblgen run - I guess libSupport and some other bits) >>> >>> On Mon, Jul 1, 2019 at 12:54 PM Zakharin, Vyacheslav P via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> [resending to the whole list] >>> >>> >>> I wonder if we can stop rebuilding TD files unconditionally, i.e. generate dependencies for TD files based on include directives and just allow the build system do its job? Would that solve most of the build time issues? >>> >>> >>> Thanks, >>> >>> Slava >>> >>> >>> <> >>> <>From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org <mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of Chris Bieneman via llvm-dev >>> Sent: Monday, July 1, 2019 10:35 AM >>> To: Joan Lluch <joan.lluch at icloud.com <mailto:joan.lluch at icloud.com>> >>> Cc: llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> >>> Subject: Re: [llvm-dev] Tablegen ridiculously slow when compiling for Debug >>> >>> >>> Hey Joan, >>> >>> >>> When looking for build support it is really useful to include a bunch of information about your build up front. Knowing that you are on macOS, and using the Xcode generator are really useful. >>> >>> >>> On macOS, BUILD_SHARED_LIBS won’t really help much because the default linker (ld64) is pretty good. Using an IDE generator and setting LLVM_USE_OPTIMIZED_TABLEGEN will kill your release builds. >>> >>> >>> In general Xcode takes 2x-3x longer than Ninja for incremental builds, and 1.5x-2x as long for clean builds. Lots of people use the Xcode generator to create a project file for navigation and editing, but most people I know doing LLVM development on macOS use Ninja for their builds. >>> >>> >>> -Chris >>> >>> >>> On Jun 30, 2019, at 12:18 PM, Joan Lluch via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Hi Greg, >>> >>> >>> I tried to setup Ninja before on my mac but I mush have done something wrong and I didn’t manage to get it work. I’m not familiarised at all with the procedures involved. I may try that again to see If I have some luck though. It’s a pity that LLVM is not particularly friendly with familiar IDEs such as xCode on macs and Visual Studio on windows. >>> >>> >>> John >>> >>> >>> On 30 Jun 2019, at 17:43, Greg Bedwell <gregbedwell at gmail.com <mailto:gregbedwell at gmail.com>> wrote: >>> >>> >>> This is also the case with the Visual Studio generators. Custom commands in a single cmake file essentially get written out line by line into a single batch file that gets processed as a custom build step. In the VS case this means that it can, for example, run X86 and Aarch64 tablegen steps in parallel with each other but all of the individual X86 invocations get processed serially. I can well imagine that the Xcode situation is similar although I've no experience with it myself to know for sure. >>> >>> >>> As previously mentioned, the best solution is probably to try to adjust your workflow to use the Ninja ( >>> >>> https://ninja-build.org <https://ninja-build.org/> >>> ) CMake generator if at all possible. It's a bit of an adjustment but it does work very nicely with the LLVM build system. >>> >>> >>> -Greg >>> >>> >>> >>> On Sun, 30 Jun 2019 at 12:08, Nicolai Hähnle via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Are you saying that the TableGen execution isn't parallelized? That >>> seems like an obvious Xcode-specific problem... >>> >>> The TableGen executions are parallelized with cmake/ninja just fine. >>> >>> Cheers, >>> Nicolai >>> >>> On 30.06.19 11:28, Joan Lluch via llvm-dev wrote: >>> > Hi Praveen, >>> > >>> > Thanks for the tip, but Xcode seems to spend all the time running >>> > tablegen "custom shell scripts", one by one at a time, not linking. >>> > Linking is actually very fast, possibly less than a second. The >>> > “scripts” that take longer are “AArch64CommonTableGen" and >>> > “AMDGPUCommonTableGen”. As said this is on LLVM 9.0. >>> > >>> > However, on LLVM 7.0.1, the same process takes just 5-6 seconds in >>> > total, with individual “scripts” taking significantly less than 1 second >>> > each. There must be some difference between LLVM 9.0 and LLVM 7.0 that >>> > might cause this (?) >>> > >>> > John >>> > >>> > >>> >> On 30 Jun 2019, at 11:17, Praveen Velliengiri >>> >> <praveenvelliengiri at gmail.com <mailto:praveenvelliengiri at gmail.com> <mailto:praveenvelliengiri at gmail.com <mailto:praveenvelliengiri at gmail.com>>> >>> >> wrote: >>> >> >>> >> * >>> >> * >>> >> cmake *BUILD_SHARED_LIBS* option, it builds llvm as .so not as .a. It >>> >> will use less memory during linking so you can increase the link >>> >> threads and your build time will be lesser. >>> >> Check this in : https://llvm.org/docs/CMake.html <https://llvm.org/docs/CMake.html> >>> >> >>> >> ** >>> >> ** >>> >> >>> >> On Sun, 30 Jun 2019 at 14:42, Joan Lluch <joan.lluch at icloud.com <mailto:joan.lluch at icloud.com> >>> >> <mailto:joan.lluch at icloud.com <mailto:joan.lluch at icloud.com>>> wrote: >>> >> >>> >> Hi Praveen, >>> >> >>> >> Please, can you elaborate on this?. What do do mean by “building >>> >> as shared objects”. >>> >> >>> >> Thanks, >>> >> >>> >> John >>> >> >>> >> >>> >> >>> >>> On 30 Jun 2019, at 07:32, Praveen Velliengiri >>> >>> <praveenvelliengiri at gmail.com <mailto:praveenvelliengiri at gmail.com> >>> >>> <mailto:praveenvelliengiri at gmail.com <mailto:praveenvelliengiri at gmail.com>>> wrote: >>> >>> >>> >>> Maybe try building llvm as a shared objects.. >>> >>> >>> >>> On Jun 30, 2019 1:30 AM, "Joan Lluch via llvm-dev" >>> >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>> wrote: >>> >>> >>> >>> Hi Florian, >>> >>> >>> >>> Ok, I ran this: >>> >>> >>> >>> cmake -S LLVM -DCMAKE_INSTALL_PREFIX=INSTALL >>> >>> -DLLVM_OPTIMIZED_TABLEGEN=On -G Xcode >>> >>> >>> >>> Compiled it again from clean, and the situation is worse than >>> >>> before. Incremental builds take an incredible amount of time >>> >>> stuck in running Tablegen scripts for all targets. Now this >>> >>> happens both in Release and Debug configurations. Just before >>> >>> this, at least Release compiled fine, but that’s no longer >>> >>> the case. >>> >>> >>> >>> Any other suggestions? What could actually cause this? >>> >>> >>> >>> Thanks >>> >>> John >>> >>> >>> >>> >>> >>> >>> >>>> On 29 Jun 2019, at 19:37, Florian Hahn >>> >>>> <florian_hahn at apple.com <mailto:florian_hahn at apple.com> <mailto:florian_hahn at apple.com <mailto:florian_hahn at apple.com>>> wrote: >>> >>>> >>> >>>> Hi, >>> >>>> >>> >>>>> On Jun 29, 2019, at 18:26, Joan Lluch via llvm-dev >>> >>>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>> >>> >>>>> wrote: >>> >>>>> >>> >>>>> Hi all, >>> >>>>> >>> >>>>> On LLVM version 7.0.1, incremental builds are very fast for >>> >>>>> both Release and Debug. I’m compiling with Xcode >>> >>>>> >>> >>>>> I recently downloaded LLVM 9.0 from the LLVM-mirror Github >>> >>>>> repository and found that Incremental "Debug” builds take a >>> >>>>> ridiculously long time due to Tablegen taking ages >>> >>>>> (literally more than 10 minutes) to generate files. This >>> >>>>> makes it totally unusable for debug purposes. However, >>> >>>>> incremental ‘Release’ builds only take a few seconds. >>> >>>>> >>> >>>>> Why is that?. Any suggestions?. >>> >>>> >>> >>>> >>> >>>> >>> >>>> You could give setting LLVM_OPTIMIZED_TABLEGEN a try >>> >>>> (https://llvm.org/docs/CMake.html <https://llvm.org/docs/CMake.html>). >>> >>>> >>> >>>> Cheers, >>> >>>> Florian >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> LLVM Developers mailing list >>> >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> >>> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> >>> >>> >>> >> >>> > >>> > >>> > _______________________________________________ >>> > LLVM Developers mailing list >>> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> >>> > >>> >>> -- >>> Lerne, wie die Welt wirklich ist, >>> Aber vergiss niemals, wie sie sein sollte. >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>_______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> >> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190702/08234842/attachment-0001.html>
Nicolai Hähnle via llvm-dev
2019-Jul-03 00:37 UTC
[llvm-dev] Tablegen ridiculously slow when compiling for Debug
On 02.07.19 10:31, Simon Pilgrim via llvm-dev wrote:> Much of this has been discussed (over many, many years) on > https://bugs.llvm.org/show_bug.cgi?id=28222 > > Some of the issues that were identified included: > > 1 - Poor tablegen dependency handling leading to unexpected rebuilds. > > 2 - Debug STL iterator checks taking an insane amount of time (might be > MSVC specific). > > 3 - Lack of parallelization of custom commands (XCode and VS builds) - > VS at least has a recent (VS2017+?) 'build custom tools in parallel' > option that can be enabled per project file - we should investigate > setting that automatically. > > 4 - A lot of O(N^2), or worse, code that has built up over the years. > > 5 - Poor STL type selection resulting it excessive iteration/access times.Let me add: 6 - The TableGen frontend is re-run for every TableGen backend invocation. It might help to invoke the TableGen executable only once for each LLVM backend, having the TableGen frontend parse everything and instantiate and resolve records only once, and then run all backends in the same process. This requires re-architecting how the backends are driven, and it does have the disadvantage that the backends can no longer run in parallel -- unless we take a careful look at making that possible -- so it's not actually an easy fix. Cheers, Nicolai> Running a profiler every so often helps find some quick improvements but > its not really fixing these core problems. > > Simon. > > On 01/07/2019 21:05, Chris Bieneman via llvm-dev wrote: > >> In CMake 3.7 and later the Ninja generator can handle depfiles which >> gives us correct and accurate dependencies for tablegen, and we do use >> that support if it is available. >> >> I'm surprised CMake has never extended that support to the Makefile >> generator, but unsurprised it isn't supported in the IDE generators. >> I'm reasonably confident that you can't add that support to Xcode >> without treating tablegen as an extra compiler, which (I believe) >> requires an Xcode plugin. Even if that isn't the case the Xcode build >> system's extensibility is largely undocumented and I'm sure it would >> be very challenging to extend it in this way. >> >> I think it would be possible to add that support to CMake's MSBuild >> generator for Visual Studio, but I'm not sure why you would. It seems >> like Microsoft's preferred approach to using CMake with Visual Studio >> is with ninja as the build tool via the CMake Server integration. >> >> -Chris >> >>> On Jul 1, 2019, at 2:57 PM, David Blaikie <dblaikie at gmail.com >>> <mailto:dblaikie at gmail.com>> wrote: >>> >>> If someone can manage it, it wouldn't be a bad thing - obviously open >>> up more parallelism (I don't know how much of LLVM can be built >>> before you hit everything that needs tblgen run - I guess libSupport >>> and some other bits) >>> >>> On Mon, Jul 1, 2019 at 12:54 PM Zakharin, Vyacheslav P via llvm-dev >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> [resending to the whole list] >>> >>> I wonder if we can stop rebuilding TD files unconditionally, i.e. >>> generate dependencies for TD files based on include directives >>> and just allow the build system do its job? Would that solve >>> most of the build time issues? >>> >>> Thanks, >>> >>> Slava >>> >>> *From:*llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org >>> <mailto:llvm-dev-bounces at lists.llvm.org>] *On Behalf Of *Chris >>> Bieneman via llvm-dev >>> *Sent:* Monday, July 1, 2019 10:35 AM >>> *To:* Joan Lluch <joan.lluch at icloud.com >>> <mailto:joan.lluch at icloud.com>> >>> *Cc:* llvm-dev <llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>> >>> *Subject:* Re: [llvm-dev] Tablegen ridiculously slow when >>> compiling for Debug >>> >>> Hey Joan, >>> >>> When looking for build support it is really useful to include a >>> bunch of information about your build up front. Knowing that you >>> are on macOS, and using the Xcode generator are really useful. >>> >>> On macOS, BUILD_SHARED_LIBS won’t really help much because the >>> default linker (ld64) is pretty good. Using an IDE generator and >>> setting LLVM_USE_OPTIMIZED_TABLEGEN will kill your release builds. >>> >>> In general Xcode takes 2x-3x longer than Ninja for incremental >>> builds, and 1.5x-2x as long for clean builds. Lots of people use >>> the Xcode generator to create a project file for navigation and >>> editing, but most people I know doing LLVM development on macOS >>> use Ninja for their builds. >>> >>> -Chris >>> >>> >>> On Jun 30, 2019, at 12:18 PM, Joan Lluch via llvm-dev >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Hi Greg, >>> >>> I tried to setup Ninja before on my mac but I mush have done >>> something wrong and I didn’t manage to get it work. I’m not >>> familiarised at all with the procedures involved. I may try >>> that again to see If I have some luck though. It’s a pity >>> that LLVM is not particularly friendly with familiar IDEs >>> such as xCode on macs and Visual Studio on windows. >>> >>> John >>> >>> On 30 Jun 2019, at 17:43, Greg Bedwell >>> <gregbedwell at gmail.com <mailto:gregbedwell at gmail.com>> wrote: >>> >>> This is also the case with the Visual Studio generators. >>> Custom commands in a single cmake file essentially get >>> written out line by line into a single batch file that >>> gets processed as a custom build step. In the VS case >>> this means that it can, for example, run X86 and Aarch64 >>> tablegen steps in parallel with each other but all of the >>> individual X86 invocations get processed serially. I can >>> well imagine that the Xcode situation is similar although >>> I've no experience with it myself to know for sure. >>> >>> As previously mentioned, the best solution is probably to >>> try to adjust your workflow to use the Ninja ( >>> >>> https://ninja-build.org <https://ninja-build.org/> >>> >>> ) CMake generator if at all possible. It's a bit of an >>> adjustment but it does work very nicely with the LLVM >>> build system. >>> >>> -Greg >>> >>> On Sun, 30 Jun 2019 at 12:08, Nicolai Hähnle via llvm-dev >>> <llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Are you saying that the TableGen execution isn't >>> parallelized? That >>> seems like an obvious Xcode-specific problem... >>> >>> The TableGen executions are parallelized with >>> cmake/ninja just fine. >>> >>> Cheers, >>> Nicolai >>> >>> On 30.06.19 11:28, Joan Lluch via llvm-dev wrote: >>> > Hi Praveen, >>> > >>> > Thanks for the tip, but Xcode seems to spend all >>> the time running >>> > tablegen "custom shell scripts", one by one at a >>> time, not linking. >>> > Linking is actually very fast, possibly less than a >>> second. The >>> > “scripts” that take longer are >>> “AArch64CommonTableGen" and >>> > “AMDGPUCommonTableGen”. As said this is on LLVM 9.0. >>> > >>> > However, on LLVM 7.0.1, the same process takes just >>> 5-6 seconds in >>> > total, with individual “scripts” taking >>> significantly less than 1 second >>> > each. There must be some difference between LLVM >>> 9.0 and LLVM 7.0 that >>> > might cause this (?) >>> > >>> > John >>> > >>> > >>> >> On 30 Jun 2019, at 11:17, Praveen Velliengiri >>> >> <praveenvelliengiri at gmail.com >>> <mailto:praveenvelliengiri at gmail.com> >>> <mailto:praveenvelliengiri at gmail.com >>> <mailto:praveenvelliengiri at gmail.com>>> >>> >> wrote: >>> >> >>> >> * >>> >> * >>> >> cmake *BUILD_SHARED_LIBS* option, it builds llvm >>> as .so not as .a. It >>> >> will use less memory during linking so you can >>> increase the link >>> >> threads and your build time will be lesser. >>> >> Check this in : https://llvm.org/docs/CMake.html >>> >> >>> >> ** >>> >> ** >>> >> >>> >> On Sun, 30 Jun 2019 at 14:42, Joan Lluch >>> <joan.lluch at icloud.com <mailto:joan.lluch at icloud.com> >>> >> <mailto:joan.lluch at icloud.com >>> <mailto:joan.lluch at icloud.com>>> wrote: >>> >> >>> >> Hi Praveen, >>> >> >>> >> Please, can you elaborate on this?. What do do >>> mean by “building >>> >> as shared objects”. >>> >> >>> >> Thanks, >>> >> >>> >> John >>> >> >>> >> >>> >> >>> >>> On 30 Jun 2019, at 07:32, Praveen Velliengiri >>> >>> <praveenvelliengiri at gmail.com >>> <mailto:praveenvelliengiri at gmail.com> >>> >>> <mailto:praveenvelliengiri at gmail.com >>> <mailto:praveenvelliengiri at gmail.com>>> wrote: >>> >>> >>> >>> Maybe try building llvm as a shared objects.. >>> >>> >>> >>> On Jun 30, 2019 1:30 AM, "Joan Lluch via >>> llvm-dev" >>> >>> <llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org> >>> <mailto:llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>>> wrote: >>> >>> >>> >>> Hi Florian, >>> >>> >>> >>> Ok, I ran this: >>> >>> >>> >>> cmake -S LLVM -DCMAKE_INSTALL_PREFIX=INSTALL >>> >>> -DLLVM_OPTIMIZED_TABLEGEN=On -G Xcode >>> >>> >>> >>> Compiled it again from clean, and the >>> situation is worse than >>> >>> before. Incremental builds take an >>> incredible amount of time >>> >>> stuck in running Tablegen scripts for all >>> targets. Now this >>> >>> happens both in Release and Debug >>> configurations. Just before >>> >>> this, at least Release compiled fine, but >>> that’s no longer >>> >>> the case. >>> >>> >>> >>> Any other suggestions? What could >>> actually cause this? >>> >>> >>> >>> Thanks >>> >>> John >>> >>> >>> >>> >>> >>> >>> >>>> On 29 Jun 2019, at 19:37, Florian Hahn >>> >>>> <florian_hahn at apple.com >>> <mailto:florian_hahn at apple.com> >>> <mailto:florian_hahn at apple.com >>> <mailto:florian_hahn at apple.com>>> wrote: >>> >>>> >>> >>>> Hi, >>> >>>> >>> >>>>> On Jun 29, 2019, at 18:26, Joan Lluch via llvm-dev >>> >>>>> <llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org> >>> <mailto:llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>>> >>> >>>>> wrote: >>> >>>>> >>> >>>>> Hi all, >>> >>>>> >>> >>>>> On LLVM version 7.0.1, incremental builds are >>> very fast for >>> >>>>> both Release and Debug. I’m compiling with Xcode >>> >>>>> >>> >>>>> I recently downloaded LLVM 9.0 from the >>> LLVM-mirror Github >>> >>>>> repository and found that Incremental "Debug” >>> builds take a >>> >>>>> ridiculously long time due to Tablegen taking ages >>> >>>>> (literally more than 10 minutes) to generate >>> files. This >>> >>>>> makes it totally unusable for debug purposes. >>> However, >>> >>>>> incremental ‘Release’ builds only take a few >>> seconds. >>> >>>>> >>> >>>>> Why is that?. Any suggestions?. >>> >>>> >>> >>>> >>> >>>> >>> >>>> You could give setting >>> LLVM_OPTIMIZED_TABLEGEN a try >>> >>>> (https://llvm.org/docs/CMake.html). >>> >>>> >>> >>>> Cheers, >>> >>>> Florian >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> LLVM Developers mailing list >>> >>> llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org> >>> <mailto:llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>> >>> >>> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> >>> >> >>> > >>> > >>> > _______________________________________________ >>> > LLVM Developers mailing list >>> > llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org> >>> > >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> > >>> >>> -- >>> Lerne, wie die Welt wirklich ist, >>> Aber vergiss niemals, wie sie sein sollte. >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-- Lerne, wie die Welt wirklich ist, Aber vergiss niemals, wie sie sein sollte.
Zakharin, Vyacheslav P via llvm-dev
2019-Jul-09 23:52 UTC
[llvm-dev] Tablegen ridiculously slow when compiling for Debug
FWIW, tablegen does not update timestamps for .inc files, if their contents is unchanged, so if you somehow affect a .td file's timestamp (without changing its contents) it will trigger rebuild of the corresponding .inc file with all subsequent incremental make builds. At the same time, this will not trigger rebuilding any targets dependent on this .inc file, since it is not modified. From time to time, I see this "overhead" in my builds, but I could not easily reproduce conditions causing change of .td file timestamp without actually modifying it. Thanks, Slava -----Original Message----- From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Nicolai Hähnle via llvm-dev Sent: Tuesday, July 2, 2019 5:38 PM To: Simon Pilgrim <llvm-dev at redking.me.uk>; llvm-dev at lists.llvm.org Subject: Re: [llvm-dev] Tablegen ridiculously slow when compiling for Debug On 02.07.19 10:31, Simon Pilgrim via llvm-dev wrote:> Much of this has been discussed (over many, many years) on > https://bugs.llvm.org/show_bug.cgi?id=28222 > > Some of the issues that were identified included: > > 1 - Poor tablegen dependency handling leading to unexpected rebuilds. > > 2 - Debug STL iterator checks taking an insane amount of time (might > be MSVC specific). > > 3 - Lack of parallelization of custom commands (XCode and VS builds) - > VS at least has a recent (VS2017+?) 'build custom tools in parallel' > option that can be enabled per project file - we should investigate > setting that automatically. > > 4 - A lot of O(N^2), or worse, code that has built up over the years. > > 5 - Poor STL type selection resulting it excessive iteration/access times.Let me add: 6 - The TableGen frontend is re-run for every TableGen backend invocation. It might help to invoke the TableGen executable only once for each LLVM backend, having the TableGen frontend parse everything and instantiate and resolve records only once, and then run all backends in the same process. This requires re-architecting how the backends are driven, and it does have the disadvantage that the backends can no longer run in parallel -- unless we take a careful look at making that possible -- so it's not actually an easy fix. Cheers, Nicolai> Running a profiler every so often helps find some quick improvements > but its not really fixing these core problems. > > Simon. > > On 01/07/2019 21:05, Chris Bieneman via llvm-dev wrote: > >> In CMake 3.7 and later the Ninja generator can handle depfiles which >> gives us correct and accurate dependencies for tablegen, and we do >> use that support if it is available. >> >> I'm surprised CMake has never extended that support to the Makefile >> generator, but unsurprised it isn't supported in the IDE generators. >> I'm reasonably confident that you can't add that support to Xcode >> without treating tablegen as an extra compiler, which (I believe) >> requires an Xcode plugin. Even if that isn't the case the Xcode build >> system's extensibility is largely undocumented and I'm sure it would >> be very challenging to extend it in this way. >> >> I think it would be possible to add that support to CMake's MSBuild >> generator for Visual Studio, but I'm not sure why you would. It seems >> like Microsoft's preferred approach to using CMake with Visual Studio >> is with ninja as the build tool via the CMake Server integration. >> >> -Chris >> >>> On Jul 1, 2019, at 2:57 PM, David Blaikie <dblaikie at gmail.com >>> <mailto:dblaikie at gmail.com>> wrote: >>> >>> If someone can manage it, it wouldn't be a bad thing - obviously >>> open up more parallelism (I don't know how much of LLVM can be built >>> before you hit everything that needs tblgen run - I guess libSupport >>> and some other bits) >>> >>> On Mon, Jul 1, 2019 at 12:54 PM Zakharin, Vyacheslav P via llvm-dev >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> [resending to the whole list] >>> >>> I wonder if we can stop rebuilding TD files unconditionally, i.e. >>> generate dependencies for TD files based on include directives >>> and just allow the build system do its job? Would that solve >>> most of the build time issues? >>> >>> Thanks, >>> >>> Slava >>> >>> *From:*llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org >>> <mailto:llvm-dev-bounces at lists.llvm.org>] *On Behalf Of *Chris >>> Bieneman via llvm-dev >>> *Sent:* Monday, July 1, 2019 10:35 AM >>> *To:* Joan Lluch <joan.lluch at icloud.com >>> <mailto:joan.lluch at icloud.com>> >>> *Cc:* llvm-dev <llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>> >>> *Subject:* Re: [llvm-dev] Tablegen ridiculously slow when >>> compiling for Debug >>> >>> Hey Joan, >>> >>> When looking for build support it is really useful to include a >>> bunch of information about your build up front. Knowing that you >>> are on macOS, and using the Xcode generator are really useful. >>> >>> On macOS, BUILD_SHARED_LIBS won’t really help much because the >>> default linker (ld64) is pretty good. Using an IDE generator and >>> setting LLVM_USE_OPTIMIZED_TABLEGEN will kill your release builds. >>> >>> In general Xcode takes 2x-3x longer than Ninja for incremental >>> builds, and 1.5x-2x as long for clean builds. Lots of people use >>> the Xcode generator to create a project file for navigation and >>> editing, but most people I know doing LLVM development on macOS >>> use Ninja for their builds. >>> >>> -Chris >>> >>> >>> On Jun 30, 2019, at 12:18 PM, Joan Lluch via llvm-dev >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Hi Greg, >>> >>> I tried to setup Ninja before on my mac but I mush have done >>> something wrong and I didn’t manage to get it work. I’m not >>> familiarised at all with the procedures involved. I may try >>> that again to see If I have some luck though. It’s a pity >>> that LLVM is not particularly friendly with familiar IDEs >>> such as xCode on macs and Visual Studio on windows. >>> >>> John >>> >>> On 30 Jun 2019, at 17:43, Greg Bedwell >>> <gregbedwell at gmail.com <mailto:gregbedwell at gmail.com>> wrote: >>> >>> This is also the case with the Visual Studio generators. >>> Custom commands in a single cmake file essentially get >>> written out line by line into a single batch file that >>> gets processed as a custom build step. In the VS case >>> this means that it can, for example, run X86 and Aarch64 >>> tablegen steps in parallel with each other but all of the >>> individual X86 invocations get processed serially. I can >>> well imagine that the Xcode situation is similar although >>> I've no experience with it myself to know for sure. >>> >>> As previously mentioned, the best solution is probably to >>> try to adjust your workflow to use the Ninja ( >>> >>> https://ninja-build.org <https://ninja-build.org/> >>> >>> ) CMake generator if at all possible. It's a bit of an >>> adjustment but it does work very nicely with the LLVM >>> build system. >>> >>> -Greg >>> >>> On Sun, 30 Jun 2019 at 12:08, Nicolai Hähnle via llvm-dev >>> <llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Are you saying that the TableGen execution isn't >>> parallelized? That >>> seems like an obvious Xcode-specific problem... >>> >>> The TableGen executions are parallelized with >>> cmake/ninja just fine. >>> >>> Cheers, >>> Nicolai >>> >>> On 30.06.19 11:28, Joan Lluch via llvm-dev wrote: >>> > Hi Praveen, >>> > >>> > Thanks for the tip, but Xcode seems to spend all >>> the time running >>> > tablegen "custom shell scripts", one by one at a >>> time, not linking. >>> > Linking is actually very fast, possibly less than a >>> second. The >>> > “scripts” that take longer are >>> “AArch64CommonTableGen" and >>> > “AMDGPUCommonTableGen”. As said this is on LLVM 9.0. >>> > >>> > However, on LLVM 7.0.1, the same process takes just >>> 5-6 seconds in >>> > total, with individual “scripts” taking >>> significantly less than 1 second >>> > each. There must be some difference between LLVM >>> 9.0 and LLVM 7.0 that >>> > might cause this (?) >>> > >>> > John >>> > >>> > >>> >> On 30 Jun 2019, at 11:17, Praveen Velliengiri >>> >> <praveenvelliengiri at gmail.com >>> <mailto:praveenvelliengiri at gmail.com> >>> <mailto:praveenvelliengiri at gmail.com >>> <mailto:praveenvelliengiri at gmail.com>>> >>> >> wrote: >>> >> >>> >> * >>> >> * >>> >> cmake *BUILD_SHARED_LIBS* option, it builds llvm >>> as .so not as .a. It >>> >> will use less memory during linking so you can >>> increase the link >>> >> threads and your build time will be lesser. >>> >> Check this in : https://llvm.org/docs/CMake.html >>> >> >>> >> ** >>> >> ** >>> >> >>> >> On Sun, 30 Jun 2019 at 14:42, Joan Lluch >>> <joan.lluch at icloud.com <mailto:joan.lluch at icloud.com> >>> >> <mailto:joan.lluch at icloud.com >>> <mailto:joan.lluch at icloud.com>>> wrote: >>> >> >>> >> Hi Praveen, >>> >> >>> >> Please, can you elaborate on this?. What do do >>> mean by “building >>> >> as shared objects”. >>> >> >>> >> Thanks, >>> >> >>> >> John >>> >> >>> >> >>> >> >>> >>> On 30 Jun 2019, at 07:32, Praveen Velliengiri >>> >>> <praveenvelliengiri at gmail.com >>> <mailto:praveenvelliengiri at gmail.com> >>> >>> <mailto:praveenvelliengiri at gmail.com >>> <mailto:praveenvelliengiri at gmail.com>>> wrote: >>> >>> >>> >>> Maybe try building llvm as a shared objects.. >>> >>> >>> >>> On Jun 30, 2019 1:30 AM, "Joan Lluch via >>> llvm-dev" >>> >>> <llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org> >>> <mailto:llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>>> wrote: >>> >>> >>> >>> Hi Florian, >>> >>> >>> >>> Ok, I ran this: >>> >>> >>> >>> cmake -S LLVM -DCMAKE_INSTALL_PREFIX=INSTALL >>> >>> -DLLVM_OPTIMIZED_TABLEGEN=On -G Xcode >>> >>> >>> >>> Compiled it again from clean, and the >>> situation is worse than >>> >>> before. Incremental builds take an >>> incredible amount of time >>> >>> stuck in running Tablegen scripts for all >>> targets. Now this >>> >>> happens both in Release and Debug >>> configurations. Just before >>> >>> this, at least Release compiled fine, but >>> that’s no longer >>> >>> the case. >>> >>> >>> >>> Any other suggestions? What could >>> actually cause this? >>> >>> >>> >>> Thanks >>> >>> John >>> >>> >>> >>> >>> >>> >>> >>>> On 29 Jun 2019, at 19:37, Florian Hahn >>> >>>> <florian_hahn at apple.com >>> <mailto:florian_hahn at apple.com> >>> <mailto:florian_hahn at apple.com >>> <mailto:florian_hahn at apple.com>>> wrote: >>> >>>> >>> >>>> Hi, >>> >>>> >>> >>>>> On Jun 29, 2019, at 18:26, Joan Lluch via llvm-dev >>> >>>>> <llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org> >>> <mailto:llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>>> >>> >>>>> wrote: >>> >>>>> >>> >>>>> Hi all, >>> >>>>> >>> >>>>> On LLVM version 7.0.1, incremental builds are >>> very fast for >>> >>>>> both Release and Debug. I’m compiling with Xcode >>> >>>>> >>> >>>>> I recently downloaded LLVM 9.0 from the >>> LLVM-mirror Github >>> >>>>> repository and found that Incremental "Debug” >>> builds take a >>> >>>>> ridiculously long time due to Tablegen taking ages >>> >>>>> (literally more than 10 minutes) to generate >>> files. This >>> >>>>> makes it totally unusable for debug purposes. >>> However, >>> >>>>> incremental ‘Release’ builds only take a few >>> seconds. >>> >>>>> >>> >>>>> Why is that?. Any suggestions?. >>> >>>> >>> >>>> >>> >>>> >>> >>>> You could give setting >>> LLVM_OPTIMIZED_TABLEGEN a try >>> >>>> (https://llvm.org/docs/CMake.html). >>> >>>> >>> >>>> Cheers, >>> >>>> Florian >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> LLVM Developers mailing list >>> >>> llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org> >>> <mailto:llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org>> >>> >>> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> >>> >> >>> > >>> > >>> > _______________________________________________ >>> > LLVM Developers mailing list >>> > llvm-dev at lists.llvm.org >>> <mailto:llvm-dev at lists.llvm.org> >>> > >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> > >>> >>> -- >>> Lerne, wie die Welt wirklich ist, >>> Aber vergiss niemals, wie sie sein sollte. >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-- Lerne, wie die Welt wirklich ist, Aber vergiss niemals, wie sie sein sollte. _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev