Matthias Braun via llvm-dev
2017-Dec-06 21:17 UTC
[llvm-dev] [cfe-dev] Who wants faster LLVM/Clang builds?
- We do indeed have a lot of unnecessary includes around in llvm (or pretty much any other C++ project for that matter). - I want faster builds. - The only way to reliably fight this is indeed automatic tools. - Having the right amount of includes also has documentation value and ideally let's you understand the structure of your project. - However relying on transitive includes works contrary to the last "undestanding/documentation" point. - (And as stated earlier to have things really clean we want `class XXX;` instead of `#include "XXX.h"` wherever possible. And if you are serious about that we also often have to reduce the amount of include code in the headers so we can move the `#include "XXX.h"` from the header to the implementation. For me personally I think the documentation/understandability we loose when relying on transitive includes weights heavier than my desire to get a faster build... - Matthias> On Dec 6, 2017, at 12:28 PM, serge guelton via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > On Wed, Dec 06, 2017 at 11:38:54AM -0800, Michael Zolotukhin via llvm-dev wrote: >> >>> On Dec 6, 2017, at 9:00 AM, mats petersson via cfe-dev <cfe-dev at lists.llvm.org> wrote: >>> >>> In my experience, a lot of time is spent on optimizing the code (assuming it's not a "-O0" build). >> The numbers were actually for the debug build (-O0 -g), so for Release build they would be different (presumably lower). >>> Also redundant includes are largely fixed by header guards, and I believe Clang [and gcc as well as MS Compilers, and probably most others too] have an include guards-cache that determines that "we've already included foo.h, and it has include guards around the whole actual content of the file, so we can just skip it”. >> By redundant here I meant that we included a file, but we didn’t use any of its content (rather than we included the same file twice). >>> >>> So I'm slightly dubious as to this being an efficient way of significantly reducing the total compilation time for the overall project - even if there are SOME cases where there is a significant improvement in a single file. The total time for a clean build [in wall-clock-time, not CPU-time] should be measured, making sure that there is enough memory. Doing a run of, say, five complete builds of the same thing [with suitable "clean" between to redo the whole build], take away the worst and the best, and perhaps also "modify one of the more common header files" (llvm/IR/Type.h for example) and build again. >> On full builds the benefit is not big (around 1%, but the noise is high), but: 1) if we only take gains more than, say, 5%, we’ll probably never see any, 2) I aim at changes that make the code strictly better (modulo David’s point about disk cache). If any change is questionable from maintenance or whatever other point of view, I’m all for dropping it. > > my 2¢ > > +1 for point 2). Even leaving aside the speed gain, removing unused > includes file just looks like good coding practice to me. > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171206/900e2590/attachment.html>
Michael Zolotukhin via llvm-dev
2017-Dec-06 21:25 UTC
[llvm-dev] [cfe-dev] Who wants faster LLVM/Clang builds?
I agree with you regarding the transitive includes. However, what’ I’m trying is different: I would like to remove headers that are included directly, but not used. Ideally, I’d like not to remove headers that are included transitively. The current patch has such removes because my tool was intended to just point us to files which can be cleaned-up and suggest a way to do it. The original idea was that we then take a look at interesting files and manually remove unnecessary headers after manual inspection. I see though how having such removes in the patch leads the conversation to transitive headers/symbols forward declaration/etc (which I don’t plan to touch at the moment), so I’ll try to improve my tools so that they produce a cleaner patch. Stay tuned! Thanks, Michael> On Dec 6, 2017, at 1:17 PM, Matthias Braun <mbraun at apple.com> wrote: > > - We do indeed have a lot of unnecessary includes around in llvm (or pretty much any other C++ project for that matter). > - I want faster builds. > - The only way to reliably fight this is indeed automatic tools. > - Having the right amount of includes also has documentation value and ideally let's you understand the structure of your project. > - However relying on transitive includes works contrary to the last "undestanding/documentation" point. > - (And as stated earlier to have things really clean we want `class XXX;` instead of `#include "XXX.h"` wherever possible. And if you are serious about that we also often have to reduce the amount of include code in the headers so we can move the `#include "XXX.h"` from the header to the implementation. > > For me personally I think the documentation/understandability we loose when relying on transitive includes weights heavier than my desire to get a faster build... > > - Matthias > >> On Dec 6, 2017, at 12:28 PM, serge guelton via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> On Wed, Dec 06, 2017 at 11:38:54AM -0800, Michael Zolotukhin via llvm-dev wrote: >>> >>>> On Dec 6, 2017, at 9:00 AM, mats petersson via cfe-dev <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote: >>>> >>>> In my experience, a lot of time is spent on optimizing the code (assuming it's not a "-O0" build). >>> The numbers were actually for the debug build (-O0 -g), so for Release build they would be different (presumably lower). >>>> Also redundant includes are largely fixed by header guards, and I believe Clang [and gcc as well as MS Compilers, and probably most others too] have an include guards-cache that determines that "we've already included foo.h, and it has include guards around the whole actual content of the file, so we can just skip it”. >>> By redundant here I meant that we included a file, but we didn’t use any of its content (rather than we included the same file twice). >>>> >>>> So I'm slightly dubious as to this being an efficient way of significantly reducing the total compilation time for the overall project - even if there are SOME cases where there is a significant improvement in a single file. The total time for a clean build [in wall-clock-time, not CPU-time] should be measured, making sure that there is enough memory. Doing a run of, say, five complete builds of the same thing [with suitable "clean" between to redo the whole build], take away the worst and the best, and perhaps also "modify one of the more common header files" (llvm/IR/Type.h for example) and build again. >>> On full builds the benefit is not big (around 1%, but the noise is high), but: 1) if we only take gains more than, say, 5%, we’ll probably never see any, 2) I aim at changes that make the code strictly better (modulo David’s point about disk cache). If any change is questionable from maintenance or whatever other point of view, I’m all for dropping it. >> >> my 2¢ >> >> +1 for point 2). Even leaving aside the speed gain, removing unused >> includes file just looks like good coding practice to me. >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171206/6efb0529/attachment.html>
Quentin Colombet via llvm-dev
2017-Dec-08 17:20 UTC
[llvm-dev] [cfe-dev] Who wants faster LLVM/Clang builds?
> On Dec 6, 2017, at 1:17 PM, Matthias Braun via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > - We do indeed have a lot of unnecessary includes around in llvm (or pretty much any other C++ project for that matter). > - I want faster builds. > - The only way to reliably fight this is indeed automatic tools. > - Having the right amount of includes also has documentation value and ideally let's you understand the structure of your project. > - However relying on transitive includes works contrary to the last "undestanding/documentation" point. > - (And as stated earlier to have things really clean we want `class XXX;` instead of `#include "XXX.h"` wherever possible. And if you are serious about that we also often have to reduce the amount of include code in the headers so we can move the `#include "XXX.h"` from the header to the implementation. > > For me personally I think the documentation/understandability we loose when relying on transitive includes weights heavier than my desire to get a faster build…+1 Q.> > - Matthias > >> On Dec 6, 2017, at 12:28 PM, serge guelton via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> On Wed, Dec 06, 2017 at 11:38:54AM -0800, Michael Zolotukhin via llvm-dev wrote: >>> >>>> On Dec 6, 2017, at 9:00 AM, mats petersson via cfe-dev <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote: >>>> >>>> In my experience, a lot of time is spent on optimizing the code (assuming it's not a "-O0" build). >>> The numbers were actually for the debug build (-O0 -g), so for Release build they would be different (presumably lower). >>>> Also redundant includes are largely fixed by header guards, and I believe Clang [and gcc as well as MS Compilers, and probably most others too] have an include guards-cache that determines that "we've already included foo.h, and it has include guards around the whole actual content of the file, so we can just skip it”. >>> By redundant here I meant that we included a file, but we didn’t use any of its content (rather than we included the same file twice). >>>> >>>> So I'm slightly dubious as to this being an efficient way of significantly reducing the total compilation time for the overall project - even if there are SOME cases where there is a significant improvement in a single file. The total time for a clean build [in wall-clock-time, not CPU-time] should be measured, making sure that there is enough memory. Doing a run of, say, five complete builds of the same thing [with suitable "clean" between to redo the whole build], take away the worst and the best, and perhaps also "modify one of the more common header files" (llvm/IR/Type.h for example) and build again. >>> On full builds the benefit is not big (around 1%, but the noise is high), but: 1) if we only take gains more than, say, 5%, we’ll probably never see any, 2) I aim at changes that make the code strictly better (modulo David’s point about disk cache). If any change is questionable from maintenance or whatever other point of view, I’m all for dropping it. >> >> my 2¢ >> >> +1 for point 2). Even leaving aside the speed gain, removing unused >> includes file just looks like good coding practice to me. >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/75183af4/attachment-0001.html>
Mikhail Zolotukhin via llvm-dev
2017-Dec-09 01:01 UTC
[llvm-dev] [cfe-dev] Who wants faster LLVM/Clang builds?
Hi, I tweaked my scripts to avoid removing includes when it doesn't give any significant benefits, which made the patches significantly smaller. This time the patches should not try to remove includes of header files, which are transitively included from other included header files. The gains mostly remained the same (plus/minus noise), the tables are in the end of the email. I also included size of preprocessed files (measured in 1000 lines of code). I suggest that from here we go as follows: maintainers/interested people take a look at files related to their components and pick the parts of the patches that they consider correct. I'll also start with some files next week if there is no objections to it. Does it sound reasonable? The most impacted files (the numbers are for Debug build): LLVM top 10 Filename TimeOld TimeNew Delta SizeOld SizeNew SizeDelta lib/CodeGen/GlobalISel/GlobalISel.cpp 0.26 0.02 -91.6% 35.0 0.3 -99.0% lib/MC/MCLabel.cpp 0.20 0.02 -88.0% 25.5 0.0 -99.9% tools/llvm-readobj/ObjDumper.cpp 0.44 0.10 -76.8% 41.0 11.8 -71.1% lib/MC/MCWinEH.cpp 0.49 0.15 -70.4% 43.9 21.4 -51.2% lib/Transforms/Vectorize/Vectorize.cpp 0.73 0.29 -60.7% 52.7 35.5 -32.6% tools/llvm-diff/DiffLog.cpp 0.59 0.27 -53.8% 50.7 33.7 -33.7% lib/Target/ARM/MCTargetDesc/ARMMachORelocationInfo.cpp 0.47 0.25 -46.7% 46.7 37.9 -18.9% lib/DebugInfo/DWARF/DWARFExpression.cpp 0.67 0.38 -43.5% 47.4 34.8 -26.7% lib/Transforms/Utils/ASanStackFrameLayout.cpp 0.52 0.32 -38.8% 41.7 33.7 -19.2% tools/llvm-dwp/llvm-dwp.cpp 2.48 1.53 -38.3% 92.5 55.2 -40.3% Full list: Clang top 10 Filename TimeOld TimeNew Delta SizeOld SizeNew SizeDelta tools/libclang/CXString.cpp 2.25 0.30 -86.9% 85.1 31.7 -62.7% lib/Tooling/CommonOptionsParser.cpp 2.33 0.68 -70.6% 87.1 34.4 -60.5% lib/AST/StmtViz.cpp 1.28 0.48 -62.5% 62.4 39.0 -37.5% tools/driver/cc1_main.cpp 3.05 1.29 -57.8% 93.7 58.6 -37.4% unittests/CodeGen/BufferSourceTest.cpp 4.12 2.62 -36.5% 145.8 103.9 -28.7% lib/CodeGen/CGLoopInfo.cpp 2.43 1.68 -30.7% 101.6 82.5 -18.8% unittests/CodeGen/CodeGenExternalTest.cpp 4.50 3.21 -28.6% 155.5 125.1 -19.5% lib/Driver/ToolChains/Contiki.cpp 0.53 0.38 -28.1% 42.4 38.0 -10.5% unittests/Tooling/RefactoringActionRulesTest.cpp 3.22 2.34 -27.5% 108.3 90.0 -16.9% lib/Serialization/GeneratePCH.cpp 2.38 1.78 -25.1% 83.8 71.1 -15.1% Full list: The updated patches: Thanks, Michael> On Dec 8, 2017, at 9:20 AM, Quentin Colombet via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > > >> On Dec 6, 2017, at 1:17 PM, Matthias Braun via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> - We do indeed have a lot of unnecessary includes around in llvm (or pretty much any other C++ project for that matter). >> - I want faster builds. >> - The only way to reliably fight this is indeed automatic tools. >> - Having the right amount of includes also has documentation value and ideally let's you understand the structure of your project. >> - However relying on transitive includes works contrary to the last "undestanding/documentation" point. >> - (And as stated earlier to have things really clean we want `class XXX;` instead of `#include "XXX.h"` wherever possible. And if you are serious about that we also often have to reduce the amount of include code in the headers so we can move the `#include "XXX.h"` from the header to the implementation. >> >> For me personally I think the documentation/understandability we loose when relying on transitive includes weights heavier than my desire to get a faster build… > > +1 > > Q. > >> >> - Matthias >> >>> On Dec 6, 2017, at 12:28 PM, serge guelton via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> On Wed, Dec 06, 2017 at 11:38:54AM -0800, Michael Zolotukhin via llvm-dev wrote: >>>> >>>>> On Dec 6, 2017, at 9:00 AM, mats petersson via cfe-dev <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote: >>>>> >>>>> In my experience, a lot of time is spent on optimizing the code (assuming it's not a "-O0" build). >>>> The numbers were actually for the debug build (-O0 -g), so for Release build they would be different (presumably lower). >>>>> Also redundant includes are largely fixed by header guards, and I believe Clang [and gcc as well as MS Compilers, and probably most others too] have an include guards-cache that determines that "we've already included foo.h, and it has include guards around the whole actual content of the file, so we can just skip it”. >>>> By redundant here I meant that we included a file, but we didn’t use any of its content (rather than we included the same file twice). >>>>> >>>>> So I'm slightly dubious as to this being an efficient way of significantly reducing the total compilation time for the overall project - even if there are SOME cases where there is a significant improvement in a single file. The total time for a clean build [in wall-clock-time, not CPU-time] should be measured, making sure that there is enough memory. Doing a run of, say, five complete builds of the same thing [with suitable "clean" between to redo the whole build], take away the worst and the best, and perhaps also "modify one of the more common header files" (llvm/IR/Type.h for example) and build again. >>>> On full builds the benefit is not big (around 1%, but the noise is high), but: 1) if we only take gains more than, say, 5%, we’ll probably never see any, 2) I aim at changes that make the code strictly better (modulo David’s point about disk cache). If any change is questionable from maintenance or whatever other point of view, I’m all for dropping it. >>> >>> my 2¢ >>> >>> +1 for point 2). Even leaving aside the speed gain, removing unused >>> includes file just looks like good coding practice to me. >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0005.html> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: llvm.txt URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0002.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0006.html> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: clang.txt URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0003.txt> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0007.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: llvm_redundant_headers.patch Type: application/octet-stream Size: 153043 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0002.obj> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0008.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: clang_redundant_headers.patch Type: application/octet-stream Size: 47574 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0003.obj> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171208/8b360593/attachment-0009.html>