Am 01.11.2011 05:59, schrieb Marc J. Driftmeyer:> Then this complaint about build times and extra CPU cycles when you're > living in a world of systems soon to average 16GB of RAM, 4-12 cores and > GPUs that would make any old Animator dream back in the '90s really > makes me laugh.Not disagreeing about the rest, but here I have to. In today's projects, full rebuilds take hours where they should take minutes, and minutes where they should take seconds. Hour-long rebuilds turn releases into things that must be organized and coordinated; minute-long rebuilds still disrupt workflow. Time and effort spent on getting the build process fast are well invested. On the reasons why make-based builds are slow, Peter Miller has some insight to offer: http://miller.emu.id.au/pmiller/books/rmch/ . I'm not sure how widely recognized that paper is. Maybe it's widely known and today's build times stem from other things than recursive make. Regards, Jo
On Tue, Nov 01, 2011 at 11:09:41AM +0100, Joachim Durchholz wrote:> On the reasons why make-based builds are slow, Peter Miller has some > insight to offer: > http://miller.emu.id.au/pmiller/books/rmch/ . > I'm not sure how widely recognized that paper is. Maybe it's widely > known and today's build times stem from other things than recursive make.Please stop pointing to that article. It should be called "GNU make considered harmful" if anything, since most of the content is a direct result of issues in that tool. As I said earlier in this thread: LLVM is nowhere big enough in terms of subdirectories that recursive make is a significant contributor to build time. Joerg
Joachim Durchholz <jo at durchholz.org> writes:> On the reasons why make-based builds are slow, Peter Miller has some > insight to offer: > http://miller.emu.id.au/pmiller/books/rmch/ . > I'm not sure how widely recognized that paper is. Maybe it's widely > known and today's build times stem from other things than recursive make.The paper is widely recognized. Its lessons, unfortunatly, are not. Chris is absolutely on-target as to why the current build is slow. It's slow because recursive make hides the parallelism. It hides the parallelism because it hides the dependencies. There is no way to get around that problem with a recursive make build system. The _only_ reason I have ever found to have a recursive make invocation is at the very beginning of the build to set up a build directory automatically. That one invocation has absolutely nothing to do with the actual build. See: http://mad-scientist.net/make/multi-arch.html -Dave
Joerg Sonnenberger <joerg at britannica.bec.de> writes:> As I said earlier in this thread: LLVM is nowhere big enough in terms > of subdirectories that recursive make is a significant contributor to > build time.But it is and many people have reported their experiences to that effect. One can't simply deny the experiences of many developers. -Dave
Am 01.11.2011 18:04, schrieb Joerg Sonnenberger:> On Tue, Nov 01, 2011 at 11:09:41AM +0100, Joachim Durchholz wrote: >> On the reasons why make-based builds are slow, Peter Miller has some >> insight to offer: >> http://miller.emu.id.au/pmiller/books/rmch/ . >> I'm not sure how widely recognized that paper is. Maybe it's widely >> known and today's build times stem from other things than recursive make. > > Please stop pointing to that article. It should be called "GNU make > considered harmful" if anything, since most of the content is a direct > result of issues in that tool.The salient point in that article is about common use of recursive make not doing what most people expect. In particular, if the dependencies are a DAG instead of a tree, you need to put everything into one big makefile since recursive make's design cannot handle the situation correctly. This is a property of the basic design of make, and cannot be fixed unless by changing what a recursive make actually does. (I have built my own make, in Rexx, as I was young and needed the tool. This otherwise unremarkable endeavour forced me to learn more about make's semantic than a mere student should be exposed to.) > As I said earlier in this thread: Sorry, I missed that post. > LLVM is> nowhere big enough in terms of subdirectories that recursive make is a > significant contributor to build time.Did you verify that this is the case? Some of the gotchas in the make semantics can cause exponential behaviour, and that's easy to overlook if the N in your O(2^N) behaviour is incremented only occasionally. Regards, Jo
greened at obbligato.org (David A. Greene) writes:> Joachim Durchholz <jo at durchholz.org> writes: > >> On the reasons why make-based builds are slow, Peter Miller has some >> insight to offer: >> http://miller.emu.id.au/pmiller/books/rmch/ . >> I'm not sure how widely recognized that paper is. Maybe it's widely >> known and today's build times stem from other things than recursive make. > > The paper is widely recognized. Its lessons, unfortunatly, are not. > > Chris is absolutely on-target as to why the current build is slow. It's > slow because recursive make hides the parallelism. It hides the > parallelism because it hides the dependencies. There is no way to get > around that problem with a recursive make build system.You keep repeating that and I say that it is wrong. Can you mention a serialization point on the LLVM build caused by recursive make? (GenLibDeps is not such example, as described on a previous message.) [snip]