llvm dev - Nov 2011 - [LLVMdev] RFC: Upcoming Build System Changes

Chris Lattner <clattner at apple.com> writes:
>> You keep repeating that and I say that it is wrong. Can you mention a
>> serialization point on the LLVM build caused by recursive make?
>> (GenLibDeps is not such example, as described on a previous message.)
>
> Any use of DIRS is a serialization point.  For example, lib/Support ->
> lib/TableGen -> utils are all built in serial before anything else is.
AFAIU DIRS is used for lib/Support an lib/TableGen because the latter
depends on the first. This is not relevant on static builds (or shared
builds on Linux, IIRC) so, if the `make' scripts were smart enough, that
case of DIRS could become something like PARALLEL_DIRS_IF_STATIC. That
would make possible to compile the source files of both libraries at the
same time. BTW, the cmake build did that (don't know if keeps doing it
after the switch to explicit dependencies.)

But you and Dave are right wrt recursive make hiding details of the
build. A non-recursive make could compile the files on both libraries
without introducing any hacks.

My insistence on Dave being wrong comes from my experience with the
cmake build. There, we turn DIRS on PARALLEL_DIRS (except when building
shared libraries on OS/X) and the amount of parallelism is huge.
Essentially, there are three phases: building Support&utils, building
the rest of the libraries, and building the tools (the final two phases
are intermingled to some extent.) I expect that even a 64core machine
will be quite busy almost all the time.

On Nov 1, 2011, at 6:41 PM, Óscar Fuentes wrote:
> Chris Lattner <clattner at apple.com> writes:
> 
>>> You keep repeating that and I say that it is wrong. Can you mention
a
>>> serialization point on the LLVM build caused by recursive make?
>>> (GenLibDeps is not such example, as described on a previous
message.)
>> 
>> Any use of DIRS is a serialization point.  For example, lib/Support
->
>> lib/TableGen -> utils are all built in serial before anything else
is.
> 
> AFAIU DIRS is used for lib/Support an lib/TableGen because the latter
> depends on the first. This is not relevant on static builds (or shared
> builds on Linux, IIRC) so, if the `make' scripts were smart enough,
that
> case of DIRS could become something like PARALLEL_DIRS_IF_STATIC. That
> would make possible to compile the source files of both libraries at the
> same time. BTW, the cmake build did that (don't know if keeps doing it
> after the switch to explicit dependencies.)
Hi Oscar,

Forgive me, but I've lost track of how any of this has to do with
Daniel's proposal. Recursive make vs not, make vs cmake all don't
actually matter to his proposal.

His proposal is basically "cmake isn't going to replace make any time
soon. We'd like to improve the make system (e.g. by eliminating autoconf)
and the first step is to factor redundancy that exists between cmake and
make".

While you might *want* cmake to replace make in the near term, it just isn't
going to happen.  I don't see how his proposal actually makes our system
cmake worse, and all of your arguments seem to trying to convince everyone that
cmake is better than make.  What is going on here?

-Chris

Chris Lattner <clattner at apple.com> writes:
> Forgive me, but I've lost track of how any of this has to do with
> Daniel's proposal. Recursive make vs not, make vs cmake all don't
> actually matter to his proposal.
One of the problems with Daniel's proposal is that besides addressing
the duplicity issue, it has no applications to back it up. The things
mentioned so far are either extremely vague, or offer dubious gains
(i.e. removing recursivity from the Makefiles) or trivially done without
his system (i.e. walking the source files, etc.)

So when it is explained that the duplicity can be addressed without his
system, those other hypothetical advantages are mentioned and then the
discussion inevitably drifts.
> His proposal is basically "cmake isn't going to replace make any
time
> soon. We'd like to improve the make system (e.g. by eliminating
> autoconf) and the first step is to factor redundancy that exists
> between cmake and make".
Does that make sense at all? How is autoconf related with duplicity in
cmake? If he thinks that cmake and autoconf platform tests are
duplicated, he will be disappointed when he discovers that cmake depends
on those tests not only for configuring the LLVM headers, but for doing
lots of stuff. Re-implementing autoconf is a daunting task (if that's
what he wants to do). Shifting to his system all the decisions that
depends on those tests is going quite a stretch further.
> While you might *want* cmake to replace make in the near term,
No, I don't want that. Somebody suggested that having two build systems
is a problem, so I retorted that if one of them must stay, is the cmake
system, for obvious reasons.
> it just isn't going to happen.  I don't see how his proposal
actually
> makes our system cmake worse, and all of your arguments seem to trying
> to convince everyone that cmake is better than make.  What is going on
> here?
I'll repeat again: Daniel's proposal means adding a lot of complexity to
the build(s), reinventing the wheel and putting a burden on the
maintainers. It is absolutely unnecessary for removing the duplicity: you
can put the info on some file or in the Makefiles themselves and cmake
will read it without problem. For the cmake maintainer, Daniel's
proposal would change his work from having to know the cmake build, to
having to know the cmake build plus the make build plus Python plus
Daniel's own system. It is a way of restricting the set of eligible
maintainers to those who know all that. I'm absolutely sure that the
amount of required changes to the cmake build for supporting his
proposal is much more complex and harder to maintain than the necessary
changes from reading the info from the common description file with
cmake's own devices.

With all due respect to Daniel: I'm convinced that he is doing this
because he loves to program Python and considers CMake "horrible"
(that's the word he used) but he's not doing a favor to LLVM with this
contribution. That's not the case of *replacing* a dinosaur like DejaGNU
with something more convenient (as he did.) He is *adding* a lot of
stuff. He is proposing a cruise missile as a means to kill a mosquito.

Related to this, I'll reiterate my opinion about the explicit library
dependency issue. What you want is to force the user to know and approve
the dependency changes his modifications to the source code
causes. That's orthogonal to maintaining the dependency info up to
date. The cmake build used to contain a system for warning the developer
about his changes to the library dependencies just after he builds and
before he commits. At the same time, the dependencies were automatically
maintained in optimal shape. That system is very easy to implement on
the `make' build.

We begin with a suboptimal decision (forcing the users to manually edit
files for declaring the dependencies instead of exploiting the automatic
system for implementing the new requirement) and this creates the
duplicity problem, which in turn prompts Daniel's proposal, which adds
even more complexity for addressing a problem that shouldn't be there to
begin with. Happens too often in engineering projects that lots of
effort is devoted to solving the problems the engineers themselves
created, but let's try to avoid doing that on this case.