llvm dev - Nov 2011 - [LLVMdev] [cfe-dev] RFC: Upcoming Build System Changes

Óscar Fuentes <ofv at wanadoo.es> writes:
> Chris Lattner <clattner at apple.com> writes:
>
>> There are different overheads in different scenarios.  The makefiles
>> get really poor utilization out of a 8 or 16-way machine because of
>> implicit synchronization between different sublibraries.
>
> BTW, adding explicit library dependencies will make the parallel builds
> worse, 
No they won't, for exactly the reason Chris stated.
> because you know when a dependency is missing (the build fails)
> but you don't know when a dependency is superfluous.
That's true but is not a show-stopper IME.  It's easier to remove
superfluous dependencies than try to track down a missing one in a
recursive make build.

                        -Dave

greened at obbligato.org (David A. Greene) writes:
> Óscar Fuentes <ofv at wanadoo.es> writes:
>
>> Chris Lattner <clattner at apple.com> writes:
>>
>>> There are different overheads in different scenarios.  The
makefiles
>>> get really poor utilization out of a 8 or 16-way machine because of
>>> implicit synchronization between different sublibraries.
>>
>> BTW, adding explicit library dependencies will make the parallel builds
>> worse, 
>
> No they won't, for exactly the reason Chris stated.
That's makes no sense. The automatic system we had generated an optimal
dependency graph: no missing or unnecesary edges. You can't do better
than than optimal, can you?. Furthermore, while building static
libraries, those didn't block one another, because there was no
dependency declarations among them. I don't know how the current system
works, but the clang build, which always used explicit dependencies,
created unnecessary block points among libraries because of that.
>> because you know when a dependency is missing (the build fails)
>> but you don't know when a dependency is superfluous.
>
> That's true but is not a show-stopper IME.  It's easier to remove
> superfluous dependencies than try to track down a missing one in a
> recursive make build.
It's quite the opposite. You'll not notice superfluous dependencies
unless you look for them. BTW, long time ago I scanned the clang library
dependencies (which used an explicit scheme all along, as mentioned
above) by comparing them with the output of GenLibDeps.pl (once tweaked
for managing the clang libs.) There was a few unnecesary/redundant
dependencies, on a set of libraries that is very small compared to LLVM
(keep in mind that the number of edges on the dependency graph grows
much faster than linear.)

To think that no unnecesary dependencies will pop up on the build is
delusional, unless you introduce some sort of automatic check. And once
you have the automatic check, why not delegate to it the handling of the
dependencies?

As mentioned elsewhere, what we want is to know when a source code
change creates a modification on the dependency graph, to decide if we
are okay with that. Manually keeping up to date the dependency files is
just an unnecessary bureaucratic burden.

Óscar Fuentes <ofv at wanadoo.es> writes:
>>>> There are different overheads in different scenarios.  The
makefiles
>>>> get really poor utilization out of a 8 or 16-way machine
because of
>>>> implicit synchronization between different sublibraries.
>>>
>>> BTW, adding explicit library dependencies will make the parallel
builds
>>> worse, 
>>
>> No they won't, for exactly the reason Chris stated.
>
> That's makes no sense. The automatic system we had generated an optimal
> dependency graph: no missing or unnecesary edges. You can't do better
> than than optimal, can you?
If you're using recursive make, it is by definitely not optimal.
> To think that no unnecesary dependencies will pop up on the build is
> delusional, unless you introduce some sort of automatic check. And once
> you have the automatic check, why not delegate to it the handling of the
> dependencies?
I never said they wouldn't show up.  I said they're not worth worrying
about until you notice them.  You notice them when you notice your build
isn't as parallel as it could be and that happens when the build takes
too long.
> As mentioned elsewhere, what we want is to know when a source code
> change creates a modification on the dependency graph, to decide if we
> are okay with that. Manually keeping up to date the dependency files is
> just an unnecessary bureaucratic burden.
I agree.  Dependency files should not be necessary.  But CMake uses such
files, doesn't it?  What's the difference between that and Daniel's
solution?  All things being equal, I prefer Daniel's solution because
CMake is just horrendous.

                                -Dave