llvm dev - Nov 2011 - [LLVMdev] [cfe-dev] RFC: Upcoming Build System Changes

Nico Weber <thakis at chromium.org> writes:
> On Tue, Nov 1, 2011 at 3:09 PM, David A. Greene <greened at
obbligato.org> wrote:
>> Óscar Fuentes <ofv at wanadoo.es> writes:
>>
>>> Okay, we can get rid of recursive make. However, as pointed out
>>> elsewhere, removing recursive make will not make a difference on
the
>>> LLVM build. What David Greene says probably is related to
difference
>>> sizes among his compiler and LLVM.
>
> I doubt that's true: Chromium's empty build is faster than
llvm's
> (only tried the make build, not the cmake build), and chromium's code
> base is considerably larger.
What David was saying is that Cray's compiler *non-empty* build ends
sooner than the LLVM *empty* build completes. Well, if only a small
translation unit needs to be compiled into a small executable (no long
compiler/linker waits) it may be possible but, again, it only speaks
about the relative sizes of LLVM/Cray's compiler.

WRT your Chromium example, the crux is not so much about what is faster,
but about what is considered slow. Chromium's empty build being faster
than LLVM is of no significance if LLVM's empty build can't be
considered annoyingly slow. And I wouldn't say that 1.3 seconds on a
rather ordinary desktop is annoyingly slow.

Óscar Fuentes <ofv at wanadoo.es> writes:
> Nico Weber <thakis at chromium.org> writes:
>
>> On Tue, Nov 1, 2011 at 3:09 PM, David A. Greene <greened at
obbligato.org> wrote:
>>> Óscar Fuentes <ofv at wanadoo.es> writes:
>>>
>>>> Okay, we can get rid of recursive make. However, as pointed out
>>>> elsewhere, removing recursive make will not make a difference
on the
>>>> LLVM build. What David Greene says probably is related to
difference
>>>> sizes among his compiler and LLVM.
>>
>> I doubt that's true: Chromium's empty build is faster than
llvm's
>> (only tried the make build, not the cmake build), and chromium's
code
>> base is considerably larger.
>
> What David was saying is that Cray's compiler *non-empty* build ends
> sooner than the LLVM *empty* build completes. Well, if only a small
> translation unit needs to be compiled into a small executable (no long
> compiler/linker waits) it may be possible but, again, it only speaks
> about the relative sizes of LLVM/Cray's compiler.
Ok, here are some hard numbers for empty builds:

LLVM empty build:
/usr/bin/time make -j16
4.32user 2.47system 0:03.21elapsed 211%CPU (0avgtext+0avgdata 13376maxresident)k
0inputs+0outputs (0major+671804minor)pagefaults 0swaps

Cray empty build:
/usr/bin/time make dynamic-developer -j16
2.88user 1.06system 0:04.94elapsed 79%CPU (0avgtext+0avgdata 64208maxresident)k
0inputs+0outputs (0major+184605minor)pagefaults 0swaps

The Cray empty build includes some really horrendous shell script
trickery known as "modules."  You can read the gory details here:

http://modules.sourceforge.net/

It's possibly the dumbest thing ever invented.  But that's what I've
got
to work with.

The point is, the Cray build includes some major shell overhead that the
LLVM build doesn't have to deal with.  And the Cray build has more
targets to build.  And the Cray build has more dependencies to examine.
And the Cray build still blows the socks off the LLVM build.

Here is actual data comparing an empty LLVM build done recursively (the
LLVM build) and non-recursively (the Cray build).

See this?

0inputs+0outputs (0major+671804minor)pagefaults 0swaps

vs. this?

0inputs+0outputs (0major+184605minor)pagefaults 0swaps

That's I/O.

                             -Dave

On Tue, Nov 01, 2011 at 06:27:35PM -0500, David A. Greene
wrote:
> Here is actual data comparing an empty LLVM build done recursively (the
> LLVM build) and non-recursively (the Cray build).
> 
> See this?
> 
> 0inputs+0outputs (0major+671804minor)pagefaults 0swaps
> 
> vs. this?
> 
> 0inputs+0outputs (0major+184605minor)pagefaults 0swaps
> 
> That's I/O.
Sorry to disappoint you, but no, this isn't I/O. You have just shown
that a LLVM build touches more mmap'd pages.

Joerg

greened at obbligato.org (David A. Greene) writes:
> Ok, here are some hard numbers for empty builds:
>
> LLVM empty build:
> /usr/bin/time make -j16
> 4.32user 2.47system 0:03.21elapsed 211%CPU (0avgtext+0avgdata
13376maxresident)k
> 0inputs+0outputs (0major+671804minor)pagefaults 0swaps
So your 16-way machine takes 2.5 times more than my 4-way cheap desktop...
> Cray empty build:
> /usr/bin/time make dynamic-developer -j16
> 2.88user 1.06system 0:04.94elapsed 79%CPU (0avgtext+0avgdata
64208maxresident)k
> 0inputs+0outputs (0major+184605minor)pagefaults 0swaps
>
> The Cray empty build includes some really horrendous shell script
> trickery known as "modules."  You can read the gory details here:
>
> http://modules.sourceforge.net/
>
> It's possibly the dumbest thing ever invented.  But that's what
I've got
> to work with.
>
> The point is, the Cray build includes some major shell overhead that the
> LLVM build doesn't have to deal with.  And the Cray build has more
> targets to build.  And the Cray build has more dependencies to examine.
> And the Cray build still blows the socks off the LLVM build.
... and you hypothesize that the shell work adds substantial time to the
Cray build, thus making even more dramatic the difference on the
LLVM-Cray build systems.
> Here is actual data comparing an empty LLVM build done recursively (the
> LLVM build) and non-recursively (the Cray build).
>
> See this?
>
> 0inputs+0outputs (0major+671804minor)pagefaults 0swaps
>
> vs. this?
>
> 0inputs+0outputs (0major+184605minor)pagefaults 0swaps
>
> That's I/O.
Let me guess: you are starting from a cold cache. Other than a
difference caused by the OS, that's the only thing that could explain
it.

The format used by your `time' is not familiar to me, so excuse me if I
make a question related to your measures:

LLVM: 4.32user 2.47system 0:03.21elapsed 211%CPU
Cray: 2.88user 1.06system 0:04.94elapsed 79%CPU

Looks like the Cray build is using less CPU but takes 1.7 seconds more
than LLVM of wall clock time. But then you note that the LLVM build uses
a lot more I/O. How can this be? It looks as if the LLVM build were much
more parallel. Is this explained because the shell scripts you mention
above introduce serialization points?

And the decisive question: is 3.2 seconds considered an annoying amount
of time for you deciding to support a substantial change on the LLVM
build, but 4.94 seconds is not annoying enough to get rid of those shell
scripts? :-)