llvm dev - Mar 2013 - [LLVMdev] llvmlab (phased buildmaster) is in production mode!

Hello LLVM Dev and Clang Dev!

David Dean and I just finished bringing up a new build master on lab.llvm.org,
llvmlab, which is located at the url http://lab.llvm.org:8013 and is in #llvm
under the username llvmlab.

llvmlab is different than the current buildbot based continuous integration
systems llvm uses; llvmlab is a phased builder based system. The high level
details of the phased builder system are as follows:

1. Builders are grouped together into phases as sub builders. Each phase is a
builder itself and triggers said grouped ``sub builders'' and assuming
that all of the sub builders complete, triggers its successor phased builder if
one exists. This creates a gating like effect.

2. All phases are gated together in a linear fashion, yielding via the linearity
the phenomena that if an earlier phase does not succeed, then no later phases
run. The key idea here is if we know that there is a fundamental issue with the
compiler why try to build 20 compilers, when performing one quick build is all
that is needed to ascertain such a fact? Also if we can not build a compiler
successfully, why try to do LNT performance runs? This gets rid of pointless
work, stops excessive emails from being sent out for 1 bad commit, and reduces
cycle time especially if certain builds take significantly longer than others to
fail.

3. Later phases do broader, longer lasting testing than earlier phases. Thus the
4 phases we currently have are:
	
	a. Phase 1 (sanity):          Phase 1 is a quick non-bootstrapped, non-lto
compiler build, to check the ``basic sanity'' of the code base and build
process. This generally takes 15-20 minutes to complete.
	b. Phase 2 (living on):      Phase 2 builds bootstrapped compilers for the
different configurations that can be used for ``living on'', i.e. good
enough for common compilation tasks. This is meant to cycle in up to an hour.
	c. Phase 3 (tree health):  Phase 3 runs performance tests, i.e., LNT (which are
not live yet) as well as other compiler builds which take a longer amount of
time (a full clang build with LTO enabled for instance).
	d. Phase 4 (validation):   Phase 4 runs longer running validation tests.
Currently we have nothing in phase 4, but I am sure that will change = p.

4. Builders in later phases rely on outputs from earlier phases. If we are doing
performance runs, why should we compile a new compiler for such a performance
run? This is duplicated work! Instead the phased build system stores
``artifacts'', i.e., built compilers, in earlier phases and uses them as
compilers for builds in later phases. Thus we could have 30 different Phase 3
LNT performance runs with different configurations all using the same compiler
artifacts built in phase 2. This significantly deduplicates work yielding a
decreased cycle time.

As time moves on we will be moving more and more builders to llvmlab including
LNT performance builders and a builder which runs the libcxx test suite.

Michael

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130327/5708ae87/attachment.html>

Most of the selections, like "console" for example, do not work when I
click on them.

On 03/27/2013 03:57 PM, Michael Gottesman wrote:
> Hello LLVM Dev and Clang Dev!
>
> David Dean and I just finished bringing up a new build master on
> lab.llvm.org <http://lab.llvm.org/>, llvmlab, which is located at the
> url http://lab.llvm.org:8013 <http://lab.llvm.org:8013/> and is in
#llvm
> under the username llvmlab.
>
> llvmlab is different than the current buildbot based continuous
> integration systems llvm uses; llvmlab is a phased builder based system.
> The high level details of the phased builder system are as follows:
>
> 1. Builders are grouped together into phases as sub builders. Each phase
> is a builder itself and triggers said grouped ``sub builders'' and
> assuming that all of the sub builders complete, triggers its successor
> phased builder if one exists. This creates a gating like effect.
>
> 2. All phases are gated together in a linear fashion, yielding via the
> linearity the phenomena that if an earlier phase does not succeed, then
> no later phases run. The key idea here is if we know that there is a
> fundamental issue with the compiler why try to build 20 compilers, when
> performing one quick build is all that is needed to ascertain such a
> fact? Also if we can not build a compiler successfully, why try to do
> LNT performance runs? This gets rid of pointless work, stops excessive
> emails from being sent out for 1 bad commit, and reduces cycle time
> especially if certain builds take significantly longer than others to fail.
>
> 3. Later phases do broader, longer lasting testing than earlier phases.
> Thus the 4 phases we currently have are:
>
> a. Phase 1 (sanity):          Phase 1 is a quick non-bootstrapped,
> non-lto compiler build, to check the ``basic sanity'' of the code
base
> and build process. This generally takes 15-20 minutes to complete.
> b. Phase 2 (living on):      Phase 2 builds bootstrapped compilers for
> the different configurations that can be used for ``living on'',
i.e.
> good enough for common compilation tasks. This is meant to cycle in up
> to an hour.
> c. Phase 3 (tree health):  Phase 3 runs performance tests, i.e., LNT
> (which are not live yet) as well as other compiler builds which take a
> longer amount of time (a full clang build with LTO enabled for instance).
> d. Phase 4 (validation):   Phase 4 runs longer running validation tests.
> Currently we have nothing in phase 4, but I am sure that will change = p.
>
> 4. Builders in later phases rely on outputs from earlier phases. If we
> are doing performance runs, why should we compile a new compiler for
> such a performance run? This is duplicated work! Instead the phased
> build system stores ``artifacts'', i.e., built compilers, in
earlier
> phases and uses them as compilers for builds in later phases. Thus we
> could have 30 different Phase 3 LNT performance runs with different
> configurations all using the same compiler artifacts built in phase 2.
> This significantly deduplicates work yielding a decreased cycle time.
>
> As time moves on we will be moving more and more builders to llvmlab
> including LNT performance builders and a builder which runs the libcxx
> test suite.
>
> Michael
>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev-Tmj1lob9twqVc3sceRu5cw at public.gmane.org
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>

Cool! This is great news.

I feel like this information should be in our documentation somewhere.
Could you start a new file ContinuousIntegration.rst and use this content
to seed it? This new page would also be a good place to mention some LLVM
idiosyncrasies like smooshlab being Apple-internal but still reporting via
IRC; these things have not had a good place to be put yet. AFAIK currently
our continuous integration infrastructure is mostly community wisdom and
besides a small mention of some of the reporting bots on index.rst there is
no documentation describing it.

-- Sean Silva
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130327/b9a4e141/attachment.html>

On Wed, Mar 27, 2013 at 3:57 PM, Michael Gottesman <mgottesman at
apple.com>wrote:
> 3. Later phases do broader, longer lasting testing than earlier phases.
> Thus the 4 phases we currently have are:
>
> a. Phase 1 (sanity):          Phase 1 is a quick non-bootstrapped, non-lto
> compiler build, to check the ``basic sanity'' of the code base and
build
> process. This generally takes 15-20 minutes to complete.
>
While most of this sounds great, this one really doesn't.

The sanity tests should be able to run *much* faster than 15-20 minutes.
Can we prioritize getting an incremental rebuild bot as the sanity phase on
reasonably fast hardware? I think it's important to get through the sanity
phase in 1-5 minutes so that phase 2 doesn't get soooo many commits piled
up on it when someone checks in code with both a miscompile and a tiny
small build break (for example).

-Chandler
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130327/26a10692/attachment.html>

Smooshlab is going away so the LLVM idiosyncrasy that you mention will be going
away soon = p. The whole idea behind bringing up this infrastructure is so that
everything that is Apple-internal but COULD be public is public and is on the
phased builders. Anything we can't bring out will (of course) stay internal.
Also I agree about the documentation.

Michael

On Mar 27, 2013, at 5:58 PM, Sean Silva <silvas at purdue.edu> wrote:
> Cool! This is great news.
> 
> I feel like this information should be in our documentation somewhere.
Could you start a new file ContinuousIntegration.rst and use this content to
seed it? This new page would also be a good place to mention some LLVM
idiosyncrasies like smooshlab being Apple-internal but still reporting via IRC;
these things have not had a good place to be put yet. AFAIK currently our
continuous integration infrastructure is mostly community wisdom and besides a
small mention of some of the reporting bots on index.rst there is no
documentation describing it.
> 
> -- Sean Silva

This is a known issue. I have actually screwed with this before it is just a
matter of getting it into zorg in the right manner.

On Mar 27, 2013, at 5:50 PM, Reed Kotler <rkotler at mips.com> wrote:
> Most of the selections, like "console" for example, do not work
when I click on them.
> 
> On 03/27/2013 03:57 PM, Michael Gottesman wrote:
>> Hello LLVM Dev and Clang Dev!
>> 
>> David Dean and I just finished bringing up a new build master on
>> lab.llvm.org <http://lab.llvm.org/>, llvmlab, which is located at
the
>> url http://lab.llvm.org:8013 <http://lab.llvm.org:8013/> and is
in #llvm
>> under the username llvmlab.
>> 
>> llvmlab is different than the current buildbot based continuous
>> integration systems llvm uses; llvmlab is a phased builder based
system.
>> The high level details of the phased builder system are as follows:
>> 
>> 1. Builders are grouped together into phases as sub builders. Each
phase
>> is a builder itself and triggers said grouped ``sub builders''
and
>> assuming that all of the sub builders complete, triggers its successor
>> phased builder if one exists. This creates a gating like effect.
>> 
>> 2. All phases are gated together in a linear fashion, yielding via the
>> linearity the phenomena that if an earlier phase does not succeed, then
>> no later phases run. The key idea here is if we know that there is a
>> fundamental issue with the compiler why try to build 20 compilers, when
>> performing one quick build is all that is needed to ascertain such a
>> fact? Also if we can not build a compiler successfully, why try to do
>> LNT performance runs? This gets rid of pointless work, stops excessive
>> emails from being sent out for 1 bad commit, and reduces cycle time
>> especially if certain builds take significantly longer than others to
fail.
>> 
>> 3. Later phases do broader, longer lasting testing than earlier phases.
>> Thus the 4 phases we currently have are:
>> 
>> a. Phase 1 (sanity):          Phase 1 is a quick non-bootstrapped,
>> non-lto compiler build, to check the ``basic sanity'' of the
code base
>> and build process. This generally takes 15-20 minutes to complete.
>> b. Phase 2 (living on):      Phase 2 builds bootstrapped compilers for
>> the different configurations that can be used for ``living
on'', i.e.
>> good enough for common compilation tasks. This is meant to cycle in up
>> to an hour.
>> c. Phase 3 (tree health):  Phase 3 runs performance tests, i.e., LNT
>> (which are not live yet) as well as other compiler builds which take a
>> longer amount of time (a full clang build with LTO enabled for
instance).
>> d. Phase 4 (validation):   Phase 4 runs longer running validation
tests.
>> Currently we have nothing in phase 4, but I am sure that will change =
p.
>> 
>> 4. Builders in later phases rely on outputs from earlier phases. If we
>> are doing performance runs, why should we compile a new compiler for
>> such a performance run? This is duplicated work! Instead the phased
>> build system stores ``artifacts'', i.e., built compilers, in
earlier
>> phases and uses them as compilers for builds in later phases. Thus we
>> could have 30 different Phase 3 LNT performance runs with different
>> configurations all using the same compiler artifacts built in phase 2.
>> This significantly deduplicates work yielding a decreased cycle time.
>> 
>> As time moves on we will be moving more and more builders to llvmlab
>> including LNT performance builders and a builder which runs the libcxx
>> test suite.
>> 
>> Michael
>> 
>> 
>> 
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev-Tmj1lob9twqVc3sceRu5cw at public.gmane.org
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Mar 27, 2013, at 6:41 PM, Chandler Carruth <chandlerc at google.com>
wrote:
> 
> On Wed, Mar 27, 2013 at 3:57 PM, Michael Gottesman <mgottesman at
apple.com> wrote:
> 3. Later phases do broader, longer lasting testing than earlier phases.
Thus the 4 phases we currently have are:
> 	
> 	a. Phase 1 (sanity):          Phase 1 is a quick non-bootstrapped, non-lto
compiler build, to check the ``basic sanity'' of the code base and build
process. This generally takes 15-20 minutes to complete.
> 
> While most of this sounds great, this one really doesn't.
> 
> The sanity tests should be able to run *much* faster than 15-20 minutes.
Agreed. I think getting the sanity time down further is possible and very
important. IIRC gribozavr has a ninja cmake bot that does clean builds in <
10 minutes. I think that that is a first step (bringing me to your question).
> Can we prioritize getting an incremental rebuild bot as the sanity phase on
reasonably fast hardware?
Incremental builds are quicker but less robust than clean builds. The nice thing
about always doing clean builds is that it simplify things by throwing out the
potential of any build failures due to a dirty build directory (but maybe I am
paranoid). If we want to do incremental builds (which note I am not averse to
btw), we should at least set up some manner to clean the build directory if we
get a build failure due to a dirty build directory that does not involve sshing
into the machine. Perhaps if a builder fails a number of times in a row, clean
it?

Also, hardware contributions are always welcome = p.

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130327/7bf388be/attachment.html>