llvm dev - Aug 2009 - [LLVMdev] Buildbots: Apology and Explanation

I need to apologize to many of you, especially Gabor, for my vitriol  
over buildbot breakages. I get very enraged by breakages that occur  
over long periods of time. Here's an attempt to explain why I fly off  
the handle at these.

Before we had buildbots, a random patch would break the system. It  
would sometimes take a whole day to determine which patch broke it. I  
came up with the idea of using builbots. We started using them  
internally at Apple (unfortunately, those machines are still  
unavailable to non-Apple employees). It quickly spread so that some  
were set up at Google. It was a huge win for us.

However, it's only a huge win if people pay attention to the error  
reports. People didn't. At first, there were a lot of "false  
negatives". Those were fixed. Then the reporting scheme wasn't great.  
Those were also fixed. Still, people would break the builds and let  
things go for hours or days at a time. This feels like a slap in the  
face to me. After so much hard work, people are ignoring the one piece  
of quality assurance that we are afforded in the LLVM community. In  
the meantime, many many patches are going in which aren't being  
tested. If something breaks after the initial breakage, then it's that  
much harder to determine what's going on. It's a lot of wasted effort.

Couple that with the fact that we at Apple (and other place, to be  
sure) rely upon ToT to be fairly stable so that we can create branches  
from ToT at a moment's notice, and you get a really bad and stressful  
situation.

So, yes, I take it personally when breakages are ignored. It's an  
unfortunate part of my personality that I'm trying to work on, but on  
occasion gets the better of me, and I lash out.

Again, my apologies. Please watch what the buildbots have to say. Even  
if you can't see the log file message, try to see if there's something  
you might have missed. Someone with access to the Apple-only machines  
would be happy to forward the error message to you if you think your  
patch might be the cause.

Sincerely,
Bill (Monkeypox)

> people would break the builds and let
>  things go for hours or days at a time.
"Hours or days".
Instead of months or years. Isn't OS wonderful?
You have high standards and I admire the importance you assign to perfection

    M

Hi Bill,
> ... Still, people would break the builds and let  
> things go for hours or days at a time.
don't forget the time-zone effect.  I regularly get build
failures in the morning, presumably because someone in the
US committed just before going to bed.  I guess they are
happily snoring away when the build-bots (and humans) start
complaining!  So when hours go by without a fix, it might be
because the committer is getting some beauty sleep (or is at
work, and can't do anything from there).

Ciao,

Duncan.

On Sat, Aug 29, 2009 at 4:15 AM, Duncan Sands<baldrick at free.fr>
wrote:
> Hi Bill,
>
>> ... Still, people would break the builds and let
>> things go for hours or days at a time.
>
> don't forget the time-zone effect.  I regularly get build
> failures in the morning, presumably because someone in the
> US committed just before going to bed.  I guess they are
> happily snoring away when the build-bots (and humans) start
> complaining!  So when hours go by without a fix, it might be
> because the committer is getting some beauty sleep (or is at
> work, and can't do anything from there).
>
> Ciao,
>
> Duncan.
At my old job, we were expected to stay after doing a commit at least
long enough for the buildbot to complain if the build was broken.  Of
course our buildbots took about 10 minutes after commit to finish a
build, so that wasn't much of a burden...

Hello,

Bill Wendling wrote:
> Before we had buildbots, a random patch would break the system. It  
> would sometimes take a whole day to determine which patch broke it. 
I see the buildbots are currently showing no problem on 32-bit linux but
I get the following build error with TOT (out-of-source autoconf build):

[...]
make[1]: Leaving directory `/home/melis/c/llvm-svn-release/tools'
make[1]: Entering directory `/home/melis/c/llvm-svn-release/runtime'
make[2]: Entering directory
`/home/melis/c/llvm-svn-release/runtime/libprofile'
llvm[2]: Compiling BasicBlockTracing.ll to BasicBlockTracing.bc for
Release build (bytecode)
/home/melis/c/llvm-svn-release/Release/bin/llvm-as:
/home/melis/c/llvm-svn-release/runtime/libprofile/Release/BasicBlockTracing.ll:1:2:
error: expected top-level entity
    .file    "BasicBlockTracing.c"
    ^
/home/melis/c/llvm-svn-release/Release/bin/opt: Invalid bitcode signature
make[2]: ***
[/home/melis/c/llvm-svn-release/runtime/libprofile/Release/BasicBlockTracing.bc]
Error 1
make[2]: Leaving directory
`/home/melis/c/llvm-svn-release/runtime/libprofile'
make[1]: *** [libprofile/.makeall] Error 2
make[1]: Leaving directory `/home/melis/c/llvm-svn-release/runtime'
make: *** [all] Error 1

It seems BasicBlockTracing.ll contains assembly instead of LLVM IR.


Also, llvm-gcc also doesn't build for me. Here the error is related to
exception handling:

make[3]: Leaving directory `/home/melis/c/llvm-gcc-svn-build/libdecnumber'
make[3]: Entering directory `/home/melis/c/llvm-gcc-svn-build/gcc'
/home/melis/local/bin/gcc -c   -g -DIN_GCC   -W -Wall -Wwrite-strings
-Wstrict-prototypes -Wmissing-prototypes -pedantic -Wno-long-long
-Wno-variadic-macros -Wno-overlength-strings -Wold-style-definition
-Wmissing-format-attribute -fno-common   -DHAVE_CONFIG_H -I. -I.
-I../../llvm-gcc-svn/gcc -I../../llvm-gcc-svn/gcc/.
-I../../llvm-gcc-svn/gcc/../include
-I../../llvm-gcc-svn/gcc/../libcpp/include 
-I../../llvm-gcc-svn/gcc/../libdecnumber -I../libdecnumber   
gtype-desc.c -o gtype-desc.o
gtype-desc.c:5862: error: 'sjlj_fc_type_node' undeclared here (not in a
function)
make[3]: *** [gtype-desc.o] Error 1
make[3]: Leaving directory `/home/melis/c/llvm-gcc-svn-build/gcc'
make[2]: *** [all-stage1-gcc] Error 2
make[2]: Leaving directory `/home/melis/c/llvm-gcc-svn-build'
make[1]: *** [stage1-bubble] Error 2
make[1]: Leaving directory `/home/melis/c/llvm-gcc-svn-build'
make: *** [all] Error 2

Is there something suddenly wrong with my system (which hasn't changed
in any way)? I used to be able to build LLVM(GCC) without problems a few
weeks ago...

Regards,
Paul

On Aug 30, 2009, at 10:18 AM, Paul Melis wrote:
> Hello,
>
> Bill Wendling wrote:
>> Before we had buildbots, a random patch would break the system. It
>> would sometimes take a whole day to determine which patch broke it.
> I see the buildbots are currently showing no problem on 32-bit linux  
> but
> I get the following build error with TOT (out-of-source autoconf  
> build):
>
> [...]
> make[1]: Leaving directory `/home/melis/c/llvm-svn-release/tools'
> make[1]: Entering directory `/home/melis/c/llvm-svn-release/runtime'
> make[2]: Entering directory
> `/home/melis/c/llvm-svn-release/runtime/libprofile'
> llvm[2]: Compiling BasicBlockTracing.ll to BasicBlockTracing.bc for
> Release build (bytecode)
> /home/melis/c/llvm-svn-release/Release/bin/llvm-as:
> /home/melis/c/llvm-svn-release/runtime/libprofile/Release/ 
> BasicBlockTracing.ll:1:2:
> error: expected top-level entity
>    .file    "BasicBlockTracing.c"
>    ^
> /home/melis/c/llvm-svn-release/Release/bin/opt: Invalid bitcode  
> signature
> make[2]: ***
> [/home/melis/c/llvm-svn-release/runtime/libprofile/Release/ 
> BasicBlockTracing.bc]
> Error 1
> make[2]: Leaving directory
> `/home/melis/c/llvm-svn-release/runtime/libprofile'
> make[1]: *** [libprofile/.makeall] Error 2
> make[1]: Leaving directory `/home/melis/c/llvm-svn-release/runtime'
> make: *** [all] Error 1
>
> It seems BasicBlockTracing.ll contains assembly instead of LLVM IR.
>
>
> Also, llvm-gcc also doesn't build for me. Here the error is related to
> exception handling:
>
> make[3]: Leaving directory `/home/melis/c/llvm-gcc-svn-build/ 
> libdecnumber'
> make[3]: Entering directory `/home/melis/c/llvm-gcc-svn-build/gcc'
> /home/melis/local/bin/gcc -c   -g -DIN_GCC   -W -Wall -Wwrite-strings
> -Wstrict-prototypes -Wmissing-prototypes -pedantic -Wno-long-long
> -Wno-variadic-macros -Wno-overlength-strings -Wold-style-definition
> -Wmissing-format-attribute -fno-common   -DHAVE_CONFIG_H -I. -I.
> -I../../llvm-gcc-svn/gcc -I../../llvm-gcc-svn/gcc/.
> -I../../llvm-gcc-svn/gcc/../include
> -I../../llvm-gcc-svn/gcc/../libcpp/include
> -I../../llvm-gcc-svn/gcc/../libdecnumber -I../libdecnumber
> gtype-desc.c -o gtype-desc.o
> gtype-desc.c:5862: error: 'sjlj_fc_type_node' undeclared here (not
> in a
> function)
> make[3]: *** [gtype-desc.o] Error 1
> make[3]: Leaving directory `/home/melis/c/llvm-gcc-svn-build/gcc'
> make[2]: *** [all-stage1-gcc] Error 2
> make[2]: Leaving directory `/home/melis/c/llvm-gcc-svn-build'
> make[1]: *** [stage1-bubble] Error 2
> make[1]: Leaving directory `/home/melis/c/llvm-gcc-svn-build'
> make: *** [all] Error 2
>
> Is there something suddenly wrong with my system (which hasn't changed
> in any way)? I used to be able to build LLVM(GCC) without problems a  
> few
> weeks ago...
>
SJLJ exception handling is only implemented on ARM Darwin. It looks  
like your target is pulling in bits of that?
> Regards,
> Paul
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev