[LLVMdev] Buildbots: Apology and Explanation

Bill Wendling
Fri, 28 Aug 2009 23:05:50 -0000

I need to apologize to many of you, especially Gabor, for my vitriol
over buildbot breakages. I get very enraged by breakages that occur
over long periods of time. Here's an attempt to explain why I fly off
the handle at these.

Before we had buildbots, a random patch would break the system. It
would sometimes take a whole day to determine which patch broke it. I
came up with the idea of using builbots. We started using them
internally at Apple (unfortunately, those machines are still
unavailable to non-Apple employees). It quickly spread so that some
were set up at Google. It was a huge win for us.

However, it's only a huge win if people pay attention to the error
reports. People didn't. At first, there were a lot of "false
negatives". Those were fixed. Then the reporting scheme wasn't great.
Those were also fixed. Still, people would break the builds and let
things go for hours or days at a time. This feels like a slap in the
face to me. After so much hard work, people are ignoring the one piece
of quality assurance that we are afforded in the LLVM community. In
the meantime, many many patches are going in which aren't being
tested. If something breaks after the initial breakage, then it's that
much harder to determine what's going on. It's a lot of wasted effort.

Couple that with the fact that we at Apple (and other place, to be
sure) rely upon ToT to be fairly stable so that we can create branches
from ToT at a moment's notice, and you get a really bad and stressful
situation.

So, yes, I take it personally when breakages are ignored. It's an
unfortunate part of my personality that I'm trying to work on, but on
occasion gets the better of me, and I lash out.

Again, my apologies. Please watch what the buildbots have to say. Even
if you can't see the log file message, try to see if there's something
you might have missed. Someone with access to the Apple-only machines
would be happy to forward the error message to you if you think your
patch might be the cause.

Sincerely,
Bill (Monkeypox)