llvm dev - Sep 2016 - [cfe-dev] Buildbot General Failure - Production Stop?

On 5 September 2016 at 23:04, Krzysztof Parzyszek
<kparzysz at codeaurora.org> wrote:
> Let's first see how bad it is once bots are fixed to build the latest
> revision.  It's only been a few days and that includes a weekend.
Of course. I'll wait until all our bots return from the first round to
know the size of the damage.

I just wanted to warn all other buildbot owners and general public
that we do have a tough situation going, and if their patches are not
essential, maybe voluntarily hold on for a while would be a good idea.
(Though, now that I read my email, it does sound alarmist. I'm in
panic mode right now, so I apologise :).

But all other affected bots won't come back until Galina reverts and
restart the master (or until their owners work around like I did).
There's no way of knowing how bad it is if the bots continue chugging
bogus green status to every commit...

cheers,
--renato

Hi Renato, do we have an idea which was the latest commit that the buildbots
(really) tested?

- V. Kalintiris
________________________________________
From: cfe-dev [cfe-dev-bounces at lists.llvm.org] on behalf of Renato Golin via
cfe-dev [cfe-dev at lists.llvm.org]
Sent: 05 September 2016 23:24
To: Krzysztof Parzyszek
Cc: LLVM Dev; Clang Dev
Subject: Re: [cfe-dev] [llvm-dev] Buildbot General Failure - Production Stop?

On 5 September 2016 at 23:04, Krzysztof Parzyszek
<kparzysz at codeaurora.org> wrote:
> Let's first see how bad it is once bots are fixed to build the latest
> revision.  It's only been a few days and that includes a weekend.
Of course. I'll wait until all our bots return from the first round to
know the size of the damage.

I just wanted to warn all other buildbot owners and general public
that we do have a tough situation going, and if their patches are not
essential, maybe voluntarily hold on for a while would be a good idea.
(Though, now that I read my email, it does sound alarmist. I'm in
panic mode right now, so I apologise :).

But all other affected bots won't come back until Galina reverts and
restart the master (or until their owners work around like I did).
There's no way of knowing how bad it is if the bots continue chugging
bogus green status to every commit...

cheers,
--renato
_______________________________________________
cfe-dev mailing list
cfe-dev at lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

On 5 September 2016 at 23:28, Vasileios Kalintiris
<Vasileios.Kalintiris at imgtec.com> wrote:
> Hi Renato, do we have an idea which was the latest commit that the
buildbots (really) tested?
No idea, as Galina said SVN was out of sync on the 2nd (18pm her local
time). I'm guessing the problem already existed by then for a long
time.

Right now I'm trying to put out the fires, not worry about the range.
Our first bot [1] broke lots of tests, and I was lucky to find commits
in those areas since the 2nd, so I already fired the email and I
expect those patches to be preemptively reverted.

Also, ARM can't self-host anymore [2], and I'm trying 280555 which is
around the time of her email. If that fails, I'll be going back ~100
commits or so at a time.

Our other bots are falling down like rotten apples [3], and I'll be
going through one at a time... :(

--renato

[1] http://lab.llvm.org:8011/builders/clang-cmake-armv7-a15/builds/14952
[2] http://lab.llvm.org:8011/builders/clang-cmake-armv7-a15-selfhost/builds/7685
[3] http://llvm.tcwglab.linaro.org/monitor/

On Mon, Sep 5, 2016 at 3:29 PM Vasileios Kalintiris via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi Renato, do we have an idea which was the latest commit that the
> buildbots (really) tested?
>
> - V. Kalintiris
>
I can tell you that at least for one of the buildbots I follow, this
<http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/15319> is
the last build that was actually tested.  You can see the slave losing
connection with the master during the "ninja check 2" stage, and if
you
change the URL to look at build 15320 (the next build) you'll see that the
"build stage 1" drops to 0 seconds and remains that way on subsequent
builds.

So, somewhere around SVN r280562 (Friday 17:38 Pacific Time) is going to be
the last tested build, but each bot will be on a slightly different
revision around that time.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160906/6f76c7d5/attachment.html>