llvm dev - Oct 2015 - Fwd: buildbot failure in LLVM on sanitizer-x86_64-linux-bootstrap

This buildbot seems to have been failing for a while (though it's hard for
me to identify the root cause in the logs, as I mentioned in another
thread, so it's hard to say if it's the same failure, or if the failure
is
consistent, etc) - anyone watching it/caring aobut it?

---------- Forwarded message ----------
From: <llvm.buildmaster at lab.llvm.org>
Date: Wed, Sep 30, 2015 at 10:16 PM
Subject: buildbot failure in LLVM on sanitizer-x86_64-linux-bootstrap
To: Adrian Prantl <aprantl at apple.com>, Ahmed Bougacha <
ahmed.bougacha at gmail.com>, Artyom Skrobov <Artyom.Skrobov at
arm.com>, Bob
Wilson <bob.wilson at apple.com>, Bruno Cardoso Lopes <bruno.cardoso at
gmail.com>,
Chad Rosier <mcrosier at codeaurora.org>, Chris Bieneman <beanz at
apple.com>,
Daniel Jasper <djasper at google.com>, David Blaikie <dblaikie at
gmail.com>,
David Majnemer <david.majnemer at gmail.com>, Davide Italiano <
davide at freebsd.org>, Dehao Chen <dehao at google.com>, Douglas
Gregor <
dgregor at apple.com>, Erik Eckstein <eeckstein at apple.com>, Evgeniy
Stepanov <
eugeni.stepanov at gmail.com>, Fiona Glaser <escha at apple.com>, Hal
Finkel <
hfinkel at anl.gov>, Ivan Krasin <krasin at chromium.org>, Jingyue Wu
<
jingyue at google.com>, Kaelyn Takata <rikka at google.com>, Kostya
Serebryany <
kcc at google.com>, Michael Zolotukhin <mzolotukhin at apple.com>, Reid
Kleckner <
rnk at google.com>, Rui Ueyama <ruiu at google.com>, Sanjay Patel <
spatel at rotateright.com>, Vedant Kumar <vsk at apple.com>
Cc: gkistanova at gmail.com


The Buildbot has detected a new failure on builder
sanitizer-x86_64-linux-bootstrap while building lld.
Full details are available at:

http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8553

Buildbot URL: http://lab.llvm.org:8011/

Buildslave for this Build: sanitizer-buildbot2

Build Reason: scheduler
Build Source Stamp: [branch trunk] 248976
Blamelist:
ab,adrian,askrobov,bruno,bwilson,cbieneman,davide,dblaikie,dehao,dgregor,djasper,eeckstein,escha,eugenis,hfinkel,jingyue,kcc,krasin,majnemer,mcrosier,mzolotukhin,rikka,rnk,ruiu,spatel,vedantk

BUILD FAILED: failed annotate failed check-llvm msan

sincerely,
 -The Buildbot
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151001/62e76e6a/attachment.html>

What is the specific concern?
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8553
shows that "check-llvm" fails under MSan, and several tests fail with
an
actual
MSan error report. Prbably you got the email notification because the slave
was restarted at
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8552,
and
you were added to blame list when the bot failed on the next iteration.


On Thu, Oct 1, 2015 at 12:06 PM, David Blaikie via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> This buildbot seems to have been failing for a while (though it's hard
for
> me to identify the root cause in the logs, as I mentioned in another
> thread, so it's hard to say if it's the same failure, or if the
failure is
> consistent, etc) - anyone watching it/caring aobut it?
>
> ---------- Forwarded message ----------
> From: <llvm.buildmaster at lab.llvm.org>
> Date: Wed, Sep 30, 2015 at 10:16 PM
> Subject: buildbot failure in LLVM on sanitizer-x86_64-linux-bootstrap
> To: Adrian Prantl <aprantl at apple.com>, Ahmed Bougacha <
> ahmed.bougacha at gmail.com>, Artyom Skrobov <Artyom.Skrobov at
arm.com>, Bob
> Wilson <bob.wilson at apple.com>, Bruno Cardoso Lopes <
> bruno.cardoso at gmail.com>, Chad Rosier <mcrosier at
codeaurora.org>, Chris
> Bieneman <beanz at apple.com>, Daniel Jasper <djasper at
google.com>, David
> Blaikie <dblaikie at gmail.com>, David Majnemer <david.majnemer at
gmail.com>,
> Davide Italiano <davide at freebsd.org>, Dehao Chen <dehao at
google.com>,
> Douglas Gregor <dgregor at apple.com>, Erik Eckstein <eeckstein at
apple.com>,
> Evgeniy Stepanov <eugeni.stepanov at gmail.com>, Fiona Glaser <
> escha at apple.com>, Hal Finkel <hfinkel at anl.gov>, Ivan Krasin
<
> krasin at chromium.org>, Jingyue Wu <jingyue at google.com>,
Kaelyn Takata <
> rikka at google.com>, Kostya Serebryany <kcc at google.com>,
Michael Zolotukhin
> <mzolotukhin at apple.com>, Reid Kleckner <rnk at google.com>,
Rui Ueyama <
> ruiu at google.com>, Sanjay Patel <spatel at rotateright.com>,
Vedant Kumar <
> vsk at apple.com>
> Cc: gkistanova at gmail.com
>
>
> The Buildbot has detected a new failure on builder
> sanitizer-x86_64-linux-bootstrap while building lld.
> Full details are available at:
>
>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8553
>
> Buildbot URL: http://lab.llvm.org:8011/
>
> Buildslave for this Build: sanitizer-buildbot2
>
> Build Reason: scheduler
> Build Source Stamp: [branch trunk] 248976
> Blamelist:
>
ab,adrian,askrobov,bruno,bwilson,cbieneman,davide,dblaikie,dehao,dgregor,djasper,eeckstein,escha,eugenis,hfinkel,jingyue,kcc,krasin,majnemer,mcrosier,mzolotukhin,rikka,rnk,ruiu,spatel,vedantk
>
> BUILD FAILED: failed annotate failed check-llvm msan
>
> sincerely,
>  -The Buildbot
>
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>

-- 
Alexey Samsonov
vonosmas at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151001/d76d426f/attachment.html>

On 1 October 2015 at 20:52, Alexey Samsonov via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> What is the specific concern?
>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8553
> shows that "check-llvm" fails under MSan, and several tests fail
with an
> actual
> MSan error report. Prbably you got the email notification because the slave
> was restarted at
>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8552,
> and
> you were added to blame list when the bot failed on the next iteration.
Yup.

We can fix that in two ways:

1. Change the buildmaster to not warn on red->exception->red.

2. Create another identical non-emailing buildmaster and move this bot there.

cheers,
--renato

On Thu, Oct 1, 2015 at 12:52 PM, Alexey Samsonov <vonosmas at gmail.com>
wrote:
> What is the specific concern?
>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8553
> shows that "check-llvm" fails under MSan, and several tests fail
with an
> actual
> MSan error report. Prbably you got the email notification because the
> slave was restarted at
>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8552,
> and
> you were added to blame list when the bot failed on the next iteration.
>
The concern is a combination of:

1) getting noisy emails (as Renato has mentioned - this could be fixed by
an improvement to buildbot (or our config - we're not sure where the issue
lies/can be addressed) to not email on red->purple->red). This is my
proximal concern, since that's the way most of us interact with the
buildbots - IRC and emails.

2) A lesser concern, is the ability to look at the buildbot dashboard
(grid/waterfall views) and have a sense of the state of open issues/points
of concern. With bots consistently red for long durations the dashboards
seem hard to use - it looks like everything's broken all the time &
it's
not clear what are known issues with ownership and what are new/untriaged
issues that need immediate attention.

- David

>
>
> On Thu, Oct 1, 2015 at 12:06 PM, David Blaikie via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> This buildbot seems to have been failing for a while (though it's
hard
>> for me to identify the root cause in the logs, as I mentioned in
another
>> thread, so it's hard to say if it's the same failure, or if the
failure is
>> consistent, etc) - anyone watching it/caring aobut it?
>>
>> ---------- Forwarded message ----------
>> From: <llvm.buildmaster at lab.llvm.org>
>> Date: Wed, Sep 30, 2015 at 10:16 PM
>> Subject: buildbot failure in LLVM on sanitizer-x86_64-linux-bootstrap
>> To: Adrian Prantl <aprantl at apple.com>, Ahmed Bougacha <
>> ahmed.bougacha at gmail.com>, Artyom Skrobov <Artyom.Skrobov at
arm.com>, Bob
>> Wilson <bob.wilson at apple.com>, Bruno Cardoso Lopes <
>> bruno.cardoso at gmail.com>, Chad Rosier <mcrosier at
codeaurora.org>, Chris
>> Bieneman <beanz at apple.com>, Daniel Jasper <djasper at
google.com>, David
>> Blaikie <dblaikie at gmail.com>, David Majnemer
<david.majnemer at gmail.com>,
>> Davide Italiano <davide at freebsd.org>, Dehao Chen <dehao at
google.com>,
>> Douglas Gregor <dgregor at apple.com>, Erik Eckstein
<eeckstein at apple.com>,
>> Evgeniy Stepanov <eugeni.stepanov at gmail.com>, Fiona Glaser
<
>> escha at apple.com>, Hal Finkel <hfinkel at anl.gov>, Ivan
Krasin <
>> krasin at chromium.org>, Jingyue Wu <jingyue at google.com>,
Kaelyn Takata <
>> rikka at google.com>, Kostya Serebryany <kcc at google.com>,
Michael
>> Zolotukhin <mzolotukhin at apple.com>, Reid Kleckner <rnk at
google.com>, Rui
>> Ueyama <ruiu at google.com>, Sanjay Patel <spatel at
rotateright.com>, Vedant
>> Kumar <vsk at apple.com>
>> Cc: gkistanova at gmail.com
>>
>>
>> The Buildbot has detected a new failure on builder
>> sanitizer-x86_64-linux-bootstrap while building lld.
>> Full details are available at:
>>
>>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/8553
>>
>> Buildbot URL: http://lab.llvm.org:8011/
>>
>> Buildslave for this Build: sanitizer-buildbot2
>>
>> Build Reason: scheduler
>> Build Source Stamp: [branch trunk] 248976
>> Blamelist:
>>
ab,adrian,askrobov,bruno,bwilson,cbieneman,davide,dblaikie,dehao,dgregor,djasper,eeckstein,escha,eugenis,hfinkel,jingyue,kcc,krasin,majnemer,mcrosier,mzolotukhin,rikka,rnk,ruiu,spatel,vedantk
>>
>> BUILD FAILED: failed annotate failed check-llvm msan
>>
>> sincerely,
>>  -The Buildbot
>>
>>
>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
>
> --
> Alexey Samsonov
> vonosmas at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151002/025ca1d5/attachment.html>