llvm dev - Sep 2020 - Flakey failure on clang-ppc64le-linux-multistage

Seems there were a couple of correlated failures that appear to be flakes
on this buildbot recently:

green:
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13974
red:
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13975
(target-override.c
during stage 1, seems to be missing the directory/symlink it just created)
red:
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13976
(same
test failure as the last, but during stage 2, not stage 1)
green:
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13977

Including Nico & Pavlov as the people who wrote/edited the test, but I'm
guessing this is something interesting going on on the buildbot itself?

powerllvm at ca.ibm.com, whoever you are on the end of that mailing list -
could you take a look at this? Possibly manually running that test in a
loop a bunch of times to see if it fails sometimes & try to help us
understand why?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200901/c91743b9/attachment.html>

<div class="socmaildefaultfont" dir="ltr"
style="font-family:Arial, Helvetica, sans-serif;font-size:10pt"
><div dir="ltr" >Interesting, thanks for bringing this to our
attention. I just took a quick look through the last 100 builds and this test
has failed 13 times. This is certainly something we need to look at. We will
investigate and see if we can make any sense of this.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Nemanja Ivanovic<br>LLVM PPC Backend
Development<br>IBM Toronto Lab<br>Email:
nemanjai@ca.ibm.com<br>Phone: 905-413-3388</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr"
style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px;
direction:ltr; margin-right:0px" >----- Original message
-----<br>From: David Blaikie <dblaikie@gmail.com><br>To:
llvm-dev <llvm-dev@lists.llvm.org>, Nico Weber
<thakis@chromium.org>, Serge Pavlov <sepavloff@gmail.com>,
powerllvm@ca.ibm.com<br>Cc:<br>Subject: [EXTERNAL] Flakey failure on
clang-ppc64le-linux-multistage<br>Date: Tue, Sep 1, 2020 6:10 PM<br>
<div dir="ltr" >Seems there were a couple of correlated failures
that appear to be flakes on this buildbot recently:<br><br>green:
<a
href="http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13974"
target="_blank">http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13974</a><br>red:
<a
href="http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13975"
target="_blank">http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13975</a>
(target-override.c during stage 1, seems to be missing the directory/symlink it
just created)<br>red: <a
href="http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13976"
target="_blank">http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13976</a>
(same test failure as the last, but during stage 2, not stage 1)<br>green:
<a
href="http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13977"
target="_blank">http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13977</a><br><br>Including
Nico & Pavlov as the people who wrote/edited the test, but I'm guessing
this is something interesting going on on the buildbot
itself?<br><br><a href="mailto:powerllvm@ca.ibm.com"
target="_blank">powerllvm@ca.ibm.com</a>, whoever you are on
the end of that mailing list - could you take a look at this? Possibly manually
running that test in a loop a bunch of times to see if it fails sometimes &
try to help us understand why?</div></blockquote>
<div dir="ltr" > </div></div><BR>

Well, I am at my wit's end. I have copied over the script and directories
for this test case and run it a few million times. First I was running one
at a time, then I switched to kicking off 1000 at a time. All the while,
the bots continued to run on the same machine. The script never failed even
once. I am not sure if this has something to do with Python as part of
llvm-lit or what is going on.
I am thinking that the best course of action for us is to mark this test
case UNSUPPORTED for PPC.

On Wed, Sep 2, 2020 at 12:41 PM Nemanja Ivanovic via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Interesting, thanks for bringing this to our attention. I just took a
> quick look through the last 100 builds and this test has failed 13 times.
> This is certainly something we need to look at. We will investigate and see
> if we can make any sense of this.
>
> Nemanja Ivanovic
> LLVM PPC Backend Development
> IBM Toronto Lab
> Email: nemanjai at ca.ibm.com
> Phone: 905-413-3388
>
>
>
> ----- Original message -----
> From: David Blaikie <dblaikie at gmail.com>
> To: llvm-dev <llvm-dev at lists.llvm.org>, Nico Weber <thakis at
chromium.org>,
> Serge Pavlov <sepavloff at gmail.com>, powerllvm at ca.ibm.com
> Cc:
> Subject: [EXTERNAL] Flakey failure on clang-ppc64le-linux-multistage
> Date: Tue, Sep 1, 2020 6:10 PM
>
> Seems there were a couple of correlated failures that appear to be flakes
> on this buildbot recently:
>
> green:
>
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13974
> red:
>
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13975
(target-override.c
> during stage 1, seems to be missing the directory/symlink it just created)
> red:
>
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13976
(same
> test failure as the last, but during stage 2, not stage 1)
> green:
>
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/13977
>
> Including Nico & Pavlov as the people who wrote/edited the test, but
I'm
> guessing this is something interesting going on on the buildbot itself?
>
> powerllvm at ca.ibm.com, whoever you are on the end of that mailing list -
> could you take a look at this? Possibly manually running that test in a
> loop a bunch of times to see if it fails sometimes & try to help us
> understand why?
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200902/7a2777d4/attachment.html>