llvm dev - Feb 2019 - Buildbots do not detect test-suite failures!!!

It has just been brought to my attention that our buildbots no longer
report failures to compile/link files in the test-suite as failures for the
build. As a result, we have at least one bot that is currently showing a
green status, yet it aborts when building 3 test cases.

I assume that something has changed with how (presumably LNT) reports these
failures and the Zorg code does not adequately detect this (I think the
buildbot scripts grep for a string such as "FAIL:" in the log or
something
along those lines). Previously, a failure to compile would show up as two
failures (a failure at compile time and a failure at link time). Now it
just shows up as NOEXE.

This really seems like something we should fix ASAP.

For an example:
http://lab.llvm.org:8011/builders/clang-s390x-linux-lnt/builds/11252/steps/test-suite/logs/stdio

The Vector-build test case shows up as NOEXE and we're clearly hitting an
UNREACHABLE while compiling the test:

Unexpected node to lower
UNREACHABLE executed at ...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190214/86edfdc7/attachment.html>

I have sent this a bit over a month ago and got no response. Does anyone
know if this issue has been resolved? I hope that we are no longer in a
situation where we may have LNT failures on build bots that aren't reported
as failures.

On Thu, Feb 14, 2019 at 6:30 AM Nemanja Ivanovic <nemanja.i.ibm at
gmail.com>
wrote:
> It has just been brought to my attention that our buildbots no longer
> report failures to compile/link files in the test-suite as failures for the
> build. As a result, we have at least one bot that is currently showing a
> green status, yet it aborts when building 3 test cases.
>
> I assume that something has changed with how (presumably LNT) reports
> these failures and the Zorg code does not adequately detect this (I think
> the buildbot scripts grep for a string such as "FAIL:" in the log
or
> something along those lines). Previously, a failure to compile would show
> up as two failures (a failure at compile time and a failure at link time).
> Now it just shows up as NOEXE.
>
> This really seems like something we should fix ASAP.
>
> For an example:
>
>
http://lab.llvm.org:8011/builders/clang-s390x-linux-lnt/builds/11252/steps/test-suite/logs/stdio
>
> The Vector-build test case shows up as NOEXE and we're clearly hitting
an
> UNREACHABLE while compiling the test:
>
> Unexpected node to lower
> UNREACHABLE executed at ...
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190320/a615b2ed/attachment.html>

Speaking of LNT, we've had a lot of trouble setting it up locally.  We
can get it to run basic pass/fail tests all right, but any performance
tracking is pretty hopeless.  From talking with others, their
experiences seem to have been the same.

It would be nice to have a non-bespoke system to test larger
applications and track performance for them.  Has anyone looked into
options?  Would such an effort be welcome, or are people happy with LNT?

                        -David

Nemanja Ivanovic via llvm-dev <llvm-dev at lists.llvm.org> writes:
> I have sent this a bit over a month ago and got no response. Does anyone
know if this issue has been resolved? I hope that we are no longer in a
situation where
> we may have LNT failures on build bots that aren't reported as
failures.
>
> On Thu, Feb 14, 2019 at 6:30 AM Nemanja Ivanovic <nemanja.i.ibm at
gmail.com> wrote:
>
>  It has just been brought to my attention that our buildbots no longer
report failures to compile/link files in the test-suite as failures for the
build. As a result, we
>  have at least one bot that is currently showing a green status, yet it
aborts when building 3 test cases.
>
>  I assume that something has changed with how (presumably LNT) reports
these failures and the Zorg code does not adequately detect this (I think the
>  buildbot scripts grep for a string such as "FAIL:" in the log or
something along those lines). Previously, a failure to compile would show up as
two failures (a
>  failure at compile time and a failure at link time). Now it just shows up
as NOEXE.
>
>  This really seems like something we should fix ASAP.
>
>  For an example:
> 
http://lab.llvm.org:8011/builders/clang-s390x-linux-lnt/builds/11252/steps/test-suite/logs/stdio
>
>  The Vector-build test case shows up as NOEXE and we're clearly hitting
an UNREACHABLE while compiling the test:
> Unexpected node to lower
> UNREACHABLE executed at ...
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev