llvm dev - Dec 2012 - [LLVMdev] FNT testers reporting success even though they failed

All the LLVM lab FNT builders were reporting failures in the same set of tests
(scimark2, LivermooreLoops, some others).  Now they've all turned green but
the
same tests are still failing.  I don't see any commits to deliberately make
the
FNT pass, so it looks like someone broke failure detection...  Note that the
FNT "report" text became empty at the same time.  Here's what
"report" was like
before:

 
http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12575/steps/make.test-suite/logs/report

and after:

http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576/steps/make.test-suite/logs/report

The first green builds are:
   http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576
   http://lab.llvm.org:8011/builders/clang-x86_64-linux-fnt/builds/57
  
http://lab.llvm.org:8011/builders/dragonegg-x86_64-linux-gcc-4.6-fnt/builds/1983

Does anyone know what might have caused this?

Best wishes, Duncan.

On Tue, Dec 11, 2012 at 1:27 AM, Duncan Sands <baldrick at free.fr>
wrote:
> All the LLVM lab FNT builders were reporting failures in the same set of
> tests
> (scimark2, LivermooreLoops, some others).  Now they've all turned green
but
> the
> same tests are still failing.  I don't see any commits to deliberately
make
> the
> FNT pass, so it looks like someone broke failure detection...  Note that
the
> FNT "report" text became empty at the same time.  Here's what
"report" was
> like
> before:
>
>
>
http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12575/steps/make.test-suite/logs/report
>
> and after:
>
>
http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576/steps/make.test-suite/logs/report
>
> The first green builds are:
>   http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576
>   http://lab.llvm.org:8011/builders/clang-x86_64-linux-fnt/builds/57
>
>
http://lab.llvm.org:8011/builders/dragonegg-x86_64-linux-gcc-4.6-fnt/builds/1983
>
> Does anyone know what might have caused this?
[Thought it might've been my change (r169769) but that doesn't look
like it intersects with this particular test task]

I think you're on the right track with the logs. If the log that
reaches the buildmaster is empty then we won't find any failures &
we'll report success.

Have you checked any of the slaves to see if the log files are empty
in the builds themselves?

Have you run a local execution of the test suite uisng roughly the
same commands & observed whether the reports are empty coming out of
that?

- David

Hello everyone,

It might make sense to start from rebooting the buildmaster, just to make sure
everything is all right on this end.
Yesterday I have tried to apply the latest changes from zorg and some of them
are broken.
Theoretically, checkconfig shouldn't affect the working instance, but the
reality could be different...
I planned to rollback to the last known-to-be-good revision and restart the
master in the evening, but could do this sooner if you feel like this is urgent.

Please let me know.

Thanks

Galina





-----Original Message-----
From: llvm-lab-wg-bounces at lists.minormatter.com [mailto:llvm-lab-wg-bounces
at lists.minormatter.com] On Behalf Of David Blaikie
Sent: Tuesday, December 11, 2012 9:12 AM
To: Duncan Sands
Cc: llvm-lab-wg at lists.minormatter.com; Galina Kistanova; llvmdev at
cs.uiuc.edu
Subject: Re: [llvm-lab-wg] FNT testers reporting success even though they failed

On Tue, Dec 11, 2012 at 1:27 AM, Duncan Sands <baldrick at free.fr>
wrote:
> All the LLVM lab FNT builders were reporting failures in the same set of
> tests
> (scimark2, LivermooreLoops, some others).  Now they've all turned green
but
> the
> same tests are still failing.  I don't see any commits to deliberately
make
> the
> FNT pass, so it looks like someone broke failure detection...  Note that
the
> FNT "report" text became empty at the same time.  Here's what
"report" was
> like
> before:
>
>
>
http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12575/steps/make.test-suite/logs/report
>
> and after:
>
>
http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576/steps/make.test-suite/logs/report
>
> The first green builds are:
>   http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576
>   http://lab.llvm.org:8011/builders/clang-x86_64-linux-fnt/builds/57
>
>
http://lab.llvm.org:8011/builders/dragonegg-x86_64-linux-gcc-4.6-fnt/builds/1983
>
> Does anyone know what might have caused this?
[Thought it might've been my change (r169769) but that doesn't look
like it intersects with this particular test task]

I think you're on the right track with the logs. If the log that
reaches the buildmaster is empty then we won't find any failures &
we'll report success.

Have you checked any of the slaves to see if the log files are empty
in the builds themselves?

Have you run a local execution of the test suite uisng roughly the
same commands & observed whether the reports are empty coming out of
that?

- David
_______________________________________________
llvm-lab-wg mailing list
llvm-lab-wg at lists.minormatter.com
http://minormatter.com/cgi-bin/mailman/listinfo/llvm-lab-wg