llvm dev - Jan 2019 - Help reproducing buildbot failures

Hi,

(Reposting the request to llvm-dev for better visibility)

I had to revert a change today, due to test errors on some buildbots. The 
errors did seem to only happen on the ppc64be and aarch64 buildbots. I 
don't have access to a ppc64be setup to test on, but I've tried to build
and run in a setup very similar to the one on the aarch64 buildbot, but I 
can't reproduce the errors there.

The failing test was present in SVN r351801 to r351810, in 
test/tools/llvm-objcopy/COFF/add-gnu-debuglink.test. The failing test 
looked like this:

http://lab.llvm.org:8011/builders/clang-ppc64be-linux/builds/29113/steps/ninja%20check%201/logs/FAIL%3A%20LLVM%3A%3Aadd-gnu-debuglink.test
http://lab.llvm.org:8011/builders/clang-cmake-armv8-quick/builds/9987/steps/ninja%20check%201/logs/FAIL%3A%20LLVM%3A%3Aadd-gnu-debuglink.test

Can someone with access to the buildbots (or similar systems) help 
reproducing the issue? The test output files and the full output of the 
commands in the tests would be helpful.

// Martin


On Tue, 22 Jan 2019, Martin Storsjö wrote:
> Hi,
>
> My newly added test seems to be failing on a few of the buildbots, but 
> working fine on others (including all my local test environments). Can any
of
> the buildbot maintainers help look into the issue? The issue on these 
> buildbots look like this:
>
>
http://lab.llvm.org:8011/builders/clang-ppc64be-linux-lnt/builds/23635/steps/ninja%20check%201/logs/FAIL%3A%20LLVM%3A%3Aadd-gnu-debuglink.test
>
> To get a grasp of the issue, are you able to provide the temporary output 
> files of the test, and the full output of the llvm-readobj and llvm-objdump
> commands in the test?
>
> Regards,
>
> // Martin
>
> On Tue, 22 Jan 2019, llvm.buildmaster at lab.llvm.org wrote:
>
>> The Buildbot has detected a new failure on builder
clang-cmake-armv8-quick
>> while building llvm.
>> Full details are available at:
>> http://lab.llvm.org:8011/builders/clang-cmake-armv8-quick/builds/9987
>> 
>> Buildbot URL: http://lab.llvm.org:8011/
>> 
>> Buildslave for this Build: linaro-armv8-01-arm-quick
>> 
>> Build Reason: scheduler
>> Build Source Stamp: [branch trunk] 351801
>> Blamelist: jhenderson,mstorsjo
>> 
>> BUILD FAILED: failed ninja check 1
>> 
>> sincerely,
>> -The Buildbot
>

On Tue, 22 Jan 2019, Martin Storsjö via llvm-dev wrote:
> Hi,
>
> (Reposting the request to llvm-dev for better visibility)
>
> I had to revert a change today, due to test errors on some buildbots. The 
> errors did seem to only happen on the ppc64be and aarch64 buildbots. I
don't
> have access to a ppc64be setup to test on, but I've tried to build and
run in
> a setup very similar to the one on the aarch64 buildbot, but I can't 
> reproduce the errors there.
>
> The failing test was present in SVN r351801 to r351810, in 
> test/tools/llvm-objcopy/COFF/add-gnu-debuglink.test. The failing test
looked
> like this:
>
>
http://lab.llvm.org:8011/builders/clang-ppc64be-linux/builds/29113/steps/ninja%20check%201/logs/FAIL%3A%20LLVM%3A%3Aadd-gnu-debuglink.test
>
http://lab.llvm.org:8011/builders/clang-cmake-armv8-quick/builds/9987/steps/ninja%20check%201/logs/FAIL%3A%20LLVM%3A%3Aadd-gnu-debuglink.test
>
> Can someone with access to the buildbots (or similar systems) help 
> reproducing the issue? The test output files and the full output of the 
> commands in the tests would be helpful.
I realized I hopefully should be able to debug the issue via the normal 
testing/buildbot infrastructure, by adding a few more commands in the test 
that print things to stdout. If the test runs successfully it shouldn't be 
visible anywhere, and if it fails, it'll give me further clues about what 
actually went wrong.

I apologize for the extra commit churn this might end up with, but 
hopefully it'll let me figure this out without waiting for help from the 
buildbot owners.

// Martin

On Wed, 23 Jan 2019, Martin Storsjö via llvm-dev wrote:
> I realized I hopefully should be able to debug the issue via the normal 
> testing/buildbot infrastructure, by adding a few more commands in the test 
> that print things to stdout. If the test runs successfully it shouldn't
be
> visible anywhere, and if it fails, it'll give me further clues about
what
> actually went wrong.
I managed to solve the issue this way, so no need to manually run tests 
and extract outputs any longer. Thanks and sorry for the noise.

// Martin