llvm dev - Jan 2018 - How to debug a test that fails only on some build bots?

I tried to land a small fix before going on vacation at the end of the
year, but I had to revert it because it broke on a few of the build bots,
specifically clang-cmake-thumbv7-a15, clang-cmake-armv7-a15, and
llvm-hexagon-elf.

https://reviews.llvm.org/D41264

It seemed to work as expected on the others.

Unfortunately, the build logs don't give many clues.  The fix included a
new test, and that test failed with an exit code of 1 but with no
messages.  For example:

FAIL: LLVM :: DebugInfo/void-typedef.ll (14416 of 23113)
******************** TEST 'LLVM :: DebugInfo/void-typedef.ll' FAILED
********************
Script:
--
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/llc
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
-o - 2>&1 |
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/FileCheck
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
--
Exit Code: 1


********************

I would have expected FileCheck messages telling me how the output
differed from expectations.  Since I don't see any of that, I'm
assuming there was a problem with the command itself, but I'm not sure
why that would happen on just a few buildbots.

I haven't been able to reproduce the failures locally on Linux or
Windows, so I'm not sure how to go about debugging this.  Any tips?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180104/574d045d/attachment.html>

Here are a couple of strategies that helped me in the past:

- Try to copy the cmake invocation from the bot's logs and reproduce it
locally.

  --> Particularly the bots you list I think all don't build an X86
backend, so maybe that's it.

- Ask for help from the maintainer of the bot
- [last resort] Commit a change that duplicates the failing command without
piping it through FileCheck, so you can see the output in the bot's log.

-- adrian
> On Jan 4, 2018, at 4:51 PM, Adrian McCarthy via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> I tried to land a small fix before going on vacation at the end of the
year, but I had to revert it because it broke on a few of the build bots,
specifically clang-cmake-thumbv7-a15, clang-cmake-armv7-a15, and
llvm-hexagon-elf.
> 
> https://reviews.llvm.org/D41264 <https://reviews.llvm.org/D41264>
> 
> It seemed to work as expected on the others.
> 
> Unfortunately, the build logs don't give many clues.  The fix included
a new test, and that test failed with an exit code of 1 but with no messages. 
For example:
> FAIL: LLVM :: DebugInfo/void-typedef.ll (14416 of 23113)
> ******************** TEST 'LLVM :: DebugInfo/void-typedef.ll'
FAILED ********************
> Script:
> --
> /local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/llc
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
-o - 2>&1 |
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/FileCheck
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
> --
> Exit Code: 1
> 
> 
> ********************
> I would have expected FileCheck messages telling me how the output differed
from expectations.  Since I don't see any of that, I'm assuming there
was a problem with the command itself, but I'm not sure why that would
happen on just a few buildbots.
> 
> I haven't been able to reproduce the failures locally on Linux or
Windows, so I'm not sure how to go about debugging this.  Any tips?
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180104/04ef38c4/attachment.html>

On 1/4/2018 4:51 PM, Adrian McCarthy via llvm-dev wrote:
> I tried to land a small fix before going on vacation at the end of the 
> year, but I had to revert it because it broke on a few of the build 
> bots, specifically clang-cmake-thumbv7-a15, clang-cmake-armv7-a15, and 
> llvm-hexagon-elf.
>
> https://reviews.llvm.org/D41264
I think you meant to put the test into test/DebugInfo/X86/ ?  Your test 
specifies an x86 target, but those buildbots don't build the x86 target.
>
> It seemed to work as expected on the others.
>
> Unfortunately, the build logs don't give many clues.  The fix included 
> a new test, and that test failed with an exit code of 1 but with no 
> messages.  For example:
> FAIL: LLVM :: DebugInfo/void-typedef.ll (14416 of 23113) 
> ******************** TEST 'LLVM :: DebugInfo/void-typedef.ll'
FAILED
> ******************** Script: -- 
> /local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/llc 
>
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
> -o - 2>&1 | 
>
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/FileCheck
>
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
> -- Exit Code: 1 ********************
llc is probably producing some error message like "error: unable to get 
target for 'i686', see --version and --triple.", but you're
redirecting
stderr, so it isn't visible.  You should try to avoid this when possible.

Your test doesn't have any positive CHECK lines, so FileCheck passes.  
You should generally try to include some positive CHECK line in 
regression tests.  And you never need to check for the string "Assertion 
failed"; the test will fail if any command on the RUN line fails or
crashes.

-Eli

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux
Foundation Collaborative Project

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180104/3b54adc2/attachment.html>

Thanks!  It looks like this is sorted out now.

On Thu, Jan 4, 2018 at 5:07 PM, Friedman, Eli <efriedma at codeaurora.org>
wrote:
> On 1/4/2018 4:51 PM, Adrian McCarthy via llvm-dev wrote:
>
> I tried to land a small fix before going on vacation at the end of the
> year, but I had to revert it because it broke on a few of the build bots,
> specifically clang-cmake-thumbv7-a15, clang-cmake-armv7-a15, and
> llvm-hexagon-elf.
>
> https://reviews.llvm.org/D41264
>
>
> I think you meant to put the test into test/DebugInfo/X86/ ?  Your test
> specifies an x86 target, but those buildbots don't build the x86
target.
>
>
> It seemed to work as expected on the others.
>
> Unfortunately, the build logs don't give many clues.  The fix included
a
> new test, and that test failed with an exit code of 1 but with no
> messages.  For example:
>
> FAIL: LLVM :: DebugInfo/void-typedef.ll (14416 of 23113)
> ******************** TEST 'LLVM :: DebugInfo/void-typedef.ll'
FAILED ********************
> Script:
> --
> /local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/llc
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
-o - 2>&1 |
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/FileCheck
/local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
> --
> Exit Code: 1
>
>
> ********************
>
>
> llc is probably producing some error message like "error: unable to
get
> target for 'i686', see --version and --triple.", but
you're redirecting
> stderr, so it isn't visible.  You should try to avoid this when
possible.
>
> Your test doesn't have any positive CHECK lines, so FileCheck passes. 
You
> should generally try to include some positive CHECK line in regression
> tests.  And you never need to check for the string "Assertion
failed"; the
> test will fail if any command on the RUN line fails or crashes.
>
> -Eli
>
> --
> Employee of Qualcomm Innovation Center, Inc.
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux
Foundation Collaborative Project
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180105/6b62516f/attachment.html>