llvm dev - Apr 2014 - [LLVMdev] Need help reproducing a sanitizer buildbot failure

I recently broke a sanitizer buildbot but I am unable to reproduce the
failure. The buildbot that failed is


http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/2959


I looked around in the logs, looking for the config/build commands that
would reproduce the failure. The problem is that the bot seems to be using
a script which I don't have access to (buildbot_bootstrap.sh). So, I looked
for cmake/ninja calls. AFAICT, the bot does a bootstrap and then runs the
testsuite with the final build configured with -DLLVM_USE_SANITIZER=Memory.

I tried that locally, but the build dies very early in tblgen:

FAILED: cd /ssd/dnovillo/llvm/bld/tools/clang/include/clang/Driver &&
/ssd/dnovillo/llvm/bld/bin/llvm-tblgen -gen-opt-parser-defs -I
/ssd/dnovillo/llvm/llvm/tools/clang/include/clang/Driver -I
/ssd/dnovillo/llvm/llvm/lib/Target -I /ssd/dnovillo/llvm/llvm/include
/ssd/dnovillo/llvm/llvm/tools/clang/include/clang/Driver/Options.td -o
/ssd/dnovillo/llvm/bld/tools/clang/include/clang/Driver/Options.inc.tmp
==12630== WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7f1ea9b01729 (/ssd/dnovillo/llvm/bld/bin/llvm-tblgen+0x8f729)
    #1 0x7f1ea9a9cbe0 (/ssd/dnovillo/llvm/bld/bin/llvm-tblgen+0x2abe0)
    #2 0x7f1ea9f7b69c (/ssd/dnovillo/llvm/bld/bin/llvm-tblgen+0x50969c)
    #3 0x7f1ea82356ff (/lib/x86_64-linux-gnu/libc.so.6+0x216ff)
    #4 0x7f1ea9b01280 (/ssd/dnovillo/llvm/bld/bin/llvm-tblgen+0x8f280)

SUMMARY: MemorySanitizer: use-of-uninitialized-value ??:0 ??
Exiting


I'm not sure how to proceed from here. The bot is clearly building things
in a different way, but I don't know how to duplicate it. Is there a  way
for me to use the same script that the bot is using? What is the general
advice on reproducing buildbot failures?


Thanks. Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140410/0ce6d771/attachment.html>

msan isn't usable without an instrumented C++ standard library.

The script in question is here:
https://code.google.com/p/address-sanitizer/source/browse/trunk/build/scripts/slave/buildbot_bootstrap.sh

They appear to use a prebuilt libstdc++ shared object.

I think the bot usually gives a readable error report, but it doesn't work
for this test because the test is passing stderr to FileCheck.  Lots of
tests do that, and we should find a way to make that work.  We might want
to pass MSAN_OPTIONS=log_path=/tmp/something.log and then cat that file
from lit if it's non-empty.


On Thu, Apr 10, 2014 at 12:45 PM, Diego Novillo <dnovillo at google.com>
wrote:
>
> I recently broke a sanitizer buildbot but I am unable to reproduce the
> failure. The buildbot that failed is
>
>
>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/2959
>
>
> I looked around in the logs, looking for the config/build commands that
> would reproduce the failure. The problem is that the bot seems to be using
> a script which I don't have access to (buildbot_bootstrap.sh). So, I
looked
> for cmake/ninja calls. AFAICT, the bot does a bootstrap and then runs the
> testsuite with the final build configured with -DLLVM_USE_SANITIZER=Memory
> .
>
> I tried that locally, but the build dies very early in tblgen:
>
> FAILED: cd /ssd/dnovillo/llvm/bld/tools/clang/include/clang/Driver
&&
> /ssd/dnovillo/llvm/bld/bin/llvm-tblgen -gen-opt-parser-defs -I
> /ssd/dnovillo/llvm/llvm/tools/clang/include/clang/Driver -I
> /ssd/dnovillo/llvm/llvm/lib/Target -I /ssd/dnovillo/llvm/llvm/include
> /ssd/dnovillo/llvm/llvm/tools/clang/include/clang/Driver/Options.td -o
> /ssd/dnovillo/llvm/bld/tools/clang/include/clang/Driver/Options.inc.tmp
> ==12630== WARNING: MemorySanitizer: use-of-uninitialized-value
>     #0 0x7f1ea9b01729 (/ssd/dnovillo/llvm/bld/bin/llvm-tblgen+0x8f729)
>     #1 0x7f1ea9a9cbe0 (/ssd/dnovillo/llvm/bld/bin/llvm-tblgen+0x2abe0)
>     #2 0x7f1ea9f7b69c (/ssd/dnovillo/llvm/bld/bin/llvm-tblgen+0x50969c)
>     #3 0x7f1ea82356ff (/lib/x86_64-linux-gnu/libc.so.6+0x216ff)
>     #4 0x7f1ea9b01280 (/ssd/dnovillo/llvm/bld/bin/llvm-tblgen+0x8f280)
>
> SUMMARY: MemorySanitizer: use-of-uninitialized-value ??:0 ??
> Exiting
>
>
> I'm not sure how to proceed from here. The bot is clearly building
things
> in a different way, but I don't know how to duplicate it. Is there a 
way
> for me to use the same script that the bot is using? What is the general
> advice on reproducing buildbot failures?
>
>
> Thanks. Diego.
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140410/a175d736/attachment.html>

On Thu, Apr 10, 2014 at 4:43 PM, Reid Kleckner <rnk at google.com> wrote:
> msan isn't usable without an instrumented C++ standard library.
>
> The script in question is here:
>
>
https://code.google.com/p/address-sanitizer/source/browse/trunk/build/scripts/slave/buildbot_bootstrap.sh
>
Thanks, Reid. I've gotten the script and I'm now running it locally.
It's
running into trouble in llvm_build2_asan, so I'll have to kick it a bit
first.


I think the bot usually gives a readable error report, but it doesn't
work
> for this test because the test is passing stderr to FileCheck.  Lots of
> tests do that, and we should find a way to make that work.  We might want
> to pass MSAN_OPTIONS=log_path=/tmp/something.log and then cat that file
> from lit if it's non-empty.
>
Yeah. From the output, I can see part of the msan failure, but I can't see
the whole thing. Worse, I can't even replicate it.


Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140410/8840c9a8/attachment.html>