llvm dev - Sep 2016 - buildbot failure in LLVM on sanitizer-x86_64-linux-fast

> On Sep 1, 2016, at 9:20 PM, Greg Parker <gparker at apple.com> wrote:
> 
>> On Sep 1, 2016, at 9:06 PM, llvm.buildmaster at lab.llvm.org wrote:
>> 
>> The Buildbot has detected a new failure on builder
sanitizer-x86_64-linux-fast while building llvm.
>> Full details are available at:
>>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/16009
>> 
>> Buildbot URL: http://lab.llvm.org:8011/
>> 
>> Buildslave for this Build: sanitizer-buildbot3
>> 
>> Build Reason: scheduler
>> Build Source Stamp: [branch trunk] 280457
>> Blamelist: gparker,hfinkel
>> 
>> BUILD FAILED: failed annotate failed check-llvm asan
>> 
>> sincerely,
>> -The Buildbot
> 
> lit.py:
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/utils/lit/lit/formats/googletest.py:39:
error: unable to discover google-tests in
'/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests':
Command
'['/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests',
'--gtest_list_tests']' returned non-zero exit status 1
> 
> This is a new lit error message introduced by r280455. I expect that the
MCTests executable is hitting an asan error when run to report what tests it
contains. Before r280455 the test silently passed as a result. Now the test
fails.
> 
> It's possible that the MCTests test has been failing to run in this
configuration for a long time. I don't think the buildbot logs for previous
builds are detailed enough to say whether MCTests was running or not.
> 
> I'll try to reproduce locally.
Didn't reproduce on my OS X asan build.

This is probably an old bug in the unittests/MC test code or possibly in the
code that it is testing. I recommend disabling unittests/MC on asan
configurations until it can be investigated, but I don't know how to do
that. Other options are to disable unittests/MC unconditionally, or roll back
r280455 until unittests/MC can be fixed.

Is there someone who can reproduce this who can take it over?


-- 
Greg Parker     gparker at apple.com <mailto:gparker at apple.com>    
Runtime Wrangler


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160902/d6fd2bdb/attachment-0001.html>

> On Sep 1, 2016, at 9:20 PM, Greg Parker <gparker at apple.com> wrote:
> 
>> On Sep 1, 2016, at 9:06 PM, llvm.buildmaster at lab.llvm.org wrote:
>> 
>> The Buildbot has detected a new failure on builder
sanitizer-x86_64-linux-fast while building llvm.
>> Full details are available at:
>>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/16009
> 
> lit.py:
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/utils/lit/lit/formats/googletest.py:39:
error: unable to discover google-tests in
'/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests':
Command
'['/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests',
'--gtest_list_tests']' returned non-zero exit status 1
> 
> This is a new lit error message introduced by r280455. I expect that the
MCTests executable is hitting an asan error when run to report what tests it
contains. Before r280455 the test silently passed as a result. Now the test
fails.
Nine tests on five builders have test failures here. Again, these are old bugs
that were previously undetected.

llvm/unittests/MC
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/12834
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/16015

tsan/tests/rtl/TsanRtlTest
tsan/tests/unit/TsanUnitTest
http://lab.llvm.org:8011/builders/sanitizer_x86_64-freebsd/builds/13492

unittests/DebugInfo/DWARF
unittests/DebugInfo/PDB
unittests/ExecutionEngine/Orc
unittests/ExecutionEngine/MCJIT
unittests/Transforms/IPO
unittests/Transforms/Utils
http://lab.llvm.org:8011/builders/llvm-mips-linux/builds/17616
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/2331

If we need to get these builders clean quickly we can temporarily downgrade the
new error to a warning. Patch:

-------------- next part --------------
A non-text attachment was scrubbed...
Name: downgrade-googletest-error.patch
Type: application/octet-stream
Size: 673 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160902/0ff3a430/attachment.obj>
-------------- next part --------------



-- 
Greg Parker     gparker at apple.com     Runtime Wrangler

> On Sep 2, 2016, at 2:01 AM, Greg Parker via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>> On Sep 1, 2016, at 9:20 PM, Greg Parker <gparker at apple.com>
wrote:
>> 
>>> On Sep 1, 2016, at 9:06 PM, llvm.buildmaster at lab.llvm.org wrote:
>>> 
>>> The Buildbot has detected a new failure on builder
sanitizer-x86_64-linux-fast while building llvm.
>>> Full details are available at:
>>>
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/16009
>> 
>> lit.py:
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/utils/lit/lit/formats/googletest.py:39:
error: unable to discover google-tests in
'/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests':
Command
'['/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests',
'--gtest_list_tests']' returned non-zero exit status 1
>> 
>> This is a new lit error message introduced by r280455. I expect that
the MCTests executable is hitting an asan error when run to report what tests it
contains. Before r280455 the test silently passed as a result. Now the test
fails.
Ivan Krasin wrote:
> 
> URL: http://llvm.org/viewvc/llvm-project?rev=280550&view=rev
> lit: print process output, if getting the list of google-tests failed.
Good idea, Ivan. 

The mips and ppc64le builders haven't rebuilt with this yet.
http://lab.llvm.org:8011/builders/llvm-mips-linux
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage
<http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage>

The freebsd builder rebuilt but provided no new output. Exit code -11 is
presumably SIGSEGV.
http://lab.llvm.org:8011/builders/sanitizer_x86_64-freebsd/builds/13509
> lit.py:
/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.src/utils/lit/lit/formats/googletest.py:41:
error: unable to discover google-tests in
'/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.obj/projects/compiler-rt/lib/tsan/tests/rtl/TsanRtlTest':
Command
'['/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.obj/projects/compiler-rt/lib/tsan/tests/rtl/TsanRtlTest',
'--gtest_list_tests']' returned non-zero exit status -11. Process
output:
> 
> lit.py:
/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.src/utils/lit/lit/formats/googletest.py:41:
error: unable to discover google-tests in
'/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.obj/projects/compiler-rt/lib/tsan/tests/unit/TsanUnitTest':
Command
'['/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.obj/projects/compiler-rt/lib/tsan/tests/unit/TsanUnitTest',
'--gtest_list_tests']' returned non-zero exit status -11. Process
output:
> 


One of the x86_64 builders has a result:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast
<http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast>
> lit.py:
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/utils/lit/lit/formats/googletest.py:41:
error: unable to discover google-tests in
'/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests':
Command
'['/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests',
'--gtest_list_tests']' returned non-zero exit status 1. Process
output:
> ================================================================>
==27367==ERROR: AddressSanitizer: initialization-order-fiasco on address
0x000002de0c90 at pc 0x000000bad579 bp 0x7ffd584afed0 sp 0x7ffd584afec8
> READ of size 8 at 0x000002de0c90 thread T0
>     #0 0xbad578 in llvm::TargetRegistry::RegisterTarget(llvm::Target&,
char const*, char const*, bool (*)(llvm::Triple::ArchType), bool)
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Support/TargetRegistry.cpp:99:9
>     #1 0x5f94ba in LLVMInitializeAArch64TargetInfo
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/AArch64/TargetInfo/AArch64TargetInfo.cpp:23:3
>     #2 0x420d2c in InitializeAllTargetInfos
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/include/llvm/Config/Targets.def:26:1
>     #3 0x420d2c in Context
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/unittests/MC/DwarfLineTables.cpp:30
>     #4 0x420d2c in __cxx_global_var_init
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/unittests/MC/DwarfLineTables.cpp:49
>     #5 0x420d2c in _GLOBAL__sub_I_DwarfLineTables.cpp
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/unittests/MC/DwarfLineTables.cpp
>     #6 0xf8e50c in __libc_csu_init
(/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests+0xf8e50c)
>     #7 0x7f7c5a46fed4 in __libc_start_main
(/lib/x86_64-linux-gnu/libc.so.6+0x21ed4)
>     #8 0x4aa6a9 in _start
(/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests+0x4aa6a9)
> 
> 0x000002de0c90 is located 16 bytes inside of global variable
'llvm::TheARM64Target' defined in
'/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/AArch64/TargetInfo/AArch64TargetInfo.cpp:17:8'
(0x2de0c80) of size 208
>   registered at:
>     #0 0x4bc87d in __asan_register_globals.part.15
/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/projects/compiler-rt/lib/asan/asan_globals.cc:338
>     #1 0x5f955b in asan.module_ctor
(/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests+0x5f955b)
TargetRegistry.cpp:99:
  // Check if this target has already been initialized, we allow this as a
  // convenience to some clients.
  if (T.Name)
    return;


-- 
Greg Parker     gparker at apple.com <mailto:gparker at apple.com>    
Runtime Wrangler


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160902/c8a8ca8b/attachment.html>