llvm dev - Jul 2015 - [LLVMdev] [cfe-dev] [3.7 Release] RC1 has been tagged, Testing Phase I begins

> Ben reports that he didn't get a tarball even after the rpath fix and
> had to disable -o pipefail to get one. Still not sure what caused
> this.
'make check' failures can cause this. I had to revert r241599 to get a
package.
> Appendix: compiler-rt test failures
> ==================================>
Just to add Mips to this: On big-endian Mips32r2:
            AddressSanitizer-mips-linux :: TestCases/Linux/kernel-area.cc
            AddressSanitizer-mips-linux ::
TestCases/Posix/coverage-direct-large.cc
            UBSan-ASan-mips :: TestCases/Float/cast-overflow.cpp
            UBSan-Standalone-mips :: TestCases/Float/cast-overflow.cpp
            libc++abi :: backtrace_test.pass.cpp
            libc++abi :: test_demangle.pass.cpp

On little-endian Mips32r2:
            AddressSanitizer-mips-linux :: TestCases/Linux/kernel-area.cc
            libc++abi :: backtrace_test.pass.cpp
            libc++abi :: test_demangle.pass.cpp

Jaydeep Patil and Nitesh Jain have shown me a patch for 'libc++abi ::
test_demangle.pass.cpp' which works for me. It should be appearing for
review shortly.
> -----Original Message-----
> From: cfe-dev-bounces at cs.uiuc.edu [mailto:cfe-dev-bounces at
cs.uiuc.edu]
> On Behalf Of Hans Wennborg
> Sent: 22 July 2015 01:53
> To: llvmdev; cfe-dev; lldb-dev at cs.uiuc.edu
> Cc: Ben Pope; Ed Maste; Sebastian Dreßler
> Subject: Re: [cfe-dev] [3.7 Release] RC1 has been tagged, Testing Phase I
> begins
> 
> On Thu, Jul 16, 2015 at 3:31 PM, Hans Wennborg <hans at chromium.org>
> wrote:
> > Dear testers,
> >
> > 3.7.0-rc1 was just tagged; please start your testing engines :-)
> 
> This has been a bumpier ride than 3.6 so far, mainly due to the cmake
> switch. We're testing stuff that didn't get tested before, which I
> think is good. Thank you very much for your efforts so far.
> 
> This is an attempt to summarize this rather long thread. TL;DR: There
> have been a bunch of issues, many addressed, some still out-standing.
> Most binaries are ready, so I plan to push RC1 to the web site soon.
> 
> Binaries on the sftp:
> ====================> clang+llvm-3.7.0-armv7a-linux-gnueabihf.tar.xz
> clang+llvm-3.7.0-rc1-aarch64-linux-gnu.tar.xz
> clang+llvm-3.7.0-rc1-amd64-unknown-freebsd10.tar.xz
> clang+llvm-3.7.0-rc1-i386-unknown-freebsd10.tar.xz
> clang+llvm-3.7.0-rc1-mipsel-linux-gnu.tar.xz
> clang+llvm-3.7.0-rc1-mips-linux-gnu.tar.xz
> clang+llvm-3.7.0-rc1-x86_64-apple-darwin.tar.xz
> clang+llvm-3.7.0-rc1-x86_64-linux-gnu-ubuntu-15.04.tar.xz
> LLVM-3.7.0-rc1-win32.exe
> LLVM-3.7.0-rc1-win64.exe
> 
> Some of these have the /usr/local prefix baked in (since fixed in
> r242706); I'll remove that manually before publishing the tarballs.
> 
> The armv7a tarball doesn't have rc1 in the name. I'll rename that
(and
> the dir within) before publishing.
> 
> Fedora and OpenSUSE isn't ready yet, but those can be added later. I
> think we have enough binaries to put RC1 on the web page.
> 
> Issues
> =====> Various compiler-rt test failures, see below.
> (Linux/signal_segv_handler.cc might be fixed by r242848.)
> 
> Dimitry says he'll use the autoconf build for FreeBSD due to errors.
> Has a patch out to do this in the script, but it hasn't landed yet.
> (Darwin currently also defaults to autoconf in the script)
> 
> Jack Howarth asked about -fopenmp currently essentially being no-op.
> As far as I know (please correct me if I'm wrong here) this is still
> the case on trunk. If it's not enabled on trunk, there is nothing to
> merge.
> 
> There is a discussion on openmp-dev about the CMake build being broken
> in Pathscale's build. The breakage does not seem to be well understood
> yet.
> 
> Nikola reports that compiler-rt fails to cmake on x86 openSUSE because
> it targets i686, but the system is i586. Same failure happens on the
> phase 2 build on x86 Fedora 22, but probably with a different cause.
> 
> Renato reported libunwind was missing from the projects list,
> preventing libcxxabi and libcxx to build. It was added in r242543, but
> there are still problems and Renato said he'll exclude them for this
> release.
> 
> Files in the tarball were prefixed with /usr/local; fixed in r242706.
> 
> The clean_rpath function was failing when no rpath was present in the
> binary; fixed in r242722
> 
> The file comparison step was broken as cmp(1) doesn't support
> --ignore-initial on all platforms; fixed in r242721.
> 
> Object/archive-extract.test failed on windows; fixed in r242527
> 
> Ben reports that he didn't get a tarball even after the rpath fix and
> had to disable -o pipefail to get one. Still not sure what caused
> this.
> 
> 
> 
> Thanks,
> Hans
> 
> 
> Appendix: compiler-rt test failures
> ==================================> From Ben on Ubuntu 14.04 x64:
> 
>     AddressSanitizer-x86_64-linux :: TestCases/Posix/readv.cc
>     MemorySanitizer :: Linux/tcgetattr.cc
>     MemorySanitizer-Unit :: Msan-x86_64-Test/MemorySanitizer.gethostent
>     MemorySanitizer-Unit :: Msan-x86_64-
> Test/MemorySanitizer.gethostent_r
>     MemorySanitizer-Unit :: Msan-x86_64-Test/MemorySanitizer.getmntent
>     MemorySanitizer-Unit :: Msan-x86_64-Test/MemorySanitizer.getmntent_r
>     MemorySanitizer-Unit ::
> Msan-x86_64-with-call-Test/MemorySanitizer.gethostent
>     MemorySanitizer-Unit ::
> Msan-x86_64-with-call-Test/MemorySanitizer.gethostent_r
>     MemorySanitizer-Unit :: Msan-x86_64-with-call-
> Test/MemorySanitizer.getmntent
>     MemorySanitizer-Unit ::
> Msan-x86_64-with-call-Test/MemorySanitizer.getmntent_r
>     SanitizerCommon-asan :: Linux/getpass.cc
>     SanitizerCommon-asan :: Posix/decorate_proc_maps.cc
>     SanitizerCommon-lsan :: Linux/getpass.cc
>     SanitizerCommon-msan :: Linux/getpass.cc
>     SanitizerCommon-msan :: Linux/signal_segv_handler.cc
>     SanitizerCommon-msan :: Posix/decorate_proc_maps.cc
>     SanitizerCommon-tsan :: Linux/getpass.cc
>     SanitizerCommon-tsan :: Posix/decorate_proc_maps.cc
> 
> From Nikola on Fedora 22 x64:
> 
>     LeakSanitizer-AddressSanitizer ::
TestCases/cleanup_in_tsd_destructor.cc
>     LeakSanitizer-AddressSanitizer :: TestCases/disabler.cc
>     LeakSanitizer-AddressSanitizer ::
TestCases/disabler_in_tsd_destructor.cc
>     LeakSanitizer-AddressSanitizer :: TestCases/ignore_object.cc
>     LeakSanitizer-Standalone :: TestCases/cleanup_in_tsd_destructor.cc
>     LeakSanitizer-Standalone :: TestCases/disabler.cc
>     LeakSanitizer-Standalone :: TestCases/disabler_in_tsd_destructor.cc
>     LeakSanitizer-Standalone :: TestCases/ignore_object.cc
> 
> From Ben on x86_64-linux-gnu-ubuntu-15.04:
> 
>    MemorySanitizer :: Linux/tcgetattr.cc
>     MemorySanitizer-Unit :: Msan-x86_64-Test/MemorySanitizer.getmntent
>     MemorySanitizer-Unit :: Msan-x86_64-Test/MemorySanitizer.getmntent_r
>     MemorySanitizer-Unit :: Msan-x86_64-with-call-
> Test/MemorySanitizer.getmntent
>     MemorySanitizer-Unit ::
> Msan-x86_64-with-call-Test/MemorySanitizer.getmntent_r
>     SanitizerCommon-asan :: Linux/getpass.cc
>     SanitizerCommon-lsan :: Linux/getpass.cc
>     SanitizerCommon-msan :: Linux/getpass.cc
>     SanitizerCommon-tsan :: Linux/getpass.cc
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

On Wed, Jul 22, 2015 at 7:03 AM, Daniel Sanders
<Daniel.Sanders at imgtec.com> wrote:
>> Ben reports that he didn't get a tarball even after the rpath fix
and
>> had to disable -o pipefail to get one. Still not sure what caused
>> this.
>
> 'make check' failures can cause this. I had to revert r241599 to
get a package.
Oh, I'm silly. Some part of my brain assumed "make -k" would not
just
"keep going" but also ignore errors. That's not the case of
course.

I suppose we could disable -o pipefail for the test step. On the other
hand, we should make these tests pass :-) Let me know what you think.
>> Appendix: compiler-rt test failures
>> ==================================>>
>
> Just to add Mips to this: On big-endian Mips32r2:
>             AddressSanitizer-mips-linux :: TestCases/Linux/kernel-area.cc
>             AddressSanitizer-mips-linux ::
TestCases/Posix/coverage-direct-large.cc
>             UBSan-ASan-mips :: TestCases/Float/cast-overflow.cpp
>             UBSan-Standalone-mips :: TestCases/Float/cast-overflow.cpp
>             libc++abi :: backtrace_test.pass.cpp
>             libc++abi :: test_demangle.pass.cpp
>
> On little-endian Mips32r2:
>             AddressSanitizer-mips-linux :: TestCases/Linux/kernel-area.cc
>             libc++abi :: backtrace_test.pass.cpp
>             libc++abi :: test_demangle.pass.cpp
>
> Jaydeep Patil and Nitesh Jain have shown me a patch for 'libc++abi ::
test_demangle.pass.cpp' which works for me. It should be appearing for
review shortly.
Great! Please let me know when it appears.

On Wed, Jul 22, 2015 at 2:46 PM, Ben Pope <benpope81 at gmail.com>
wrote:
> On Thursday, July 23, 2015 12:42 AM, Hans Wennborg wrote:
>>
>> On Wed, Jul 22, 2015 at 7:03 AM, Daniel Sanders
>> <Daniel.Sanders at imgtec.com> wrote:
>>>>
>>>> Ben reports that he didn't get a tarball even after the
rpath fix and
>>>> had to disable -o pipefail to get one. Still not sure what
caused
>>>> this.
>>>
>>>
>>> 'make check' failures can cause this. I had to revert
r241599 to get a
>>> package.
>>
>>
>> Oh, I'm silly. Some part of my brain assumed "make -k"
would not just
>> "keep going" but also ignore errors. That's not the case
of course.
>>
>> I suppose we could disable -o pipefail for the test step. On the other
>> hand, we should make these tests pass :-) Let me know what you think.
>
>
> :)
>
> That's a +1, or what are the tests for.  On the other hand, it's
technically
> not a regression and it's your call.
It's annoying that a test failure blocks building the tarball though.
I'd like to get these tests fixed, but for the RC's, and maybe even
for the release it can be useful to be able to ignore benign test
failures and still get a tarball.

On the other hand, without -o pipefail, there is a real risk that the
test results drown in the rest of the output.

I'll try to cook up a patch that doesn't stop a build on test errors,
but still surfaces them at the end of the script run.

 - Hans

> -----Original Message-----
> From: hwennborg at google.com [mailto:hwennborg at google.com] On Behalf
> Of Hans Wennborg
> Sent: 22 July 2015 17:42
> To: Daniel Sanders
> Cc: llvmdev; cfe-dev; lldb-dev at cs.uiuc.edu; Ben Pope; Ed Maste;
Sebastian
> Dreßler
> Subject: Re: [cfe-dev] [3.7 Release] RC1 has been tagged, Testing Phase I
> begins
> 
> On Wed, Jul 22, 2015 at 7:03 AM, Daniel Sanders
> <Daniel.Sanders at imgtec.com> wrote:
> >> Ben reports that he didn't get a tarball even after the rpath
fix and
> >> had to disable -o pipefail to get one. Still not sure what caused
> >> this.
> >
> > 'make check' failures can cause this. I had to revert r241599
to get a
> package.
> 
> Oh, I'm silly. Some part of my brain assumed "make -k" would
not just
> "keep going" but also ignore errors. That's not the case of
course.
> 
> I suppose we could disable -o pipefail for the test step. On the other
> hand, we should make these tests pass :-) Let me know what you think.
Generally speaking, I agree that we should fix these tests although I do have
one failure in the mipsel-linux-gnu package which isn't fixable without
either replacing the machine or enabling atime on an NFS partition (there's
no local storage). In that particular case, I think it's fine to accept the
expected failure. It would be nice to automatically XFAIL that test on noatime
filesystems though.

I think the release script shouldn't give up at the first sign of problems.
We're going to get the occasional failure during release testing and we
should always get a package to enable further testing beyond what the release
script does. The ideal would be to have the release script detect whether there
were failures and print something like 'Errors occurred, please check the
logs for details' to highlight the problem but otherwise behave as if
everything was fine.

This is completely untested and may be bash/Linux specific but we could have
something like:
	defer_fail() {
	  local exit_code
	  set +e
	  "$@"
	  exit_code=$?
	  set -e
	  if [ "$exit_code" != 0 ]; then
	   SOMETHING_FAILED=1
	  fi
	}
	
	set -e
	set -o pipefail
	command1            # The script will stop if command1 fails
	command2 | command3 # The script will stop if either command2/command3 fails
	defer_fail command4 # The script will report an error later if command4 fails
but will proceed
	defer_fail command5 | defer_fail command6 # The script will report an error
later if either command5/command6 fails but will proceed
	
	if [ "$SOMETHING_FAILED" = 1 ]; then
	  echo "Something failed, please check the logs"
	fi