llvm dev - Jul 2013 - [LLVMdev] [RFC] Switching make check to use 'set -o pipefail'

> Hi Rafael,
>
> Did this discussion ever get a conclusion?  I support enabling
> pipefail.  Fallout for out of tree users should be easy to fix.  As we
> learned from LLVM tests, almost all tests that start to fail actually
> indicate a real problem that was hidden.
So far I got some positive feedback, but no strong LGTM from someone
in the area :-(
> Dmitri
>

Cheers,
Rafael

On Wed, Jul 17, 2013 at 9:48 PM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:
> > Hi Rafael,
> >
> > Did this discussion ever get a conclusion?  I support enabling
> > pipefail.  Fallout for out of tree users should be easy to fix.  As we
> > learned from LLVM tests, almost all tests that start to fail actually
> > indicate a real problem that was hidden.
>
> So far I got some positive feedback, but no strong LGTM from someone
> in the area :-(
>
+1 more for pipefail, if that helps.  :)

The only standing objection has to do with out-of-tree target maintainers,
and honestly I think they'll be fine.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130719/d711ffc5/attachment.html>

On Fri, Jul 19, 2013 at 6:34 AM, Reid Kleckner <rnk at google.com>
wrote:
> On Wed, Jul 17, 2013 at 9:48 PM, Rafael Espíndola
> <rafael.espindola at gmail.com> wrote:
>>
>> > Hi Rafael,
>> >
>> > Did this discussion ever get a conclusion?  I support enabling
>> > pipefail.  Fallout for out of tree users should be easy to fix. 
As we
>> > learned from LLVM tests, almost all tests that start to fail
actually
>> > indicate a real problem that was hidden.
>>
>> So far I got some positive feedback, but no strong LGTM from someone
>> in the area :-(
>
>
> +1 more for pipefail, if that helps.  :)
Another +1.
> The only standing objection has to do with out-of-tree target maintainers,
> and honestly I think they'll be fine.
I'm sure they will be. This is not the worst thing they have to live
with (the API churn is massive) & that's a tradeoff we/they
deliberately make for this project: trunk moves forward.

On Wed, Jul 17, 2013 at 6:48 PM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:
> > Hi Rafael,
> >
> > Did this discussion ever get a conclusion?  I support enabling
> > pipefail.  Fallout for out of tree users should be easy to fix.  As we
> > learned from LLVM tests, almost all tests that start to fail actually
> > indicate a real problem that was hidden.
>
> So far I got some positive feedback, but no strong LGTM from someone
> in the area :-(
>
Ok, here is a strong LGTM. =]

Please make the change, and do the following things to aid out-of-tree
maintainers:

1) Add a flag to lit and an option to configure/make (I don't care about
CMake here as that is much less frequently used for out-of-tree work) to
disable pipefail.

2) Add a way to disable this behavior using the lit '.cfg' files,
especially the 'lit.local.cfg' files. This way out-of-tree targets can
put
such a .cfg file in their target's test directory and ignore this change.

3) Add some significant documentation for what this means to both the lit
documentation and to the release notes so that folks are aware of the test
infrastructure change when they pick up a giant pile of changes in the
release

Also, please send a note (in a new thread) to llvmdev when the switch is
flipped with a reminder about how to disable the new behavior for folks
that can't update their test suite. You'll probably want to flip the
switch
when you have time to track down lots of build bot failures. =D

Thanks for making the test infrastructure better,
-Chandler
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130720/9946a289/attachment.html>

> Ok, here is a strong LGTM. =]
>
> Please make the change, and do the following things to aid out-of-tree
> maintainers:
>
> 1) Add a flag to lit and an option to configure/make (I don't care
about
> CMake here as that is much less frequently used for out-of-tree work) to
> disable pipefail.
>
I have just fixed the last failures on windows. I have also added
documentation and the support for disabling pipefail in a directory.
The one thing I have not implemented yet is the configure change.

The reason is that after thinking a bit about it it looks like
something we don't want to have. What we want to provide is an easy
way for people doing out of tree work to get their tests passing after
an upgrade. We do want upstream tests to fail for them if they, for
example, break opt so that it crashes on exit. This is exactly what
the lit.local.cfg provides.

A new patch is attached. What do you think?
> Also, please send a note (in a new thread) to llvmdev when the switch is
> flipped with a reminder about how to disable the new behavior for folks
that
> can't update their test suite. You'll probably want to flip the
switch when
> you have time to track down lots of build bot failures. =D
OK.
> Thanks for making the test infrastructure better,
> -Chandler
Cheers,
Rafael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: t.patch
Type: application/octet-stream
Size: 3650 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130726/5aeba473/attachment.obj>