llvm dev - Jul 2017 - Swallowing of input in FileCheck

> On Jul 7, 2017, at 2:19 PM, Reid Kleckner <rnk at google.com> wrote:
> 
> On Fri, Jul 7, 2017 at 1:20 PM, George Karpenkov via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Thus, I propose modifying FileCheck default behavior to dump all swallowed
output on stderr when the test has failed.
> Would there be any objections to such a change?
> 
> Yes.
>  
> I understand the concern that log files might become unnecessarily large,
but since it would only be done for failed
> test I think the added readability would be worth it.
> 
> I disagree, it would be too much output. During development, it's
pretty common to cause tens of tests to fail. I don't really want 10 entire
assembly files dumped into my console during incremental development. Our test
output is already long, and I wish it were shorter.
Could this be solved by having lit be intelligent about showing less output when
there are large numbers of test failures (w/o other output), and truncating very
large outputs?

I do think there are situations where having the output just show up by default
locally could prevent needing to rerun a command, which is handy.
> I agree that this is a real problem when remote buildbots in different
configurations get involved. Locally debugging FileCheck failures is easy, you
just copy-paste the command like you said and pipe it to less. It's only a
pain when you aren't sure if a failure on a bot will reproduce locally. So,
I would be in favor of an option to lit that we enable on buildslaves that dumps
the output. We already have a '\bFileCheck\b' substitution in lit.
We'd just expand it to 'FileCheck --dump-on-failure' or something on
bots.
This sounds reasonable to me, no matter what on the above question.

 - Daniel

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170707/84d7266c/attachment.html>

What about having an environment variable FILECHECKER_VERBOSE=1?
This would not require substitutions, and could be even set automatically by
“lit” when launched with “-v”.
At least to me that would make debugging tests much
easier.
> On Jul 7, 2017, at 3:05 PM, Daniel Dunbar <daniel_dunbar at
apple.com> wrote:
> 
> 
>> On Jul 7, 2017, at 2:19 PM, Reid Kleckner <rnk at google.com
<mailto:rnk at google.com>> wrote:
>> 
>> On Fri, Jul 7, 2017 at 1:20 PM, George Karpenkov via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>> Thus, I propose modifying FileCheck default behavior to dump all
swallowed output on stderr when the test has failed.
>> Would there be any objections to such a change?
>> 
>> Yes.
>>  
>> I understand the concern that log files might become unnecessarily
large, but since it would only be done for failed
>> test I think the added readability would be worth it.
>> 
>> I disagree, it would be too much output. During development, it's
pretty common to cause tens of tests to fail. I don't really want 10 entire
assembly files dumped into my console during incremental development. Our test
output is already long, and I wish it were shorter.
> 
> Could this be solved by having lit be intelligent about showing less output
when there are large numbers of test failures (w/o other output), and truncating
very large outputs?
> 
> I do think there are situations where having the output just show up by
default locally could prevent needing to rerun a command, which is handy.
> 
>> I agree that this is a real problem when remote buildbots in different
configurations get involved. Locally debugging FileCheck failures is easy, you
just copy-paste the command like you said and pipe it to less. It's only a
pain when you aren't sure if a failure on a bot will reproduce locally. So,
I would be in favor of an option to lit that we enable on buildslaves that dumps
the output. We already have a '\bFileCheck\b' substitution in lit.
We'd just expand it to 'FileCheck --dump-on-failure' or something on
bots.
> 
> This sounds reasonable to me, no matter what on the above question.
> 
>  - Daniel
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170707/3d530df9/attachment.html>

Ideally/the better integration with Buildbot would be to have these outputs
referenced as "associated files" (it's been a while since I played
with
buildbot - I remember finding this and considering how it could be done,
but not getting all the way through) so they'd come back as actual files on
the build master, linked from the results page that you could click on to
view/download.

(similarly in the local output, having these files written to disk and the
name of the file mentioned in the output would seem nice to me - some tests
use the %t, etc, to create temporary files and you can see their names in
the output, but some stream directly - it'd be great if the direct
streaming still allowed the user to inspect the files along pipe chain)

(aside: I'd love it if lit would tell me /which/ of the RUN commands it was
running when it failed, or which one the output came from (in the case of a
single RUN line having multiple commands... ) somehow - would simplify
things a bit too)

On Fri, Jul 7, 2017 at 4:24 PM George Karpenkov via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> What about having an environment variable FILECHECKER_VERBOSE=1?
> This would not require substitutions, and could be even set automatically
> by “lit” when launched with “-v”.
> At least to me that would make debugging tests much easier.
>
> On Jul 7, 2017, at 3:05 PM, Daniel Dunbar <daniel_dunbar at
apple.com> wrote:
>
>
> On Jul 7, 2017, at 2:19 PM, Reid Kleckner <rnk at google.com> wrote:
>
> On Fri, Jul 7, 2017 at 1:20 PM, George Karpenkov via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>>
>> Thus, I propose modifying FileCheck default behavior to dump all
>> swallowed output on stderr when the test has failed.
>> Would there be any objections to such a change?
>>
>
> Yes.
>
>
>> I understand the concern that log files might become unnecessarily
large,
>> but since it would only be done for failed
>> test I think the added readability would be worth it.
>>
>
> I disagree, it would be too much output. During development, it's
pretty
> common to cause tens of tests to fail. I don't really want 10 entire
> assembly files dumped into my console during incremental development. Our
> test output is already long, and I wish it were shorter.
>
>
> Could this be solved by having lit be intelligent about showing less
> output when there are large numbers of test failures (w/o other output),
> and truncating very large outputs?
>
> I do think there are situations where having the output just show up by
> default locally could prevent needing to rerun a command, which is handy.
>
> I agree that this is a real problem when remote buildbots in different
> configurations get involved. Locally debugging FileCheck failures is easy,
> you just copy-paste the command like you said and pipe it to less. It's
> only a pain when you aren't sure if a failure on a bot will reproduce
> locally. So, I would be in favor of an option to lit that we enable on
> buildslaves that dumps the output. We already have a
'\bFileCheck\b'
> substitution in lit. We'd just expand it to 'FileCheck
--dump-on-failure'
> or something on bots.
>
>
> This sounds reasonable to me, no matter what on the above question.
>
>  - Daniel
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170708/4483658c/attachment.html>