llvm dev - Jul 2018 - Super-verbose failure mode for FileCheck

Hi All,

FileCheck is a great tool for writing tests.
However, it can be also a great tool for hiding the error cause from you,
especially if a very common idiom

RUN: ./myprog | FileCheck %s

is used.
If you need to know the output of “myprog” to figure out why your test is
failing —  then well, good luck,
you would need to manually copy’n paste that line, run it, and then note the
output.
(produced line “expected/got” are rarely enough to debug the problem)

Would it be useful to have an optional, off-by-default mode for FileCheck,
turned on by an environment variable,
which would save all the input it gets, and then dump it on failure?
I also wish we could get a 

The test case I imagine would be to run:

ninja check-X

note the failure,
and then rerun:

env FILECHECK_SUPER_VERBOSE=1 ninja check-X

Thoughts?
George

George Karpenkov via llvm-dev <llvm-dev at lists.llvm.org> writes:
> Would it be useful to have an optional, off-by-default mode for
> FileCheck, turned on by an environment variable, which would save all
> the input it gets, and then dump it on failure?
Yes please!  I'd also like an option to dump it to stdout.  I run tests
inside an IDE and it would be convenient to be able to look at the
output there rather than having to open new files.

                         -David

Could be useful - as Paul mentioned, having it on standard output could be
good - especially for buildbots (where rerunning the command is
impractical). Though a more generalized solution for 'lit' where it
dumps
all the inputs and outputs (well, I guess, the textually representable ones
- maybe that's basically just FileCheck) in a way that buildbot could split
them out into separate files (easier to read, download/use, etc than one
big stream), would be extra super awesome.

On Thu, Jul 12, 2018 at 6:31 PM George Karpenkov via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi All,
>
> FileCheck is a great tool for writing tests.
> However, it can be also a great tool for hiding the error cause from you,
> especially if a very common idiom
>
> RUN: ./myprog | FileCheck %s
>
> is used.
> If you need to know the output of “myprog” to figure out why your test is
> failing —  then well, good luck,
> you would need to manually copy’n paste that line, run it, and then note
> the output.
> (produced line “expected/got” are rarely enough to debug the problem)
>
> Would it be useful to have an optional, off-by-default mode for FileCheck,
> turned on by an environment variable,
> which would save all the input it gets, and then dump it on failure?
> I also wish we could get a
>
> The test case I imagine would be to run:
>
> ninja check-X
>
> note the failure,
> and then rerun:
>
> env FILECHECK_SUPER_VERBOSE=1 ninja check-X
>
> Thoughts?
> George
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180713/e08a89bb/attachment.html>

https://reviews.llvm.org/D49328
> On Jul 13, 2018, at 9:16 AM, David Blaikie <dblaikie at gmail.com>
wrote:
> 
> Could be useful - as Paul mentioned, having it on standard output could be
good - especially for buildbots (where rerunning the command is impractical).
Though a more generalized solution for 'lit' where it dumps all the
inputs and outputs (well, I guess, the textually representable ones - maybe
that's basically just FileCheck) in a way that buildbot could split them out
into separate files (easier to read, download/use, etc than one big stream),
would be extra super awesome.
> 
> On Thu, Jul 12, 2018 at 6:31 PM George Karpenkov via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi All,
> 
> FileCheck is a great tool for writing tests.
> However, it can be also a great tool for hiding the error cause from you,
especially if a very common idiom
> 
> RUN: ./myprog | FileCheck %s
> 
> is used.
> If you need to know the output of “myprog” to figure out why your test is
failing —  then well, good luck,
> you would need to manually copy’n paste that line, run it, and then note
the output.
> (produced line “expected/got” are rarely enough to debug the problem)
> 
> Would it be useful to have an optional, off-by-default mode for FileCheck,
turned on by an environment variable,
> which would save all the input it gets, and then dump it on failure?
> I also wish we could get a 
> 
> The test case I imagine would be to run:
> 
> ninja check-X
> 
> note the failure,
> and then rerun:
> 
> env FILECHECK_SUPER_VERBOSE=1 ninja check-X
> 
> Thoughts?
> George
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180716/38809522/attachment.html>