llvm dev - Jan 2013 - [LLVMdev] [PATCH] A "very verbose" mode for FileCheck

Hello,

When someone breaks a FileCheck-based test on some buildbot, sometimes
it may not be obvious *why* did it fail.  If the failure can not be
reproduced locally, it can be very hard to fix.

I propose adding a "very verbose" mode to FileCheck.  In this mode
FileCheck will dump the input file in case of failure.  This mode will
be enabled by an environment variable "FILECHECK_VERY_VERBOSE".  If we
chose a command line option, we would have to edit all FileCheck-based
tests to use %FileCheck.

Attached is the proposed patch.  A patch for zorg will follow if this
is accepted.  Normal output for 'make check' is unchanged.

I am not attached to the name "FILECHECK_VERY_VERBOSE", suggestions
for better names are welcome.

Dmitri

-- 
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at
gmail.com>*/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: filecheck-very-verbose-v1.patch
Type: application/octet-stream
Size: 3688 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130116/eccadb06/attachment.obj>

On Wed, Jan 16, 2013 at 10:32 AM, Dmitri Gribenko <gribozavr at gmail.com>
wrote:
> Hello,
>
> When someone breaks a FileCheck-based test on some buildbot, sometimes
> it may not be obvious *why* did it fail.  If the failure can not be
> reproduced locally, it can be very hard to fix.
>
> I propose adding a "very verbose" mode to FileCheck.  In this
mode
> FileCheck will dump the input file in case of failure.  This mode will
> be enabled by an environment variable "FILECHECK_VERY_VERBOSE". 
If we
> chose a command line option, we would have to edit all FileCheck-based
> tests to use %FileCheck.
>
> Attached is the proposed patch.  A patch for zorg will follow if this
> is accepted.  Normal output for 'make check' is unchanged.
>
> I am not attached to the name "FILECHECK_VERY_VERBOSE",
suggestions
> for better names are welcome.
>
Could you describe a situation where this helps in more detail? It's
quite easy to reproduce the input file to FileCheck locally.

Eli

On Wed, Jan 16, 2013 at 8:38 PM, Eli Bendersky <eliben at google.com>
wrote:
> On Wed, Jan 16, 2013 at 10:32 AM, Dmitri Gribenko <gribozavr at
gmail.com> wrote:
>> Hello,
>>
>> When someone breaks a FileCheck-based test on some buildbot, sometimes
>> it may not be obvious *why* did it fail.  If the failure can not be
>> reproduced locally, it can be very hard to fix.
>>
>> I propose adding a "very verbose" mode to FileCheck.  In this
mode
>> FileCheck will dump the input file in case of failure.  This mode will
>> be enabled by an environment variable
"FILECHECK_VERY_VERBOSE".  If we
>> chose a command line option, we would have to edit all FileCheck-based
>> tests to use %FileCheck.
>>
>> Attached is the proposed patch.  A patch for zorg will follow if this
>> is accepted.  Normal output for 'make check' is unchanged.
>>
>> I am not attached to the name "FILECHECK_VERY_VERBOSE",
suggestions
>> for better names are welcome.
>>
>
> Could you describe a situation where this helps in more detail? It's
> quite easy to reproduce the input file to FileCheck locally.
Imagine a situation when you have opt | FileCheck.  And 'opt' produces
something weird only on a buildbot, while on your machine the output
is correct.

See, for example, this thread
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20130114/162207.html

Dmitri

-- 
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at
gmail.com>*/

On Jan 16, 2013, at 10:32 AM, Dmitri Gribenko <gribozavr at gmail.com>
wrote:
> Hello,
> 
> When someone breaks a FileCheck-based test on some buildbot, sometimes
> it may not be obvious *why* did it fail.  If the failure can not be
> reproduced locally, it can be very hard to fix.
> 
> I propose adding a "very verbose" mode to FileCheck.  In this
mode
> FileCheck will dump the input file in case of failure.  This mode will
> be enabled by an environment variable "FILECHECK_VERY_VERBOSE". 
If we
> chose a command line option, we would have to edit all FileCheck-based
> tests to use %FileCheck.
I think that this idea is good, but I'd prefer it be implemented a different
way:

 - Filecheck should take a new flag -dump-input-on-error that causes it to...
dump the input file on error.
 - Lit should be the thing that checks the environment (or perhaps add a new
option to lit), and adds the flag to FileCheck invocations.

I don't like it when the behavior of such a low-level tool like this changes
based on environment variables.  It isn't discoverable in --help.  If for
some reason, it is bad for lit to implicitly pass the option, I'd rather
have a standard FILECHECK_COMMANDLINE environment variable, and have filecheck
parse arbitrary options out of it using the cl::ParseEnvironmentOptions
function.

-Chris

On Wed, Jan 16, 2013 at 9:24 PM, Chris Lattner <clattner at apple.com>
wrote:
>
> On Jan 16, 2013, at 10:32 AM, Dmitri Gribenko <gribozavr at
gmail.com> wrote:
>
>> Hello,
>>
>> When someone breaks a FileCheck-based test on some buildbot, sometimes
>> it may not be obvious *why* did it fail.  If the failure can not be
>> reproduced locally, it can be very hard to fix.
>>
>> I propose adding a "very verbose" mode to FileCheck.  In this
mode
>> FileCheck will dump the input file in case of failure.  This mode will
>> be enabled by an environment variable
"FILECHECK_VERY_VERBOSE".  If we
>> chose a command line option, we would have to edit all FileCheck-based
>> tests to use %FileCheck.
>
> I think that this idea is good, but I'd prefer it be implemented a
different way:
>
>  - Filecheck should take a new flag -dump-input-on-error that causes it
to... dump the input file on error.
>  - Lit should be the thing that checks the environment (or perhaps add a
new option to lit), and adds the flag to FileCheck invocations.
>
> I don't like it when the behavior of such a low-level tool like this
changes based on environment variables.  It isn't discoverable in --help. 
If for some reason, it is bad for lit to implicitly pass the option, I'd
rather have a standard FILECHECK_COMMANDLINE environment variable, and have
filecheck parse arbitrary options out of it using the
cl::ParseEnvironmentOptions function.
I agree that a command line option would be better.  But in that case
all tests should be updated.  It is not an issue for me -- it is
mostly mechanical.  So should I change tests to use %FileCheck?

Dmitri

-- 
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at
gmail.com>*/