llvm dev - Mar 2017 - FileCheck feature request- by default ignore IR-"headers"

Hi all-
I hope this is the right place for this.

Anyway, the primary usage for the utility "FileCheck" is to pattern
match for specific values in a stream.  This is perfectly consistent and
deterministic for the most part!  However, when validating clang's LLVM-IR
generation, it is possible to make an invalid match against the top few
generated lines (the LLVM-IR Headers), which are inconsistent.

For example (from my personal pain), we have a test:
clang/test/CodeGen/debug-info-macro.c  that has a 'check-not' on the
word "macros".  Unfortunately, my SVN workspace name was
"ms_macros", which gave me a false match.  I'm a bit ashamed to
say how long this took me to figure out :)

Anywya, to my feature request: to prevent someone else from going through this
pain again, would it be possible to have FileCheck ignore these first 4 lines as
default behavior?  I suspect there might be a few tests that would want to
validate those headers, though I expect those are the vast minority, so I would
think this should be opt in.

-Erich

"Keane, Erich via llvm-dev" <llvm-dev at lists.llvm.org>
writes:
> I hope this is the right place for this.
>
> Anyway, the primary usage for the utility "FileCheck" is to
pattern
> match for specific values in a stream.  This is perfectly consistent
> and deterministic for the most part!  However, when validating clang's
> LLVM-IR generation, it is possible to make an invalid match against
> the top few generated lines (the LLVM-IR Headers), which are
> inconsistent.
>
> For example (from my personal pain), we have a test:
> clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the
> word "macros".  Unfortunately, my SVN workspace name was
"ms_macros",
> which gave me a false match.  I'm a bit ashamed to say how long this
> took me to figure out :)
This is a bug in the test and an annoyingly common pitfall when it comes
to writing "CHECK-NOT" checks - it should be checking for something
more
specific.
> Anywya, to my feature request: to prevent someone else from going
> through this pain again, would it be possible to have FileCheck ignore
> these first 4 lines as default behavior?  I suspect there might be a
> few tests that would want to validate those headers, though I expect
> those are the vast minority, so I would think this should be opt in.
Unfortunately this isn't really practical to do - FileCheck is
completely unaware of what kind of file it's checking, so it wouldn't
know when to skip those couple of lines (many tests are not IR!).

Yeah, this is a documented shortcoming of FileCheck:
http://llvm.org/docs/TestingGuide.html#fragile-tests

I think the most practical thing to do would be to make FileCheck reject
test cases that consist only CHECK-NOT directives, unless a flag is passed
to disable this behavior.

On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi all-
> I hope this is the right place for this.
>
> Anyway, the primary usage for the utility "FileCheck" is to
pattern match
> for specific values in a stream.  This is perfectly consistent and
> deterministic for the most part!  However, when validating clang's
LLVM-IR
> generation, it is possible to make an invalid match against the top few
> generated lines (the LLVM-IR Headers), which are inconsistent.
>
> For example (from my personal pain), we have a test:
> clang/test/CodeGen/debug-info-macro.c  that has a 'check-not' on
the word
> "macros".  Unfortunately, my SVN workspace name was
"ms_macros", which gave
> me a false match.  I'm a bit ashamed to say how long this took me to
figure
> out :)
>
> Anywya, to my feature request: to prevent someone else from going through
> this pain again, would it be possible to have FileCheck ignore these first
> 4 lines as default behavior?  I suspect there might be a few tests that
> would want to validate those headers, though I expect those are the vast
> minority, so I would think this should be opt in.
>
> -Erich
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170330/db44289f/attachment.html>

Alright, I guess it isn’t just my pain then, it makes it feel better ☺  I think
that proposed feature would be really nice, since it would encourage people to
write tests that have a //CHECK: some-thing-after-header first!

From: Reid Kleckner [mailto:rnk at google.com]
Sent: Thursday, March 30, 2017 11:15 AM
To: Keane, Erich <erich.keane at intel.com>
Cc: llvm-dev at lists.llvm.org
Subject: Re: [llvm-dev] FileCheck feature request- by default ignore
IR-"headers"

Yeah, this is a documented shortcoming of FileCheck:
http://llvm.org/docs/TestingGuide.html#fragile-tests

I think the most practical thing to do would be to make FileCheck reject test
cases that consist only CHECK-NOT directives, unless a flag is passed to disable
this behavior.

On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Hi all-
I hope this is the right place for this.

Anyway, the primary usage for the utility "FileCheck" is to pattern
match for specific values in a stream.  This is perfectly consistent and
deterministic for the most part!  However, when validating clang's LLVM-IR
generation, it is possible to make an invalid match against the top few
generated lines (the LLVM-IR Headers), which are inconsistent.

For example (from my personal pain), we have a test:
clang/test/CodeGen/debug-info-macro.c  that has a 'check-not' on the
word "macros".  Unfortunately, my SVN workspace name was
"ms_macros", which gave me a false match.  I'm a bit ashamed to
say how long this took me to figure out :)

Anywya, to my feature request: to prevent someone else from going through this
pain again, would it be possible to have FileCheck ignore these first 4 lines as
default behavior?  I suspect there might be a few tests that would want to
validate those headers, though I expect those are the vast minority, so I would
think this should be opt in.

-Erich
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170330/12dcd35f/attachment.html>