Keane, Erich via llvm-dev
2017-Mar-30 18:17 UTC
[llvm-dev] FileCheck feature request- by default ignore IR-"headers"
Alright, I guess it isn’t just my pain then, it makes it feel better ☺ I think that proposed feature would be really nice, since it would encourage people to write tests that have a //CHECK: some-thing-after-header first! From: Reid Kleckner [mailto:rnk at google.com] Sent: Thursday, March 30, 2017 11:15 AM To: Keane, Erich <erich.keane at intel.com> Cc: llvm-dev at lists.llvm.org Subject: Re: [llvm-dev] FileCheck feature request- by default ignore IR-"headers" Yeah, this is a documented shortcoming of FileCheck: http://llvm.org/docs/TestingGuide.html#fragile-tests I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior. On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: Hi all- I hope this is the right place for this. Anyway, the primary usage for the utility "FileCheck" is to pattern match for specific values in a stream. This is perfectly consistent and deterministic for the most part! However, when validating clang's LLVM-IR generation, it is possible to make an invalid match against the top few generated lines (the LLVM-IR Headers), which are inconsistent. For example (from my personal pain), we have a test: clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the word "macros". Unfortunately, my SVN workspace name was "ms_macros", which gave me a false match. I'm a bit ashamed to say how long this took me to figure out :) Anywya, to my feature request: to prevent someone else from going through this pain again, would it be possible to have FileCheck ignore these first 4 lines as default behavior? I suspect there might be a few tests that would want to validate those headers, though I expect those are the vast minority, so I would think this should be opt in. -Erich _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170330/12dcd35f/attachment.html>
Robinson, Paul via llvm-dev
2017-Mar-30 21:50 UTC
[llvm-dev] FileCheck feature request- by default ignore IR-"headers"
re-add llvm-dev… From: Robinson, Paul Sent: Thursday, March 30, 2017 2:50 PM To: 'Keane, Erich'; Reid Kleckner Subject: RE: [llvm-dev] FileCheck feature request- by default ignore IR-"headers" I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior. That might help a certain class of tests, but I'm not so sure that's such a common case. It's about as easy to write a positive CHECK[-LABEL] that will unexpectedly match something that you didn't expect. At least the test with only CHECK-NOT in it *will* fail if you get it wrong, which is certainly better than passing when it should fail. So, I am reluctant to encourage yet another wart/hack on how FileCheck works. --paulr From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Keane, Erich via llvm-dev Sent: Thursday, March 30, 2017 11:18 AM To: Reid Kleckner Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] FileCheck feature request- by default ignore IR-"headers" Alright, I guess it isn’t just my pain then, it makes it feel better ☺ I think that proposed feature would be really nice, since it would encourage people to write tests that have a //CHECK: some-thing-after-header first! From: Reid Kleckner [mailto:rnk at google.com] Sent: Thursday, March 30, 2017 11:15 AM To: Keane, Erich <erich.keane at intel.com<mailto:erich.keane at intel.com>> Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] FileCheck feature request- by default ignore IR-"headers" Yeah, this is a documented shortcoming of FileCheck: http://llvm.org/docs/TestingGuide.html#fragile-tests I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior. On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: Hi all- I hope this is the right place for this. Anyway, the primary usage for the utility "FileCheck" is to pattern match for specific values in a stream. This is perfectly consistent and deterministic for the most part! However, when validating clang's LLVM-IR generation, it is possible to make an invalid match against the top few generated lines (the LLVM-IR Headers), which are inconsistent. For example (from my personal pain), we have a test: clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the word "macros". Unfortunately, my SVN workspace name was "ms_macros", which gave me a false match. I'm a bit ashamed to say how long this took me to figure out :) Anywya, to my feature request: to prevent someone else from going through this pain again, would it be possible to have FileCheck ignore these first 4 lines as default behavior? I suspect there might be a few tests that would want to validate those headers, though I expect those are the vast minority, so I would think this should be opt in. -Erich _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170330/68e10f3e/attachment.html>
David Blaikie via llvm-dev
2017-Mar-31 15:06 UTC
[llvm-dev] FileCheck feature request- by default ignore IR-"headers"
FWIW - a -NOT only test looks especially problematic to me. The test would pass if the program produced no output, basically - which sounds overly loose (a shade less problematic than the "run this and don't check anything - not crashing is all we're testing for here" - which should have some positive test of the behavior that is expected other than crashing). So I'd see such a constraint as potentially interesting on that basis - if it fixes this other issue as well, eh, helpful. But I'm also OK with it being a thing we document, and every new contributor eventually learns once... it doesn't seem to have a lot of overhead. On Thu, Mar 30, 2017 at 2:50 PM Robinson, Paul via llvm-dev < llvm-dev at lists.llvm.org> wrote:> re-add llvm-dev… > > > > *From:* Robinson, Paul > *Sent:* Thursday, March 30, 2017 2:50 PM > *To:* 'Keane, Erich'; Reid Kleckner > *Subject:* RE: [llvm-dev] FileCheck feature request- by default ignore > IR-"headers" > > > > I think the most practical thing to do would be to make FileCheck reject > test cases that consist only CHECK-NOT directives, unless a flag is passed > to disable this behavior. > > > > That might help a certain class of tests, but I'm not so sure that's such > a common case. It's about as easy to write a positive CHECK[-LABEL] that > will unexpectedly match something that you didn't expect. At least the > test with only CHECK-NOT in it *will* fail if you get it wrong, which is > certainly better than passing when it should fail. > > So, I am reluctant to encourage yet another wart/hack on how FileCheck > works. > > --paulr > > > > *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org > <llvm-dev-bounces at lists.llvm.org>] *On Behalf Of *Keane, Erich via > llvm-dev > *Sent:* Thursday, March 30, 2017 11:18 AM > *To:* Reid Kleckner > > > *Cc:* llvm-dev at lists.llvm.org > *Subject:* Re: [llvm-dev] FileCheck feature request- by default ignore > IR-"headers" > > > > Alright, I guess it isn’t just my pain then, it makes it feel better J I > think that proposed feature would be really nice, since it would encourage > people to write tests that have a //CHECK: some-thing-after-header first! > > > > *From:* Reid Kleckner [mailto:rnk at google.com <rnk at google.com>] > *Sent:* Thursday, March 30, 2017 11:15 AM > *To:* Keane, Erich <erich.keane at intel.com> > *Cc:* llvm-dev at lists.llvm.org > *Subject:* Re: [llvm-dev] FileCheck feature request- by default ignore > IR-"headers" > > > > Yeah, this is a documented shortcoming of FileCheck: > > http://llvm.org/docs/TestingGuide.html#fragile-tests > > > > I think the most practical thing to do would be to make FileCheck reject > test cases that consist only CHECK-NOT directives, unless a flag is passed > to disable this behavior. > > > > On Thu, Mar 30, 2017 at 8:57 AM, Keane, Erich via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > Hi all- > I hope this is the right place for this. > > Anyway, the primary usage for the utility "FileCheck" is to pattern match > for specific values in a stream. This is perfectly consistent and > deterministic for the most part! However, when validating clang's LLVM-IR > generation, it is possible to make an invalid match against the top few > generated lines (the LLVM-IR Headers), which are inconsistent. > > For example (from my personal pain), we have a test: > clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the word > "macros". Unfortunately, my SVN workspace name was "ms_macros", which gave > me a false match. I'm a bit ashamed to say how long this took me to figure > out :) > > Anywya, to my feature request: to prevent someone else from going through > this pain again, would it be possible to have FileCheck ignore these first > 4 lines as default behavior? I suspect there might be a few tests that > would want to validate those headers, though I expect those are the vast > minority, so I would think this should be opt in. > > -Erich > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170331/22f022a3/attachment.html>