llvm dev - Mar 2016 - FileCheck: combining -DAG and -NOT

If you look at the FileCheck documentation page:
http://llvm.org/docs/CommandGuide/FileCheck.html
you'll find this intriguing example of combining -DAG with -NOT
(slightly amended to avoid some potential confusion):

; CHECK-DAG: BEFORE
; CHECK-NOT: BETWEEN
; CHECK-DAG: AFTER

The page says this will reject the sequence "AFTER BEFORE", which is
correct.  It's intuitively obvious that it will also reject the text
"BEFORE BETWEEN AFTER" as well.

Pop Quiz:  Will it accept or reject "BEFORE AFTER BETWEEN"?
No, don't go try it, that's cheating; what do you *think* will happen?
Take a minute, I won't mind.
---

So, here's the story.

Normally a -NOT line will scan the text between the points where the
preceding and following CHECK lines match.  By that mental model, you'd
expect the first -DAG to find BEFORE, the second -DAG to find AFTER, and
then -NOT would verify the absence of "BETWEEN" between those two
points.
All very intuitive, and I've certainly seen tests written to expect that.

However, things get a little funky when you have -NOT followed by -DAG.
What's the endpoint of the -NOT search?  The thing that follows isn't
a CHECK, it's a CHECK-DAG, except that -NOT is already kind of DAG-ish
so you have two DAG-ish groups staring at each other wondering who will
go first.

In this case, turns out, it's the -NOT who blinks first.  The endpoint
of the search is implicitly the end-of-input.  The Pop Quiz answer is:
*Rejected.*  The BETWEEN occurs after BEFORE, and before the end-of-input.

Here's the real question:  *Should* FileCheck run the second -DAG group
before it runs the -NOT group?  Then the range for the -NOT would be
bounded by the matching points for the surrounding -DAG lines, which
probably matches what basically everybody expects to happen.

Or, leave things as they are, and add a cautionary tale to the FileCheck
documentation page?  Given there's an actual documentation example, it
would seem inappropriate to make it an *error* if a -NOT is followed
by a -DAG!

Thanks,
--paulr

I'd probably have expected the behavior it has - that -DAG and -NOT are not
ordered with respect to each other, and form a bag of things ordered with
respect to enclosing CHECK:s.

If you want to try turning it into an error, you could find all the places
we do use interspersed -DAG and -NOT and have a discussion about whether
they're more buggy than they are useful/correct.

On Mon, Mar 7, 2016 at 5:39 PM, Robinson, Paul via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> If you look at the FileCheck documentation page:
> http://llvm.org/docs/CommandGuide/FileCheck.html
> you'll find this intriguing example of combining -DAG with -NOT
> (slightly amended to avoid some potential confusion):
>
> ; CHECK-DAG: BEFORE
> ; CHECK-NOT: BETWEEN
> ; CHECK-DAG: AFTER
>
> The page says this will reject the sequence "AFTER BEFORE", which
is
> correct.  It's intuitively obvious that it will also reject the text
> "BEFORE BETWEEN AFTER" as well.
>
> Pop Quiz:  Will it accept or reject "BEFORE AFTER BETWEEN"?
> No, don't go try it, that's cheating; what do you *think* will
happen?
> Take a minute, I won't mind.
> ---
>
> So, here's the story.
>
> Normally a -NOT line will scan the text between the points where the
> preceding and following CHECK lines match.  By that mental model, you'd
> expect the first -DAG to find BEFORE, the second -DAG to find AFTER, and
> then -NOT would verify the absence of "BETWEEN" between those two
points.
> All very intuitive, and I've certainly seen tests written to expect
that.
>
> However, things get a little funky when you have -NOT followed by -DAG.
> What's the endpoint of the -NOT search?  The thing that follows
isn't
> a CHECK, it's a CHECK-DAG, except that -NOT is already kind of DAG-ish
> so you have two DAG-ish groups staring at each other wondering who will
> go first.
>
> In this case, turns out, it's the -NOT who blinks first.  The endpoint
> of the search is implicitly the end-of-input.  The Pop Quiz answer is:
> *Rejected.*  The BETWEEN occurs after BEFORE, and before the end-of-input.
>
> Here's the real question:  *Should* FileCheck run the second -DAG group
> before it runs the -NOT group?  Then the range for the -NOT would be
> bounded by the matching points for the surrounding -DAG lines, which
> probably matches what basically everybody expects to happen.
>
> Or, leave things as they are, and add a cautionary tale to the FileCheck
> documentation page?  Given there's an actual documentation example, it
> would seem inappropriate to make it an *error* if a -NOT is followed
> by a -DAG!
>
> Thanks,
> --paulr
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160307/6848136f/attachment.html>

Nope.  Sorry, your expectation is incorrect.
"BETWEEN BEFORE AFTER" would be accepted, because: (1) the first –DAG
matches BEFORE; (2) the –NOT range starts at the preceding match-point, i.e. the
(end of the) BEFORE, thus does not find BETWEEN; (3) the second –DAG starts at
the same point as the –NOT.
That is, the first –DAG and the following –NOT *are* ordered; the –NOT and the
subsequent –DAG are *not* ordered.
You most certainly cannot intermix them freely and expect them all to look at
the same range; that is explicitly not the documented (or actual) behavior.
--paulr

From: David Blaikie [mailto:dblaikie at gmail.com]
Sent: Monday, March 07, 2016 8:43 PM
To: Robinson, Paul
Cc: llvm-dev at lists.llvm.org
Subject: Re: [llvm-dev] FileCheck: combining -DAG and -NOT

I'd probably have expected the behavior it has - that -DAG and -NOT are not
ordered with respect to each other, and form a bag of things ordered with
respect to enclosing CHECK:s.

If you want to try turning it into an error, you could find all the places we do
use interspersed -DAG and -NOT and have a discussion about whether they're
more buggy than they are useful/correct.

On Mon, Mar 7, 2016 at 5:39 PM, Robinson, Paul via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
If you look at the FileCheck documentation page:
http://llvm.org/docs/CommandGuide/FileCheck.html
you'll find this intriguing example of combining -DAG with -NOT
(slightly amended to avoid some potential confusion):

; CHECK-DAG: BEFORE
; CHECK-NOT: BETWEEN
; CHECK-DAG: AFTER

The page says this will reject the sequence "AFTER BEFORE", which is
correct.  It's intuitively obvious that it will also reject the text
"BEFORE BETWEEN AFTER" as well.

Pop Quiz:  Will it accept or reject "BEFORE AFTER BETWEEN"?
No, don't go try it, that's cheating; what do you *think* will happen?
Take a minute, I won't mind.
---

So, here's the story.

Normally a -NOT line will scan the text between the points where the
preceding and following CHECK lines match.  By that mental model, you'd
expect the first -DAG to find BEFORE, the second -DAG to find AFTER, and
then -NOT would verify the absence of "BETWEEN" between those two
points.
All very intuitive, and I've certainly seen tests written to expect that.

However, things get a little funky when you have -NOT followed by -DAG.
What's the endpoint of the -NOT search?  The thing that follows isn't
a CHECK, it's a CHECK-DAG, except that -NOT is already kind of DAG-ish
so you have two DAG-ish groups staring at each other wondering who will
go first.

In this case, turns out, it's the -NOT who blinks first.  The endpoint
of the search is implicitly the end-of-input.  The Pop Quiz answer is:
*Rejected.*  The BETWEEN occurs after BEFORE, and before the end-of-input.

Here's the real question:  *Should* FileCheck run the second -DAG group
before it runs the -NOT group?  Then the range for the -NOT would be
bounded by the matching points for the surrounding -DAG lines, which
probably matches what basically everybody expects to happen.

Or, leave things as they are, and add a cautionary tale to the FileCheck
documentation page?  Given there's an actual documentation example, it
would seem inappropriate to make it an *error* if a -NOT is followed
by a -DAG!

Thanks,
--paulr

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160308/9274e505/attachment-0001.html>