llvm dev - May 2018 - RFC: [FileCheck] CHECK-DAG for multiple occurrences of string

Hi Jessica,

This time I'm replying all....

On Fri, May 4, 2018 at 12:45 PM, Jessica Paquette <jpaquette at apple.com>
wrote:
> I would personally like a feature like that in FileCheck because it would
> make it a lot easier to write MachineOutliner tests, and would make the
> tests significantly smaller and easier to understand.
>
How do MachineOutliner tests accomplish this now?  Can you point me to an
example?

Thanks.

Joel

>
> - Jessica
>
> > On May 4, 2018, at 8:40 AM, Joel E. Denny via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > Hi,
> >
> > Using FileCheck, I have not found a way to make a group of CHECK-DAG
> directives match multiple occurrences of a string.  For example, I naively
> thought the following would match successfully:
> >
> > ```
> > $ cat checks.txt
> > // CHECK: start
> > // CHECK-DAG: foo
> > // CHECK-DAG: foo
> > // CHECK-DAG: bar
> > // CHECK-NEXT: end
> >
> > $ cat input.txt
> > start
> > foo
> > bar
> > foo
> > end
> >
> > $ FileCheck --input-file=input.txt checks.txt
> > checks.txt:5:16: error: CHECK-NEXT: is not on the line after the
> previous match
> > // CHECK-NEXT: end
> >                ^
> > input.txt:5:1: note: 'next' match was here
> > end
> > ^
> > input.txt:3:4: note: previous match ended here
> > bar
> >    ^
> > input.txt:4:1: note: non-matching line after previous match is here
> > foo
> > ^
> > ```
> >
> > The trouble is that both "CHECK-DAG: foo" directives match
the first
> "foo".
> >
> > I'd like this ability for testing a parallel program that outputs
a
> series of non-unique strings in non-deterministic order.  Am I trying to
> push FileCheck beyond its intended domain?  Is there some existing feature
> for this purpose that I've overlooked?  If not, I see two potential
> solutions:
> >
> > 1. In a CHECK-DAG group, don't let the matches for patterns
overlap.
> >
> > 2. Add a new CHECK-DAG-N directive, where N is some integer, to
express
> that a pattern must have N non-overlapping matches.
> >
> > An advantage of #1 that the intuitive way (at least in my mind) of
> expressing multiple occurrences of a string, as in the example above, would
> work.  An advantage of #2 is that existing CHECK-DAG functionality would
> not change, and so there should be no chance of impacting existing well
> formed tests.
> >
> > To understand the issue better, I've prototyped #2.  It still
needs test
> cases and documentation, so it's not ready for a formal patch review. 
If
> people like the idea, I'll polish it up.
> >
> > Thanks.
> >
> > Joel
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180504/ebf6bd06/attachment.html>

Right now, the tests try to accomplish the following

1. Define a sequence of instructions (e.g a,b,c)
2. Insert that sequence into k places with an unique instruction between them to
make sure the outliner will yank them out *without overlaps*
3. Check for k calls to an outlined function
4. Check that the outlined sequence still exists in the program

This can result in some pretty long tests which would be a lot easier to define
if we could say something like “this function must contain k instances of an
outlined call, and not contain this sequence of instructions”.

test/CodeGen/machine-outliner.mir is a pretty good example of a test that would
benefit from this sort of thing.

- Jessica
> On May 4, 2018, at 10:05 AM, Joel E. Denny <jdenny.ornl at gmail.com>
wrote:
> 
> Hi Jessica,
> 
> This time I'm replying all....
> 
> On Fri, May 4, 2018 at 12:45 PM, Jessica Paquette <jpaquette at
apple.com <mailto:jpaquette at apple.com>> wrote:
> I would personally like a feature like that in FileCheck because it would
make it a lot easier to write MachineOutliner tests, and would make the tests
significantly smaller and easier to understand.
> 
> How do MachineOutliner tests accomplish this now?  Can you point me to an
example?
> 
> Thanks.
> 
> Joel
>  
> 
> - Jessica
> 
> > On May 4, 2018, at 8:40 AM, Joel E. Denny via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> > 
> > Hi,
> > 
> > Using FileCheck, I have not found a way to make a group of CHECK-DAG
directives match multiple occurrences of a string.  For example, I naively
thought the following would match successfully:
> > 
> > ```
> > $ cat checks.txt 
> > // CHECK: start
> > // CHECK-DAG: foo
> > // CHECK-DAG: foo
> > // CHECK-DAG: bar
> > // CHECK-NEXT: end
> > 
> > $ cat input.txt 
> > start
> > foo
> > bar
> > foo
> > end
> > 
> > $ FileCheck --input-file=input.txt checks.txt
> > checks.txt:5:16: error: CHECK-NEXT: is not on the line after the
previous match
> > // CHECK-NEXT: end
> >                ^
> > input.txt:5:1: note: 'next' match was here
> > end
> > ^
> > input.txt:3:4: note: previous match ended here
> > bar
> >    ^
> > input.txt:4:1: note: non-matching line after previous match is here
> > foo
> > ^
> > ```
> > 
> > The trouble is that both "CHECK-DAG: foo" directives match
the first "foo".
> > 
> > I'd like this ability for testing a parallel program that outputs
a series of non-unique strings in non-deterministic order.  Am I trying to push
FileCheck beyond its intended domain?  Is there some existing feature for this
purpose that I've overlooked?  If not, I see two potential solutions:
> > 
> > 1. In a CHECK-DAG group, don't let the matches for patterns
overlap.
> > 
> > 2. Add a new CHECK-DAG-N directive, where N is some integer, to
express that a pattern must have N non-overlapping matches.
> > 
> > An advantage of #1 that the intuitive way (at least in my mind) of
expressing multiple occurrences of a string, as in the example above, would
work.  An advantage of #2 is that existing CHECK-DAG functionality would not
change, and so there should be no chance of impacting existing well formed
tests.
> > 
> > To understand the issue better, I've prototyped #2.  It still
needs test cases and documentation, so it's not ready for a formal patch
review.  If people like the idea, I'll polish it up.
> > 
> > Thanks.
> > 
> > Joel
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180507/37c034e7/attachment.html>

On Mon, May 7, 2018 at 1:23 PM, Jessica Paquette <jpaquette at apple.com>
wrote:
> Right now, the tests try to accomplish the following
>
> 1. Define a sequence of instructions (e.g a,b,c)
> 2. Insert that sequence into k places with an unique instruction between
> them to make sure the outliner will yank them out *without overlaps*
> 3. Check for k calls to an outlined function
> 4. Check that the outlined sequence still exists in the program
>
> This can result in some pretty long tests which would be a lot easier to
> define if we could say something like “this function must contain k
> instances of an outlined call, and not contain this sequence of
> instructions”.
>
> test/CodeGen/machine-outliner.mir is a pretty good example of a test that
> would benefit from this sort of thing.
>
I'm not sure if what I'm proposing will do what you need.  CHECK-DAG is
for
cases where a set of expected strings is unordered.  I'm trying to extend
it to handle cases where those strings aren't unique either.  In your
example, I see the non-unique property but not the unordered property.  Did
I miss it?

Moreover, it seems your example could be handled now by two FileCheck calls
with different prefixes, say OUT and INS.  The first would use a sequence
of k "OUT:" directives to ensure that there are k outlined calls.  The
second would use "INS-NOT:" directives to ensure other instructions
don't
occur.  The calls could also have a common prefix, say CHECK, to use for
the function labels.  Does that do what you want?

Thanks

Joel


>
> - Jessica
>
> On May 4, 2018, at 10:05 AM, Joel E. Denny <jdenny.ornl at gmail.com>
wrote:
>
> Hi Jessica,
>
> This time I'm replying all....
>
> On Fri, May 4, 2018 at 12:45 PM, Jessica Paquette <jpaquette at
apple.com>
> wrote:
>
>> I would personally like a feature like that in FileCheck because it
would
>> make it a lot easier to write MachineOutliner tests, and would make the
>> tests significantly smaller and easier to understand.
>>
>
> How do MachineOutliner tests accomplish this now?  Can you point me to an
> example?
>
> Thanks.
>
> Joel
>
>
>>
>> - Jessica
>>
>> > On May 4, 2018, at 8:40 AM, Joel E. Denny via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>> >
>> > Hi,
>> >
>> > Using FileCheck, I have not found a way to make a group of
CHECK-DAG
>> directives match multiple occurrences of a string.  For example, I
naively
>> thought the following would match successfully:
>> >
>> > ```
>> > $ cat checks.txt
>> > // CHECK: start
>> > // CHECK-DAG: foo
>> > // CHECK-DAG: foo
>> > // CHECK-DAG: bar
>> > // CHECK-NEXT: end
>> >
>> > $ cat input.txt
>> > start
>> > foo
>> > bar
>> > foo
>> > end
>> >
>> > $ FileCheck --input-file=input.txt checks.txt
>> > checks.txt:5:16: error: CHECK-NEXT: is not on the line after the
>> previous match
>> > // CHECK-NEXT: end
>> >                ^
>> > input.txt:5:1: note: 'next' match was here
>> > end
>> > ^
>> > input.txt:3:4: note: previous match ended here
>> > bar
>> >    ^
>> > input.txt:4:1: note: non-matching line after previous match is
here
>> > foo
>> > ^
>> > ```
>> >
>> > The trouble is that both "CHECK-DAG: foo" directives
match the first
>> "foo".
>> >
>> > I'd like this ability for testing a parallel program that
outputs a
>> series of non-unique strings in non-deterministic order.  Am I trying
to
>> push FileCheck beyond its intended domain?  Is there some existing
feature
>> for this purpose that I've overlooked?  If not, I see two potential
>> solutions:
>> >
>> > 1. In a CHECK-DAG group, don't let the matches for patterns
overlap.
>> >
>> > 2. Add a new CHECK-DAG-N directive, where N is some integer, to
express
>> that a pattern must have N non-overlapping matches.
>> >
>> > An advantage of #1 that the intuitive way (at least in my mind) of
>> expressing multiple occurrences of a string, as in the example above,
would
>> work.  An advantage of #2 is that existing CHECK-DAG functionality
would
>> not change, and so there should be no chance of impacting existing well
>> formed tests.
>> >
>> > To understand the issue better, I've prototyped #2.  It still
needs
>> test cases and documentation, so it's not ready for a formal patch
review.
>> If people like the idea, I'll polish it up.
>> >
>> > Thanks.
>> >
>> > Joel
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180507/ee52c3c3/attachment.html>