llvm dev - Jul 2020 - [RFC] Compiled regression tests.

Michael Kruse via llvm-dev <llvm-dev at lists.llvm.org> writes:
> Am Mi., 1. Juli 2020 um 10:18 Uhr schrieb Robinson, Paul <
> paul.robinson at sony.com>:
>
>> The test as written is fragile because it requires a certain ordering. 
If
>> the output order is not important, use CHECK-DAG rather than CHECK. 
This
>> would be a failure to understand the testing tool.
>>
>
> CHECK-DAG does not help here since what changes is within a list on the
> same line, and we have no CHECK-SAME-DAG or CHECK-DAG-SAME. Even if we had
> it, the actual line that changed is textually the same and FileCheck would
> need to backtrack deep into the following lines for alternative placeholder
> substitutions. It would look like
>
> CHECK-SAME-DAG: ![[ACCESS_GROUP_INNER:[0-9]+]]
> CHECK-SAME-DAG: ,
> CHECK-SAME-DAG: ![[ACCESS_GROUP_OUTER:[0-9]+]]
Would this not work?

CHECK-SAME: ![[ACCESS_GROUP_INNER:[0-9]+]]
CHECK-SAME: ![[ACCESS_GROUP_OUTER:[0-9]+]]

I don't think CHECK-SAME is sensitive to order within the line.  This
works for me in my metadata tests but maybe I've just been lucky.
> IMHO having a tool that allows to better express what is intended to be
> tested is already worth a lot.
Exactly.
> For instance, we usually don't care about SSA value names or MDNode
> numbers, but we have to put extra work into regex-ing away those names
> in FileCheck tests and as a result, most tests we have do still expect
> the exact number for metadata nodes. This is a problem if we we want
> to emit new metadata nodes in that all those tests need to be updated.
IME, many of our tests check way too much.  Codegen tests are
particularly egregious in this regard because many are (re-)generated
and match whole functions of asm.

One of the major problems with using FileCheck regexps in generated
tests is that it's very difficult to take an update_*_test_checks.py
output and whittle it down because the FileCheck regexps define and use
variables which need to be massaged while altering the test.  I have
some patches that help alleviate this.
> This problem goes away if the test method by default ignored value
> names/MDNode numbers and software development people had to put extra
> work if they actually want to verify this.
If the test method is appropriate for the test, sure.  But I suspect
we'll have FileCheck tests for a good long while and also making that
experience better is well worth the effort.

                  -David

Am Mi., 1. Juli 2020 um 13:59 Uhr schrieb David Greene <david.greene at
hpe.com>:
> > CHECK-DAG does not help here since what changes is within a list on
the
> > same line, and we have no CHECK-SAME-DAG or CHECK-DAG-SAME. Even if we
had
> > it, the actual line that changed is textually the same and FileCheck
would
> > need to backtrack deep into the following lines for alternative
placeholder
> > substitutions. It would look like
> >
> > CHECK-SAME-DAG: ![[ACCESS_GROUP_INNER:[0-9]+]]
> > CHECK-SAME-DAG: ,
> > CHECK-SAME-DAG: ![[ACCESS_GROUP_OUTER:[0-9]+]]
>
> Would this not work?
>
> CHECK-SAME: ![[ACCESS_GROUP_INNER:[0-9]+]]
> CHECK-SAME: ![[ACCESS_GROUP_OUTER:[0-9]+]]
>
> I don't think CHECK-SAME is sensitive to order within the line.  This
> works for me in my metadata tests but maybe I've just been lucky.
AFAIU this will assume ACCESS_GROUP_INNER to appear before
ACCESS_GROUP_OUTER since CHECK-SAME will continue matching at the
column where the previous match stopped, thus have the same result as
written in the same line. If CHECK-SAME starts over at the beginning
of the line  ACCESS_GROUP_OUTER/ACCESS_GROUP_INNER will both match the
same.


Michael

Michael Kruse via llvm-dev <llvm-dev at lists.llvm.org> writes:
>> Would this not work?
>>
>> CHECK-SAME: ![[ACCESS_GROUP_INNER:[0-9]+]]
>> CHECK-SAME: ![[ACCESS_GROUP_OUTER:[0-9]+]]
>>
>> I don't think CHECK-SAME is sensitive to order within the line. 
This
>> works for me in my metadata tests but maybe I've just been lucky.
>
> AFAIU this will assume ACCESS_GROUP_INNER to appear before
> ACCESS_GROUP_OUTER since CHECK-SAME will continue matching at the
> column where the previous match stopped, thus have the same result as
> written in the same line. If CHECK-SAME starts over at the beginning
> of the line  ACCESS_GROUP_OUTER/ACCESS_GROUP_INNER will both match the
> same.
Is the behavior of CHECK-SAME documented anywhere?  It's not on the main
FileCheck web page.  I assumed it started over from the beginning of the
line but now I'm not sure.

              -David