llvm dev - Jan 2020 - [RFC][FileCheck] New option to negate check patterns

Hi all,

There have been a few cases recently where I've noticed two test cases in
the same lit test that do the same thing except invert the CHECK, to show
that something is NOT present. I'm talking about something like the
following:

# RUN: llvm-sometool --print-string | FileCheck %s --check-prefix=STRING
# RUN: llvm-sometool --no-print-string | FileCheck %s
--check-prefix=NO-STRING
# STRING: This is the string
# NO-STRING-NOT: This is the string

In such cases, as can be seen, the CHECK line effectively has to be
duplicated (either in an explicit check like in the above example, or via
--implicit-check-not). Duplication is generally bad, especially in this
sort of case, as it only takes a typo in the NOT pattern, or a careless
developer/reviewer pair changing the output to cause the NOT pattern to no
longer be useful.

I'd like to propose a new FileCheck option (e.g.
--check-not-prefix/--check-not-prefixes) which allows implicitly converting
a check prefix to a -NOT version of the same prefix. That would allow
writing the above example as:

# RUN: llvm-sometool --print-string | FileCheck %s --check-prefix=STRING
# RUN: llvm-sometool --no-print-string | FileCheck %s
--check-not-prefix=STRING
# STRING: This is the string

If there was a typo or somebody changed the string output, this mechanism
would ensure there is no chance of the pattern rotting.

Caveat: I don't know what would be the appropriate way of handling
non-trivial checks, i.e. existing CHECK-NOT/CHECK-NEXT/CHECK-SAME etc. I'd
appreciate any ideas on this.

Thoughts?

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200131/83c34049/attachment.html>

Hi James,

I feel it might be confusing to have a CHECK becomes effectively a CHECK-NOT,
especially if the RUN line is far from the CHECK line (which is often the case
when a single RUN line drives several groups of CHECK directives (e.g. code
generation tested for several functions for a specific feature, like PIC). You
also loose control on where the NOT should be: it would have to be at the same
location as the CHECK even though for the NOT case you might want to check it
somewhere else.

How about having a concept of regex variables where you give a name to a given
directive's pattern which you could reuse as another pattern. Something like
(syntax TBD):

CHECK<NAME>: mov [[REG:r[0-9]+]], #42
CHECK-NOT: <NAME>

Best regards,

Thomas
________________________________
From: James Henderson <jh7370.2008 at my.bristol.ac.uk>
Sent: 31 January 2020 09:14
To: llvm-dev <llvm-dev at lists.llvm.org>; Thomas Preud'homme
<thomasp at graphcore.ai>; Paul Robinson <paul.robinson at
sony.com>; George Rimar <grimar at accesssoftek.com>
Subject: [RFC][FileCheck] New option to negate check patterns


     [This message was sent from somebody outside of your organisation]



Hi all,

There have been a few cases recently where I've noticed two test cases in
the same lit test that do the same thing except invert the CHECK, to show that
something is NOT present. I'm talking about something like the following:

# RUN: llvm-sometool --print-string | FileCheck %s --check-prefix=STRING
# RUN: llvm-sometool --no-print-string | FileCheck %s --check-prefix=NO-STRING
# STRING: This is the string
# NO-STRING-NOT: This is the string

In such cases, as can be seen, the CHECK line effectively has to be duplicated
(either in an explicit check like in the above example, or via
--implicit-check-not). Duplication is generally bad, especially in this sort of
case, as it only takes a typo in the NOT pattern, or a careless
developer/reviewer pair changing the output to cause the NOT pattern to no
longer be useful.

I'd like to propose a new FileCheck option (e.g.
--check-not-prefix/--check-not-prefixes) which allows implicitly converting a
check prefix to a -NOT version of the same prefix. That would allow writing the
above example as:

# RUN: llvm-sometool --print-string | FileCheck %s --check-prefix=STRING
# RUN: llvm-sometool --no-print-string | FileCheck %s --check-not-prefix=STRING
# STRING: This is the string

If there was a typo or somebody changed the string output, this mechanism would
ensure there is no chance of the pattern rotting.

Caveat: I don't know what would be the appropriate way of handling
non-trivial checks, i.e. existing CHECK-NOT/CHECK-NEXT/CHECK-SAME etc. I'd
appreciate any ideas on this.

Thoughts?

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200131/5be22620/attachment-0001.html>

​Hi all,

> I feel it might be confusing to have a CHECK becomes effectively a
CHECK-NOT,
> especially if the RUN line is far from the CHECK line (which is often the
case when
> a single RUN line drives several groups of CHECK directives (e.g. code
generation
> tested for several functions for a specific feature, like PIC). You also
loose control
> on where the NOT should be: it would have to be at the same location as the
> CHECK even though for the NOT case you might want to check it somewhere
else.

I think I agree with Thomas.

+ the relationship with "CHECK-NOT/CHECK-NEXT/CHECK-SAME" might make
things overcomplicated probably.
> How about having a concept of regex variables where you give a name
> to a given directive's pattern which you could reuse as another
pattern. Something like (syntax TBD):
>
> CHECK<NAME>: mov [[REG:r[0-9]+]], #42
> CHECK-NOT: <NAME>
I.e. without adding a new optinons for FileCheck, something like the following?
# RUN: llvm-sometool --print-string | FileCheck %s --check-prefix=CHECK1
# RUN: llvm-sometool --no-print-string | FileCheck %s --check-prefix=CHECK2

CHECK1<NAME>: mov [[REG:r[0-9]+]], #42
CHECK2-NOT: <NAME>

It might work probably.  We already have the ability to name parts of
the output checked:

// CHECK:      Dynamic Relocations {
// CHECK-NEXT:   {{.*}} R_AARCH64_RELATIVE - [[BAR_ADDR:.*]]

// CHECK:      Symbols [
// CHECK-NEXT:     Value: [[BAR_ADDR]]

So adding a way for naming the whole line does not look
an unreasonable/inconsistent extention to me I think.

Best regards,​
George | Developer | Access Softek, Inc​​​​​​​

________________________________
From: James Henderson <jh7370.2008 at my.bristol.ac.uk>
Sent: 31 January 2020 09:14
To: llvm-dev <llvm-dev at lists.llvm.org>; Thomas Preud'homme
<thomasp at graphcore.ai>; Paul Robinson <paul.robinson at
sony.com>; George Rimar <grimar at accesssoftek.com>
Subject: [RFC][FileCheck] New option to negate check patterns


     [This message was sent from somebody outside of your organisation]



Hi all,

There have been a few cases recently where I've noticed two test cases in
the same lit test that do the same thing except invert the CHECK, to show that
something is NOT present. I'm talking about something like the following:

# RUN: llvm-sometool --print-string | FileCheck %s --check-prefix=STRING
# RUN: llvm-sometool --no-print-string | FileCheck %s --check-prefix=NO-STRING
# STRING: This is the string
# NO-STRING-NOT: This is the string

In such cases, as can be seen, the CHECK line effectively has to be duplicated
(either in an explicit check like in the above example, or via
--implicit-check-not). Duplication is generally bad, especially in this sort of
case, as it only takes a typo in the NOT pattern, or a careless
developer/reviewer pair changing the output to cause the NOT pattern to no
longer be useful.

I'd like to propose a new FileCheck option (e.g.
--check-not-prefix/--check-not-prefixes) which allows implicitly converting a
check prefix to a -NOT version of the same prefix. That would allow writing the
above example as:

# RUN: llvm-sometool --print-string | FileCheck %s --check-prefix=STRING
# RUN: llvm-sometool --no-print-string | FileCheck %s --check-not-prefix=STRING
# STRING: This is the string

If there was a typo or somebody changed the string output, this mechanism would
ensure there is no chance of the pattern rotting.

Caveat: I don't know what would be the appropriate way of handling
non-trivial checks, i.e. existing CHECK-NOT/CHECK-NEXT/CHECK-SAME etc. I'd
appreciate any ideas on this.

Thoughts?

James


** We have updated our privacy policy, which contains important information
about how we collect and process your personal data. To read the policy, please
click here<http://www.graphcore.ai/privacy> **

This email and its attachments are intended solely for the addressed recipients
and may contain confidential or legally privileged information.
If you are not the intended recipient you must not copy, distribute or
disseminate this email in any way; to do so may be unlawful.

Any personal data/special category personal data herein are processed in
accordance with UK data protection legislation.
All associated feasible security measures are in place. Further details are
available from the Privacy Notice on the website and/or from the Company.

Graphcore Limited (registered in England and Wales with registration number
10185006) is registered at 107 Cheapside, London, UK, EC2V 6DN.
This message was scanned for viruses upon transmission. However Graphcore
accepts no liability for any such transmission.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200131/6d23e49d/attachment.html>