llvm dev - Jul 2020 - [FileCheck] RFC: Add support for line anchors.

Hello,

I was wondering about extending FileCheck to enable creating line
anchors. These are numeric variables that hold the value of the line
number that where they were defined.

The motivation for this comes from test cases using clang-based
diagnostics which often include notes attached to source locations in
different parts of the file. In order to test for the correct location
of the note, the line number has to be written explicitly or as an,
often large, offset to the current line. This harms both readability
and maintainability. Using this new system one could append a line of
interest with an anchor-comment and refer back to it inside FileCheck.

I have created a basic patch that implements this here 
https://reviews.llvm.org/D84037 but it definitely needs a few looks
over by people who are more clued up on the internal of FileCheck.

The current syntax, based off this patch, is as follows:
  - Added a command line option called `anchor-prefix` which is a
comma-seperated list of prefixes to be used when declaring anchors.
This is defaulted to `LINE-ANCHOR`
  - To declare a anchor in the test file use
    `LINE-ANCHOR: ANCHOR_NAME`
    note: If you specify a different anchor-prefix using the command
line, use that name instead of `LINE-ANCHOR`
    ANCHOR_NAME Follows the rules all other variable names aside from
the fact it can't start with '$'.
  - When referring to an anchor in a check use the same numeric
variable syntax that FileCheck already supports:
    `CHECK: [[#ANCHOR_NAME]][[#ANCHOR_NAME+1]]`

Here is a brief (contrived) example of the usage of this:
```
#define BAD_FUNCTION() badFunction() // LINE-ANCHOR: BAD_FUNC
// Further down in the file
  BAD_FUNCTION();
CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
CHECK-NOTES :[[#BAD-FUNC]]:3: note: expanded from macro 'BAD_FUNCTION'
```

Regards,
Nathan James

Hi Nathan,

On Fri, Jul 17, 2020 at 12:23 PM Nathan James via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Hello,
>
> I was wondering about extending FileCheck to enable creating line
> anchors. These are numeric variables that hold the value of the line
> number that where they were defined.
>
I think something like this could be useful.  However, I think it would be
more useful to have a general directive for defining FileCheck variables
inline without trying to capture from input.  For example:

```
#define BAD_FUNCTION() badFunction() // CHECK-DEFINE: [[#BAD_FUNC:@LINE]]
// Further down in the file
  BAD_FUNCTION();
CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
CHECK-NOTES :[[#BAD_FUNC]]:3: note: expanded from macro 'BAD_FUNCTION'
```

The exact syntax is debatable.

I think this form is more useful because it can also define strings or
numerics (or maybe even patterns) to be reused in multiple FileCheck
directives across multiple FileCheck calls from different RUN lines.
Currently, you either have to capture such a variable from the input or
specify it with -D on every FileCheck call that needs it.

James Henderson: Weren't you working on something like this?

Thanks.

Joel

>
> The motivation for this comes from test cases using clang-based
> diagnostics which often include notes attached to source locations in
> different parts of the file. In order to test for the correct location
> of the note, the line number has to be written explicitly or as an,
> often large, offset to the current line. This harms both readability
> and maintainability. Using this new system one could append a line of
> interest with an anchor-comment and refer back to it inside FileCheck.
>
> I have created a basic patch that implements this here
> https://reviews.llvm.org/D84037 but it definitely needs a few looks
> over by people who are more clued up on the internal of FileCheck.
>
> The current syntax, based off this patch, is as follows:
>   - Added a command line option called `anchor-prefix` which is a
> comma-seperated list of prefixes to be used when declaring anchors.
> This is defaulted to `LINE-ANCHOR`
>   - To declare a anchor in the test file use
>     `LINE-ANCHOR: ANCHOR_NAME`
>     note: If you specify a different anchor-prefix using the command
> line, use that name instead of `LINE-ANCHOR`
>     ANCHOR_NAME Follows the rules all other variable names aside from
> the fact it can't start with '$'.
>   - When referring to an anchor in a check use the same numeric
> variable syntax that FileCheck already supports:
>     `CHECK: [[#ANCHOR_NAME]][[#ANCHOR_NAME+1]]`
>
> Here is a brief (contrived) example of the usage of this:
> ```
> #define BAD_FUNCTION() badFunction() // LINE-ANCHOR: BAD_FUNC
> // Further down in the file
>   BAD_FUNCTION();
> CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
> CHECK-NOTES :[[#BAD-FUNC]]:3: note: expanded from macro
'BAD_FUNCTION'
> ```
>
> Regards,
> Nathan James
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200717/8bd74217/attachment.html>

On Fri, Jul 17, 2020 at 2:52 PM Joel E. Denny <jdenny.ornl at gmail.com>
wrote:
> Hi Nathan,
>
> On Fri, Jul 17, 2020 at 12:23 PM Nathan James via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> Hello,
>>
>> I was wondering about extending FileCheck to enable creating line
>> anchors. These are numeric variables that hold the value of the line
>> number that where they were defined.
>>
>
> I think something like this could be useful.  However, I think it would be
> more useful to have a general directive for defining FileCheck variables
> inline without trying to capture from input.  For example:
>
> ```
> #define BAD_FUNCTION() badFunction() // CHECK-DEFINE: [[#BAD_FUNC:@LINE]]
> // Further down in the file
>   BAD_FUNCTION();
> CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
> CHECK-NOTES :[[#BAD_FUNC]]:3: note: expanded from macro
'BAD_FUNCTION'
> ```
>
> The exact syntax is debatable.
>
> I think this form is more useful because it can also define strings or
> numerics (or maybe even patterns) to be reused in multiple FileCheck
> directives across multiple FileCheck calls from different RUN lines.
> Currently, you either have to capture such a variable from the input or
> specify it with -D on every FileCheck call that needs it.
>
> James Henderson: Weren't you working on something like this?
>
> Thanks.
>
> Joel
>
>
>>
>> The motivation for this comes from test cases using clang-based
>> diagnostics which often include notes attached to source locations in
>> different parts of the file. In order to test for the correct location
>> of the note, the line number has to be written explicitly or as an,
>> often large, offset to the current line. This harms both readability
>> and maintainability. Using this new system one could append a line of
>> interest with an anchor-comment and refer back to it inside FileCheck.
>>
>
Also, just in case you weren't aware, there is a Clang facility for
checking its own diagnostics:

http://clang.llvm.org/doxygen/classclang_1_1VerifyDiagnosticConsumer.html#details

You'd put a directive for each diagnostic on or near the line it cites.

Joel

>
>> I have created a basic patch that implements this here
>> https://reviews.llvm.org/D84037 but it definitely needs a few looks
>> over by people who are more clued up on the internal of FileCheck.
>>
>> The current syntax, based off this patch, is as follows:
>>   - Added a command line option called `anchor-prefix` which is a
>> comma-seperated list of prefixes to be used when declaring anchors.
>> This is defaulted to `LINE-ANCHOR`
>>   - To declare a anchor in the test file use
>>     `LINE-ANCHOR: ANCHOR_NAME`
>>     note: If you specify a different anchor-prefix using the command
>> line, use that name instead of `LINE-ANCHOR`
>>     ANCHOR_NAME Follows the rules all other variable names aside from
>> the fact it can't start with '$'.
>>   - When referring to an anchor in a check use the same numeric
>> variable syntax that FileCheck already supports:
>>     `CHECK: [[#ANCHOR_NAME]][[#ANCHOR_NAME+1]]`
>>
>> Here is a brief (contrived) example of the usage of this:
>> ```
>> #define BAD_FUNCTION() badFunction() // LINE-ANCHOR: BAD_FUNC
>> // Further down in the file
>>   BAD_FUNCTION();
>> CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
>> CHECK-NOTES :[[#BAD-FUNC]]:3: note: expanded from macro
'BAD_FUNCTION'
>> ```
>>
>> Regards,
>> Nathan James
>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200717/1147afaf/attachment.html>

Hi Joel,

That sounds like a very nice idea and definitely a direction I could
get behind. However I feel that outside the use case I suggested, this
functionality would only be used to compress CHECK lines that contain
repeated text, not saying its a bad or good thing though. WDYT?

~Nathan

On Fri, 2020-07-17 at 14:52 -0400, Joel E. Denny via cfe-dev
wrote:
> Hi Nathan,
> 
> On Fri, Jul 17, 2020 at 12:23 PM Nathan James via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
> > Hello,
> > 
> > I was wondering about extending FileCheck to enable creating line
> > anchors. These are numeric variables that hold the value of the
> > line
> > number that where they were defined.
> 
> I think something like this could be useful.  However, I think it
> would be more useful to have a general directive for defining
> FileCheck variables inline without trying to capture from input.  For
> example:
> 
> ```
> #define BAD_FUNCTION() badFunction() // CHECK-DEFINE:
> [[#BAD_FUNC:@LINE]]
> // Further down in the file
>   BAD_FUNCTION();
> CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
> CHECK-NOTES :[[#BAD_FUNC]]:3: note: expanded from macro
> 'BAD_FUNCTION'
> ```
> 
> The exact syntax is debatable.
> 
> I think this form is more useful because it can also define strings
> or numerics (or maybe even patterns) to be reused in multiple
> FileCheck directives across multiple FileCheck calls from different
> RUN lines.  Currently, you either have to capture such a variable
> from the input or specify it with -D on every FileCheck call that
> needs it.
> 
> James Henderson: Weren't you working on something like this?
> 
> Thanks.
> 
> Joel
>  
> > The motivation for this comes from test cases using clang-based
> > diagnostics which often include notes attached to source locations
> > in
> > different parts of the file. In order to test for the correct
> > location
> > of the note, the line number has to be written explicitly or as an,
> > often large, offset to the current line. This harms both
> > readability
> > and maintainability. Using this new system one could append a line
> > of
> > interest with an anchor-comment and refer back to it inside
> > FileCheck.
> > 
> > I have created a basic patch that implements this here 
> > https://reviews.llvm.org/D84037 but it definitely needs a few looks
> > over by people who are more clued up on the internal of FileCheck.
> > 
> > The current syntax, based off this patch, is as follows:
> >   - Added a command line option called `anchor-prefix` which is a
> > comma-seperated list of prefixes to be used when declaring anchors.
> > This is defaulted to `LINE-ANCHOR`
> >   - To declare a anchor in the test file use
> >     `LINE-ANCHOR: ANCHOR_NAME`
> >     note: If you specify a different anchor-prefix using the
> > command
> > line, use that name instead of `LINE-ANCHOR`
> >     ANCHOR_NAME Follows the rules all other variable names aside
> > from
> > the fact it can't start with '$'.
> >   - When referring to an anchor in a check use the same numeric
> > variable syntax that FileCheck already supports:
> >     `CHECK: [[#ANCHOR_NAME]][[#ANCHOR_NAME+1]]`
> > 
> > Here is a brief (contrived) example of the usage of this:
> > ```
> > #define BAD_FUNCTION() badFunction() // LINE-ANCHOR: BAD_FUNC
> > // Further down in the file
> >   BAD_FUNCTION();
> > CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
> > CHECK-NOTES :[[#BAD-FUNC]]:3: note: expanded from macro
> > 'BAD_FUNCTION'
> > ```
> > 
> > Regards,
> > Nathan James
> > 
> > 
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev