llvm dev - Jul 2020 - [cfe-dev] [FileCheck] RFC: Add support for line anchors.

Hi Joel,

That sounds like a very nice idea and definitely a direction I could
get behind. However I feel that outside the use case I suggested, this
functionality would only be used to compress CHECK lines that contain
repeated text, not saying its a bad or good thing though. WDYT?

~Nathan

On Fri, 2020-07-17 at 14:52 -0400, Joel E. Denny via cfe-dev
wrote:
> Hi Nathan,
> 
> On Fri, Jul 17, 2020 at 12:23 PM Nathan James via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
> > Hello,
> > 
> > I was wondering about extending FileCheck to enable creating line
> > anchors. These are numeric variables that hold the value of the
> > line
> > number that where they were defined.
> 
> I think something like this could be useful.  However, I think it
> would be more useful to have a general directive for defining
> FileCheck variables inline without trying to capture from input.  For
> example:
> 
> ```
> #define BAD_FUNCTION() badFunction() // CHECK-DEFINE:
> [[#BAD_FUNC:@LINE]]
> // Further down in the file
>   BAD_FUNCTION();
> CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
> CHECK-NOTES :[[#BAD_FUNC]]:3: note: expanded from macro
> 'BAD_FUNCTION'
> ```
> 
> The exact syntax is debatable.
> 
> I think this form is more useful because it can also define strings
> or numerics (or maybe even patterns) to be reused in multiple
> FileCheck directives across multiple FileCheck calls from different
> RUN lines.  Currently, you either have to capture such a variable
> from the input or specify it with -D on every FileCheck call that
> needs it.
> 
> James Henderson: Weren't you working on something like this?
> 
> Thanks.
> 
> Joel
>  
> > The motivation for this comes from test cases using clang-based
> > diagnostics which often include notes attached to source locations
> > in
> > different parts of the file. In order to test for the correct
> > location
> > of the note, the line number has to be written explicitly or as an,
> > often large, offset to the current line. This harms both
> > readability
> > and maintainability. Using this new system one could append a line
> > of
> > interest with an anchor-comment and refer back to it inside
> > FileCheck.
> > 
> > I have created a basic patch that implements this here 
> > https://reviews.llvm.org/D84037 but it definitely needs a few looks
> > over by people who are more clued up on the internal of FileCheck.
> > 
> > The current syntax, based off this patch, is as follows:
> >   - Added a command line option called `anchor-prefix` which is a
> > comma-seperated list of prefixes to be used when declaring anchors.
> > This is defaulted to `LINE-ANCHOR`
> >   - To declare a anchor in the test file use
> >     `LINE-ANCHOR: ANCHOR_NAME`
> >     note: If you specify a different anchor-prefix using the
> > command
> > line, use that name instead of `LINE-ANCHOR`
> >     ANCHOR_NAME Follows the rules all other variable names aside
> > from
> > the fact it can't start with '$'.
> >   - When referring to an anchor in a check use the same numeric
> > variable syntax that FileCheck already supports:
> >     `CHECK: [[#ANCHOR_NAME]][[#ANCHOR_NAME+1]]`
> > 
> > Here is a brief (contrived) example of the usage of this:
> > ```
> > #define BAD_FUNCTION() badFunction() // LINE-ANCHOR: BAD_FUNC
> > // Further down in the file
> >   BAD_FUNCTION();
> > CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
> > CHECK-NOTES :[[#BAD-FUNC]]:3: note: expanded from macro
> > 'BAD_FUNCTION'
> > ```
> > 
> > Regards,
> > Nathan James
> > 
> > 
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

On Fri, Jul 17, 2020 at 4:23 PM Nathan James <n.james93 at hotmail.co.uk>
wrote:
> Hi Joel,
>
> That sounds like a very nice idea and definitely a direction I could
> get behind. However I feel that outside the use case I suggested, this
> functionality would only be used to compress CHECK lines that contain
> repeated text, not saying its a bad or good thing though. WDYT?
>
I can think of a few use cases:

1. Capturing `@LINE`, as you suggest.
2. Making a single directive's complex pattern more readable by extracting
and naming components.
3. Avoiding the maintenance burden of repeating the same string, numeric,
or pattern expression multiple times within a single directive or across
multiple directives.

`-D` can sometimes help with #2 and #3.  However, it still has to be
repeated across multiple RUN lines in some cases.  Moreover, it's sometimes
better to define variables near their uses, but the RUN lines can be far
away.

As an example of #2 and #3, I've occasionally written subtle patterns, such
as `{{$[[:space:]]}}` for newline, that I wanted to use in multiple
places.  It's not immediately obvious what that pattern does, but `[[NL]]`
would be much clearer and could be documented once at its definition.

It just occurred to me that the new directive might require a
"CHECK-DEF-CONST" form that is processed before all other directives.
Otherwise, forward-references wouldn't be possible.  You might want
forward-references for #1, for example.

Joel

>
> ~Nathan
>
> On Fri, 2020-07-17 at 14:52 -0400, Joel E. Denny via cfe-dev wrote:
> > Hi Nathan,
> >
> > On Fri, Jul 17, 2020 at 12:23 PM Nathan James via cfe-dev <
> > cfe-dev at lists.llvm.org> wrote:
> > > Hello,
> > >
> > > I was wondering about extending FileCheck to enable creating line
> > > anchors. These are numeric variables that hold the value of the
> > > line
> > > number that where they were defined.
> >
> > I think something like this could be useful.  However, I think it
> > would be more useful to have a general directive for defining
> > FileCheck variables inline without trying to capture from input.  For
> > example:
> >
> > ```
> > #define BAD_FUNCTION() badFunction() // CHECK-DEFINE:
> > [[#BAD_FUNC:@LINE]]
> > // Further down in the file
> >   BAD_FUNCTION();
> > CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
> > CHECK-NOTES :[[#BAD_FUNC]]:3: note: expanded from macro
> > 'BAD_FUNCTION'
> > ```
> >
> > The exact syntax is debatable.
> >
> > I think this form is more useful because it can also define strings
> > or numerics (or maybe even patterns) to be reused in multiple
> > FileCheck directives across multiple FileCheck calls from different
> > RUN lines.  Currently, you either have to capture such a variable
> > from the input or specify it with -D on every FileCheck call that
> > needs it.
> >
> > James Henderson: Weren't you working on something like this?
> >
> > Thanks.
> >
> > Joel
> >
> > > The motivation for this comes from test cases using clang-based
> > > diagnostics which often include notes attached to source
locations
> > > in
> > > different parts of the file. In order to test for the correct
> > > location
> > > of the note, the line number has to be written explicitly or as
an,
> > > often large, offset to the current line. This harms both
> > > readability
> > > and maintainability. Using this new system one could append a
line
> > > of
> > > interest with an anchor-comment and refer back to it inside
> > > FileCheck.
> > >
> > > I have created a basic patch that implements this here
> > > https://reviews.llvm.org/D84037 but it definitely needs a few
looks
> > > over by people who are more clued up on the internal of
FileCheck.
> > >
> > > The current syntax, based off this patch, is as follows:
> > >   - Added a command line option called `anchor-prefix` which is a
> > > comma-seperated list of prefixes to be used when declaring
anchors.
> > > This is defaulted to `LINE-ANCHOR`
> > >   - To declare a anchor in the test file use
> > >     `LINE-ANCHOR: ANCHOR_NAME`
> > >     note: If you specify a different anchor-prefix using the
> > > command
> > > line, use that name instead of `LINE-ANCHOR`
> > >     ANCHOR_NAME Follows the rules all other variable names aside
> > > from
> > > the fact it can't start with '$'.
> > >   - When referring to an anchor in a check use the same numeric
> > > variable syntax that FileCheck already supports:
> > >     `CHECK: [[#ANCHOR_NAME]][[#ANCHOR_NAME+1]]`
> > >
> > > Here is a brief (contrived) example of the usage of this:
> > > ```
> > > #define BAD_FUNCTION() badFunction() // LINE-ANCHOR: BAD_FUNC
> > > // Further down in the file
> > >   BAD_FUNCTION();
> > > CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
> > > CHECK-NOTES :[[#BAD-FUNC]]:3: note: expanded from macro
> > > 'BAD_FUNCTION'
> > > ```
> > >
> > > Regards,
> > > Nathan James
> > >
> > >
> > > _______________________________________________
> > > cfe-dev mailing list
> > > cfe-dev at lists.llvm.org
> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> >
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200717/4a89cc0c/attachment.html>

As Joel mentioned, I was at one point looking at implementing a
CHECK-DEFINE. The idea was to define a variable inline without it actually
matching any text. My use-case was where a string might want reusing in
different runs of FileCheck, perhaps sometimes with a -NOT after it. The
separate runs meant that using -D was not particularly appealing, as it
just meant defining the string in two different places, whilst the -NOT
suffix meant reusing the CHECK line itself wasn't viable either. Repeating
the string also meant an update in one place might not get done in the
other place, and since CHECK-NOT patterns are brittle, the test might start
passing spuriously.

Simple example:

# RUN: tool --success | FileCheck %s --check-prefix=SUCCESS
# RUN: tool --fail | FileCheck %s --check-prefix=FAIL

# SUCCESS: success message
# FAIL-NOT: success message

Could become:

# RUN: tool --success | FileCheck %s --check-prefixes=DEF,SUCCESS
# RUN: tool --fail | FileCheck %s --check-prefixes=DEF,FAIL

# DEF-DEFINE: [[MSG:success message]]
# SUCCESS: [[MSG]]
# FAIL-NOT: [[MSG]]

Unfortunately, this wasn't something I could sink much time into, and
I've
now put it on the back shelf for the foreseeable future. I'd be happy to
put a patch up on Phabricator for others to pick up and work with, if they
want?

I got a fair distance towards a prototype solution, but I wasn't
particularly happy with the internal implementation, plus there were
several cases where it either didn't work, or broke something else. The
biggest problem I faced was that you probably don't want DEFINE directives
defining variables that appear before it, but FileCheck's two-pass design
made it tricky to handle that.

Back to the original problem: I haven't thought this through completely,
but I wonder if you could use @Fāng-ruì Sòng <maskray at google.com>'s
recent
"extract" tool to split the pattern file up used by FileCheck somehow
(see
https://reviews.llvm.org/D83834). By discarding the leading lines (using
--no-leading-lines IIRC), you could make the @LINE directives more
localised. Essentially it would look something like:

# RUN: extract input1.c --no-leading-lines %s > %t1.c
# RUN: do-a-thing %s | FileCheck %t1.c
#--- input1.c
this line causes some output # CHECK: [[@LINE]]: ...

Hope that helps!

James

On Fri, 17 Jul 2020 at 21:57, Joel E. Denny via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> On Fri, Jul 17, 2020 at 4:23 PM Nathan James <n.james93 at
hotmail.co.uk>
> wrote:
>
>> Hi Joel,
>>
>> That sounds like a very nice idea and definitely a direction I could
>> get behind. However I feel that outside the use case I suggested, this
>> functionality would only be used to compress CHECK lines that contain
>> repeated text, not saying its a bad or good thing though. WDYT?
>>
>
> I can think of a few use cases:
>
> 1. Capturing `@LINE`, as you suggest.
> 2. Making a single directive's complex pattern more readable by
extracting
> and naming components.
> 3. Avoiding the maintenance burden of repeating the same string, numeric,
> or pattern expression multiple times within a single directive or across
> multiple directives.
>
> `-D` can sometimes help with #2 and #3.  However, it still has to be
> repeated across multiple RUN lines in some cases.  Moreover, it's
sometimes
> better to define variables near their uses, but the RUN lines can be far
> away.
>
> As an example of #2 and #3, I've occasionally written subtle patterns,
> such as `{{$[[:space:]]}}` for newline, that I wanted to use in multiple
> places.  It's not immediately obvious what that pattern does, but
`[[NL]]`
> would be much clearer and could be documented once at its definition.
>
> It just occurred to me that the new directive might require a
> "CHECK-DEF-CONST" form that is processed before all other
directives.
> Otherwise, forward-references wouldn't be possible.  You might want
> forward-references for #1, for example.
>
> Joel
>
>
>>
>> ~Nathan
>>
>> On Fri, 2020-07-17 at 14:52 -0400, Joel E. Denny via cfe-dev wrote:
>> > Hi Nathan,
>> >
>> > On Fri, Jul 17, 2020 at 12:23 PM Nathan James via cfe-dev <
>> > cfe-dev at lists.llvm.org> wrote:
>> > > Hello,
>> > >
>> > > I was wondering about extending FileCheck to enable creating
line
>> > > anchors. These are numeric variables that hold the value of
the
>> > > line
>> > > number that where they were defined.
>> >
>> > I think something like this could be useful.  However, I think it
>> > would be more useful to have a general directive for defining
>> > FileCheck variables inline without trying to capture from input. 
For
>> > example:
>> >
>> > ```
>> > #define BAD_FUNCTION() badFunction() // CHECK-DEFINE:
>> > [[#BAD_FUNC:@LINE]]
>> > // Further down in the file
>> >   BAD_FUNCTION();
>> > CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
>> > CHECK-NOTES :[[#BAD_FUNC]]:3: note: expanded from macro
>> > 'BAD_FUNCTION'
>> > ```
>> >
>> > The exact syntax is debatable.
>> >
>> > I think this form is more useful because it can also define
strings
>> > or numerics (or maybe even patterns) to be reused in multiple
>> > FileCheck directives across multiple FileCheck calls from
different
>> > RUN lines.  Currently, you either have to capture such a variable
>> > from the input or specify it with -D on every FileCheck call that
>> > needs it.
>> >
>> > James Henderson: Weren't you working on something like this?
>> >
>> > Thanks.
>> >
>> > Joel
>> >
>> > > The motivation for this comes from test cases using
clang-based
>> > > diagnostics which often include notes attached to source
locations
>> > > in
>> > > different parts of the file. In order to test for the correct
>> > > location
>> > > of the note, the line number has to be written explicitly or
as an,
>> > > often large, offset to the current line. This harms both
>> > > readability
>> > > and maintainability. Using this new system one could append a
line
>> > > of
>> > > interest with an anchor-comment and refer back to it inside
>> > > FileCheck.
>> > >
>> > > I have created a basic patch that implements this here
>> > > https://reviews.llvm.org/D84037 but it definitely needs a few
looks
>> > > over by people who are more clued up on the internal of
FileCheck.
>> > >
>> > > The current syntax, based off this patch, is as follows:
>> > >   - Added a command line option called `anchor-prefix` which
is a
>> > > comma-seperated list of prefixes to be used when declaring
anchors.
>> > > This is defaulted to `LINE-ANCHOR`
>> > >   - To declare a anchor in the test file use
>> > >     `LINE-ANCHOR: ANCHOR_NAME`
>> > >     note: If you specify a different anchor-prefix using the
>> > > command
>> > > line, use that name instead of `LINE-ANCHOR`
>> > >     ANCHOR_NAME Follows the rules all other variable names
aside
>> > > from
>> > > the fact it can't start with '$'.
>> > >   - When referring to an anchor in a check use the same
numeric
>> > > variable syntax that FileCheck already supports:
>> > >     `CHECK: [[#ANCHOR_NAME]][[#ANCHOR_NAME+1]]`
>> > >
>> > > Here is a brief (contrived) example of the usage of this:
>> > > ```
>> > > #define BAD_FUNCTION() badFunction() // LINE-ANCHOR: BAD_FUNC
>> > > // Further down in the file
>> > >   BAD_FUNCTION();
>> > > CHECK-NOTES: [[@LINE-1]]:3: warning: called a bad function
>> > > CHECK-NOTES :[[#BAD-FUNC]]:3: note: expanded from macro
>> > > 'BAD_FUNCTION'
>> > > ```
>> > >
>> > > Regards,
>> > > Nathan James
>> > >
>> > >
>> > > _______________________________________________
>> > > cfe-dev mailing list
>> > > cfe-dev at lists.llvm.org
>> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>> >
>> > _______________________________________________
>> > cfe-dev mailing list
>> > cfe-dev at lists.llvm.org
>> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200720/ba129505/attachment.html>