thr3ads.net - llvm dev - [llvm-dev] FileCheck idiom difficulties [Nov 2019]

If this information is useful, please help other people find it:
Share via:

James Henderson via llvm-dev

2019-Nov-06 12:17 UTC

[llvm-dev] FileCheck idiom difficulties

Hi all,

Many of our lit tests use FileCheck and a tool like llvm-readobj to check
properties of a section header/symbol/etc. A typical (pseudoised for
brevity) output to match against might be something like the following:

Symbols [
  Symbol {
    Name: foo
    Value: 0
    Type: Function
    Section: .foo (1)
  }
  Symbol {
    Name: bar
    Value: 1
    Type: Object
    Section: .foo (1)
  }
]

and your lit test might want to check the properties of the foo symbol like
so:

# CHECK:      Name: foo
# CHECK-NEXT: Value: 0
# CHECK-NEXT: Type: Function
# CHECK-NEXT: Section: .foo (1)

This is fine. But what if you only care about the section of a symbol, and
not the value or type etc? You could do the following:

# CHECK: Name: foo
# CHECK: Section: .foo (1)

Hopefully some of you will already notice the problem with this approach:
if foo was in, say, the .baz section, the test will spuriously pass,
because the Section line will match the Section line for .bar. One
alternative to this is to explicitly match each field in between, using
CHECK-NEXT:

# CHECK:      Name: foo
# CHECK-NEXT: Value:
# CHECK-NEXT: Type:
# CHECK-NEXT: Section: .foo (1)

This works, but somewhat hides what is really being tested by adding extra
noise to the checks. In reality, there are actually other fields too that
need to be listed, meaning the "interesting" parts of the test are
even
more hidden.

I recently started using yet another approach:

# CHECK: Name: foo
# CHECK: Section:
# CHECK-SAME:     .foo (1)

This works because the Section: matched will be the first one found, i.e.
the one belonging to foo, and then .foo will be looked for on the same
line. However, I noticed today that this pattern has its own problem,
namely that there could be something between the Section tag and .foo. In
other words, the above pattern would match "Section: .bar.foo". A
couple of
solutions to this are:

# CHECK: Section:
# CHECK-NOT: {{[:graph:]}}
# CHECK-SAME: .foo (1)

# CHECK: Section:
# CHECK-SAME: {{^}} .foo (1)

The first one ensures that there's no non-whitespace between the end of
"Section:" and the start of ".foo (1)". The second ensures
that the start
of the CHECK-SAME match is the "start of line", and since the first
half of
the line has already been consumed, it means " .foo (1)" must
immediately
follow "Section:". However, the first is even less readable than the
current CHECK-SAME approach, whilst the second is somewhat confusing if you
don't realise that FileCheck effectively consumes the things it has matched
already, so that they effectively don't exist any more.

Does anybody have any other suggestions/thoughts/comments? One idea I had
was for a new directive something like "CHECK-IMMEDIATE" which is
implicitly the same as the final approach I suggested above, but maybe
adding a new directive to achieve this isn't the right approach?

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/055f9807/attachment.html>

George Rimar via llvm-dev

2019-Nov-06 13:01 UTC

head link

[llvm-dev] FileCheck idiom difficulties

> One idea I had was for a new directive something like
"CHECK-IMMEDIATE" which
> is implicitly the same as the final approach I suggested above, but maybe
adding a new
> directive to achieve this isn't the right approach?

"CHECK-IMMEDIATE"? (or "CHECK-CONTINUE"/better name) sound
like a clean and fine approach to me.

?I'd go with it probably.


Best regards,
George | Developer | Access Softek, Inc
________________________________
От: James Henderson <jh7370.2008 at my.bristol.ac.uk>
Отправлено: 6 ноября 2019 г. 15:17
Кому: llvm-dev; George Rimar; Fāng-ruì Sòng; Paul Robinson; thomasp at
graphcore.ai
Тема: FileCheck idiom difficulties

CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.  If you suspect potential phishing or spam email, report it to
ReportSpam at accesssoftek.com

Hi all,

Many of our lit tests use FileCheck and a tool like llvm-readobj to check
properties of a section header/symbol/etc. A typical (pseudoised for brevity)
output to match against might be something like the following:

Symbols [
  Symbol {
    Name: foo
    Value: 0
    Type: Function
    Section: .foo (1)
  }
  Symbol {
    Name: bar
    Value: 1
    Type: Object
    Section: .foo (1)
  }
]

and your lit test might want to check the properties of the foo symbol like so:

# CHECK:      Name: foo
# CHECK-NEXT: Value: 0
# CHECK-NEXT: Type: Function
# CHECK-NEXT: Section: .foo (1)

This is fine. But what if you only care about the section of a symbol, and not
the value or type etc? You could do the following:

# CHECK: Name: foo
# CHECK: Section: .foo (1)

Hopefully some of you will already notice the problem with this approach: if foo
was in, say, the .baz section, the test will spuriously pass, because the
Section line will match the Section line for .bar. One alternative to this is to
explicitly match each field in between, using CHECK-NEXT:

# CHECK:      Name: foo
# CHECK-NEXT: Value:
# CHECK-NEXT: Type:
# CHECK-NEXT: Section: .foo (1)

This works, but somewhat hides what is really being tested by adding extra noise
to the checks. In reality, there are actually other fields too that need to be
listed, meaning the "interesting" parts of the test are even more
hidden.

I recently started using yet another approach:

# CHECK: Name: foo
# CHECK: Section:
# CHECK-SAME:     .foo (1)

This works because the Section: matched will be the first one found, i.e. the
one belonging to foo, and then .foo will be looked for on the same line.
However, I noticed today that this pattern has its own problem, namely that
there could be something between the Section tag and .foo. In other words, the
above pattern would match "Section: .bar.foo". A couple of solutions
to this are:

# CHECK: Section:
# CHECK-NOT: {{[:graph:]}}
# CHECK-SAME: .foo (1)

# CHECK: Section:
# CHECK-SAME: {{^}} .foo (1)

The first one ensures that there's no non-whitespace between the end of
"Section:" and the start of ".foo (1)". The second ensures
that the start of the CHECK-SAME match is the "start of line", and
since the first half of the line has already been consumed, it means " .foo
(1)" must immediately follow "Section:". However, the first is
even less readable than the current CHECK-SAME approach, whilst the second is
somewhat confusing if you don't realise that FileCheck effectively consumes
the things it has matched already, so that they effectively don't exist any
more.

Does anybody have any other suggestions/thoughts/comments? One idea I had was
for a new directive something like "CHECK-IMMEDIATE" which is
implicitly the same as the final approach I suggested above, but maybe adding a
new directive to achieve this isn't the right approach?

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/361e5b62/attachment.html>

Thomas Preud'homme via llvm-dev

2019-Nov-06 13:54 UTC

head link

[llvm-dev] FileCheck idiom difficulties

Aren't the name lines unique? If they are you could use the good old
CHECK-LABEL:

# CHECK-LABEL: Name: foo
# CHECK: Section: .foo (1)
________________________________
From: George Rimar <grimar at accesssoftek.com>
Sent: 06 November 2019 13:01
To: jh7370.2008 at my.bristol.ac.uk <jh7370.2008 at my.bristol.ac.uk>
Cc: Fāng-ruì Sòng <maskray at google.com>; llvm-dev <llvm-dev at
lists.llvm.org>; Thomas Preud'homme <thomasp at graphcore.ai>; Paul
Robinson <paul.robinson at sony.com>
Subject: Re: FileCheck idiom difficulties

> One idea I had was for a new directive something like
"CHECK-IMMEDIATE" which
> is implicitly the same as the final approach I suggested above, but maybe
adding a new
> directive to achieve this isn't the right approach?

"CHECK-IMMEDIATE" (or "CHECK-CONTINUE"/better name) sound
like a clean and fine approach to me.

I'd go with it probably.

Best regards,
George | Developer | Access Softek, Inc
________________________________
От: James Henderson <jh7370.2008 at my.bristol.ac.uk>
Отправлено: 6 ноября 2019 г. 15:17
Кому: llvm-dev; George Rimar; Fāng-ruì Sòng; Paul Robinson; thomasp at
graphcore.ai
Тема: FileCheck idiom difficulties

CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and know the content
is safe.  If you suspect potential phishing or spam email, report it to
ReportSpam at accesssoftek.com

Hi all,

Many of our lit tests use FileCheck and a tool like llvm-readobj to check
properties of a section header/symbol/etc. A typical (pseudoised for brevity)
output to match against might be something like the following:

Symbols [
  Symbol {
    Name: foo
    Value: 0
    Type: Function
    Section: .foo (1)
  }
  Symbol {
    Name: bar
    Value: 1
    Type: Object
    Section: .foo (1)
  }
]

and your lit test might want to check the properties of the foo symbol like so:

# CHECK:      Name: foo
# CHECK-NEXT: Value: 0
# CHECK-NEXT: Type: Function
# CHECK-NEXT: Section: .foo (1)

This is fine. But what if you only care about the section of a symbol, and not
the value or type etc? You could do the following:

# CHECK: Name: foo
# CHECK: Section: .foo (1)

Hopefully some of you will already notice the problem with this approach: if foo
was in, say, the .baz section, the test will spuriously pass, because the
Section line will match the Section line for .bar. One alternative to this is to
explicitly match each field in between, using CHECK-NEXT:

# CHECK:      Name: foo
# CHECK-NEXT: Value:
# CHECK-NEXT: Type:
# CHECK-NEXT: Section: .foo (1)

This works, but somewhat hides what is really being tested by adding extra noise
to the checks. In reality, there are actually other fields too that need to be
listed, meaning the "interesting" parts of the test are even more
hidden.

I recently started using yet another approach:

# CHECK: Name: foo
# CHECK: Section:
# CHECK-SAME:     .foo (1)

This works because the Section: matched will be the first one found, i.e. the
one belonging to foo, and then .foo will be looked for on the same line.
However, I noticed today that this pattern has its own problem, namely that
there could be something between the Section tag and .foo. In other words, the
above pattern would match "Section: .bar.foo". A couple of solutions
to this are:

# CHECK: Section:
# CHECK-NOT: {{[:graph:]}}
# CHECK-SAME: .foo (1)

# CHECK: Section:
# CHECK-SAME: {{^}} .foo (1)

The first one ensures that there's no non-whitespace between the end of
"Section:" and the start of ".foo (1)". The second ensures
that the start of the CHECK-SAME match is the "start of line", and
since the first half of the line has already been consumed, it means " .foo
(1)" must immediately follow "Section:". However, the first is
even less readable than the current CHECK-SAME approach, whilst the second is
somewhat confusing if you don't realise that FileCheck effectively consumes
the things it has matched already, so that they effectively don't exist any
more.

Does anybody have any other suggestions/thoughts/comments? One idea I had was
for a new directive something like "CHECK-IMMEDIATE" which is
implicitly the same as the final approach I suggested above, but maybe adding a
new directive to achieve this isn't the right approach?

James

** We have updated our privacy policy, which contains important information
about how we collect and process your personal data. To read the policy, please
click here<http://www.graphcore.ai/privacy> **

This email and its attachments are intended solely for the addressed recipients
and may contain confidential or legally privileged information.
If you are not the intended recipient you must not copy, distribute or
disseminate this email in any way; to do so may be unlawful.

Any personal data/special category personal data herein are processed in
accordance with UK data protection legislation.
All associated feasible security measures are in place. Further details are
available from the Privacy Notice on the website and/or from the Company.

Graphcore Limited (registered in England and Wales with registration number
10185006) is registered at 107 Cheapside, London, UK, EC2V 6DN.
This message was scanned for viruses upon transmission. However Graphcore
accepts no liability for any such transmission.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/2517bbc6/attachment.html>

David Blaikie via llvm-dev

2019-Nov-06 15:48 UTC

head link

[llvm-dev] FileCheck idiom difficulties

I'd usually write this sort of test (& there are many in the DWARF
emission
tests):


# CHECK:      Name: foo
# CHECK-NOT: }
# CHECK: Section: .foo (1)

Or, if you're checking every section, you could use a
--implicit-check-not=Symbol
or similar.

On Wed, Nov 6, 2019 at 4:18 AM James Henderson via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
> Hi all,
>
> Many of our lit tests use FileCheck and a tool like llvm-readobj to check
> properties of a section header/symbol/etc. A typical (pseudoised for
> brevity) output to match against might be something like the following:
>
> Symbols [
>   Symbol {
>     Name: foo
>     Value: 0
>     Type: Function
>     Section: .foo (1)
>   }
>   Symbol {
>     Name: bar
>     Value: 1
>     Type: Object
>     Section: .foo (1)
>   }
> ]
>
> and your lit test might want to check the properties of the foo symbol
> like so:
>
> # CHECK:      Name: foo
> # CHECK-NEXT: Value: 0
> # CHECK-NEXT: Type: Function
> # CHECK-NEXT: Section: .foo (1)
>
> This is fine. But what if you only care about the section of a symbol, and
> not the value or type etc? You could do the following:
>
> # CHECK: Name: foo
> # CHECK: Section: .foo (1)
>
> Hopefully some of you will already notice the problem with this approach:
> if foo was in, say, the .baz section, the test will spuriously pass,
> because the Section line will match the Section line for .bar. One
> alternative to this is to explicitly match each field in between, using
> CHECK-NEXT:
>
> # CHECK:      Name: foo
> # CHECK-NEXT: Value:
> # CHECK-NEXT: Type:
> # CHECK-NEXT: Section: .foo (1)
>
> This works, but somewhat hides what is really being tested by adding extra
> noise to the checks. In reality, there are actually other fields too that
> need to be listed, meaning the "interesting" parts of the test
are even
> more hidden.
>
> I recently started using yet another approach:
>
> # CHECK: Name: foo
> # CHECK: Section:
> # CHECK-SAME:     .foo (1)
>
> This works because the Section: matched will be the first one found, i.e.
> the one belonging to foo, and then .foo will be looked for on the same
> line. However, I noticed today that this pattern has its own problem,
> namely that there could be something between the Section tag and .foo. In
> other words, the above pattern would match "Section: .bar.foo". A
couple of
> solutions to this are:
>
> # CHECK: Section:
> # CHECK-NOT: {{[:graph:]}}
> # CHECK-SAME: .foo (1)
>
> # CHECK: Section:
> # CHECK-SAME: {{^}} .foo (1)
>
> The first one ensures that there's no non-whitespace between the end of
> "Section:" and the start of ".foo (1)". The second
ensures that the start
> of the CHECK-SAME match is the "start of line", and since the
first half of
> the line has already been consumed, it means " .foo (1)" must
immediately
> follow "Section:". However, the first is even less readable than
the
> current CHECK-SAME approach, whilst the second is somewhat confusing if you
> don't realise that FileCheck effectively consumes the things it has
matched
> already, so that they effectively don't exist any more.
>
> Does anybody have any other suggestions/thoughts/comments? One idea I had
> was for a new directive something like "CHECK-IMMEDIATE" which is
> implicitly the same as the final approach I suggested above, but maybe
> adding a new directive to achieve this isn't the right approach?
>
> James
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/c2e3fb60/attachment.html>

James Henderson via llvm-dev

2019-Nov-06 16:44 UTC

head link

[llvm-dev] FileCheck idiom difficulties

Thanks, I had considered using the CHECK-NOT approach. That would work in
most cases, I believe, but I'm always nervous to use it because CHECK-NOTs
can start failing to check the right thing if output changes. For example,
George Rimar recently made changes that turned some list-like structs into
dictionaries (using curly brackets instead of square brackets) in the
readobj output, which could theoretically have affected the '}' style.

However, the implicit-check-not approach probably works for many cases.
Thanks.

On Wed, 6 Nov 2019 at 15:49, David Blaikie <dblaikie at gmail.com> wrote:
> I'd usually write this sort of test (& there are many in the DWARF
> emission tests):
>
>
> # CHECK:      Name: foo
> # CHECK-NOT: }
> # CHECK: Section: .foo (1)
>
> Or, if you're checking every section, you could use a
--implicit-check-not=Symbol
> or similar.
>
> On Wed, Nov 6, 2019 at 4:18 AM James Henderson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>> Hi all,
>>
>> Many of our lit tests use FileCheck and a tool like llvm-readobj to
check
>> properties of a section header/symbol/etc. A typical (pseudoised for
>> brevity) output to match against might be something like the following:
>>
>> Symbols [
>>   Symbol {
>>     Name: foo
>>     Value: 0
>>     Type: Function
>>     Section: .foo (1)
>>   }
>>   Symbol {
>>     Name: bar
>>     Value: 1
>>     Type: Object
>>     Section: .foo (1)
>>   }
>> ]
>>
>> and your lit test might want to check the properties of the foo symbol
>> like so:
>>
>> # CHECK:      Name: foo
>> # CHECK-NEXT: Value: 0
>> # CHECK-NEXT: Type: Function
>> # CHECK-NEXT: Section: .foo (1)
>>
>> This is fine. But what if you only care about the section of a symbol,
>> and not the value or type etc? You could do the following:
>>
>> # CHECK: Name: foo
>> # CHECK: Section: .foo (1)
>>
>> Hopefully some of you will already notice the problem with this
approach:
>> if foo was in, say, the .baz section, the test will spuriously pass,
>> because the Section line will match the Section line for .bar. One
>> alternative to this is to explicitly match each field in between, using
>> CHECK-NEXT:
>>
>> # CHECK:      Name: foo
>> # CHECK-NEXT: Value:
>> # CHECK-NEXT: Type:
>> # CHECK-NEXT: Section: .foo (1)
>>
>> This works, but somewhat hides what is really being tested by adding
>> extra noise to the checks. In reality, there are actually other fields
too
>> that need to be listed, meaning the "interesting" parts of
the test are
>> even more hidden.
>>
>> I recently started using yet another approach:
>>
>> # CHECK: Name: foo
>> # CHECK: Section:
>> # CHECK-SAME:     .foo (1)
>>
>> This works because the Section: matched will be the first one found,
i.e.
>> the one belonging to foo, and then .foo will be looked for on the same
>> line. However, I noticed today that this pattern has its own problem,
>> namely that there could be something between the Section tag and .foo.
In
>> other words, the above pattern would match "Section:
.bar.foo". A couple of
>> solutions to this are:
>>
>> # CHECK: Section:
>> # CHECK-NOT: {{[:graph:]}}
>> # CHECK-SAME: .foo (1)
>>
>> # CHECK: Section:
>> # CHECK-SAME: {{^}} .foo (1)
>>
>> The first one ensures that there's no non-whitespace between the
end of
>> "Section:" and the start of ".foo (1)". The second
ensures that the start
>> of the CHECK-SAME match is the "start of line", and since the
first half of
>> the line has already been consumed, it means " .foo (1)" must
immediately
>> follow "Section:". However, the first is even less readable
than the
>> current CHECK-SAME approach, whilst the second is somewhat confusing if
you
>> don't realise that FileCheck effectively consumes the things it has
matched
>> already, so that they effectively don't exist any more.
>>
>> Does anybody have any other suggestions/thoughts/comments? One idea I
had
>> was for a new directive something like "CHECK-IMMEDIATE"
which is
>> implicitly the same as the final approach I suggested above, but maybe
>> adding a new directive to achieve this isn't the right approach?
>>
>> James
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/3aa8b98f/attachment-0001.html>

Reid Kleckner via llvm-dev

2019-Nov-06 19:12 UTC

head link

[llvm-dev] FileCheck idiom difficulties

I think we should think about changing llvm-readobj to produce more
filecheck friendly output.

This was a problem we used to have with LLVM debug info metadata: it wasn't
very structured, and it was printed in some arbitrary order. Then Duncan (I
think) added the DI* classes, which made it easier to match something
semantic, like `DILocalVariable.*name: "foo"`, and standardized on a
topological output ordering, so you could start from the variable, then
match the type, and then find the type metadata number later.

If we printed `Section (.foo) {` for example, that would help some.

It still doesn't help establish delimited regions for properties printed
across multiple lines for readability, though... Should we add some kind of
ad-hoc delimiter balancing to FileCheck? Something like:

CHECK-SCOPE: Section {
CHECK: Name: asdf
CHECK: Name: asdf
CHECK-ENDSCOPE: }

A SCOPE directive line would have to end in a known delimiter, '(',
'{', or
'['. The ENDSCOPE directive would only match lines with delimiters that
balance with the opening delimiter. It would kind of work for filechecking
JSON, for example.

The scope idea here is pretty half-baked, but it's food for thought.

On Wed, Nov 6, 2019 at 4:18 AM James Henderson via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
> Hi all,
>
> Many of our lit tests use FileCheck and a tool like llvm-readobj to check
> properties of a section header/symbol/etc. A typical (pseudoised for
> brevity) output to match against might be something like the following:
>
> Symbols [
>   Symbol {
>     Name: foo
>     Value: 0
>     Type: Function
>     Section: .foo (1)
>   }
>   Symbol {
>     Name: bar
>     Value: 1
>     Type: Object
>     Section: .foo (1)
>   }
> ]
>
> and your lit test might want to check the properties of the foo symbol
> like so:
>
> # CHECK:      Name: foo
> # CHECK-NEXT: Value: 0
> # CHECK-NEXT: Type: Function
> # CHECK-NEXT: Section: .foo (1)
>
> This is fine. But what if you only care about the section of a symbol, and
> not the value or type etc? You could do the following:
>
> # CHECK: Name: foo
> # CHECK: Section: .foo (1)
>
> Hopefully some of you will already notice the problem with this approach:
> if foo was in, say, the .baz section, the test will spuriously pass,
> because the Section line will match the Section line for .bar. One
> alternative to this is to explicitly match each field in between, using
> CHECK-NEXT:
>
> # CHECK:      Name: foo
> # CHECK-NEXT: Value:
> # CHECK-NEXT: Type:
> # CHECK-NEXT: Section: .foo (1)
>
> This works, but somewhat hides what is really being tested by adding extra
> noise to the checks. In reality, there are actually other fields too that
> need to be listed, meaning the "interesting" parts of the test
are even
> more hidden.
>
> I recently started using yet another approach:
>
> # CHECK: Name: foo
> # CHECK: Section:
> # CHECK-SAME:     .foo (1)
>
> This works because the Section: matched will be the first one found, i.e.
> the one belonging to foo, and then .foo will be looked for on the same
> line. However, I noticed today that this pattern has its own problem,
> namely that there could be something between the Section tag and .foo. In
> other words, the above pattern would match "Section: .bar.foo". A
couple of
> solutions to this are:
>
> # CHECK: Section:
> # CHECK-NOT: {{[:graph:]}}
> # CHECK-SAME: .foo (1)
>
> # CHECK: Section:
> # CHECK-SAME: {{^}} .foo (1)
>
> The first one ensures that there's no non-whitespace between the end of
> "Section:" and the start of ".foo (1)". The second
ensures that the start
> of the CHECK-SAME match is the "start of line", and since the
first half of
> the line has already been consumed, it means " .foo (1)" must
immediately
> follow "Section:". However, the first is even less readable than
the
> current CHECK-SAME approach, whilst the second is somewhat confusing if you
> don't realise that FileCheck effectively consumes the things it has
matched
> already, so that they effectively don't exist any more.
>
> Does anybody have any other suggestions/thoughts/comments? One idea I had
> was for a new directive something like "CHECK-IMMEDIATE" which is
> implicitly the same as the final approach I suggested above, but maybe
> adding a new directive to achieve this isn't the right approach?
>
> James
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/6d1ada97/attachment-0001.html>

Robinson, Paul via llvm-dev

2019-Nov-06 20:12 UTC

head link

[llvm-dev] FileCheck idiom difficulties

I have to say, I’m not super excited about the answer always being “let’s add
another FileCheck directive.”  FileCheck is basically a meta-grep, that knows
how to execute some sub-searches in different orders.  Asking it to do things
like implicit paren balancing is really out of scope for the tool.
--paulr

From: Reid Kleckner <rnk at google.com>
Sent: Wednesday, November 6, 2019 2:13 PM
To: James Henderson <jh7370.2008 at my.bristol.ac.uk>
Cc: llvm-dev <llvm-dev at lists.llvm.org>; George Rimar <grimar at
accesssoftek.com>; Fāng-ruì Sòng <maskray at google.com>; Robinson,
Paul <paul.robinson at sony.com>; thomasp at graphcore.ai
Subject: Re: [llvm-dev] FileCheck idiom difficulties

I think we should think about changing llvm-readobj to produce more filecheck
friendly output.

This was a problem we used to have with LLVM debug info metadata: it wasn't
very structured, and it was printed in some arbitrary order. Then Duncan (I
think) added the DI* classes, which made it easier to match something semantic,
like `DILocalVariable.*name: "foo"`, and standardized on a topological
output ordering, so you could start from the variable, then match the type, and
then find the type metadata number later.

If we printed `Section (.foo) {` for example, that would help some.

It still doesn't help establish delimited regions for properties printed
across multiple lines for readability, though... Should we add some kind of
ad-hoc delimiter balancing to FileCheck? Something like:

CHECK-SCOPE: Section {
CHECK: Name: asdf
CHECK: Name: asdf
CHECK-ENDSCOPE: }

A SCOPE directive line would have to end in a known delimiter, '(',
'{', or '['. The ENDSCOPE directive would only match lines with
delimiters that balance with the opening delimiter. It would kind of work for
filechecking JSON, for example.

The scope idea here is pretty half-baked, but it's food for thought.

On Wed, Nov 6, 2019 at 4:18 AM James Henderson via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

Hi all,

Many of our lit tests use FileCheck and a tool like llvm-readobj to check
properties of a section header/symbol/etc. A typical (pseudoised for brevity)
output to match against might be something like the following:

Symbols [
  Symbol {
    Name: foo
    Value: 0
    Type: Function
    Section: .foo (1)
  }
  Symbol {
    Name: bar
    Value: 1
    Type: Object
    Section: .foo (1)
  }
]

and your lit test might want to check the properties of the foo symbol like so:

# CHECK:      Name: foo
# CHECK-NEXT: Value: 0
# CHECK-NEXT: Type: Function
# CHECK-NEXT: Section: .foo (1)

This is fine. But what if you only care about the section of a symbol, and not
the value or type etc? You could do the following:

# CHECK: Name: foo
# CHECK: Section: .foo (1)

Hopefully some of you will already notice the problem with this approach: if foo
was in, say, the .baz section, the test will spuriously pass, because the
Section line will match the Section line for .bar. One alternative to this is to
explicitly match each field in between, using CHECK-NEXT:

# CHECK:      Name: foo
# CHECK-NEXT: Value:
# CHECK-NEXT: Type:
# CHECK-NEXT: Section: .foo (1)

This works, but somewhat hides what is really being tested by adding extra noise
to the checks. In reality, there are actually other fields too that need to be
listed, meaning the "interesting" parts of the test are even more
hidden.

I recently started using yet another approach:

# CHECK: Name: foo
# CHECK: Section:
# CHECK-SAME:     .foo (1)

This works because the Section: matched will be the first one found, i.e. the
one belonging to foo, and then .foo will be looked for on the same line.
However, I noticed today that this pattern has its own problem, namely that
there could be something between the Section tag and .foo. In other words, the
above pattern would match "Section: .bar.foo". A couple of solutions
to this are:

# CHECK: Section:
# CHECK-NOT: {{[:graph:]}}
# CHECK-SAME: .foo (1)

# CHECK: Section:
# CHECK-SAME: {{^}} .foo (1)

The first one ensures that there's no non-whitespace between the end of
"Section:" and the start of ".foo (1)". The second ensures
that the start of the CHECK-SAME match is the "start of line", and
since the first half of the line has already been consumed, it means " .foo
(1)" must immediately follow "Section:". However, the first is
even less readable than the current CHECK-SAME approach, whilst the second is
somewhat confusing if you don't realise that FileCheck effectively consumes
the things it has matched already, so that they effectively don't exist any
more.

Does anybody have any other suggestions/thoughts/comments? One idea I had was
for a new directive something like "CHECK-IMMEDIATE" which is
implicitly the same as the final approach I suggested above, but maybe adding a
new directive to achieve this isn't the right approach?

James
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/f85cbb92/attachment.html>

David Blaikie via llvm-dev

2019-Nov-06 20:19 UTC

head link

[llvm-dev] FileCheck idiom difficulties

On Wed, Nov 6, 2019 at 11:12 AM Reid Kleckner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I think we should think about changing llvm-readobj to produce more
> filecheck friendly output.
>
> This was a problem we used to have with LLVM debug info metadata: it
> wasn't very structured, and it was printed in some arbitrary order.
Then
> Duncan (I think) added the DI* classes, which made it easier to match
> something semantic, like `DILocalVariable.*name: "foo"`, and
standardized
> on a topological output ordering, so you could start from the variable,
> then match the type, and then find the type metadata number later.
>
> If we printed `Section (.foo) {` for example, that would help some.
>
> It still doesn't help establish delimited regions for properties
printed
> across multiple lines for readability, though... Should we add some kind of
> ad-hoc delimiter balancing to FileCheck? Something like:
>
> CHECK-SCOPE: Section {
> CHECK: Name: asdf
> CHECK: Name: asdf
> CHECK-ENDSCOPE: }
>
There's some overlap between ^ and this existing construct \/

CHECK: Section {
CHECK-DAG: Name: asdf
CHECK-DAG: Name: asdf
CHECK: }

>
> A SCOPE directive line would have to end in a known delimiter, '(',
'{',
> or '['. The ENDSCOPE directive would only match lines with
delimiters that
> balance with the opening delimiter. It would kind of work for filechecking
> JSON, for example.
>
> The scope idea here is pretty half-baked, but it's food for thought.
>
> On Wed, Nov 6, 2019 at 4:18 AM James Henderson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>> Hi all,
>>
>> Many of our lit tests use FileCheck and a tool like llvm-readobj to
check
>> properties of a section header/symbol/etc. A typical (pseudoised for
>> brevity) output to match against might be something like the following:
>>
>> Symbols [
>>   Symbol {
>>     Name: foo
>>     Value: 0
>>     Type: Function
>>     Section: .foo (1)
>>   }
>>   Symbol {
>>     Name: bar
>>     Value: 1
>>     Type: Object
>>     Section: .foo (1)
>>   }
>> ]
>>
>> and your lit test might want to check the properties of the foo symbol
>> like so:
>>
>> # CHECK:      Name: foo
>> # CHECK-NEXT: Value: 0
>> # CHECK-NEXT: Type: Function
>> # CHECK-NEXT: Section: .foo (1)
>>
>> This is fine. But what if you only care about the section of a symbol,
>> and not the value or type etc? You could do the following:
>>
>> # CHECK: Name: foo
>> # CHECK: Section: .foo (1)
>>
>> Hopefully some of you will already notice the problem with this
approach:
>> if foo was in, say, the .baz section, the test will spuriously pass,
>> because the Section line will match the Section line for .bar. One
>> alternative to this is to explicitly match each field in between, using
>> CHECK-NEXT:
>>
>> # CHECK:      Name: foo
>> # CHECK-NEXT: Value:
>> # CHECK-NEXT: Type:
>> # CHECK-NEXT: Section: .foo (1)
>>
>> This works, but somewhat hides what is really being tested by adding
>> extra noise to the checks. In reality, there are actually other fields
too
>> that need to be listed, meaning the "interesting" parts of
the test are
>> even more hidden.
>>
>> I recently started using yet another approach:
>>
>> # CHECK: Name: foo
>> # CHECK: Section:
>> # CHECK-SAME:     .foo (1)
>>
>> This works because the Section: matched will be the first one found,
i.e.
>> the one belonging to foo, and then .foo will be looked for on the same
>> line. However, I noticed today that this pattern has its own problem,
>> namely that there could be something between the Section tag and .foo.
In
>> other words, the above pattern would match "Section:
.bar.foo". A couple of
>> solutions to this are:
>>
>> # CHECK: Section:
>> # CHECK-NOT: {{[:graph:]}}
>> # CHECK-SAME: .foo (1)
>>
>> # CHECK: Section:
>> # CHECK-SAME: {{^}} .foo (1)
>>
>> The first one ensures that there's no non-whitespace between the
end of
>> "Section:" and the start of ".foo (1)". The second
ensures that the start
>> of the CHECK-SAME match is the "start of line", and since the
first half of
>> the line has already been consumed, it means " .foo (1)" must
immediately
>> follow "Section:". However, the first is even less readable
than the
>> current CHECK-SAME approach, whilst the second is somewhat confusing if
you
>> don't realise that FileCheck effectively consumes the things it has
matched
>> already, so that they effectively don't exist any more.
>>
>> Does anybody have any other suggestions/thoughts/comments? One idea I
had
>> was for a new directive something like "CHECK-IMMEDIATE"
which is
>> implicitly the same as the final approach I suggested above, but maybe
>> adding a new directive to achieve this isn't the right approach?
>>
>> James
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191106/8b2630b3/attachment.html>

George Rimar via llvm-dev

2019-Nov-07 08:05 UTC

head link

[llvm-dev] FileCheck idiom difficulties

> Or, if you're checking every section, you could use a
--implicit-check-not=Symbol or similar.
'implicit-check-not' should be used with caution, btw. It can cause
matches against the file path in some situations.
See: https://reviews.llvm.org/rL361621, https://reviews.llvm.org/rL361563?

George.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191107/ba3387ce/attachment.html>

James Henderson via llvm-dev

2019-Nov-07 10:06 UTC

head link

[llvm-dev] FileCheck idiom difficulties

FWIW, I did a BoF on the llvm binutils earlier this year at Euro LLVM, to
discuss the direction of the binutils and one of the suggestions that
seemed to get broad appeal was a more machine-readable output format option
(e.g. JSON/XML). llvm-readobj's output is relatively human readable, but
isn't quite JSON. You could possibly then write a straightforward python
script to load the json and test it's properties exactly, if you're
interested in the property values rather than how they are printed. I
haven't got around to this yet, and probably won't for quite some time,
so
if anybody else is interested in the idea, I'd be happy for them to take it
forward.

James

On Wed, 6 Nov 2019 at 19:12, Reid Kleckner <rnk at google.com> wrote:
> I think we should think about changing llvm-readobj to produce more
> filecheck friendly output.
>
> This was a problem we used to have with LLVM debug info metadata: it
> wasn't very structured, and it was printed in some arbitrary order.
Then
> Duncan (I think) added the DI* classes, which made it easier to match
> something semantic, like `DILocalVariable.*name: "foo"`, and
standardized
> on a topological output ordering, so you could start from the variable,
> then match the type, and then find the type metadata number later.
>
> If we printed `Section (.foo) {` for example, that would help some.
>
> It still doesn't help establish delimited regions for properties
printed
> across multiple lines for readability, though... Should we add some kind of
> ad-hoc delimiter balancing to FileCheck? Something like:
>
> CHECK-SCOPE: Section {
> CHECK: Name: asdf
> CHECK: Name: asdf
> CHECK-ENDSCOPE: }
>
> A SCOPE directive line would have to end in a known delimiter, '(',
'{',
> or '['. The ENDSCOPE directive would only match lines with
delimiters that
> balance with the opening delimiter. It would kind of work for filechecking
> JSON, for example.
>
> The scope idea here is pretty half-baked, but it's food for thought.
>
> On Wed, Nov 6, 2019 at 4:18 AM James Henderson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>> Hi all,
>>
>> Many of our lit tests use FileCheck and a tool like llvm-readobj to
check
>> properties of a section header/symbol/etc. A typical (pseudoised for
>> brevity) output to match against might be something like the following:
>>
>> Symbols [
>>   Symbol {
>>     Name: foo
>>     Value: 0
>>     Type: Function
>>     Section: .foo (1)
>>   }
>>   Symbol {
>>     Name: bar
>>     Value: 1
>>     Type: Object
>>     Section: .foo (1)
>>   }
>> ]
>>
>> and your lit test might want to check the properties of the foo symbol
>> like so:
>>
>> # CHECK:      Name: foo
>> # CHECK-NEXT: Value: 0
>> # CHECK-NEXT: Type: Function
>> # CHECK-NEXT: Section: .foo (1)
>>
>> This is fine. But what if you only care about the section of a symbol,
>> and not the value or type etc? You could do the following:
>>
>> # CHECK: Name: foo
>> # CHECK: Section: .foo (1)
>>
>> Hopefully some of you will already notice the problem with this
approach:
>> if foo was in, say, the .baz section, the test will spuriously pass,
>> because the Section line will match the Section line for .bar. One
>> alternative to this is to explicitly match each field in between, using
>> CHECK-NEXT:
>>
>> # CHECK:      Name: foo
>> # CHECK-NEXT: Value:
>> # CHECK-NEXT: Type:
>> # CHECK-NEXT: Section: .foo (1)
>>
>> This works, but somewhat hides what is really being tested by adding
>> extra noise to the checks. In reality, there are actually other fields
too
>> that need to be listed, meaning the "interesting" parts of
the test are
>> even more hidden.
>>
>> I recently started using yet another approach:
>>
>> # CHECK: Name: foo
>> # CHECK: Section:
>> # CHECK-SAME:     .foo (1)
>>
>> This works because the Section: matched will be the first one found,
i.e.
>> the one belonging to foo, and then .foo will be looked for on the same
>> line. However, I noticed today that this pattern has its own problem,
>> namely that there could be something between the Section tag and .foo.
In
>> other words, the above pattern would match "Section:
.bar.foo". A couple of
>> solutions to this are:
>>
>> # CHECK: Section:
>> # CHECK-NOT: {{[:graph:]}}
>> # CHECK-SAME: .foo (1)
>>
>> # CHECK: Section:
>> # CHECK-SAME: {{^}} .foo (1)
>>
>> The first one ensures that there's no non-whitespace between the
end of
>> "Section:" and the start of ".foo (1)". The second
ensures that the start
>> of the CHECK-SAME match is the "start of line", and since the
first half of
>> the line has already been consumed, it means " .foo (1)" must
immediately
>> follow "Section:". However, the first is even less readable
than the
>> current CHECK-SAME approach, whilst the second is somewhat confusing if
you
>> don't realise that FileCheck effectively consumes the things it has
matched
>> already, so that they effectively don't exist any more.
>>
>> Does anybody have any other suggestions/thoughts/comments? One idea I
had
>> was for a new directive something like "CHECK-IMMEDIATE"
which is
>> implicitly the same as the final approach I suggested above, but maybe
>> adding a new directive to achieve this isn't the right approach?
>>
>> James
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191107/921ac9d6/attachment.html>

llvm dev - Nov 2019 - FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties

[llvm-dev] FileCheck idiom difficulties