llvm dev - Oct 2020 - [RFC] FileCheck: (dis)allowing unused prefixes

Hello all,

TL;DR; if you used FileCheck --check-prefixes and you missed (misspelled,
for instance) one of the prefixes in your test, FileCheck silently ignores
that and the test passes.

1579 tests have this property.


*The details*
*=========*
Please refer to https://reviews.llvm.org/D90281 and the discussion there
for more details (make sure you open "older changes" for full context)

The problem is covered by the TL;DR;.

The proposal is to add an explicit flag to FileCheck,
--allow-unused-prefixes, to indicate whether the current behavior is
intended (for instance, jdoerfert contributed a scenario where that is the
case).

We want the default behavior to be 'strict', i.e.
--allow-unused-prefixes=false. Doing that right now would lead to 1500 test
failures.

To get there (thanks, maskray, for suggestion), we propose we:
* land D90281 where the flag is introduced, but is flipped to match today's
behavior
* employ a 'busy beavers' approach, where test maintainers patch their
tests:
  - either leveraging the flag, to explicitly indicate that unused prefixes
is intended (i.e. add --allow-unused-patches=true); or
  - fix the test (e.g. maybe there was a misspelling issue/omission/etc).

A spreadsheet with the failing tests is available here
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>
[1].

The request to the community members is to please sign up for their
respective area in the spreadsheet, and then mark it completed when that's
the case (yes/no in the respective column).

When all the tests are fixed, we will then flip --allow-unused-prefixes to
false by default.

Meanwhile, please consider leveraging the flag explicitly when you author
new tests that use --check-prefixes. That can be then cleaned up easily
after we switch to the 'strict' behavior.

Thanks!

[1]
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/fdb099cc/attachment.html>

Thanks for doing this!  I am impressed that the test count is so high; what is
that, around 2% of all tests?  I wonder how many are typos (bad) versus
genuinely unused (benign), but that would be tedious to determine.
Example of a typo: clang/test/AST/ast-dump-decl.c, MODULE vs MODULES
Example of unused: llvm/test/DebugInfo/X86/dbg-value-inlined-parameter.ll,
specifies LINUX and DARWIN but neither is used.

I’ll take this opportunity to mention that I’m planning a 10% project to look at
the inverse question: What directives are in the test files but aren’t exercised
by any FileCheck run?  This is trickier as you need to collect data across
multiple runs, and tests run in parallel.  Also it’s a bit ambiguous what “looks
like” a directive, they aren’t really all intended to be directives.  Should be
fun.
If anyone is already delving into this, I’d be curious to hear about the
approach.

Thanks,
--paulr

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Mircea
Trofin via llvm-dev
Sent: Thursday, October 29, 2020 1:28 PM
To: LLVM Dev <llvm-dev at lists.llvm.org>
Subject: [llvm-dev] [RFC] FileCheck: (dis)allowing unused prefixes

Hello all,

TL;DR; if you used FileCheck --check-prefixes and you missed (misspelled, for
instance) one of the prefixes in your test, FileCheck silently ignores that and
the test passes.

1579 tests have this property.


The details
========Please refer to
https://reviews.llvm.org/D90281<https://urldefense.com/v3/__https:/reviews.llvm.org/D90281__;!!JmoZiZGBv3RvKRSx!sQiWDnLaUAuuZLnJnmDYlIXvXwHr_j61zk-FLPNO_AhEEHEXZo_4yewxrqdv8DIPHA$>
and the discussion there for more details (make sure you open "older
changes" for full context)

The problem is covered by the TL;DR;.

The proposal is to add an explicit flag to FileCheck, --allow-unused-prefixes,
to indicate whether the current behavior is intended (for instance, jdoerfert
contributed a scenario where that is the case).

We want the default behavior to be 'strict', i.e.
--allow-unused-prefixes=false. Doing that right now would lead to 1500 test
failures.

To get there (thanks, maskray, for suggestion), we propose we:
* land D90281 where the flag is introduced, but is flipped to match today's
behavior
* employ a 'busy beavers' approach, where test maintainers patch their
tests:
  - either leveraging the flag, to explicitly indicate that unused prefixes is
intended (i.e. add --allow-unused-patches=true); or
  - fix the test (e.g. maybe there was a misspelling issue/omission/etc).

A spreadsheet with the failing tests is available
here<https://urldefense.com/v3/__https:/docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing__;!!JmoZiZGBv3RvKRSx!sQiWDnLaUAuuZLnJnmDYlIXvXwHr_j61zk-FLPNO_AhEEHEXZo_4yewxrqeR_nOyIg$>
[1].

The request to the community members is to please sign up for their respective
area in the spreadsheet, and then mark it completed when that's the case
(yes/no in the respective column).

When all the tests are fixed, we will then flip --allow-unused-prefixes to false
by default.

Meanwhile, please consider leveraging the flag explicitly when you author new
tests that use --check-prefixes. That can be then cleaned up easily after we
switch to the 'strict' behavior.

Thanks!

[1]
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing<https://urldefense.com/v3/__https:/docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing__;!!JmoZiZGBv3RvKRSx!sQiWDnLaUAuuZLnJnmDYlIXvXwHr_j61zk-FLPNO_AhEEHEXZo_4yewxrqeR_nOyIg$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/fb1c8765/attachment.html>

On Oct 29, 2020, at 10:28 AM, Mircea Trofin via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hello all,
> 
> TL;DR; if you used FileCheck --check-prefixes and you missed (misspelled,
for instance) one of the prefixes in your test, FileCheck silently ignores that
and the test passes.
> 
> 1579 tests have this property. 
Wow, that is quite a “whoops”.  Thank you for catching this, your plan sounds
really great.

-Chris

> 
> 
> The details
> ========> Please refer to https://reviews.llvm.org/D90281
<https://reviews.llvm.org/D90281> and the discussion there for more
details (make sure you open "older changes" for full context)
> 
> The problem is covered by the TL;DR;. 
> 
> The proposal is to add an explicit flag to FileCheck,
--allow-unused-prefixes, to indicate whether the current behavior is intended
(for instance, jdoerfert contributed a scenario where that is the case).
> 
> We want the default behavior to be 'strict', i.e.
--allow-unused-prefixes=false. Doing that right now would lead to 1500 test
failures.
> 
> To get there (thanks, maskray, for suggestion), we propose we:
> * land D90281 where the flag is introduced, but is flipped to match
today's behavior
> * employ a 'busy beavers' approach, where test maintainers patch
their tests:
>   - either leveraging the flag, to explicitly indicate that unused prefixes
is intended (i.e. add --allow-unused-patches=true); or
>   - fix the test (e.g. maybe there was a misspelling issue/omission/etc).
> 
> A spreadsheet with the failing tests is available here
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>
[1].
> 
> The request to the community members is to please sign up for their
respective area in the spreadsheet, and then mark it completed when that's
the case (yes/no in the respective column).
> 
> When all the tests are fixed, we will then flip --allow-unused-prefixes to
false by default.
> 
> Meanwhile, please consider leveraging the flag explicitly when you author
new tests that use --check-prefixes. That can be then cleaned up easily after we
switch to the 'strict' behavior.
> 
> Thanks!
> 
> [1]
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>_______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/4d940649/attachment.html>

On Thu, Oct 29, 2020 at 11:40 AM Chris Lattner <clattner at nondot.org>
wrote:
> On Oct 29, 2020, at 10:28 AM, Mircea Trofin via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
> Hello all,
>
> TL;DR; if you used FileCheck --check-prefixes and you missed (misspelled,
> for instance) one of the prefixes in your test, FileCheck silently ignores
> that and the test passes.
>
> 1579 tests have this property.
>
>
> Wow, that is quite a “whoops”.  Thank you for catching this, your plan
> sounds really great.
>
Thanks, credit also goes to maskray for helping flesh out the plan.

> -Chris
>
>
>
>
> *The details*
> *=========*
> Please refer to https://reviews.llvm.org/D90281 and the discussion there
> for more details (make sure you open "older changes" for full
context)
>
> The problem is covered by the TL;DR;.
>
> The proposal is to add an explicit flag to FileCheck,
> --allow-unused-prefixes, to indicate whether the current behavior is
> intended (for instance, jdoerfert contributed a scenario where that is the
> case).
>
> We want the default behavior to be 'strict', i.e.
> --allow-unused-prefixes=false. Doing that right now would lead to 1500 test
> failures.
>
> To get there (thanks, maskray, for suggestion), we propose we:
> * land D90281 where the flag is introduced, but is flipped to match
> today's behavior
> * employ a 'busy beavers' approach, where test maintainers patch
their
> tests:
>   - either leveraging the flag, to explicitly indicate that unused
> prefixes is intended (i.e. add --allow-unused-patches=true); or
>   - fix the test (e.g. maybe there was a misspelling issue/omission/etc).
>
> A spreadsheet with the failing tests is available here
>
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>
> [1].
>
> The request to the community members is to please sign up for their
> respective area in the spreadsheet, and then mark it completed when
that's
> the case (yes/no in the respective column).
>
> When all the tests are fixed, we will then flip --allow-unused-prefixes to
> false by default.
>
> Meanwhile, please consider leveraging the flag explicitly when you author
> new tests that use --check-prefixes. That can be then cleaned up easily
> after we switch to the 'strict' behavior.
>
> Thanks!
>
> [1]
>
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/6d098aeb/attachment.html>

On Thu, Oct 29, 2020 at 10:28 AM Mircea Trofin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello all,
>
> TL;DR; if you used FileCheck --check-prefixes and you missed (misspelled,
> for instance) one of the prefixes in your test, FileCheck silently ignores
> that and the test passes.
>
> 1579 tests have this property.
>
>
> *The details*
> *=========*
> Please refer to https://reviews.llvm.org/D90281 and the discussion there
> for more details (make sure you open "older changes" for full
context)
>
> The problem is covered by the TL;DR;.
>
> The proposal is to add an explicit flag to FileCheck,
> --allow-unused-prefixes, to indicate whether the current behavior is
> intended (for instance, jdoerfert contributed a scenario where that is the
> case).
>
> We want the default behavior to be 'strict', i.e.
> --allow-unused-prefixes=false. Doing that right now would lead to 1500 test
> failures.
>
> To get there (thanks, maskray, for suggestion), we propose we:
> * land D90281 where the flag is introduced, but is flipped to match
> today's behavior
> * employ a 'busy beavers' approach, where test maintainers patch
their
> tests:
>   - either leveraging the flag, to explicitly indicate that unused
> prefixes is intended (i.e. add --allow-unused-patches=true); or
>   - fix the test (e.g. maybe there was a misspelling issue/omission/etc).
>
Something that can be done to progressively roll-out the feature to more
and more directories is to use the lit configuration.
For example I cleaned up MLIR to be compliant with the FileCheck stricter
flag `-enable-var-scope` which catches another class of bugs, but couldn't
practically go through the thousands of failures in the monorepo, so the
flag is enabled by default in the MLIR test folder:
https://github.com/llvm/llvm-project/blob/master/mlir/test/lit.cfg.py#L84-L87
You can do the same on a per-directory basis anywhere. If you get to change
the default, you can use the same strategy to opt-out entire subtrees.

(I'll likely try to do the same as enable-var-scope in MLIR and enable it
by default there)

Thanks!

-- 
Mehdi

>
> A spreadsheet with the failing tests is available here
>
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>
> [1].
>
> The request to the community members is to please sign up for their
> respective area in the spreadsheet, and then mark it completed when
that's
> the case (yes/no in the respective column).
>
> When all the tests are fixed, we will then flip --allow-unused-prefixes to
> false by default.
>
> Meanwhile, please consider leveraging the flag explicitly when you author
> new tests that use --check-prefixes. That can be then cleaned up easily
> after we switch to the 'strict' behavior.
>
> Thanks!
>
> [1]
>
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/2d466b0b/attachment.html>

On 10/29/20 1:40 PM, Chris Lattner via llvm-dev wrote:
> On Oct 29, 2020, at 10:28 AM, Mircea Trofin via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> Hello all,
>>
>> TL;DR; if you used FileCheck --check-prefixes and you missed
(misspelled, for instance) one of the prefixes in your test, FileCheck silently
ignores that and the test passes.
>>
>> 1579 tests have this property.
> Wow, that is quite a “whoops”.  Thank you for catching this, your plan
sounds really great.
It has yet to be seen how many are actual "bugs" or
"oversights" vs.
intentional.
We should try to track that as well ;)

~ Johannes

> -Chris
>
>
>>
>> The details
>> ========>> Please refer to https://reviews.llvm.org/D90281
<https://reviews.llvm.org/D90281> and the discussion there for more
details (make sure you open "older changes" for full context)
>>
>> The problem is covered by the TL;DR;.
>>
>> The proposal is to add an explicit flag to FileCheck,
--allow-unused-prefixes, to indicate whether the current behavior is intended
(for instance, jdoerfert contributed a scenario where that is the case).
>>
>> We want the default behavior to be 'strict', i.e.
--allow-unused-prefixes=false. Doing that right now would lead to 1500 test
failures.
>>
>> To get there (thanks, maskray, for suggestion), we propose we:
>> * land D90281 where the flag is introduced, but is flipped to match
today's behavior
>> * employ a 'busy beavers' approach, where test maintainers
patch their tests:
>>    - either leveraging the flag, to explicitly indicate that unused
prefixes is intended (i.e. add --allow-unused-patches=true); or
>>    - fix the test (e.g. maybe there was a misspelling
issue/omission/etc).
>>
>> A spreadsheet with the failing tests is available here
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>
[1].
>>
>> The request to the community members is to please sign up for their
respective area in the spreadsheet, and then mark it completed when that's
the case (yes/no in the respective column).
>>
>> When all the tests are fixed, we will then flip --allow-unused-prefixes
to false by default.
>>
>> Meanwhile, please consider leveraging the flag explicitly when you
author new tests that use --check-prefixes. That can be then cleaned up easily
after we switch to the 'strict' behavior.
>>
>> Thanks!
>>
>> [1]
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>_______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

This seems like a great change! In order to find the specific unused
prefixes it would be helpful if you landed the patch with the `true`
default (I applied the patch locally in the meantime)
--
Keith Smiley


On Thu, Oct 29, 2020 at 10:28 AM Mircea Trofin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello all,
>
> TL;DR; if you used FileCheck --check-prefixes and you missed (misspelled,
> for instance) one of the prefixes in your test, FileCheck silently ignores
> that and the test passes.
>
> 1579 tests have this property.
>
>
> *The details*
> *=========*
> Please refer to https://reviews.llvm.org/D90281 and the discussion there
> for more details (make sure you open "older changes" for full
context)
>
> The problem is covered by the TL;DR;.
>
> The proposal is to add an explicit flag to FileCheck,
> --allow-unused-prefixes, to indicate whether the current behavior is
> intended (for instance, jdoerfert contributed a scenario where that is the
> case).
>
> We want the default behavior to be 'strict', i.e.
> --allow-unused-prefixes=false. Doing that right now would lead to 1500 test
> failures.
>
> To get there (thanks, maskray, for suggestion), we propose we:
> * land D90281 where the flag is introduced, but is flipped to match
> today's behavior
> * employ a 'busy beavers' approach, where test maintainers patch
their
> tests:
>   - either leveraging the flag, to explicitly indicate that unused
> prefixes is intended (i.e. add --allow-unused-patches=true); or
>   - fix the test (e.g. maybe there was a misspelling issue/omission/etc).
>
> A spreadsheet with the failing tests is available here
>
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>
> [1].
>
> The request to the community members is to please sign up for their
> respective area in the spreadsheet, and then mark it completed when
that's
> the case (yes/no in the respective column).
>
> When all the tests are fixed, we will then flip --allow-unused-prefixes to
> false by default.
>
> Meanwhile, please consider leveraging the flag explicitly when you author
> new tests that use --check-prefixes. That can be then cleaned up easily
> after we switch to the 'strict' behavior.
>
> Thanks!
>
> [1]
>
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/287c4861/attachment.html>

An update: as of 871d658c9ceb, the flag is now available, if folks need to
use it.

On Thu, Oct 29, 2020 at 10:28 AM Mircea Trofin <mtrofin at google.com>
wrote:
> Hello all,
>
> TL;DR; if you used FileCheck --check-prefixes and you missed (misspelled,
> for instance) one of the prefixes in your test, FileCheck silently ignores
> that and the test passes.
>
> 1579 tests have this property.
>
>
> *The details*
> *=========*
> Please refer to https://reviews.llvm.org/D90281 and the discussion there
> for more details (make sure you open "older changes" for full
context)
>
> The problem is covered by the TL;DR;.
>
> The proposal is to add an explicit flag to FileCheck,
> --allow-unused-prefixes, to indicate whether the current behavior is
> intended (for instance, jdoerfert contributed a scenario where that is the
> case).
>
> We want the default behavior to be 'strict', i.e.
> --allow-unused-prefixes=false. Doing that right now would lead to 1500 test
> failures.
>
> To get there (thanks, maskray, for suggestion), we propose we:
> * land D90281 where the flag is introduced, but is flipped to match
> today's behavior
> * employ a 'busy beavers' approach, where test maintainers patch
their
> tests:
>   - either leveraging the flag, to explicitly indicate that unused
> prefixes is intended (i.e. add --allow-unused-patches=true); or
>   - fix the test (e.g. maybe there was a misspelling issue/omission/etc).
>
> A spreadsheet with the failing tests is available here
>
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>
> [1].
>
> The request to the community members is to please sign up for their
> respective area in the spreadsheet, and then mark it completed when
that's
> the case (yes/no in the respective column).
>
> When all the tests are fixed, we will then flip --allow-unused-prefixes to
> false by default.
>
> Meanwhile, please consider leveraging the flag explicitly when you author
> new tests that use --check-prefixes. That can be then cleaned up easily
> after we switch to the 'strict' behavior.
>
> Thanks!
>
> [1]
>
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201030/9b06296a/attachment.html>

There are currently 1350 owner-less failures in the spreadsheet
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit#gid=0>.
These seem to be the larger areas there.

If you see an area you have ownership or expertise in, please sign up for
fixing the tests by Monday, Nov. 9.

Otherwise, I will "blanket-add" --allow-unused-prefixes=true to the
remaining failing tests.


On Fri, Oct 30, 2020 at 12:48 PM Mircea Trofin <mtrofin at google.com>
wrote:
> An update: as of 871d658c9ceb, the flag is now available, if folks need to
> use it.
>
> On Thu, Oct 29, 2020 at 10:28 AM Mircea Trofin <mtrofin at
google.com> wrote:
>
>> Hello all,
>>
>> TL;DR; if you used FileCheck --check-prefixes and you missed
(misspelled,
>> for instance) one of the prefixes in your test, FileCheck silently
ignores
>> that and the test passes.
>>
>> 1579 tests have this property.
>>
>>
>> *The details*
>> *=========*
>> Please refer to https://reviews.llvm.org/D90281 and the discussion
there
>> for more details (make sure you open "older changes" for full
context)
>>
>> The problem is covered by the TL;DR;.
>>
>> The proposal is to add an explicit flag to FileCheck,
>> --allow-unused-prefixes, to indicate whether the current behavior is
>> intended (for instance, jdoerfert contributed a scenario where that is
the
>> case).
>>
>> We want the default behavior to be 'strict', i.e.
>> --allow-unused-prefixes=false. Doing that right now would lead to 1500
test
>> failures.
>>
>> To get there (thanks, maskray, for suggestion), we propose we:
>> * land D90281 where the flag is introduced, but is flipped to match
>> today's behavior
>> * employ a 'busy beavers' approach, where test maintainers
patch their
>> tests:
>>   - either leveraging the flag, to explicitly indicate that unused
>> prefixes is intended (i.e. add --allow-unused-patches=true); or
>>   - fix the test (e.g. maybe there was a misspelling
issue/omission/etc).
>>
>> A spreadsheet with the failing tests is available here
>>
<https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing>
>> [1].
>>
>> The request to the community members is to please sign up for their
>> respective area in the spreadsheet, and then mark it completed when
that's
>> the case (yes/no in the respective column).
>>
>> When all the tests are fixed, we will then flip --allow-unused-prefixes
>> to false by default.
>>
>> Meanwhile, please consider leveraging the flag explicitly when you
author
>> new tests that use --check-prefixes. That can be then cleaned up easily
>> after we switch to the 'strict' behavior.
>>
>> Thanks!
>>
>> [1]
>>
https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit?usp=sharing
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201105/7d2f4f72/attachment.html>