[llvm-dev] [RFC] Require PRs for XFAILing tests

Matthias Braun via llvm-dev
Wed, 28 Sep 2016 20:16:43 -0000

> On Sep 28, 2016, at 1:01 PM, Chris Bieneman via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>
> I think there are a few interesting things that could follow from
solidifying a policy requiring PRs for XFAILs.
>
> First and foremost bugs can have way more context than than you would often
find in a test case comment. That would make it a lot easier to audit XFAILs in
the future and help keep the number of XFAILs to a minimum. I think this is
important because many of our XFAILs are really old, and I’m not convinced that
we shouldn’t just be deleting some of these tests.
>
> For example, 2008-12-14-StrideAndSigned.ll and 7 other tests were marked
with “XFAIL: *” in 2009, and the commit message doesn’t really explain what was
going on:
>
>> commit 789558db70d9513a017c11c5be30945839fdff1c
>> Author: Nick Lewycky <nicholas at mxc.ca>
>> Date:   Tue Jan 13 09:18:58 2009 +0000
>>
>>    Wind SCEV back in time, to Nov 18th. This 'fixes' PR3275,
PR3294, PR3295,
>>    PR3296 and PR3302.
>>
>>
>>    git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk at 62160
91177308-0d34-0410-b5e6-96231b3b80d8
>
>
> Requiring a PR doesn’t necessarily fix this problem because you could list
the wrong PR, or even just a generic stub PR that didn’t add meaningful value,
but I think our community is pretty good at keeping people honest via code
reviews.

Other important aspects:
- We can immediately start a specific and meaningful conversation about the
problem
- Progress/blockers in solving the problem can be documented if necessary
- It is actually easy to find the corresponding discussion to a problem (the
alternative of looking up the commit adding the XFAIL and then searching the
commits mailing list for threads on that commit is more trouble)

- Matthias