llvm dev - Oct 2016 - [RFC] Require PRs for XFAILing tests

> -----Original Message-----
> From: Alex Bradbury [mailto:asb at asbradbury.org]
> Sent: Saturday, October 01, 2016 1:06 PM
> To: Robinson, Paul
> Cc: Renato Golin; Chris Bieneman; llvm-dev at lists.llvm.org
> Subject: Re: [llvm-dev] [RFC] Require PRs for XFAILing tests
> 
> On 28 September 2016 at 19:58, Robinson, Paul via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > On 28 September 2016 at 10:08, Chris Bieneman via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> >> I cannot think of any situation where a universally failing test
> >> should be in-tree unless it is a bug that someone is expecting to
fix.
> >
> > It seems moderately common to mark something XFAIL temporarily to get
> > the bots green while then going ahead to fix the issue.  Your proposal
> > would add extra overhead to that flow by requiring a PR as well.  This
> > has value when it turns out that fix can't happen in the short
term for
> > any reason.  I don't have a feel for how common that is, although
I'm
> > sure it does happen.
> > I think the overhead is worth the added value, but then I'm a
process
> > kind of guy.
> 
> I'm not saying I _like_ this solution, but if that were an issue we
> could always have an open issue e.g. "PRNNNN: Some tests are marked
> XFAIL but only have this generic PR listed as the reason", for use in
> these "quick fix" cases. It would also be easy to track if these
> "quick fixes" didn't happen shortly.
As David Blaikie mentioned, our bug hygiene is not really that good.
It would be easy to find the set of tests citing the generic PR, but
somebody would have to take it upon themselves to go looking for them.
By the time that happened, the kinds of details we'd want to see in a
bug would be just as missing as if we had no XFAIL-to-PR link at all.

Conversely, requiring short-term XFAILs to have their own PR means
that if somebody fixed the test and forgot to close the PR, that
dangling PR would be easy to recognize as something that could be
summarily closed if anybody decided to go look at all the XFAIL-linked
PRs.  This scenario leaves an open PR kicking around, O the horror,
but we have not lost any useful information.

Now, I think it would be a great and useful thing for somebody to take
on the role of PR Czar, to do that kind of sanitization of the bugs,
but I don't see it happening as an ongoing role.  Therefore I prefer
a process that is a bit more tedious but doesn't lose information
over a simpler but lossy process.
--paulr
> 
> Alex

On 10/3/2016 12:21 PM, Robinson, Paul via llvm-dev
wrote:
> As David Blaikie mentioned, our bug hygiene is not really that good.
Adding a process to create more PRs is not going to change that. Until 
we have a plan to address that, forcing more bug reports is not really 
going to accomplish much.

-Krzysztof

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Krzysztof Parzyszek via llvm-dev
> Sent: Monday, October 03, 2016 10:40 AM
> To: llvm-dev at lists.llvm.org
> Subject: Re: [llvm-dev] [RFC] Require PRs for XFAILing tests
> 
> On 10/3/2016 12:21 PM, Robinson, Paul via llvm-dev wrote:
> > As David Blaikie mentioned, our bug hygiene is not really that good.
> 
> Adding a process to create more PRs is not going to change that.
Of course not.  The point of my remark is that linking an XFAIL test
to a generic content-free PR is no better than having no link at all,
*because* our hygiene is not good.
> Until
> we have a plan to address that, forcing more bug reports is not really
> going to accomplish much.
If the XFAIL-linked PRs are content-free, that is worse than useless.
The point is to have those PRs say something useful about the specific
XFAIL case, which is a vast improvement over what we have today (i.e.,
almost nothing).

According to numbers thrown around on this thread, the net increase
in PRs would be around 1%.
--paulr
> 
> -Krzysztof
> 
> --
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted by The Linux Foundation
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev