On 16 October 2015 at 15:17, Robinson, Paul <Paul_Robinson at playstation.sony.com> wrote:> But if > there are new fails, the blame mailer can do a set-difference and report > only the new ones. That would reduce the noise a bit, hmm?Hi Paul, The danger there is that it'd be easier to "get used" to having some failures as long as you don't have "new" failures. Every place I worked that supported that philosophy, ended up with all bots "orange". It's never the intention, but it's almost always the inevitable consequence. In a small team, or a single company, it may be a lot easier to move them back to green, but in an open community, it's not that easy, nor that quick. The way we work with the same concept, as David mentioned repeatedly, is to use XFAILs. It is essentially the same thing, except that "it hurts more" to mark an XFAIL than to see a different shade of red, so we're more reluctant to ignore them. Plus, an orange bot that becomes red (new failures) will itself become orange as time passes, or new failures show up. If we end up with that many shades of red, understanding the difference will become harder, and the value will decrease. cheers, --renato
> -----Original Message----- > From: Renato Golin [mailto:renato.golin at linaro.org] > Sent: Friday, October 16, 2015 7:29 AM > To: Robinson, Paul > Cc: David Blaikie; LLVM Dev; Galina Kistanova > Subject: Re: [cfe-dev] Buildbot Noise > > On 16 October 2015 at 15:17, Robinson, Paul > <Paul_Robinson at playstation.sony.com> wrote: > > But if > > there are new fails, the blame mailer can do a set-difference and report > > only the new ones. That would reduce the noise a bit, hmm? > > Hi Paul, > > The danger there is that it'd be easier to "get used" to having some > failures as long as you don't have "new" failures. Every place I > worked that supported that philosophy, ended up with all bots > "orange". It's never the intention, but it's almost always the > inevitable consequence. In a small team, or a single company, it may > be a lot easier to move them back to green, but in an open community, > it's not that easy, nor that quick. > > The way we work with the same concept, as David mentioned repeatedly, > is to use XFAILs. It is essentially the same thing, except that "it > hurts more" to mark an XFAIL than to see a different shade of red, so > we're more reluctant to ignore them.Hmmm which is manually telling the bot not to worry about it any more, rather than automatically having the bot figure it out. But from "outside" the bot now looks green, which is less embarrassing to the perpetrator, if embarrassment is part of the social process of getting people to fix things. That part is cultural. And an XFAIL for "test will never work in this configuration" is not obviously different from "somebody broke this test and ought to fix it" from the outside either; on my x86_64 Linux I get 148 XFAILs. Which ones are "will never work" and which are "ought to get fixed soon"? For the people who now get impatient with red bots, there is no barrier to having them be impatient with orange bots. And for the people who complain about irrelevant noise, there would be less of it, which will make them less irritating the rest of us. :-)> > Plus, an orange bot that becomes red (new failures) will itself become > orange as time passes, or new failures show up. If we end up with that > many shades of red, understanding the difference will become harder, > and the value will decrease.My internal dashboard has 4 values that would be meaningful upstream: pass, infra-failure, fail, no-new-fail. Buildbot seems to have the first three already; one more does not seem like an excessive conceptual burden. --paulr> > cheers, > --renato
On 16 October 2015 at 19:10, Robinson, Paul <Paul_Robinson at playstation.sony.com> wrote:> That part is cultural.It is... :)> Which ones are "will never work" and which are "ought to get fixed soon"?In theory, the ones that have bugs associated with it, which is not easy to find unless someone adds a comment with the XFAIL, which not many people do. I'm not saying it's better, I'm just saying how it is.> For the people who now get impatient with red bots, there is no barrier > to having them be impatient with orange bots. And for the people who > complain about irrelevant noise, there would be less of it, which will > make them less irritating the rest of us. :-)I honestly don't see either way as problems, so I can't tell you how much better it would be. :) Maybe you should talk to David directly. I really don't mind whatever you guys choose.> My internal dashboard has 4 values that would be meaningful upstream: > pass, infra-failure, fail, no-new-fail. Buildbot seems to have the first > three already; one more does not seem like an excessive conceptual burden.We don't have the infra problem totally nailed yet, only if the bots crash/restart, not if it's SVN or server issue. But you're right, it's not binary. I personally don't treat orange and red bots differently, for me the only colour I want to see is green. But some people complain that if they have to distinguish between different shades of red, they just won't. I find it an odd behaviour in a community that wants to be diverse and "safe", to openly state you don't care about other people's problems and think that it's the other person that should "fix", but anyway. cheers, --renato