llvm dev - May 2015 - [LLVMdev] [cfe-dev] LLVM IRC channel flooded?

On 05/19/2015 10:40 AM, James Y Knight wrote:
> Yes, I also find the amount of bot spam in #llvm is basically 
> intolerable. It makes it difficult to see actual people talking. At 
> first, I just put all the bots on /ignore. Now I have an xchat script 
> to move the botspam to another tab (tabify-004.pl 
> <http://tabify-004.pl/>). I'd recommend that the bots should just
be
> moved to #llvm-bots and fix the problem for everyone. Those who are 
> committing changes can join that channel, too, and others don't care.
>
> While we're on this subject, I also find the official buildbot page 
> (lab.llvm.org:8011 <http://lab.llvm.org:8011/>) almost unusable,
since
> so many columns are either always red, or else are so flaky that they 
> basically randomly alternate between passing and failing. So, at a 
> glance, it's impossible to tell whether the current state of the tree 
> is good. (I certainly haven't memorized which ones are
"supposed" to
> be red, and which are not. Maybe others have). Having flaky and 
> always-failing builds show up on the buildbot pages, and notifying 
> IRC, really has negative utility, since it not only is not providing 
> useful information, but is serving to obscure the actual important 
> failures, and causing people to spend time investigating non-problems.
>
> Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page 
> instead, which was a great recommendation -- that page shows problems 
> much more clearly. But it's unfortunate that there *needs* to be a 
> separate "sane builders only" buildmaster.
>
> E.g. (and not to pick on this particular bot, this is just one example 
> of many): 
> http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655
--
> passed, while the previous failed. But, it's not caused by the commit, 
> it's just arbitrary.
>
> Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this
> bot failed? 
> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017"
> -- answer: because it's randomly broken. Wasted the questioner's
time
> trying to investigate the failure.
>
>
> If all the flaky or always-broken builder configurations got hidden 
> from the main pages of buildbot, and stopped sending emails/IRC 
> notifications to anyone but their "owner", that would be a
substantial
> improvement.
+1, this is a really good summary of issues.  I'm in full support of any 
and all efforts to reduce noise here.

I've gotten to the point where I only watch a small handful bots. 
Anything other than that, I pretty much ignore unless someone emails me 
directly or replies to a commit.  I'm not quite to point of marking 
buildbot emails as spam, but I'm definitely not paying them much 
attention either.  One particular irritant is getting emails 12-24 hours 
later about someone else's breakage that has *already been fixed*.  The 
long cycling bots are really irritating in that respect.
>
> On Tue, May 19, 2015 at 10:15 AM, Renato Golin 
> <renato.golin at linaro.org <mailto:renato.golin at
linaro.org>> wrote:
>
>     Folks,
>
>     I know it's a reasonably valuable thing to have the buildbot IRC
bot
>     publishing results, but the channel is kind of flooded with the
>     messages, and the more bots we put up, the worse it will be.
>
>     I think we still need the NOC warnings, but not over IRC. The Buildbot
>     NOC page is horrible and useless, since it doesn't know the
difference
>     between "it's red and I know it" from "it's
broken".
>
>     For that reason, I have built my own NOC page:
>
>     http://people.linaro.org/~renato.golin/llvm/arm-bots/
>     <http://people.linaro.org/%7Erenato.golin/llvm/arm-bots/>
>
>     But that machine is too slow to cope with all bots. We may need a
>     project to build such a system on a larger scale.
>
>     However, for now, I think not printing the green results in IRC would
>     go a long way of cleaning the channel up.
>
>     Any thoughts?
>
>     cheers,
>     --renato
>     _______________________________________________
>     cfe-dev mailing list
>     cfe-dev at cs.uiuc.edu <mailto:cfe-dev at cs.uiuc.edu>
>     http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150520/085b0b11/attachment.html>

On 20 May 2015 at 18:47, Philip Reames <listmail at philipreames.com>
wrote:
> One particular irritant is getting emails 12-24 hours later about someone
else's
> breakage that has *already been fixed*.  The long cycling bots are really
> irritating in that respect.
That's not that easy to fix, and I think we'll have to cope with that
forever. Not all machines are fast, and some buildbots do a full
self-host, with compiler-rt and running all tests. Others do a full
benchmark run of LNT, running it 5-8 times, which can take several
hours on an ARM box.

The benchmark bots should be marked not to spam, since they're not
there to pick up errors, but the full self-hosting ones do need to
warn on errors. For example, right now I have a bug only on a thumbv7a
self-hosting bot, and not on others. I'm now bisecting it to find the
culprit, but this is not always clear, as the longer it takes for me
to realise, the harder it will be to fix it.

The only way out of it is for people to look at the fast bots, and if
they're fixed, check the commit that did it and see if the slow bot
has been fixed by the same commit later.

Buildbot owners will eventually pick those problems up, but as I said,
the longer it takes, the harder it is to get to the bottom of it, and
the higher is the probability of getting more regressions introduced
because the bot is red and won't warn.

cheers,
--renato

On 05/20/2015 11:04 AM, Renato Golin wrote:
> On 20 May 2015 at 18:47, Philip Reames <listmail at philipreames.com>
wrote:
>> One particular irritant is getting emails 12-24 hours later about
someone else's
>> breakage that has *already been fixed*.  The long cycling bots are
really
>> irritating in that respect.
> That's not that easy to fix, and I think we'll have to cope with
that
> forever. Not all machines are fast, and some buildbots do a full
> self-host, with compiler-rt and running all tests. Others do a full
> benchmark run of LNT, running it 5-8 times, which can take several
> hours on an ARM box.
I agree it's not easy, but it's not something we should just live with 
either. There are ways to address the problem and we should consider them.

As a randomly chosen example, one thing we could do would be to have the 
notion of a "last good commit".  Fast builders would cycle off ToT, if
any one (or some subset) passed, that advances the notion of last good 
commit.  Slower builders should cycle off the last good commit, not 
ToT.  We have all the mechanisms to implement this today.  It could be 
as simple as parsing the JSON output of buildbot in the script that runs 
the slower build bots and sync to a particular revision rather than
ToT.
>
> The benchmark bots should be marked not to spam, since they're not
> there to pick up errors, but the full self-hosting ones do need to
> warn on errors. For example, right now I have a bug only on a thumbv7a
> self-hosting bot, and not on others. I'm now bisecting it to find the
> culprit, but this is not always clear, as the longer it takes for me
> to realise, the harder it will be to fix it.
At this point, you're long past the point I was grossing about.  I'm not
arguing that long running bots shouldn't notify; I'm arguing they 
shouldn't report *obvious* false positives.

Also, the bisect step really should be automated... :)
>
> The only way out of it is for people to look at the fast bots, and if
> they're fixed, check the commit that did it and see if the slow bot
> has been fixed by the same commit later.
You've now wasted 10 minutes or more my time per slow noisy bot. When I 
routinely get 10+ builder failure emails for changes that are clean, 
that's not worthwhile investment.
>
> Buildbot owners will eventually pick those problems up, but as I said,
> the longer it takes, the harder it is to get to the bottom of it, and
> the higher is the probability of getting more regressions introduced
> because the bot is red and won't warn.
I agree.  All I'm suggesting is reducing noise so that real failures are 
likely to be noticed quickly.
>
> cheers,
> --renato