[LLVMdev] [cfe-dev] LLVM IRC channel flooded?

Chris Matthews
Tue, 19 May 2015 18:28:17 -0000

When we built green dragon we tried to be really accountable for this sort of
cruft, with a goal of 99% useful notifications, or nothing.

On green dragon we curate both which builds notify the IRC and which builds show
up on the main page.  Anything that fails for reasons unrelated to the commit is
not allowed to do either. We use the phased build approach to make sure we
notify only once per failure.  Builds that are red for more than a week are
disabled, if we can’t fix it in a week, its not worth building anymore.
Because of that, libcxx builds, LLDB and performance builds do not notify and
some are disabled.  When we email the blamelist, I am CCed on every email, and
they are not filtered from my inbox.  If the blame list is long, it only emails
me, and I track down who broke it.

Of course green dragon only runs a small proportion of the total builds.

If you can’t look at the build page, and know that everything that is red is a
real problem, we have a real problem.  Even within builds, if most of the steps
are marked as failures, you don’t know what when wrong.

> On May 19, 2015, at 10:40 AM, James Y Knight <jyknight at google.com>
wrote:
>
> Yes, I also find the amount of bot spam in #llvm is basically intolerable.
It makes it difficult to see actual people talking. At first, I just put all the
bots on /ignore. Now I have an xchat script to move the botspam to another tab
(tabify-004.pl <http://tabify-004.pl/>). I'd recommend that the bots
should just be moved to #llvm-bots and fix the problem for everyone. Those who
are committing changes can join that channel, too, and others don't care.
>
> While we're on this subject, I also find the official buildbot page
(lab.llvm.org:8011 <http://lab.llvm.org:8011/>) almost unusable, since so
many columns are either always red, or else are so flaky that they basically
randomly alternate between passing and failing. So, at a glance, it's
impossible to tell whether the current state of the tree is good. (I certainly
haven't memorized which ones are "supposed" to be red, and which
are not. Maybe others have). Having flaky and always-failing builds show up on
the buildbot pages, and notifying IRC, really has negative utility, since it not
only is not providing useful information, but is serving to obscure the actual
important failures, and causing people to spend time investigating non-problems.
>
> Someone gave me the hint to use the http://bb.pgr.jp/
<http://bb.pgr.jp/> buildbot page instead, which was a great
recommendation -- that page shows problems much more clearly. But it's
unfortunate that there *needs* to be a separate "sane builders only"
buildmaster.
>
> E.g. (and not to pick on this particular bot, this is just one example of
many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655
<http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655>
-- passed, while the previous failed. But, it's not caused by the commit,
it's just arbitrary.
>
> Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this
bot failed?
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017
<http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017>"
-- answer: because it's randomly broken. Wasted the questioner's time
trying to investigate the failure.
>
>
> If all the flaky or always-broken builder configurations got hidden from
the main pages of buildbot, and stopped sending emails/IRC notifications to
anyone but their "owner", that would be a substantial improvement.
>
> On Tue, May 19, 2015 at 10:15 AM, Renato Golin <renato.golin at
linaro.org <mailto:renato.golin at linaro.org>> wrote:
> Folks,
>
> I know it's a reasonably valuable thing to have the buildbot IRC bot
> publishing results, but the channel is kind of flooded with the
> messages, and the more bots we put up, the worse it will be.
>
> I think we still need the NOC warnings, but not over IRC. The Buildbot
> NOC page is horrible and useless, since it doesn't know the difference
> between "it's red and I know it" from "it's
broken".
>
> For that reason, I have built my own NOC page:
>
> http://people.linaro.org/~renato.golin/llvm/arm-bots/
<http://people.linaro.org/~renato.golin/llvm/arm-bots/>
>
> But that machine is too slow to cope with all bots. We may need a
> project to build such a system on a larger scale.
>
> However, for now, I think not printing the green results in IRC would
> go a long way of cleaning the channel up.
>
> Any thoughts?
>
> cheers,
> --renato
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu <mailto:cfe-dev at cs.uiuc.edu>
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
<http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150519/5f25867e/attachment.html>