On Tue, May 19, 2015 at 10:40 AM, James Y Knight <jyknight at google.com> wrote:> Yes, I also find the amount of bot spam in #llvm is basically intolerable. > It makes it difficult to see actual people talking. At first, I just put > all the bots on /ignore. Now I have an xchat script to move the botspam to > another tab (tabify-004.pl). I'd recommend that the bots should just be > moved to #llvm-bots and fix the problem for everyone. Those who are > committing changes can join that channel, too, and others don't care. > > While we're on this subject, I also find the official buildbot page ( > lab.llvm.org:8011) almost unusable, since so many columns are either > always red, or else are so flaky that they basically randomly alternate > between passing and failing. So, at a glance, it's impossible to tell > whether the current state of the tree is good. (I certainly haven't > memorized which ones are "supposed" to be red, and which are not. Maybe > others have). Having flaky and always-failing builds show up on the > buildbot pages, and notifying IRC, really has negative utility, since it > not only is not providing useful information, but is serving to obscure the > actual important failures, and causing people to spend time investigating > non-problems. > > Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page > instead, which was a great recommendation -- that page shows problems much > more clearly. But it's unfortunate that there *needs* to be a separate > "sane builders only" buildmaster. > > E.g. (and not to pick on this particular bot, this is just one example of > many): > http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- > passed, while the previous failed. But, it's not caused by the commit, it's > just arbitrary. > > Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot > failed? > http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" -- > answer: because it's randomly broken. Wasted the questioner's time trying > to investigate the failure. >Whenever you get crappy fail-mail, please forward it to llvm-dev, cc'ing the bot owner and request the issue be addressed or the bot be removed. Yeah, I know it's not an ideal process, but it's something to keep issues visible/pushed on. But, yes, having some more formal process to deal with this sort of thing would be nice (I can imagine some process along the lines of "bots start in experimental and need a track record of low flake/false positive results for some period of time before being promoted out of experimental so they can send mail to blame lists and IRC, etc" coupled with some mechanism for demoting a buildbot back into experimental if it starts behaving poorly) - David> > > If all the flaky or always-broken builder configurations got hidden from > the main pages of buildbot, and stopped sending emails/IRC notifications to > anyone but their "owner", that would be a substantial improvement. > > On Tue, May 19, 2015 at 10:15 AM, Renato Golin <renato.golin at linaro.org> > wrote: > >> Folks, >> >> I know it's a reasonably valuable thing to have the buildbot IRC bot >> publishing results, but the channel is kind of flooded with the >> messages, and the more bots we put up, the worse it will be. >> >> I think we still need the NOC warnings, but not over IRC. The Buildbot >> NOC page is horrible and useless, since it doesn't know the difference >> between "it's red and I know it" from "it's broken". >> >> For that reason, I have built my own NOC page: >> >> http://people.linaro.org/~renato.golin/llvm/arm-bots/ >> >> But that machine is too slow to cope with all bots. We may need a >> project to build such a system on a larger scale. >> >> However, for now, I think not printing the green results in IRC would >> go a long way of cleaning the channel up. >> >> Any thoughts? >> >> cheers, >> --renato >> _______________________________________________ >> cfe-dev mailing list >> cfe-dev at cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev >> > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150519/f2716b6f/attachment.html>
Just some stats, after looking through lab.llvm.org <http://lab.llvm.org/>:8011 Maybe these should be marked as experimental, and removed from the builders link on the main page. Never passed at all: libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03 libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan libcxx-libcxxabi-x86_64-linux-ubuntu-tsan libcxx-libcxxabi-x86_64-linux-ubuntu-gcc libcxx-libcxxabi-x86_64-apple-darwin14-system-lib lldb-x86_64-ubuntu-14.04-android llgo-x86_64-linux Not pass in at least a month: llvm-clang-lld-x86_64-debian-fast clang-native-mingw32-win7 clang-x86_64-linux-selfhost-abi-test clang-x64-ninja-win7-debug perf-x86_64-penryn-O3-polly-detect-only sanitizer-x86_64-linux-bootstrap sanitizer_x86_64-freebsd sanitizer-windows libcxx-libcxxabi-x86_64-apple-darwin14-tot-clang clang-amd64-openbsd lldb-x86_64-debian-clang lldb-x86_64-freebsd lldb-x86_64-ubuntu-14.10> On May 19, 2015, at 11:32 AM, David Blaikie <dblaikie at gmail.com> wrote: > > > > On Tue, May 19, 2015 at 10:40 AM, James Y Knight <jyknight at google.com <mailto:jyknight at google.com>> wrote: > Yes, I also find the amount of bot spam in #llvm is basically intolerable. It makes it difficult to see actual people talking. At first, I just put all the bots on /ignore. Now I have an xchat script to move the botspam to another tab (tabify-004.pl <http://tabify-004.pl/>). I'd recommend that the bots should just be moved to #llvm-bots and fix the problem for everyone. Those who are committing changes can join that channel, too, and others don't care. > > While we're on this subject, I also find the official buildbot page (lab.llvm.org:8011 <http://lab.llvm.org:8011/>) almost unusable, since so many columns are either always red, or else are so flaky that they basically randomly alternate between passing and failing. So, at a glance, it's impossible to tell whether the current state of the tree is good. (I certainly haven't memorized which ones are "supposed" to be red, and which are not. Maybe others have). Having flaky and always-failing builds show up on the buildbot pages, and notifying IRC, really has negative utility, since it not only is not providing useful information, but is serving to obscure the actual important failures, and causing people to spend time investigating non-problems. > > Someone gave me the hint to use the http://bb.pgr.jp/ <http://bb.pgr.jp/> buildbot page instead, which was a great recommendation -- that page shows problems much more clearly. But it's unfortunate that there *needs* to be a separate "sane builders only" buildmaster. > > E.g. (and not to pick on this particular bot, this is just one example of many): http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 <http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655> -- passed, while the previous failed. But, it's not caused by the commit, it's just arbitrary. > > Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this bot failed? http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017 <http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017>" -- answer: because it's randomly broken. Wasted the questioner's time trying to investigate the failure. > > Whenever you get crappy fail-mail, please forward it to llvm-dev, cc'ing the bot owner and request the issue be addressed or the bot be removed. Yeah, I know it's not an ideal process, but it's something to keep issues visible/pushed on. > > But, yes, having some more formal process to deal with this sort of thing would be nice (I can imagine some process along the lines of "bots start in experimental and need a track record of low flake/false positive results for some period of time before being promoted out of experimental so they can send mail to blame lists and IRC, etc" coupled with some mechanism for demoting a buildbot back into experimental if it starts behaving poorly) > > - David > > > > If all the flaky or always-broken builder configurations got hidden from the main pages of buildbot, and stopped sending emails/IRC notifications to anyone but their "owner", that would be a substantial improvement. > > On Tue, May 19, 2015 at 10:15 AM, Renato Golin <renato.golin at linaro.org <mailto:renato.golin at linaro.org>> wrote: > Folks, > > I know it's a reasonably valuable thing to have the buildbot IRC bot > publishing results, but the channel is kind of flooded with the > messages, and the more bots we put up, the worse it will be. > > I think we still need the NOC warnings, but not over IRC. The Buildbot > NOC page is horrible and useless, since it doesn't know the difference > between "it's red and I know it" from "it's broken". > > For that reason, I have built my own NOC page: > > http://people.linaro.org/~renato.golin/llvm/arm-bots/ <http://people.linaro.org/~renato.golin/llvm/arm-bots/> > > But that machine is too slow to cope with all bots. We may need a > project to build such a system on a larger scale. > > However, for now, I think not printing the green results in IRC would > go a long way of cleaning the channel up. > > Any thoughts? > > cheers, > --renato > _______________________________________________ > cfe-dev mailing list > cfe-dev at cs.uiuc.edu <mailto:cfe-dev at cs.uiuc.edu> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev <http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev> > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev> > > > _______________________________________________ > cfe-dev mailing list > cfe-dev at cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150519/f4891d7c/attachment.html>
On 05/19/2015 08:50 PM, Chris Matthews wrote:> Just some stats, after looking through lab.llvm.org > <http://lab.llvm.org>:8011 > > Maybe these should be marked as experimental, and removed from the > builders link on the main page. > > Never passed at all: > libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03 > libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan > libcxx-libcxxabi-x86_64-linux-ubuntu-tsan > libcxx-libcxxabi-x86_64-linux-ubuntu-gcc > libcxx-libcxxabi-x86_64-apple-darwin14-system-lib > lldb-x86_64-ubuntu-14.04-android > llgo-x86_64-linux > > Not pass in at least a month: > > llvm-clang-lld-x86_64-debian-fast > clang-native-mingw32-win7 > clang-x86_64-linux-selfhost-abi-test > clang-x64-ninja-win7-debug > perf-x86_64-penryn-O3-polly-detect-only > sanitizer-x86_64-linux-bootstrap > sanitizer_x86_64-freebsd > sanitizer-windows > libcxx-libcxxabi-x86_64-apple-darwin14-tot-clang > clang-amd64-openbsd > lldb-x86_64-debian-clang > lldb-x86_64-freebsd > lldb-x86_64-ubuntu-14.10Chris, thanks for going through this! I am all in favor of removing/disabling these bots (and would be OK with being even more aggressive). The one Polly bot listed is a performance buildbot which has emails or IRC messages disabled. I now removed it completely from the buildbot list to also keep the web interface clean. Best, Tobias
On Tue, May 19, 2015 at 11:50 AM, Chris Matthews <chris.matthews at apple.com> wrote:> sanitizer-windows >So, I tried to track down what went wrong here, and the oldest build I can find is: http://lab.llvm.org:8011/builders/sanitizer-windows/builds/3916 This raises a different problem: the buildmaster doesn't hold onto enough logs. That build is from five days ago, and already I can't find the relevant blamelist causing the breakage. =/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150519/9ad7d27f/attachment.html>
On 19 May 2015 at 19:50, Chris Matthews <chris.matthews at apple.com> wrote:> Maybe these should be marked as experimental, and removed from the builders > link on the main page.Right now, I have two "failing" LNT bots. One of them is a known LNT server instability, and I brought the bot down myself. If you shut down the bot gracefully, no one gets an email, so if you fix it and bring it back, no one gets annoyed. If the bot owners are not willing to do that kind of management, or are unresponsive, we should take the bots out of the "official" list and not report anything from them. If anyone wants to put a bot up and not care about it, they can also put up their own buildmaster, so that we don't have to mix lost bots with production bots. No emails, no reds on the production page. cheers, --renato
On Wed, 20 May 2015 at 02:55 Chris Matthews <chris.matthews at apple.com> wrote:> Just some stats, after looking through lab.llvm.org:8011 > > Maybe these should be marked as experimental, and removed from the > builders link on the main page. > > Never passed at all: > libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03 > libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan > libcxx-libcxxabi-x86_64-linux-ubuntu-tsan > libcxx-libcxxabi-x86_64-linux-ubuntu-gcc > libcxx-libcxxabi-x86_64-apple-darwin14-system-lib > lldb-x86_64-ubuntu-14.04-android > llgo-x86_64-linux >Hi, llgo-x86_64-linux is mine. Sorry, I had disabled the slave agent to avoid spurious emails, but hadn't considered its impact on the status pages. I'm fine with disabling it altogether for now; I'm waiting on a fix to Ninja to be merged. Cheers, Andrew> Not pass in at least a month: > > llvm-clang-lld-x86_64-debian-fast > clang-native-mingw32-win7 > clang-x86_64-linux-selfhost-abi-test > clang-x64-ninja-win7-debug > perf-x86_64-penryn-O3-polly-detect-only > sanitizer-x86_64-linux-bootstrap > sanitizer_x86_64-freebsd > sanitizer-windows > libcxx-libcxxabi-x86_64-apple-darwin14-tot-clang > clang-amd64-openbsd > lldb-x86_64-debian-clang > lldb-x86_64-freebsd > lldb-x86_64-ubuntu-14.10 > > > On May 19, 2015, at 11:32 AM, David Blaikie <dblaikie at gmail.com> wrote: > > > > On Tue, May 19, 2015 at 10:40 AM, James Y Knight <jyknight at google.com> > wrote: > >> Yes, I also find the amount of bot spam in #llvm is basically >> intolerable. It makes it difficult to see actual people talking. At first, >> I just put all the bots on /ignore. Now I have an xchat script to move the >> botspam to another tab (tabify-004.pl). I'd recommend that the bots >> should just be moved to #llvm-bots and fix the problem for everyone. Those >> who are committing changes can join that channel, too, and others don't >> care. >> >> While we're on this subject, I also find the official buildbot page ( >> lab.llvm.org:8011) almost unusable, since so many columns are either >> always red, or else are so flaky that they basically randomly alternate >> between passing and failing. So, at a glance, it's impossible to tell >> whether the current state of the tree is good. (I certainly haven't >> memorized which ones are "supposed" to be red, and which are not. Maybe >> others have). Having flaky and always-failing builds show up on the >> buildbot pages, and notifying IRC, really has negative utility, since it >> not only is not providing useful information, but is serving to obscure the >> actual important failures, and causing people to spend time investigating >> non-problems. >> >> Someone gave me the hint to use the http://bb.pgr.jp/ buildbot page >> instead, which was a great recommendation -- that page shows problems much >> more clearly. But it's unfortunate that there *needs* to be a separate >> "sane builders only" buildmaster. >> >> E.g. (and not to pick on this particular bot, this is just one example of >> many): >> http://lab.llvm.org:8011/builders/clang-native-arm-cortex-a9/builds/27655 -- >> passed, while the previous failed. But, it's not caused by the commit, it's >> just arbitrary. >> >> Or, yesterday, on #llvm: "Anyone want to give me a clue as to why this >> bot failed? >> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/18017" >> -- answer: because it's randomly broken. Wasted the questioner's time >> trying to investigate the failure. >> > > Whenever you get crappy fail-mail, please forward it to llvm-dev, cc'ing > the bot owner and request the issue be addressed or the bot be removed. > Yeah, I know it's not an ideal process, but it's something to keep issues > visible/pushed on. > > But, yes, having some more formal process to deal with this sort of thing > would be nice (I can imagine some process along the lines of "bots start in > experimental and need a track record of low flake/false positive results > for some period of time before being promoted out of experimental so they > can send mail to blame lists and IRC, etc" coupled with some mechanism for > demoting a buildbot back into experimental if it starts behaving poorly) > > - David > > >> >> >> If all the flaky or always-broken builder configurations got hidden from >> the main pages of buildbot, and stopped sending emails/IRC notifications to >> anyone but their "owner", that would be a substantial improvement. >> >> On Tue, May 19, 2015 at 10:15 AM, Renato Golin <renato.golin at linaro.org> >> wrote: >> >>> Folks, >>> >>> I know it's a reasonably valuable thing to have the buildbot IRC bot >>> publishing results, but the channel is kind of flooded with the >>> messages, and the more bots we put up, the worse it will be. >>> >>> I think we still need the NOC warnings, but not over IRC. The Buildbot >>> NOC page is horrible and useless, since it doesn't know the difference >>> between "it's red and I know it" from "it's broken". >>> >>> For that reason, I have built my own NOC page: >>> >>> http://people.linaro.org/~renato.golin/llvm/arm-bots/ >>> >>> But that machine is too slow to cope with all bots. We may need a >>> project to build such a system on a larger scale. >>> >>> However, for now, I think not printing the green results in IRC would >>> go a long way of cleaning the channel up. >>> >>> Any thoughts? >>> >>> cheers, >>> --renato >>> _______________________________________________ >>> cfe-dev mailing list >>> cfe-dev at cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev >>> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> >> > _______________________________________________ > cfe-dev mailing list > cfe-dev at cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150519/983f5151/attachment.html>