llvm dev - Aug 2015 - buildbot failure in LLVM on clang-native-arm-cortex-a9

On 08/26/2015 04:12 PM, Renato Golin via llvm-dev wrote:
> On 26 August 2015 at 15:07, Davide Italiano <davide at freebsd.org>
wrote:
>> I see frequent timeouts on this bot -- can the timeout be increased?
>
> Hi Davide,
>
> Increasing the timeout is not ideal. We need to fix the problem in a
> different way.
>
> Our other ARM bots are using CMake 3.2 + Ninja 1.5, which gets rid of
> one source of time outs (slow testing), so that may be one way out.
>
> For now, please ignore any timeout on bots, as they happen not just
> with this one. :)
What's the problem with increasing the timeout? Asking people to ignore
buildbot mails does not seem right. If the buildbot is flaky I believe
the buildbot owner should ensure it shuts up until the problems have
been resolved and the buildbot has a low false positive rate again.

Best,
Tobias

On Wed, Aug 26, 2015 at 10:32 AM, Tobias Grosser via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> On 08/26/2015 04:12 PM, Renato Golin via llvm-dev wrote:
>>
>> On 26 August 2015 at 15:07, Davide Italiano <davide at
freebsd.org> wrote:
>>>
>>> I see frequent timeouts on this bot -- can the timeout be
increased?
>>
>>
>> Hi Davide,
>>
>> Increasing the timeout is not ideal. We need to fix the problem in a
>> different way.
>>
>> Our other ARM bots are using CMake 3.2 + Ninja 1.5, which gets rid of
>> one source of time outs (slow testing), so that may be one way out.
>>
>> For now, please ignore any timeout on bots, as they happen not just
>> with this one. :)
>
>
> What's the problem with increasing the timeout? Asking people to ignore
> buildbot mails does not seem right. If the buildbot is flaky I believe
> the buildbot owner should ensure it shuts up until the problems have
> been resolved and the buildbot has a low false positive rate again.
Yes, please. I would go one step further and ask that authors of flaky
tests with non-deterministic failures should revert those tests until
the problems have been resolved as well. The number of false positives
currently makes it very hard to know what the state of the product is.

~Aaron
>
> Best,
> Tobias
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 26 August 2015 at 15:32, Tobias Grosser <tobias at grosser.es>
wrote:
> What's the problem with increasing the timeout? Asking people to ignore
> buildbot mails does not seem right. If the buildbot is flaky I believe
> the buildbot owner should ensure it shuts up until the problems have
> been resolved and the buildbot has a low false positive rate again.
That's the point I make about solving the real issue, not increase the
timeout.

CMake + Ninja has fixed virtually all our flakiness on all other ARM
bots, so I think we should give it a try first.

--renato

On 08/26/2015 04:38 PM, Renato Golin via llvm-dev wrote:
> On 26 August 2015 at 15:32, Tobias Grosser <tobias at grosser.es>
wrote:
>> What's the problem with increasing the timeout? Asking people to
ignore
>> buildbot mails does not seem right. If the buildbot is flaky I believe
>> the buildbot owner should ensure it shuts up until the problems have
>> been resolved and the buildbot has a low false positive rate again.
>
> That's the point I make about solving the real issue, not increase the
timeout.
>
> CMake + Ninja has fixed virtually all our flakiness on all other ARM
> bots, so I think we should give it a try first.
What time-line do you have in mind for this fix? If you are in charge
and can make this happen within a day, giving cmake + ninja a chance seems
OK.

However, if the owner of the buildbot is not known or the fix can not come
soon, I am in favor of disabling the noise and (re)enabling it when someone
found time to address the problem and verify the solution. The cost of
buildbot noise is very high, both in terms of developer time spent, but
more importantly due to people starting to ignore them when monitoring them
becomes costly.

Best,
Tobias