llvm dev - Aug 2015 - buildbot failure in LLVM on clang-native-arm-cortex-a9

On 26 August 2015 at 17:21, David Blaikie <dblaikie at gmail.com>
wrote:
> (oh, and add long cycle times to the list of issues - people do have a
> tendency to ignore bots that come back with giant blame lists & no
obvious
> determination as to who's patch caused the problem, if any)
Yes, but remember, not all hardware is as fast as a multi-core Xeon
server. Build times can't always be controlled.

But I agree with you on all accounts. The bot owner should bear the
responsibility of his/her own unstable bots. If it brings less value
than it adds cost to the community, it should be moved to a separate
buildmaster that doesn't email people around, but can be accessed, so
the owner can point breakages to devs.

cheers,
--renato

On Wed, Aug 26, 2015 at 9:27 AM, Renato Golin <renato.golin at linaro.org>
wrote:
> On 26 August 2015 at 17:21, David Blaikie <dblaikie at gmail.com>
wrote:
> > (oh, and add long cycle times to the list of issues - people do have a
> > tendency to ignore bots that come back with giant blame lists & no
> obvious
> > determination as to who's patch caused the problem, if any)
>
> Yes, but remember, not all hardware is as fast as a multi-core Xeon
> server. Build times can't always be controlled.
>
Small blame lists can still be acquired by having more hardware. Certainly
not always possible/in the budget for those who want to verify these things.

> But I agree with you on all accounts. The bot owner should bear the
> responsibility of his/her own unstable bots. If it brings less value
> than it adds cost to the community, it should be moved to a separate
> buildmaster that doesn't email people around, but can be accessed, so
> the owner can point breakages to devs.
>
More than just not emailing, it'd be great to have a generally-green
dashboard to look at. For now it's hard to get a sense of what's
'really'
broken'.

But yeah, all of that stuff.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150826/21f94c70/attachment.html>

On 08/26/2015 09:27 AM, Renato Golin via llvm-dev wrote:
> On 26 August 2015 at 17:21, David Blaikie <dblaikie at gmail.com>
wrote:
>> (oh, and add long cycle times to the list of issues - people do have a
>> tendency to ignore bots that come back with giant blame lists & no
obvious
>> determination as to who's patch caused the problem, if any)
> Yes, but remember, not all hardware is as fast as a multi-core Xeon
> server. Build times can't always be controlled.
No, but there are many known partial solutions.  We can explore some of 
them if build time is the only issue.

Some options (I don't know which if any have already been explored)
- Kick off builds on slow bots at revisions which have passed a fast bot
- Overlap builds on different pieces of hardware so that revision ranges 
are smaller (i.e. "that change was in the previous build and didn't 
fail, so it can't be that")
- Build incrementally with occasional clean builds at already known good 
revisions (or for failure recovery)
- Separate build and test.  Cross build on a faster machine then 
transfer the binaries to a slower test machine.  Run only a build on the 
slow machine.  (i.e. decrease latency of build+test to latency of build 
OR test)
- Consider using an emulator on a faster machine to get initial 
results.  Only kick off builds on hardware for changes that pass the 
emulator.  (This is a particular variant of (1) above.)
>
> But I agree with you on all accounts. The bot owner should bear the
> responsibility of his/her own unstable bots. If it brings less value
> than it adds cost to the community, it should be moved to a separate
> buildmaster that doesn't email people around, but can be accessed, so
> the owner can point breakages to devs.
>
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 26 August 2015 at 17:32, David Blaikie <dblaikie at gmail.com>
wrote:
> Small blame lists can still be acquired by having more hardware. Certainly
> not always possible/in the budget for those who want to verify these
things.
More unstable hardware is more unstable. :)

cheers,
--renato

On Wed, Aug 26, 2015 at 9:38 AM, Philip Reames <listmail at
philipreames.com>
wrote:
> On 08/26/2015 09:27 AM, Renato Golin via llvm-dev wrote:
>
>> On 26 August 2015 at 17:21, David Blaikie <dblaikie at gmail.com>
wrote:
>>
>>> (oh, and add long cycle times to the list of issues - people do
have a
>>> tendency to ignore bots that come back with giant blame lists &
no
>>> obvious
>>> determination as to who's patch caused the problem, if any)
>>>
>> Yes, but remember, not all hardware is as fast as a multi-core Xeon
>> server. Build times can't always be controlled.
>>
> No, but there are many known partial solutions.  We can explore some of
> them if build time is the only issue.
>
> Some options (I don't know which if any have already been explored)
> - Kick off builds on slow bots at revisions which have passed a fast bot
>
This ^

> - Overlap builds on different pieces of hardware so that revision ranges
> are smaller (i.e. "that change was in the previous build and
didn't fail,
> so it can't be that")
> - Build incrementally with occasional clean builds at already known good
> revisions (or for failure recovery)
>
this ^

> - Separate build and test.  Cross build on a faster machine then transfer
> the binaries to a slower test machine.  Run only a build on the slow
> machine.  (i.e. decrease latency of build+test to latency of build OR test)
>
and this ^ would all be nice, but require a bigger investment by someone in
terms of major buildbot overhaul/improvements - Apple had an internal
system (their "phase based builder") that they upstreamed some of, but
then
switched to Jenkins. Not sure if their Jenkins config (public or private)
does this, but might do.

> - Consider using an emulator on a faster machine to get initial results.
> Only kick off builds on hardware for changes that pass the emulator.  (This
> is a particular variant of (1) above.)
>
>>
>> But I agree with you on all accounts. The bot owner should bear the
>> responsibility of his/her own unstable bots. If it brings less value
>> than it adds cost to the community, it should be moved to a separate
>> buildmaster that doesn't email people around, but can be accessed,
so
>> the owner can point breakages to devs.
>>
>> cheers,
>> --renato
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150826/0a34ded9/attachment-0001.html>