llvm dev - Feb 2017 - Build status expectations for experimental targets

Hey all,

Every few weeks, a change is committed to trunk that breaks the AVR
buildbot.

A problem presents when commit authors do not fix the build, and just leave
it because it passes on the core buildbots. The build stays red for a few
days until I go and check it. In the meantime, it likely causes spam for
most if not all developers when they commit new code.

All commits should keep master green, but what is the expectation for
experimental backends? Is it reasonable to expect all developers who commit
code to ensure tests pass on the AVR backend?

On top of this, is there any way to notify maintainers of a backend when a
buildbot has been failing for some time? I imagine other experimental
backends have run into the same problems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170203/01efdd4e/attachment.html>

> On 3 Feb 2017, at 10:37, Dylan McKay via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hey all,
> 
> Every few weeks, a change is committed to trunk that breaks the AVR
buildbot.
> 
> A problem presents when commit authors do not fix the build, and just leave
it because it passes on the core buildbots. The build stays red for a few days
until I go and check it. In the meantime, it likely causes spam for most if not
all developers when they commit new code.
> 
> All commits should keep master green, but what is the expectation for
experimental backends? Is it reasonable to expect all developers who commit code
to ensure tests pass on the AVR backend?
Hi,

The builder isn't marked as experimental so I think the expectation is that
people keep it green and contact the bot owner if they need help figuring out
why their change makes it red. That said, it sounds a bit odd to have a
non-experimental builder for an experimental backend.
> On top of this, is there any way to notify maintainers of a backend when a
buildbot has been failing for some time? I imagine other experimental backends
have run into the same problems.
I used to have all the MIPS buildbots send me an email on every failure and
filter those emails into a folder. I'd then use the unread mail count to
keep track of how long it's been failing and investigate if it was more than
the occasional build.

If you want to do the same, then you'll need to add an
InformativeMailNotifier to
http://llvm.org/svn/llvm-project/zorg/trunk/buildbot/osuosl/master/config/status.py
<http://llvm.org/svn/llvm-project/zorg/trunk/buildbot/osuosl/master/config/status.py>.
If you search for 'clang-cmake-mips' you'll find the one I used to
use.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170203/7f4add2b/attachment.html>

On Fri, Feb 3, 2017, at 11:37 AM, Dylan McKay via llvm-dev
wrote:
> Hey all,
> 
> Every few weeks, a change is committed to trunk that breaks the AVR
> buildbot.
> 
> A problem presents when commit authors do not fix the build, and just
> leave
> it because it passes on the core buildbots. The build stays red for a few
> days until I go and check it. In the meantime, it likely causes spam for
> most if not all developers when they commit new code.
> 
> All commits should keep master green, but what is the expectation for
> experimental backends? Is it reasonable to expect all developers who
> commit
> code to ensure tests pass on the AVR backend?
>
> On top of this, is there any way to notify maintainers of a backend when
> a
> buildbot has been failing for some time? I imagine other experimental
> backends have run into the same problems.
Hi Dylan,

this probably depends on what we want experimental targets to be and how
they are different from normal targets. From my developer perspective,
anything that is not enabled by default and is not checked by default in
a normal LLVM build is something a normal developer should not need to
worry about. As such, I do not
expect committers to investigate bugs triggered in the AVR backend. If
we would believe the AVR backend is stable enough, such that users can
rely on it and such that other developers are unlikely to trigger bugs
in the AVR backend, the AVR backend should most likely be promoted to a
stable backend.

As a result of this, I would also expect buildbots of the AVR backend to
not send any emails to the general public, but to instead send emails to
the buildbot owner and maintainer of the AVR backend.

In my perspective, it is the job of the buildbot owner of an
experimental backend to investigate failures in this backend and to
clarify if a failure is caused by a bug in a recent change or rather by
a bug in the AVR backend.
If it seems as if the original change committed is incorrect, the
buildbot owner should work with the committer of the corresponding patch
to get the bug resolved.

Obviously, the answer I gave above is very black and white. In reality,
the real answer depends on the maturity of the backend and the quality
of the availability of the buildbot owner. What I wrote above has a
rather new and unstable backend in mind -- especially the category of
backend we would expect to become experimental in the first place. The
more stable a backend is, the more rare breakages are, the faster bugs
are fixed, and the more likely it is that breakages are due to bugs in
LLVM commits, the more it makes sense to have a buildbot that sends out
emails to the actual committers.

In Polly I have two kinds of buildbots. A set of buildbots which are
experimental and do not send emails ever and a set of bots for which I
know they have a very low rate of failures (every couple of weeks) and
for which I also make sure that I myself can either fix or XFAIL any
issues very quickly (commonly at most 2-3 failing builds during working
hours). I do not expect anybody to fix failures in polly. However, I
made the experience that people are kind enough to help fixing problems,
if the bot generally has a reputation of being green most of the time.

In your case I suggest to make the buildbot experimental, disable emails
to committers and add yourself as email receiver. When you can make sure
the bot sends very rarely emails and you yourself can make sure bugs are
addressed / XFAILED within a very small delay and this has been proven
to work for a couple of weeks, you could carefully consider of enabling
emails to other again.

Best,
Tobias

On 3 February 2017 at 12:18, Tobias Grosser via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> In your case I suggest to make the buildbot experimental, disable emails
> to committers and add yourself as email receiver. When you can make sure
> the bot sends very rarely emails and you yourself can make sure bugs are
> addressed / XFAILED within a very small delay and this has been proven
> to work for a couple of weeks, you could carefully consider of enabling
> emails to other again.
+1. The silent staging buildbot is what you want I believe
http://lists.llvm.org/pipermail/lldb-commits/Week-of-Mon-20151012/024214.html

Best,

Alex

> On Feb 3, 2017, at 4:18 AM, Tobias Grosser via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On Fri, Feb 3, 2017, at 11:37 AM, Dylan McKay via llvm-dev wrote:
>> Hey all,
>> 
>> Every few weeks, a change is committed to trunk that breaks the AVR
>> buildbot.
>> 
>> A problem presents when commit authors do not fix the build, and just
>> leave
>> it because it passes on the core buildbots. The build stays red for a
few
>> days until I go and check it. In the meantime, it likely causes spam
for
>> most if not all developers when they commit new code.
>> 
>> All commits should keep master green, but what is the expectation for
>> experimental backends? Is it reasonable to expect all developers who
>> commit
>> code to ensure tests pass on the AVR backend?
>> 
>> On top of this, is there any way to notify maintainers of a backend
when
>> a
>> buildbot has been failing for some time? I imagine other experimental
>> backends have run into the same problems.
> 
> Hi Dylan,
> 
> this probably depends on what we want experimental targets to be and how
> they are different from normal targets. From my developer perspective,
> anything that is not enabled by default and is not checked by default in
> a normal LLVM build is something a normal developer should not need to
> worry about. As such, I do not
> expect committers to investigate bugs triggered in the AVR backend. If
> we would believe the AVR backend is stable enough, such that users can
> rely on it and such that other developers are unlikely to trigger bugs
> in the AVR backend, the AVR backend should most likely be promoted to a
> stable backend.
> 
> As a result of this, I would also expect buildbots of the AVR backend to
> not send any emails to the general public, but to instead send emails to
> the buildbot owner and maintainer of the AVR backend.
> 
> In my perspective, it is the job of the buildbot owner of an
> experimental backend to investigate failures in this backend and to
> clarify if a failure is caused by a bug in a recent change or rather by
> a bug in the AVR backend.
> If it seems as if the original change committed is incorrect, the
> buildbot owner should work with the committer of the corresponding patch
> to get the bug resolved.
> 
> Obviously, the answer I gave above is very black and white. In reality,
> the real answer depends on the maturity of the backend and the quality
> of the availability of the buildbot owner. What I wrote above has a
> rather new and unstable backend in mind -- especially the category of
> backend we would expect to become experimental in the first place. The
> more stable a backend is, the more rare breakages are, the faster bugs
> are fixed, and the more likely it is that breakages are due to bugs in
> LLVM commits, the more it makes sense to have a buildbot that sends out
> emails to the actual committers.
> 
> In Polly I have two kinds of buildbots. A set of buildbots which are
> experimental and do not send emails ever and a set of bots for which I
> know they have a very low rate of failures (every couple of weeks) and
> for which I also make sure that I myself can either fix or XFAIL any
> issues very quickly (commonly at most 2-3 failing builds during working
> hours). I do not expect anybody to fix failures in polly. However, I
> made the experience that people are kind enough to help fixing problems,
> if the bot generally has a reputation of being green most of the time.
> 
> In your case I suggest to make the buildbot experimental, disable emails
> to committers and add yourself as email receiver. When you can make sure
> the bot sends very rarely emails and you yourself can make sure bugs are
> addressed / XFAILED within a very small delay and this has been proven
> to work for a couple of weeks, you could carefully consider of enabling
> emails to other again.
If I was in this position, I’d also configure the bot to build *only* the AVR
backend. That’s help make sure that an email does get send when a test fails in
the X86 backend.

Best,

— 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170203/f08c46c6/attachment.html>