llvm dev - Feb 2017 - Build status expectations for experimental targets

> On Feb 3, 2017, at 4:18 AM, Tobias Grosser via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On Fri, Feb 3, 2017, at 11:37 AM, Dylan McKay via llvm-dev wrote:
>> Hey all,
>> 
>> Every few weeks, a change is committed to trunk that breaks the AVR
>> buildbot.
>> 
>> A problem presents when commit authors do not fix the build, and just
>> leave
>> it because it passes on the core buildbots. The build stays red for a
few
>> days until I go and check it. In the meantime, it likely causes spam
for
>> most if not all developers when they commit new code.
>> 
>> All commits should keep master green, but what is the expectation for
>> experimental backends? Is it reasonable to expect all developers who
>> commit
>> code to ensure tests pass on the AVR backend?
>> 
>> On top of this, is there any way to notify maintainers of a backend
when
>> a
>> buildbot has been failing for some time? I imagine other experimental
>> backends have run into the same problems.
> 
> Hi Dylan,
> 
> this probably depends on what we want experimental targets to be and how
> they are different from normal targets. From my developer perspective,
> anything that is not enabled by default and is not checked by default in
> a normal LLVM build is something a normal developer should not need to
> worry about. As such, I do not
> expect committers to investigate bugs triggered in the AVR backend. If
> we would believe the AVR backend is stable enough, such that users can
> rely on it and such that other developers are unlikely to trigger bugs
> in the AVR backend, the AVR backend should most likely be promoted to a
> stable backend.
> 
> As a result of this, I would also expect buildbots of the AVR backend to
> not send any emails to the general public, but to instead send emails to
> the buildbot owner and maintainer of the AVR backend.
> 
> In my perspective, it is the job of the buildbot owner of an
> experimental backend to investigate failures in this backend and to
> clarify if a failure is caused by a bug in a recent change or rather by
> a bug in the AVR backend.
> If it seems as if the original change committed is incorrect, the
> buildbot owner should work with the committer of the corresponding patch
> to get the bug resolved.
> 
> Obviously, the answer I gave above is very black and white. In reality,
> the real answer depends on the maturity of the backend and the quality
> of the availability of the buildbot owner. What I wrote above has a
> rather new and unstable backend in mind -- especially the category of
> backend we would expect to become experimental in the first place. The
> more stable a backend is, the more rare breakages are, the faster bugs
> are fixed, and the more likely it is that breakages are due to bugs in
> LLVM commits, the more it makes sense to have a buildbot that sends out
> emails to the actual committers.
> 
> In Polly I have two kinds of buildbots. A set of buildbots which are
> experimental and do not send emails ever and a set of bots for which I
> know they have a very low rate of failures (every couple of weeks) and
> for which I also make sure that I myself can either fix or XFAIL any
> issues very quickly (commonly at most 2-3 failing builds during working
> hours). I do not expect anybody to fix failures in polly. However, I
> made the experience that people are kind enough to help fixing problems,
> if the bot generally has a reputation of being green most of the time.
> 
> In your case I suggest to make the buildbot experimental, disable emails
> to committers and add yourself as email receiver. When you can make sure
> the bot sends very rarely emails and you yourself can make sure bugs are
> addressed / XFAILED within a very small delay and this has been proven
> to work for a couple of weeks, you could carefully consider of enabling
> emails to other again.
If I was in this position, I’d also configure the bot to build *only* the AVR
backend. That’s help make sure that an email does get send when a test fails in
the X86 backend.

Best,

— 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170203/f08c46c6/attachment.html>

The builder isn’t marked as experimental so I think the expectation is that
people keep it green and contact the bot owner if they need help figuring
out why their change makes it red. That said, it sounds a bit odd to have a
non-experimental builder for an experimental backend.

I see. I had followed the generic How to add a builder
<http://llvm.org/docs/HowToAddABuilder.html> docs, which doesn’t mention
the concept of an experimental buildbot. I’ll send a patch to mention it.

If you want to do the same, then you’ll need to add an
InformativeMailNotifier to
http://llvm.org/svn/llvm-project/zorg/trunk/buildbot/osuosl/master/config/status.py
.

Nice! Exactly what I was looking for.

If we would believe the AVR backend is stable enough, such that users can
rely on it and such that other developers are unlikely to trigger bugs in
the AVR backend, the AVR backend should most likely be promoted to a stable
backend.

In general, I’ve found that almost all of the time that the AVR build
breaks, it’s been something pretty small which also caused a bunch of other
targets to fail also, which I suppose is a good sign. On the topic, I plan
on following up on promoting the backend to stable once the current effort
of enabling AVR in Rust is complete and we’ve ironed out any bugs found in
usage.

As a result of this, I would also expect buildbots of the AVR backend to
not send any emails to the general public, but to instead send emails to
the buildbot owner and maintainer of the AVR backend.

Agree with this

+1. The silent staging buildbot is what you want I believe
http://lists.llvm.org/pipermail/lldb-commits/Week-of-Mon-20151012/024214.html

That sounds good. My plan is to make the buildbot a staging bot, and then
be the sole receiver of emails from it.

If I was in this position, I’d also configure the bot to build *only* the
AVR backend. That’s help make sure that an email does get send when a test
fails in the X86 backend.

I would love to do this, but there’s a bug in the backend which causes a
few of the Generic CodeGen tests to fail. To work around this, I leave X86
as the default target for now. I’m definitely planning on updating this
once I’ve fixed the bug.

Thanks for all of the feedback!
​

On Sat, Feb 4, 2017 at 4:18 AM, Mehdi Amini via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
> On Feb 3, 2017, at 4:18 AM, Tobias Grosser via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> On Fri, Feb 3, 2017, at 11:37 AM, Dylan McKay via llvm-dev wrote:
>
> Hey all,
>
> Every few weeks, a change is committed to trunk that breaks the AVR
> buildbot.
>
> A problem presents when commit authors do not fix the build, and just
> leave
> it because it passes on the core buildbots. The build stays red for a few
> days until I go and check it. In the meantime, it likely causes spam for
> most if not all developers when they commit new code.
>
> All commits should keep master green, but what is the expectation for
> experimental backends? Is it reasonable to expect all developers who
> commit
> code to ensure tests pass on the AVR backend?
>
> On top of this, is there any way to notify maintainers of a backend when
> a
> buildbot has been failing for some time? I imagine other experimental
> backends have run into the same problems.
>
>
> Hi Dylan,
>
> this probably depends on what we want experimental targets to be and how
> they are different from normal targets. From my developer perspective,
> anything that is not enabled by default and is not checked by default in
> a normal LLVM build is something a normal developer should not need to
> worry about. As such, I do not
> expect committers to investigate bugs triggered in the AVR backend. If
> we would believe the AVR backend is stable enough, such that users can
> rely on it and such that other developers are unlikely to trigger bugs
> in the AVR backend, the AVR backend should most likely be promoted to a
> stable backend.
>
> As a result of this, I would also expect buildbots of the AVR backend to
> not send any emails to the general public, but to instead send emails to
> the buildbot owner and maintainer of the AVR backend.
>
> In my perspective, it is the job of the buildbot owner of an
> experimental backend to investigate failures in this backend and to
> clarify if a failure is caused by a bug in a recent change or rather by
> a bug in the AVR backend.
> If it seems as if the original change committed is incorrect, the
> buildbot owner should work with the committer of the corresponding patch
> to get the bug resolved.
>
> Obviously, the answer I gave above is very black and white. In reality,
> the real answer depends on the maturity of the backend and the quality
> of the availability of the buildbot owner. What I wrote above has a
> rather new and unstable backend in mind -- especially the category of
> backend we would expect to become experimental in the first place. The
> more stable a backend is, the more rare breakages are, the faster bugs
> are fixed, and the more likely it is that breakages are due to bugs in
> LLVM commits, the more it makes sense to have a buildbot that sends out
> emails to the actual committers.
>
> In Polly I have two kinds of buildbots. A set of buildbots which are
> experimental and do not send emails ever and a set of bots for which I
> know they have a very low rate of failures (every couple of weeks) and
> for which I also make sure that I myself can either fix or XFAIL any
> issues very quickly (commonly at most 2-3 failing builds during working
> hours). I do not expect anybody to fix failures in polly. However, I
> made the experience that people are kind enough to help fixing problems,
> if the bot generally has a reputation of being green most of the time.
>
> In your case I suggest to make the buildbot experimental, disable emails
> to committers and add yourself as email receiver. When you can make sure
> the bot sends very rarely emails and you yourself can make sure bugs are
> addressed / XFAILED within a very small delay and this has been proven
> to work for a couple of weeks, you could carefully consider of enabling
> emails to other again.
>
>
> If I was in this position, I’d also configure the bot to build *only* the
> AVR backend. That’s help make sure that an email does get send when a test
> fails in the X86 backend.
>
> Best,
>
> —
> Mehdi
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170204/aa3b8fe1/attachment.html>

> On Feb 3, 2017, at 12:45 PM, Dylan McKay via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> The builder isn’t marked as experimental so I think the expectation is that
people keep it green and contact the bot owner if they need help figuring out
why their change makes it red. That said, it sounds a bit odd to have a
non-experimental builder for an experimental backend.
> 
> I see. I had followed the generic How to add a builder
<http://llvm.org/docs/HowToAddABuilder.html> docs, which doesn’t mention
the concept of an experimental buildbot. I’ll send a patch to mention it.
> 
> If you want to do the same, then you’ll need to add an
InformativeMailNotifier to
http://llvm.org/svn/llvm-project/zorg/trunk/buildbot/osuosl/master/config/status.py
<http://llvm.org/svn/llvm-project/zorg/trunk/buildbot/osuosl/master/config/status.py>.
> 
> Nice! Exactly what I was looking for.
> 
> If we would believe the AVR backend is stable enough, such that users can
rely on it and such that other developers are unlikely to trigger bugs in the
AVR backend, the AVR backend should most likely be promoted to a stable backend.
> 
> In general, I’ve found that almost all of the time that the AVR build
breaks, it’s been something pretty small which also caused a bunch of other
targets to fail also, which I suppose is a good sign. On the topic, I plan on
following up on promoting the backend to stable once the current effort of
enabling AVR in Rust is complete and we’ve ironed out any bugs found in usage.
> 
> As a result of this, I would also expect buildbots of the AVR backend to
not send any emails to the general public, but to instead send emails to the
buildbot owner and maintainer of the AVR backend.
> 
> Agree with this
> 
> +1. The silent staging buildbot is what you want I believe
>
http://lists.llvm.org/pipermail/lldb-commits/Week-of-Mon-20151012/024214.html
<http://lists.llvm.org/pipermail/lldb-commits/Week-of-Mon-20151012/024214.html>
> That sounds good. My plan is to make the buildbot a staging bot, and then
be the sole receiver of emails from it.
> 
> If I was in this position, I’d also configure the bot to build only the AVR
backend. That’s help make sure that an email does get send when a test fails in
the X86 backend.
> 
> I would love to do this, but there’s a bug in the backend which causes a
few of the Generic CodeGen tests to fail. To work around this, I leave X86 as
the default target for now. I’m definitely planning on updating this once I’ve
fixed the bug.
> 
> 
This usually happens when LLVM_DEFAULT_TARGET_TRIPLE is not explicitely set and
you end up with your host machine as default while not building the x86 target.
If you set LLVM_DEFAULT_TARGET_TRIPLE to some AVR ones the failure should go
away (otherwise complain and file bugs).

- Matthias

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170203/3332a7db/attachment.html>