llvm dev - Apr 2016 - Wanted: a way to test changes before breaking all the build bots.

So, I broke a few build modes with a recent commit (r266002). I had been
developing with clang in debug mode, which worked and passed tests fine.
But upon committing, I discovered that I had broken:
1) Bots building with GCC. (fixed subsequently in r266011)
2) Bots doing an NDEBUG build with -Werror. (fixed subsequently in r266016)
3) Bots building llvm with msan. (fixed via revert r266062)

Obviously, that was not an ideal outcome.

I'd love to have some reasonable way that I could try out a commit *other
than* submitting it and hoping for the best, so that sort of thing can be
avoided.

Ideally, I could run a change through the actual buildbots and get back a
report. I'm not sure if that's possible or not. Or, if it would overload
the build machines, since many of them seem barely even able keep up with
the submit rate. (it would still be valuable even if the feature was
restricted to platforms easily obtainable at arbitrary scale on public
clouds, though.)

But, barring that, even just having a script that would automatically run
all of the builds on my own machine, equivalent to what the bots running
the same host platform would run, would be a great help. With that, I could
at least be fairly sure my commit wouldn't break any Linux/x86-64 bots,
which isn't really reasonably possible right now.

Is anyone working on anything to make this situation better?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160412/a936e24a/attachment.html>

On 12 April 2016 at 16:13, James Y Knight via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> I'd love to have some reasonable way that I could try out a commit
*other
> than* submitting it and hoping for the best, so that sort of thing can be
> avoided.
Hi James,

The proposal for pre-commit buildbots is a long lived one...

The accepted reality (not a consensus), is that it's more
cost-effective to spend time and money on real builders for new
configurations than duplicating builders for pre-commit. Given that we
have a lax policy on reverting patches anyway, people feel comfortable
in breaking a few things here and there.

Personally, I would love some pre-commit buildbot. I really dislike
the series of fix-up patches that come after breaking bots, making
them impossible to revert.

I'm assuming that the number of pre-commit patches are larger than
commit ones, given that people will be trying them out as they bake.
Therefore, such a pre-commit infrastructure would have to be bigger,
per configuration, than our current buildbot infrastructure. If we
have one builder per architecture, but multiple slaves, that could be
scalable, but they wouldn't catch all bugs (which is ok).

We'd also want all those pre-commit builders to be monitoring the same
branch across all projects. And that's where testing patches that
cross project boundaries become complicated. LLVM has around 12 major
repositories, that get built and tested regularly, and the algorithm
that would pick a branch and test in Buildbot is not trivial.

The alternative would be Jenkins, uploading a patch (that could
potentially span across multiple repositories), and then test. The
infrastructure would be *much* simpler, but we'd have to create a
parallel infrastructure, not necessarily synchronised with the main
one. Given that Apple is moving to Jenkins internally, I don't see
this as a bad thing, at least not for a completely separated
infrastructure.

But both solutions will have the volume issue. You may have to wait
longer to have your patch validated on a tree that might be, now,
outdated, than to commit, break & fix. This is a somewhat contentious
issue in this community.

My opinion is that we should have less haste, more quality. So,
pre-commit bots (BB or Jk or whatever) is a way forward, and I support
it. But I have been frequently outnumbered in the past... :)

cheers,
--renato

I don't think we need perfect/complete coverage in a pre-commit
scenario, limiting it to the high-value cases would be enough.

I haven't tried to collect data but my intuition is that the most
common breakages are Windows, followed by release builds.  The latter
is something people can do themselves (although it's tedious); the
former, not so much.
--paulr
> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Renato Golin via llvm-dev
> Sent: Tuesday, April 12, 2016 8:44 AM
> To: James Y Knight
> Cc: llvm-dev
> Subject: Re: [llvm-dev] Wanted: a way to test changes before breaking all
> the build bots.
> 
> On 12 April 2016 at 16:13, James Y Knight via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > I'd love to have some reasonable way that I could try out a commit
> *other
> > than* submitting it and hoping for the best, so that sort of thing can
> be
> > avoided.
> 
> Hi James,
> 
> The proposal for pre-commit buildbots is a long lived one...
> 
> The accepted reality (not a consensus), is that it's more
> cost-effective to spend time and money on real builders for new
> configurations than duplicating builders for pre-commit. Given that we
> have a lax policy on reverting patches anyway, people feel comfortable
> in breaking a few things here and there.
> 
> Personally, I would love some pre-commit buildbot. I really dislike
> the series of fix-up patches that come after breaking bots, making
> them impossible to revert.
> 
> I'm assuming that the number of pre-commit patches are larger than
> commit ones, given that people will be trying them out as they bake.
> Therefore, such a pre-commit infrastructure would have to be bigger,
> per configuration, than our current buildbot infrastructure. If we
> have one builder per architecture, but multiple slaves, that could be
> scalable, but they wouldn't catch all bugs (which is ok).
> 
> We'd also want all those pre-commit builders to be monitoring the same
> branch across all projects. And that's where testing patches that
> cross project boundaries become complicated. LLVM has around 12 major
> repositories, that get built and tested regularly, and the algorithm
> that would pick a branch and test in Buildbot is not trivial.
> 
> The alternative would be Jenkins, uploading a patch (that could
> potentially span across multiple repositories), and then test. The
> infrastructure would be *much* simpler, but we'd have to create a
> parallel infrastructure, not necessarily synchronised with the main
> one. Given that Apple is moving to Jenkins internally, I don't see
> this as a bad thing, at least not for a completely separated
> infrastructure.
> 
> But both solutions will have the volume issue. You may have to wait
> longer to have your patch validated on a tree that might be, now,
> outdated, than to commit, break & fix. This is a somewhat contentious
> issue in this community.
> 
> My opinion is that we should have less haste, more quality. So,
> pre-commit bots (BB or Jk or whatever) is a way forward, and I support
> it. But I have been frequently outnumbered in the past... :)
> 
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev