llvm dev - Aug 2014 - [LLVMdev] Plans for the Apple supported Darwin buildbot cluster

There as been a bit of talk about zorg recently, so I thought this would be a
good time to share our plans for the darwin buildbots that live here:

http://lab.llvm.org:8013/builders <http://lab.llvm.org:8013/builders>

To be clear, those are not these buildbots:

http://lab.llvm.org:8011/builders <http://lab.llvm.org:8011/builders>

We are not talking about changing the main buildbots.


We are working on a redesign of the Darwin build cluster that will include some
hardware upgrades.

The most notable change will be a switch from Buildbot to Jenkins.  We will be
replacing the current phased system with a Jenkins analog and reproducing the
current builds.  The Jenkins will execute scripts for each “project” that we
will be checked in to zorg.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140812/a583e8e0/attachment.html>

On 12 August 2014 19:34, Chris Matthews <chris.matthews at apple.com>
wrote:
> The most notable change will be a switch from Buildbot to Jenkins.  We will
> be replacing the current phased system with a Jenkins analog and
reproducing
> the current builds.  The Jenkins will execute scripts for each “project”
> that we will be checked in to zorg.
Hi Chris,

I'm very interested in this. Can you share some of the decisions on
moving to Jenkins?

I ask that because we have our GCC testing on Jenkins and LLVM testing
on Buildbots, and to be honest, buildbots are more stable and
reliable, not to mention having more readable web pages, error
messages and emailing committers, etc. Can Jenkins (at least in your
setup) do the same?

I really don't care much how I run the tests, as long as they keep me
informed and don't hassle me, which has been an almost free ride with
the buildbots so far.

If you guys add Jenking support I may also have a go here internally,
and if you guys could also share the Jenkins configuration, I could
update our own copy with a decent setup. :D

cheers,
--renato

On Tue, Aug 12, 2014 at 11:34:17AM -0700, Chris Matthews
wrote:
> There as been a bit of talk about zorg recently, so I thought this
> would be a good time to share our plans for the darwin buildbots that live
here:
> 
> http://lab.llvm.org:8013/builders <http://lab.llvm.org:8013/builders>
> 
> To be clear, those are not these buildbots:
> 
> http://lab.llvm.org:8011/builders <http://lab.llvm.org:8011/builders>
> 
> We are not talking about changing the main buildbots.
Would it be possible to drop using random high ports and just hook them
all up under http://lab.llvm.org directly?

Joerg

I think that would involve setting up a reverse proxy inside the cluster that
dispatches requests to the different systems.  We use a similar proxy internally
here.

I think Galina knows best if that would be reasonable?
> On Aug 12, 2014, at 1:53 PM, Joerg Sonnenberger <joerg at
britannica.bec.de> wrote:
> 
> On Tue, Aug 12, 2014 at 11:34:17AM -0700, Chris Matthews wrote:
>> There as been a bit of talk about zorg recently, so I thought this
>> would be a good time to share our plans for the darwin buildbots that
live here:
>> 
>> http://lab.llvm.org:8013/builders
<http://lab.llvm.org:8013/builders>
>> 
>> To be clear, those are not these buildbots:
>> 
>> http://lab.llvm.org:8011/builders
<http://lab.llvm.org:8011/builders>
>> 
>> We are not talking about changing the main buildbots.
> 
> Would it be possible to drop using random high ports and just hook them
> all up under http://lab.llvm.org directly?
> 
> Joerg
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

It was mostly pragmatic concerns that made us want to switch.

We are using both Jenkins and Buildbot internally.  We found more people had
Jenkins experience than Buildbot.  We wrote custom code for a lot of features
buildbot was missing, that seem to come standard in Jenkins.  Our extensions to
buildbot internally were getting out of control, and sucking up a lot of
maintenance time.

As a project, Jenkins seems to be moving faster. Jenkins also makes it very easy
to setup new jobs, no checking out zorg repos!

We have a selection of plugins which make some of these problems you mentioned
go away.  We are using the failure cause management plugin to extract errors
from logs, and the email-ext plugin make nicer emails.  One thing we have been
working towards is isolating infrastructure failures (svn, disk space etc) from
build and test failures.

We have not found one to be ‘more’ reliable than the other.
> On Aug 12, 2014, at 1:49 PM, Renato Golin <renato.golin at
linaro.org> wrote:
> 
> On 12 August 2014 19:34, Chris Matthews <chris.matthews at apple.com>
wrote:
>> The most notable change will be a switch from Buildbot to Jenkins.  We
will
>> be replacing the current phased system with a Jenkins analog and
reproducing
>> the current builds.  The Jenkins will execute scripts for each
“project”
>> that we will be checked in to zorg.
> 
> Hi Chris,
> 
> I'm very interested in this. Can you share some of the decisions on
> moving to Jenkins?
> 
> I ask that because we have our GCC testing on Jenkins and LLVM testing
> on Buildbots, and to be honest, buildbots are more stable and
> reliable, not to mention having more readable web pages, error
> messages and emailing committers, etc. Can Jenkins (at least in your
> setup) do the same?
> 
> I really don't care much how I run the tests, as long as they keep me
> informed and don't hassle me, which has been an almost free ride with
> the buildbots so far.
> 
> If you guys add Jenking support I may also have a go here internally,
> and if you guys could also share the Jenkins configuration, I could
> update our own copy with a decent setup. :D
> 
> cheers,
> --renato

Sounds good. I'm sick of buildbot. I'd be happy to see it replaced with
something better. We'll see how it goes and can evaluate what we want to do
with the lab.llvm.org:8011 bots. I also wish we weren't using random high
ports for this stuff...


On Tue, Aug 12, 2014 at 11:34 AM, Chris Matthews <chris.matthews at
apple.com>
wrote:
> There as been a bit of talk about zorg recently, so I thought this would
> be a good time to share our plans for the darwin buildbots that live here:
>
> http://lab.llvm.org:8013/builders
>
> To be clear, those are not these buildbots:
>
> http://lab.llvm.org:8011/builders
>
> We are not talking about changing the main buildbots.
>
>
> We are working on a redesign of the Darwin build cluster that will include
> some hardware upgrades.
>
> The most notable change will be a switch from Buildbot to Jenkins.  We
> will be replacing the current phased system with a Jenkins analog and
> reproducing the current builds.  The Jenkins will execute scripts for each
> “project” that we will be checked in to zorg.
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140812/4d04bf61/attachment.html>

On 08/13/14 07:42 AM, Reid Kleckner wrote:
> Sounds good. I'm sick of buildbot. I'd be happy to see it replaced 
> with something better. We'll see how it goes and can evaluate what we 
> want to do with the lab.llvm.org:8011 <http://lab.llvm.org:8011>
bots.
> I also wish we weren't using random high ports for this stuff...
Sorry to troll this thread, but jenkins sucks compared to bamboo and 
they probably would give you an open source license.

specifically - the web interface is more clean/intuitive, reporting for 
bamboo, configuration and overall clean/professional. etc

On 12 Aug 2014, at 19:34, Chris Matthews <chris.matthews at apple.com>
wrote:
> The most notable change will be a switch from Buildbot to Jenkins.
We're currently running LLVM test builds in Jenkins and I have patched lit
to produce JUnit XML output that Jenkins can consume (it's not pretty, but
it does at least let you produce the graphs of tests / failures for each build).
If there's general interest in this feature, I'd be happy to upstream
it.  The current implementation writes the JUnit XML to a file specified on the
command line, so you also get the normal output.  It could be improved, but
it's probably a sensible starting point for anyone wanting to work on this.

David

On 13 August 2014 11:09, David Chisnall <David.Chisnall at cl.cam.ac.uk>
wrote:
> We're currently running LLVM test builds in Jenkins and I have patched
lit to produce JUnit XML output that Jenkins can consume (it's not pretty,
but it does at least let you produce the graphs of tests / failures for each
build).  If there's general interest in this feature, I'd be happy to
upstream it.  The current implementation writes the JUnit XML to a file
specified on the command line, so you also get the normal output.  It could be
improved, but it's probably a sensible starting point for anyone wanting to
work on this.
I think that'd be interesting.

--renato

Yes! That would be really great! 
> On Aug 13, 2014, at 3:09 AM, David Chisnall <David.Chisnall at
cl.cam.ac.uk> wrote:
> 
>> On 12 Aug 2014, at 19:34, Chris Matthews <chris.matthews at
apple.com> wrote:
>> 
>> The most notable change will be a switch from Buildbot to Jenkins.
> 
> We're currently running LLVM test builds in Jenkins and I have patched
lit to produce JUnit XML output that Jenkins can consume (it's not pretty,
but it does at least let you produce the graphs of tests / failures for each
build).  If there's general interest in this feature, I'd be happy to
upstream it.  The current implementation writes the JUnit XML to a file
specified on the command line, so you also get the normal output.  It could be
improved, but it's probably a sensible starting point for anyone wanting to
work on this.
> 
> David
>

I'd be interested in seeing what you have planned upstreamed. I've used 
Jenkins on several projects.

Will Zorg be a vehicle for transition, where buildbot slaves become 
Jenkins nodes defined in the project files?

Do you see a majority of this this as a jenkins matrix project, or some 
other organization?

Rick

On 08/13/2014 05:09 AM, David Chisnall wrote:
> On 12 Aug 2014, at 19:34, Chris Matthews <chris.matthews at
apple.com> wrote:
>
>> The most notable change will be a switch from Buildbot to Jenkins.
> We're currently running LLVM test builds in Jenkins and I have patched
lit to produce JUnit XML output that Jenkins can consume (it's not pretty,
but it does at least let you produce the graphs of tests / failures for each
build).  If there's general interest in this feature, I'd be happy to
upstream it.  The current implementation writes the JUnit XML to a file
specified on the command line, so you also get the normal output.  It could be
improved, but it's probably a sensible starting point for anyone wanting to
work on this.
>
> David
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev