llvm dev - Aug 2014 - [LLVMdev] Documentation for adding builders and slaves to zorg?

Hi all,

I'm working on adding a new buildbot. I'm don't understand what is
required
when patching zorg's builders.py and slaves.py.

Is there any documentation on this?
Would anybody be willing to help?

Best,

Eric Fiselier
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140731/93c86c1a/attachment.html>

On 01/08/2014 03:04, Eric Fiselier wrote:
> Hi all,
>
> I'm working on adding a new buildbot. I'm don't understand what
is required
> when patching zorg's builders.py and slaves.py.
>
> Is there any documentation on this?
> Would anybody be willing to help?
Something like the following?

http://llvm.org/docs/HowToAddABuilder.html


Tobias

On 01/08/2014 08:40, Eric Fiselier wrote:
> Not quite, That is the documentation I was following.
> I'm hoping to find more information on step #9.
I don't think there is other documentation. I would just look at the 
last commits that added builders. They should show exactly the changes 
needed.

Cheers,
Tobias

P.S: Re-adding llvmdev@ mailing list. This keeps our answers and 
discussions available to everyone.

Hi Eric,

There is documentation here [1].

I actually started (but have not finished) looking at doing something
with the buildbot recently. I was going to actually email llvmdev
after I finished because so far the experience has been a little
painful in terms of lack of documentation and zorg's organisation but
seeing as you've brought this issue up now might be a good time for me
to start the discussion.

What I wanted to do was

- create a new Factory which defines how to build the LLVM Sphinx documentation
- create a builder that uses this factory
- Run the a buildbot master and my own slave locally on my own machine
to test this works
- Once I created a configuration that works correctly, create patches
to zorg and submit.

I basically followed a guide I found at [2] to help me with this and
so far what I've managed to do is use some of the exisiting zorg code
to run a completely local buildbot master and slave which has a single
builder to build LLVM. The pain points I found were

- I had to install a really old version of buildbot and then had to
manually patch it so it would actually work (not such a big deal)
- I had to setup quite a few symlinks to setup a working master
configuration that would actually start locally on my machine. For
this use case the way zorg code is organised seems a bit broken.
- I had to rip out most of builders.py and slaves.py (I don't want my
machine contacting the public slaves while I'm testing, also it can't
anyway without passwords) and modify master.cfg.

My builders.py current looks like this.

```
from zorg.buildbot.builders import LLVMBuilder
reload(LLVMBuilder)
from zorg.buildbot.builders import LLVMBuilder

# HACK
def _get_llvm_builders():
    return [
            { 'name': 'llvm_buildtest',
              'slavenames': ['dan-sputnik'],
              'builddir': 'llvm_x86_buildtest',
              'factory':
LLVMBuilder.getLLVMBuildFactory("x86_64-linux-gnu", jobs=1)
            }

           ]

# HACK
def get_builders():
    for b in _get_llvm_builders():
        b['category'] = 'llvm'
        yield b
```

and slaves.py currently looks like this

```
import buildbot
import buildbot.buildslave
import os

import config

def create_slave(name, *args, **kwargs):
    password = config.options.get('Slave Passwords', name)
    return buildbot.buildslave.BuildSlave(name, password=password,
*args, **kwargs)

# HACK
def get_build_slaves():
    return [
        # My slave
        buildbot.buildslave.BuildSlave("dan-sputnik",
                                       "password",
                                       properties={'jobs':1},
                                       max_builds=1)
           ]
```

My buildbot master was basically created by doing this...

# Get buildbot
$ virtualenv2 --no-site-packages venv2
$ . venv2/bin/activate
$ pip install sqlalchemy==0.7.10
$ pip install buildbot==0.8.5

# patch broken buildbot code just downloaded into virtual environment (see [2])
....

# Create initial master code and get zorg code
$ buildbot create-master -r master
$ git clone http://llvm.org/git/zorg.git

# Now for some hacky symlinks
$ cd master
$ ln -s zorg ../zorg/zorg
$ ln -s config ../zorg/buildbot/osuosl/master/config/
$ ln -s master.cfg ../zorg/buildbot/osuosl/master/master.cfg

# Hack master.cfg and builders.py so it only does things locally and
present a web interface (see [2])
...

# Hack config/slaves.py so it only contains my local slave.
...

# Finally start the buildbot master
$ cd ../
$ buildbot start master

I could then easily create and start my local slave by doing

$ pip install buildbot-slave==0.8.5
$ buildslave create-slave slave localhost:9990 dan-sputnik password
$ buildslave start slave/


The last two points are the really annoying things because I really
want to test any modifications I make the zorg code before I send a
patch. However I'm not going to fully succeed with this because when
I've finished the changes I've made to zorg code will contain the
useful changes (i.e. a new build configuration and a new builder) and
not useful changes (the hacks I had to make to run a buildbot master
locally on my machine). So I'll then have to clean up the code so it
only contains the relevant stuff (and not my hacks) but then I won't
be able to test any of this locally so then I will have to hope that I
didn't break anything and just submit that patch.

I'd quite happily write some  documentation for this because LLVM's
buildbot stuff doesn't seem well documented. It would be good to know
if what I've done above is the right way to test adding a new Factory
(it doesn't feel like it... so many hacks :( ).

I hope this is some what useful to you Eric, even if we're not trying
to do exactly the same thing with the build bot.

[1] http://llvm.org/docs/HowToAddABuilder.html
[2]
http://llvm.lyngvig.org/Articles/How-to-Setup-an-Arch-Linux-Buildbot-for-LLVM#9

Thanks,
Dan.