llvm dev - May 2014 - [LLVMdev] Buildbot try builds to get pre-commit performance number

Hi,

as we got again more activity regarding performance testing and 
especially LNT, I wanted to get out an idea I had since a while, but 
where I neither had nor have the time or hardware to implement it.

The buildbot infrastructure provides so called 'try' builds, that allow 
a developer to submit patches they want to check for correctness or 
performance before actually committing them.

It would be nice to enable such a feature for our developers using our 
buildbot and LNT infrastructure. With the current -O3 builders, we 
should have enough hardware resources to draft a first version of this. 
So in case anybody is interested to try to set this up, that would be 
amazing and I am happy to discuss ideas here.

Also, with the LLVM Foundation being set up, my hope is that we get at 
some point a larger set of recent machines which will allow us to both 
increase the bandwidth for such a feature as well as the reliability of 
the performance results.

Cheers,
Tobias

On 2 May 2014 09:27, Tobias Grosser <tobias at grosser.es>
wrote:
> The buildbot infrastructure provides so called 'try' builds, that
allow a
> developer to submit patches they want to check for correctness or
> performance before actually committing them.
Hi Tobi,

I like this idea very much, but I worry about possible abuses of web
input. Is this feature a patch upload form?

If they are, it'd help to have a git project (say on github, so we can
have more control), where those buildbots should pull the code from
(given specific branch names), so they can only build from an already
merged patchset and allow for more complex merges, too.

cheers,
--renato

On 02/05/2014 11:13, Renato Golin wrote:
> On 2 May 2014 09:27, Tobias Grosser <tobias at grosser.es> wrote:
>> The buildbot infrastructure provides so called 'try' builds,
that allow a
>> developer to submit patches they want to check for correctness or
>> performance before actually committing them.
>
> Hi Tobi,
>
> I like this idea very much, but I worry about possible abuses of web
> input. Is this feature a patch upload form?
http://docs.buildbot.net/0.8.1/try.html

We could probably also make the buildbot pull from a git repository. One 
of the issues could be that we currently have different SVN repositories 
that get checked out.

Cheers,
Tobias

I really love this idea!

LNT has great functionality for A/B comparison runs, which can send you a nice
report email.  I have been discussing with a few people how we could make it
easier to try out performance patches on random hardware.

I don’t know if using buildbot’s try feature would be compatible with the
current buildbot pipeline?

On May 2, 2014, at 1:27 AM, Tobias Grosser <tobias at grosser.es> wrote:
> Hi,
> 
> as we got again more activity regarding performance testing and especially
LNT, I wanted to get out an idea I had since a while, but where I neither had
nor have the time or hardware to implement it.
> 
> The buildbot infrastructure provides so called 'try' builds, that
allow a developer to submit patches they want to check for correctness or
performance before actually committing them.
> 
> It would be nice to enable such a feature for our developers using our
buildbot and LNT infrastructure. With the current -O3 builders, we should have
enough hardware resources to draft a first version of this. So in case anybody
is interested to try to set this up, that would be amazing and I am happy to
discuss ideas here.
> 
> Also, with the LLVM Foundation being set up, my hope is that we get at some
point a larger set of recent machines which will allow us to both increase the
bandwidth for such a feature as well as the reliability of the performance
results.
> 
> Cheers,
> Tobias