llvm dev - Oct 2020 - Help needed setting up a new buildbot for linux-sparcv9 and other targets

Hello John,
> I would like to set up some additional buildbots for LLVM and Clang that
are currently
> missing in the current list builders [1], namely linux-sparcv9,
linux-ppc32 and linux-mips
> (and later linux-m68k).
Thanks for working on this!
> I'm wondering what to use as the
"buildbot-worker-access-name" username.
Any distinct unused yet name which represents the build worker you are
about to introduce is fine. For example, "linux-sparcv9",
"linux-ppc32",
"linux-mips", and "linux-m68k" look good. You could specify
builddir to be
the same as the worker name. This would keep things simple.

I can assign passwords to your workers or you can choose whatever password
you want and send directly to me. Please let me know if you need my public
pgp key for secure communication.

We do not accept changes to zorg at this time because of the upgrade to a
newer version of buildbot. The repository is locked and I'm in the middle
of preparing a set of patches you will need to base your changes on. I hope
this will be done early next week.

Anyway, once the llvm-zorg has all the changes for the new buildbot, you
will need to propose a patch. Please note that the below explanation is
based on the new file names which are not available yet in the github.

To add your workers to LLVM buildbot, you will need to propose a patch for
buildbot/osuosl/master/config/workers.py which adds the workers, and for
buildbot/osuosl/master/config/builders.py to add your build configurations
which use your workers. If you are up to building self hosting clang and
llvm, then please use UnifiedTreeBuilder.

If you will elaborate what build configuration you are after for your bots,
I might give you a good example to follow or will help you with the patch.

I assume you already made sure you can build locally on each of those
workers what you are after.

Thanks

Galina




On Wed, Oct 21, 2020 at 11:25 AM David Blaikie <dblaikie at gmail.com>
wrote:
> Hey Galina - perhaps you could help John here?
>
> On Wed, Oct 21, 2020 at 3:22 AM John Paul Adrian Glaubitz via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hello!
>>
>> I would like to set up some additional buildbots for LLVM and Clang
that
>> are currently
>> missing in the current list builders [1], namely linux-sparcv9,
>> linux-ppc32 and linux-mips
>> (and later linux-m68k).
>>
>> I'm currently following the guide in [2] and I have installed the
build
>> dependencies for
>> LLVM as well as the buildbot package on a Debian sparc64 unstable
machine
>> (Sun T5220).
>>
>> I'm currently stuck at this step:
>>
>> > $ buildbot-worker create-worker
<buildbot-worker-root-directory> \
>> >              lab.llvm.org:9990 \
>> >              <buildbot-worker-access-name> \
>> >              <buildbot-worker-access-password>
>>
>> I'm wondering what to use as the
"buildbot-worker-access-name" username.
>>
>> And what builddir does the sentence
>>
>>         "Please make sure your builder name and its builddir are
unique
>> through the file."
>>
>> refer to? Isn't the build dir local to the builder machine only?
>>
>> Thanks,
>> Adrian
>>
>> > [1] http://lab.llvm.org:8011/#/builders
>> > [2] http://llvm.org/docs/HowToAddABuilder.html
>>
>> --
>>  .''`.  John Paul Adrian Glaubitz
>> : :' :  Debian Developer - glaubitz at debian.org
>> `. `'   Freie Universitaet Berlin - glaubitz at physik.fu-berlin.de
>>   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201021/a719f70e/attachment.html>

Hi Galina!

On 10/22/20 7:41 AM, Galina Kistanova wrote:
> Any distinct unused yet name which represents the build worker you are
> about to introduce is fine. For example, "linux-sparcv9",
"linux-ppc32",
> "linux-mips", and "linux-m68k" look good. You could
specify builddir to be
> the same as the worker name. This would keep things simple.
Is there any naming scheme that the other buildbot-workers are using or are the
names just random?

And why does the buildddir have to be unique? Isn't that local on the
buildbot-
worker?
> I can assign passwords to your workers or you can choose whatever password
> you want and send directly to me. Please let me know if you need my public
> pgp key for secure communication.
Yes, your GPG key would be appreciated although I guess I can find it on the
public servers assigned to your email?
> We do not accept changes to zorg at this time because of the upgrade to a
> newer version of buildbot. The repository is locked and I'm in the
middle
> of preparing a set of patches you will need to base your changes on. I hope
> this will be done early next week.
OK, noted.
> Anyway, once the llvm-zorg has all the changes for the new buildbot, you
> will need to propose a patch. Please note that the below explanation is
> based on the new file names which are not available yet in the github.
> 
> To add your workers to LLVM buildbot, you will need to propose a patch for
> buildbot/osuosl/master/config/workers.py which adds the workers, and for
> buildbot/osuosl/master/config/builders.py to add your build configurations
> which use your workers. If you are up to building self hosting clang and
> llvm, then please use UnifiedTreeBuilder.
OK.
> If you will elaborate what build configuration you are after for your bots,
> I might give you a good example to follow or will help you with the patch.
Selfhosting LLVM and clang would be my goal on 64-bit SPARC on Linux.

The machine has 64 vCPUs and 32 GiB of RAM. Running Debian unstable with
the latest versions of GCC, CMake and Binutils. Ninja is available, too.
> I assume you already made sure you can build locally on each of those
> workers what you are after.
Yep. That works fine. Although there is currently one regression on SPARC
which I already reported to the list.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz at debian.org
`. `'   Freie Universitaet Berlin - glaubitz at physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Hi John,
> Is there any naming scheme that the other buildbot-workers are using or
are the names just random?

There is no enforced naming scheme. This is more up to the bot owner. If
you want, you could use <machine>-<operatingsystem> or vice versa.
> And why does the buildddir have to be unique? Isn't that local on the
buildbot-worker?

Here is the quote from the buildbot documentation:
builddirSpecifies the name of a subdirectory of the master’s basedir in
which everything related to this builder will be stored. This holds build
status information. If not set, this parameter defaults to the builder
name, with some characters escaped. Each builder must have a unique build
directory.workerbuilddirSpecifies the name of a subdirectory (under the
worker’s configured base directory) in which everything related to this
builder will be placed on the worker. This is where checkouts, compiles,
and tests are run. If not set, defaults to builddir. If a worker is
connected to multiple builders that share the same workerbuilddir, make
sure the worker is set to run one build at a time or ensure this is fine to
run multiple builds from the same directory simultaneously.

Thanks

Galina

On Thu, Oct 22, 2020 at 12:44 AM John Paul Adrian Glaubitz <
glaubitz at physik.fu-berlin.de> wrote:
> Hi Galina!
>
> On 10/22/20 7:41 AM, Galina Kistanova wrote:
> > Any distinct unused yet name which represents the build worker you are
> > about to introduce is fine. For example, "linux-sparcv9",
"linux-ppc32",
> > "linux-mips", and "linux-m68k" look good. You
could specify builddir to
> be
> > the same as the worker name. This would keep things simple.
>
> Is there any naming scheme that the other buildbot-workers are using or
> are the
> names just random?
>
> And why does the buildddir have to be unique? Isn't that local on the
> buildbot-
> worker?
>
> > I can assign passwords to your workers or you can choose whatever
> password
> > you want and send directly to me. Please let me know if you need my
> public
> > pgp key for secure communication.
>
> Yes, your GPG key would be appreciated although I guess I can find it on
> the
> public servers assigned to your email?
>
> > We do not accept changes to zorg at this time because of the upgrade
to a
> > newer version of buildbot. The repository is locked and I'm in the
middle
> > of preparing a set of patches you will need to base your changes on. I
> hope
> > this will be done early next week.
>
> OK, noted.
>
> > Anyway, once the llvm-zorg has all the changes for the new buildbot,
you
> > will need to propose a patch. Please note that the below explanation
is
> > based on the new file names which are not available yet in the github.
> >
> > To add your workers to LLVM buildbot, you will need to propose a patch
> for
> > buildbot/osuosl/master/config/workers.py which adds the workers, and
for
> > buildbot/osuosl/master/config/builders.py to add your build
> configurations
> > which use your workers. If you are up to building self hosting clang
and
> > llvm, then please use UnifiedTreeBuilder.
>
> OK.
>
> > If you will elaborate what build configuration you are after for your
> bots,
> > I might give you a good example to follow or will help you with the
> patch.
>
> Selfhosting LLVM and clang would be my goal on 64-bit SPARC on Linux.
>
> The machine has 64 vCPUs and 32 GiB of RAM. Running Debian unstable with
> the latest versions of GCC, CMake and Binutils. Ninja is available, too.
>
> > I assume you already made sure you can build locally on each of those
> > workers what you are after.
> Yep. That works fine. Although there is currently one regression on SPARC
> which I already reported to the list.
>
> Adrian
>
> --
>  .''`.  John Paul Adrian Glaubitz
> : :' :  Debian Developer - glaubitz at debian.org
> `. `'   Freie Universitaet Berlin - glaubitz at physik.fu-berlin.de
>   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201022/752abc79/attachment.html>