llvm dev - Nov 2013 - [LLVMdev] Quad-Core ARMv7 Build Slave Seeks Noble Purpose

On 14 November 2013 02:13, Sean Silva <chisophugis at gmail.com> wrote:
> I love Arch, but it is probably a bit too unstable for a long-term
> buildbot. CC'ing Renato who might have some suggestions.
>
Mikael, Sean,

I think having an Arch buildbot is a great idea. At Linaro, we normally
test on Debian-derived distros, and having something else entirely is a
good stress test for the compiler, the build configuration, the buildbot
scripts and the test infrastructure.

Regarding what to test, I'd begin with Clang+LLVM check-all. There are
already configurations on Zorg for that kind of setup. You can copy from
the A9 bots and change to "A15". I recommend you to disable make clean
(Clean=false), so that the cycle time becomes minutes, not hours. You
should set this up locally, on your own build master, and let it run for a
few days, and if it's stable, you can go to stage 2.

Stage 2 is either to put it in production (ie. moving the configuration to
Zorg and connecting your bot to LLVM's master), or enhance the testing
capabilities of your bot. The former is *very* simple, but the latter
depends on what you want. Stage 3 would be to put it into production.

Normally, the rules of thumb for ARM bots:
 * I woulnd't have bots running check-all AND the test-suite/lldb, because
I want them to be orthogonal, ie. I don't want the test-suite bot stopping
short of testing because of a silly breakage in a new test.
 * I wouldn't test lldb if you don't care about it. (I don't, yet).
lldb is
a separate project and I had trouble setting it up to run on ARM before.
 * Always have more than one buildbot on any configuration. Build time can
be huge, and dev boards are notoriously faulty. I had huge problems with
Panda boards in the past, to the point where I removed them all from the
build rota. The odroid U2 seems more stable, but the XU has some
hardware/kernel problems (randomly re-mounting partitions read-only,
disabling CPUs and never re-enabling them again, cache flush between every
big.LITTLE switch, amongst others).
 * Create boot scripts to check for those problems, plus set the CPU
scheduler to "performance" on ALL CPUs. This eases most CPU problems.
 * Create a stable configuration and save the image as it will run in
production, to make it easier to re-create bots on the spot
 * Have extra spare boards to replace a broken bot, as most of the time,
the easiest path is to re-flash, but you need something running while you
do it
 * Running the build on SDcards is ok, but they are more prone to failures
than good quality USB sticks, and those are more prone to failures than
external hard-drives (those are also a lot faster). So, at least, I'd
recommend you to buy a SanDisk Ultra USB stick.
 * Make sure you have a decent power supply (dozens of dollars worth) that
can provide *at least* 4amps.

All that may seem daunting, but there is one critical issue of hosting a
buildbot: reliability of the test is equals to reliability of the platform.

In the beginning, there were only a handful of ARM boards, and they were
broken most of the time, and ARM was not considered a stable target. We
changed it by introducing lots of new bots, test-suite and fixing all the
bugs, but once my Pandas starting to fail randomly, the popular belief was
that "failures on ARM are due to the board stability, not my commits",
and
sure enough, bugs started to creep in. We don't want that.

So, while I welcome new buildbots for ARM, we must do it right, from the
beginning. We still don't have enough critical mass to be able to have a
faulty bot, unfortunatelly.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/1ef1a19a/attachment.html>

Renato, thanks for your elaborate walk-through of the issues with ARM
boards.  I'm trying to add some of this to the "How to Build on
ARM"
document and will submit a patch later on.

I already ran into the problem of cores disappearing, but on Arch Linux
(which uses a fairly recent kernel), the missing cores come back as soon as
the load falls to zero.

Unfortunately, my personal budget does not allow me more than a single
Odroid XU board for the time being.  So I'll have to do with only one
board.  I happen to have an eMMC card and it is fairly fast.

By the way, how do you set the CPU scheduler to "performance" (procfs
something?).  Just so that it can be added to the docs.

-- Mikael


2013/11/14 Renato Golin <renato.golin at linaro.org>
> On 14 November 2013 02:13, Sean Silva <chisophugis at gmail.com>
wrote:
>
>> I love Arch, but it is probably a bit too unstable for a long-term
>> buildbot. CC'ing Renato who might have some suggestions.
>>
>
> Mikael, Sean,
>
> I think having an Arch buildbot is a great idea. At Linaro, we normally
> test on Debian-derived distros, and having something else entirely is a
> good stress test for the compiler, the build configuration, the buildbot
> scripts and the test infrastructure.
>
> Regarding what to test, I'd begin with Clang+LLVM check-all. There are
> already configurations on Zorg for that kind of setup. You can copy from
> the A9 bots and change to "A15". I recommend you to disable make
clean
> (Clean=false), so that the cycle time becomes minutes, not hours. You
> should set this up locally, on your own build master, and let it run for a
> few days, and if it's stable, you can go to stage 2.
>
> Stage 2 is either to put it in production (ie. moving the configuration to
> Zorg and connecting your bot to LLVM's master), or enhance the testing
> capabilities of your bot. The former is *very* simple, but the latter
> depends on what you want. Stage 3 would be to put it into production.
>
> Normally, the rules of thumb for ARM bots:
>  * I woulnd't have bots running check-all AND the test-suite/lldb,
because
> I want them to be orthogonal, ie. I don't want the test-suite bot
stopping
> short of testing because of a silly breakage in a new test.
>  * I wouldn't test lldb if you don't care about it. (I don't,
yet). lldb
> is a separate project and I had trouble setting it up to run on ARM before.
>  * Always have more than one buildbot on any configuration. Build time can
> be huge, and dev boards are notoriously faulty. I had huge problems with
> Panda boards in the past, to the point where I removed them all from the
> build rota. The odroid U2 seems more stable, but the XU has some
> hardware/kernel problems (randomly re-mounting partitions read-only,
> disabling CPUs and never re-enabling them again, cache flush between every
> big.LITTLE switch, amongst others).
>  * Create boot scripts to check for those problems, plus set the CPU
> scheduler to "performance" on ALL CPUs. This eases most CPU
problems.
>  * Create a stable configuration and save the image as it will run in
> production, to make it easier to re-create bots on the spot
>  * Have extra spare boards to replace a broken bot, as most of the time,
> the easiest path is to re-flash, but you need something running while you
> do it
>  * Running the build on SDcards is ok, but they are more prone to failures
> than good quality USB sticks, and those are more prone to failures than
> external hard-drives (those are also a lot faster). So, at least, I'd
> recommend you to buy a SanDisk Ultra USB stick.
>  * Make sure you have a decent power supply (dozens of dollars worth) that
> can provide *at least* 4amps.
>
> All that may seem daunting, but there is one critical issue of hosting a
> buildbot: reliability of the test is equals to reliability of the platform.
>
> In the beginning, there were only a handful of ARM boards, and they were
> broken most of the time, and ARM was not considered a stable target. We
> changed it by introducing lots of new bots, test-suite and fixing all the
> bugs, but once my Pandas starting to fail randomly, the popular belief was
> that "failures on ARM are due to the board stability, not my
commits", and
> sure enough, bugs started to creep in. We don't want that.
>
> So, while I welcome new buildbots for ARM, we must do it right, from the
> beginning. We still don't have enough critical mass to be able to have
a
> faulty bot, unfortunatelly.
>
> cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/3708eed6/attachment.html>

On 14 November 2013 17:43, Mikael Lyngvig <mikael at lyngvig.org> wrote:
> Renato, thanks for your elaborate walk-through of the issues with ARM
> boards.  I'm trying to add some of this to the "How to Build on
ARM"
> document and will submit a patch later on.
>
Nice, thanks! That would be great!


Unfortunately, my personal budget does not allow me more than a
single
> Odroid XU board for the time being.  So I'll have to do with only one
> board.  I happen to have an eMMC card and it is fairly fast.
>
That's good. Still, I'd have it running locally on your master for a
week
or so, just to make sure it's stable enough.

To create a local master you need to read this doc:

https://buildbot.readthedocs.org/en/v0.8.6/manual/installation.html

And add Zorg to your master. The way I did was to create a master on
buildbot's user dir, (buildbot create-master ~/buildmaster/llvm), checkout
zorg, symlink the master/builders configs from zorg into the buildmaster
dir, symlink the zorg's Python module into /lib/python2.7/whatever
(can't
remember), and start the master via (buildbot master start).

Then, I'd create two buildslaves on the board, one in ~/localbot and one in
~/buildbot, with the former pointing to your local master and the latter
pointing to LLVM's official master, so that it's easy for you to swap. (
http://llvm.org/docs/HowToAddABuilder.html)

On the master, I stripped down all builders from the config and only left
my own boards, so that I don't clutter the waterfall/builders page.


 By the way, how do you set the CPU scheduler to "performance"
(procfs
> something?).  Just so that it can be added to the docs.
>
for ((cpu=0; cpu<`grep -c proc /proc/cpuinfo`; cpu++)); do
  sudo sudo cpufreq-set -c $cpu -g performance
done

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/a45d77c8/attachment.html>