llvm dev - Jul 2016 - [cfe-dev] AArch64 testsuite buildbots timeout

Hello,

Some of our AArch64 bots have started timing out while compiling
SingleSource/UnitTests/Vector/AArch64/aarch64_neon_intrinsics.

On clang-cmake-aarch64-quick the failure first appears between r275337
and r275351, and on clang-cmake-aarch64-full it appears after r275352,
so there's isn't a clear culprit for this. I suspect we have been
slowly approaching that threshold for a while.

Both Renato and I are currently travelling, so we can't investigate
this until Monday. I can't really think of any temporary solution
other than bumping the timeout threshold, but IIUC that would affect
all the bots.

Sorry about the inconvenience,
Diana

On 14 July 2016 at 06:50, Diana Picus via cfe-dev
<cfe-dev at lists.llvm.org> wrote:
> Both Renato and I are currently travelling, so we can't investigate
> this until Monday. I can't really think of any temporary solution
> other than bumping the timeout threshold, but IIUC that would affect
> all the bots.
We could split the file in half? Maybe a script based on int/float
would be roughly half and at least slightly principled.

Tim.

On 14 July 2016 at 14:57, Tim Northover via cfe-dev
<cfe-dev at lists.llvm.org> wrote:
> We could split the file in half? Maybe a script based on int/float
> would be roughly half and at least slightly principled.
That's another idea. Though, at the same time, Tramp3D-v4 is *also*
timing out, and only when building all tests on all cores, but not
when building it alone.

This could be a recent point change, but we're not tracking build time
individually. Though, if you see the total build time on:

http://llvm.org/perf/db_default/v4/nts/machine/46

Until build 275348, it was ~4h50, then jumped to ~5h10, which points
at some commit in that range that made compilation time much worse.

On the x86_64 board, it also seem to have jumped:

http://llvm.org/perf/db_default/v4/nts/machine/47

275310 was ~5h20 and 275356 was 5h55.

This could be affecting many tests, but tramp3d and unit tests more,
or just those, but it seems to be target independent.

The ARM builder hasn't got there yet, due to another bug adding a
build failure, but I expect to see the same problems...

cheers,
--renato

On 07/14/16 08:50, Diana Picus via llvm-dev wrote:
> Some of our AArch64 bots have started timing out while compiling
> SingleSource/UnitTests/Vector/AArch64/aarch64_neon_intrinsics.
>
> On clang-cmake-aarch64-quick the failure first appears between r275337
> and r275351, and on clang-cmake-aarch64-full it appears after r275352,
> so there's isn't a clear culprit for this. I suspect we have been
> slowly approaching that threshold for a while.
>
> Both Renato and I are currently travelling, so we can't investigate
> this until Monday. I can't really think of any temporary solution
> other than bumping the timeout threshold, but IIUC that would affect
> all the bots.
I have a couple of powerpc bots that on occasion time out as well.  It 
doesn't happen very often and I have been trying to tweak things to 
avoid it but it is still an ongoing issue.

At least a few of the factories have the ability to override the timeout 
setting on a per-slave basis.  See the SanitizerBuildFactory and 
LLVMCMakeBuildFactory factories for instance.  Perhaps it is time to add 
this to all (or at least more) of the factories.

-- 

-Bill Seurer

On 15 Jul 2016 8:36 p.m., "Bill Seurer via llvm-dev" <
llvm-dev at lists.llvm.org> wrote:
>
> On 07/14/16 08:50, Diana Picus via llvm-dev wrote:
>>
>> Some of our AArch64 bots have started timing out while compiling
>> SingleSource/UnitTests/Vector/AArch64/aarch64_neon_intrinsics.
>>
>> On clang-cmake-aarch64-quick the failure first appears between r275337
>> and r275351, and on clang-cmake-aarch64-full it appears after r275352,
>> so there's isn't a clear culprit for this. I suspect we have
been
>> slowly approaching that threshold for a while.
>>
>> Both Renato and I are currently travelling, so we can't investigate
>> this until Monday. I can't really think of any temporary solution
>> other than bumping the timeout threshold, but IIUC that would affect
>> all the bots.
>
>
> I have a couple of powerpc bots that on occasion time out as well.  It
doesn't happen very often and I have been trying to tweak things to avoid
it but it is still an ongoing issue.
>
> At least a few of the factories have the ability to override the timeout
setting on a per-slave basis.  See the SanitizerBuildFactory and
LLVMCMakeBuildFactory factories for instance.  Perhaps it is time to add
this to all (or at least more) of the factories.

It'd also be interesting to know why we're slower on that range. I
wouldn't
like to make it easier for people to tune the timeout, or we'll end up
regressing on compile time too much without noticing.

Cheers,
Renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160715/59db9575/attachment.html>