freebsd stable - Jun 2020 - Buildworld and buildkernel with very slow compilation, recently

On 21 Jun 2020, at 14:36, Chris Nehren <cnehren at pobox.com>
wrote:
> 
> On Sunday, June 21, 2020 8:11:15 AM EDT Michael Grimm wrote:
>> Hi,
>> 
>> I am following FreeBSD 12.1-STABLE.
>> 
>> Clang has been upgraded to version 10.0.0 on May, 1st, and ever since
that
>> time, I do observe a dramatic increase in compilation times of building
>> world, kernel and ports. I didn't benchmark the exact times, but
>> compilation times are at least increased by a factor of 1.5. Nothing
has
>> changed of the last month besides upgrading 12.1-Stable every other
week.
>> 
>> Has anyone else been bitten by this?
> 
> I don't have measurements to corroborate this, but here's a mailing
list
> thread where folks are talking about it (split across two URLs, the OP
posted
> in April and then there was a followup in May):
> 
> http://lists.llvm.org/pipermail/llvm-dev/2020-April/140938.html
> https://lists.llvm.org/pipermail/llvm-dev/2020-May/141482.html
> 
> So there definitely seems to be *something* going on, and you are indeed
not
> crazy. :)
Indeed, there is some upstream discussion going on about this issue.
There are some scenarios where people see non-negligible performance
loss, but apparently not everybody suffers from it. If you build the
whole ports collection, it is rather likely you may bump into it. I have
personally not seen much performance difference in building world,
kernel etc.

One of the upstream problems is that there is not really any
authoritative performance regression log being built up, so it is hard
to see where such regressions were introduced. Somebody then has to
spend a lot of time tracking down each and every regression, and then
attempt to untangle it from the dozens of commits made around the same
time. :)

In any case, there is at least some attention on it now, so hopefully
this will improve again. I don't think such fixes will be trivial
though, so it is not likely they will land in 10.0.1.

-Dimitry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 223 bytes
Desc: Message signed with OpenPGP
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20200621/48dccc0f/attachment.sig>

Dimitry Andric <dim at FreeBSD.org> wrote:
> 
> On 21 Jun 2020, at 14:36, Chris Nehren <cnehren at pobox.com> wrote:
>> On Sunday, June 21, 2020 8:11:15 AM EDT Michael Grimm wrote:
>>> Hi,
>>> 
>>> I am following FreeBSD 12.1-STABLE.
>>> 
>>> Clang has been upgraded to version 10.0.0 on May, 1st, and ever
since that
>>> time, I do observe a dramatic increase in compilation times of
building
>>> world, kernel and ports. I didn't benchmark the exact times,
but
>>> compilation times are at least increased by a factor of 1.5.
Nothing has
>>> changed of the last month besides upgrading 12.1-Stable every other
week.
>>> 
>>> Has anyone else been bitten by this?
>> 
>> I don't have measurements to corroborate this, but here's a
mailing list
>> thread where folks are talking about it (split across two URLs, the OP
posted
>> in April and then there was a followup in May):
>> 
>> http://lists.llvm.org/pipermail/llvm-dev/2020-April/140938.html
>> https://lists.llvm.org/pipermail/llvm-dev/2020-May/141482.html
>> 
>> So there definitely seems to be *something* going on, and you are
indeed not
>> crazy. :)
> 
> Indeed, there is some upstream discussion going on about this issue.
> There are some scenarios where people see non-negligible performance
> loss, but apparently not everybody suffers from it. If you build the
> whole ports collection, it is rather likely you may bump into it. I have
> personally not seen much performance difference in building world,
> kernel etc.
> 
> One of the upstream problems is that there is not really any
> authoritative performance regression log being built up, so it is hard
> to see where such regressions were introduced. Somebody then has to
> spend a lot of time tracking down each and every regression, and then
> attempt to untangle it from the dozens of commits made around the same
> time. :)
> 
> In any case, there is at least some attention on it now, so hopefully
> this will improve again. I don't think such fixes will be trivial
> though, so it is not likely they will land in 10.0.1.
> 
> -Dimitry
ok, so others see some performance loss as well.

In the meantime I activated META_MODE and hope, that will speed-up my weekly
compilations.

Thanks to all who answered, and regards,
Michael

22.06.2020 3:56, Dimitry Andric wrote:
> One of the upstream problems is that there is not really any
> authoritative performance regression log being built up, so it is hard
> to see where such regressions were introduced. Somebody then has to
> spend a lot of time tracking down each and every regression, and then
> attempt to untangle it from the dozens of commits made around the same
> time. :)
Such project as FreeBSD could become valuable source of information to track
such an issue
and it's easy to obtain, for example:

make buildworld 2>&1 | logger -t buildworld -p user.info

This generates timestamps for every line called in process with help of syslogd.
It should be easy to compare outputs obtained with same FreeBSD revision
but different clang versions, calculate time diffs and sort by diffs descending
:-)

Same for ports builds, maybe even quicker.