freebsd stable - Jul 2019 - Buildworld times (was Re: svn commit: r350256 - in stable/12: . contrib/compiler-rt/lib/sanitizer_common contrib/libunwind/src contrib/llvm/lib/DebugInfo/DWARF contrib/llvm/lib/MC contrib/llvm/lib/Object contrib/llvm/lib/Ta...)

(Please retain CCs, I am not subscribed to the list)

Below is hard evidence of 3 things on stable/11 (not 12) after r350259:

1. r350259 adds *substantial* time to buildworld.

2. WITHOUT_CLANG_EXTRAS+WITHOUT_CLANG_FULL+WITHOUT_LLDB can help improve
the situation after r350259, but it is still no where near as fast as
pre-r350259.

3. Kernel build times are fine; issue is with world.

TL;DR for lazy folks:

stable/11 r350330 world + minimal clang = 1:29:34
stable/11 r350330 world + full clang    = 1:46:31
stable/11 r350252 world + minimal clang =   56:52
stable/11 r350252 world + full clang    = 1:14:30

I cannot even begin to tell you how big of an impact this has on my
low-end dual-core VPS box (world takes hours upon hours).

We've been down this road before, many many times, since the
introduction of clang/LLVM.  Here's just a few that went no where.  I
couldn't find the more-useful one that had some concrete numbers in it,
dating back to pre-2016 (maybe sometime in 2014 or 2015?):

https://lists.freebsd.org/pipermail/freebsd-current/2017-January/thread.html#64431
https://lists.freebsd.org/pipermail/freebsd-stable/2017-January/thread.html#86646
https://lists.freebsd.org/pipermail/freebsd-questions/2016-November/thread.html#274684

Does anyone have a good/recent write-up on how to switch to gcc?  :-)


System
=====* Intel Core 2 Quad Q9550 @ 2.83GHz
* 8GB ECC RAM
* Samsung SSD 840 EVO 250GB filesystem (UFS2 + SU (not SUJ) + TRIM) + 32GB swap
* Running stable/11 r349226
* Misc notes
  - r350330 happened to be what was "master" at the time of my test
  - r350252 was the commit on stable/11 immediately before r350259
  - Switching to r350252 accomplished via: cd /usr/src && svnlite up
-r350252
  - System uses kern.maxvnodes=856944, last tuned 2018/06/07


Test #1, building r350330 minimal clang
======================================# cat /etc/src.conf
WITHOUT_ATM=true
WITHOUT_BLUETOOTH=true
WITHOUT_DEBUG_FILES=true
WITHOUT_FLOPPY=true
WITHOUT_FREEBSD_UPDATE=true
WITHOUT_IPFILTER=true
WITHOUT_IPX=true
WITHOUT_LIB32=true
WITHOUT_NDIS=true
WITHOUT_NETGRAPH=true
WITHOUT_PPP=true
WITHOUT_SENDMAIL=true
WITHOUT_TESTS=true
WITHOUT_WIRELESS=true
WITH_OPENSSH_NONE_CIPHER=true
WITHOUT_CLANG_EXTRAS=true
WITHOUT_CLANG_FULL=true
WITHOUT_LLDB=true
WITHOUT_LLVM_TARGET_AARCH64=true
WITHOUT_LLVM_TARGET_ARM=true
WITHOUT_LLVM_TARGET_MIPS=true
WITHOUT_LLVM_TARGET_POWERPC=true
WITHOUT_LLVM_TARGET_SPARC=true
WITHOUT_REPRODUCIBLE_BUILD=true
# cat /etc/make.conf
KERNCONF=X7SBA_RELENG_11_amd64
CPUTYPE?=core2
SVN_UPDATE=yes
STRIPCFLAGS+=	-fno-omit-frame-pointer

Result:
# rm -fr /usr/obj/*
# cd /usr/src
# time make -j4 buildworld
19906.874u 1280.928s 1:29:33.51 394.3%  57966+778k 23504+14200io 13867pf+0w
# time make -j4 buildkernel
1592.460u 196.047s 7:36.61 391.6%       48704+614k 6627+18158io 7361pf+0w


Test #2, building r350330 full clang
==================================="full clang" means same as Test #1
but with these 3 src.conf lines
commented out, i.e. CLANG_EXTRAS, CLANG_FULL, and LLDB are ENABLED:

WITHOUT_CLANG_EXTRAS=true
WITHOUT_CLANG_FULL=true
WITHOUT_LLDB=true

Result:
# rm -fr /usr/obj/*
# cd /usr/src
# time make -j4 buildworld
23779.674u 1463.156s 1:46:30.75 394.9%  57621+783k 20093+15423io 7283pf+0w
# time make -j4 buildkernel
1594.079u 194.345s 7:36.48 391.7%       48707+614k 5301+18013io 5342pf+0w


Test #3, building r350252 minimal clang
======================================Same configs as Test #1

Result:
# rm -fr /usr/obj/*
# cd /usr/src
# time make -j4 buildworld
12582.693u 882.543s 56:52.35 394.6%     62698+760k 21432+9694io 6923pf+0w
# time make -j4 buildkernel
1649.559u 184.934s 7:48.01 391.9%       57053+622k 7566+18291io 5402pf+0w


Test #4, building r350252 full clang
===================================Same configs as Test #2

# rm -fr /usr/obj/*
# cd /usr/src
# time make -j4 buildworld
16600.975u 1068.754s 1:14:29.53 395.3%  63271+774k 8683+10876io 4707pf+0w
# time make -j4 buildkernel
1650.654u 183.966s 7:47.47 392.4%       57117+623k 2829+17951io 1926pf+0w

-- 
| Jeremy Chadwick                                 jdc at koitsu.org |
| UNIX Systems Administrator                      PGP 0x2A389531 |
| Making life hard for others since 1977.                        |

On Fri, 26 Jul 2019 at 22:44, Jeremy Chadwick via freebsd-stable
<freebsd-stable at freebsd.org> wrote:
>
> TL;DR for lazy folks:
>
> stable/11 r350330 world + minimal clang = 1:29:34
> stable/11 r350330 world + full clang    = 1:46:31
> stable/11 r350252 world + minimal clang =   56:52
> stable/11 r350252 world + full clang    = 1:14:30
Yes, because you're currently running r349226, and the world build
thus first builds its own toolchain. Try the same experiment running
on a r349226 or later installed world.

On 7/26/2019 10:38 PM, Jeremy Chadwick via freebsd-stable
wrote:
> (Please retain CCs, I am not subscribed to the list)
>
> Below is hard evidence of 3 things on stable/11 (not 12) after r350259:
>
> 1. r350259 adds *substantial* time to buildworld.
Are you sure this is not the same as the issue in RELENG12 ? ie. the new
version of clang is built as part of world since it differs from whats
installed.? I had a RELENG11 box sitting around from July 4th

(nanobsd2)# time make -j8 buildworld > /var/log/build.out
13421.587u 906.968s 1:01:36.13 387.6%?? 75193+900k 285907+3240194io
108879pf+0w
0(nanobsd2)#

Updated to revision 350414.
0(nanobsd2)# time make -j8 buildworld > /var/log/build.out
18656.513u 1117.246s 1:23:53.24 392.8%? 70122+916k 255703+3772612io
104343pf+0w
0(nanobsd2)#

Did installworld/kernel.

buildworld after that

0(nanobsd2)# time make -j8 buildworld > /var/log/build.out
12998.964u 649.697s 58:48.61 386.7%???? 75748+907k 358314+3247016io
108534pf+0w
0(nanobsd2)#

Which is probably about the same as before the update build time wise

> 2. WITHOUT_CLANG_EXTRAS+WITHOUT_CLANG_FULL+WITHOUT_LLDB can help improve
> the situation after r350259, but it is still no where near as fast as
> pre-r350259.
>
> 3. Kernel build times are fine; issue is with world.
>
> TL;DR for lazy folks:
>
> stable/11 r350330 world + minimal clang = 1:29:34
> stable/11 r350330 world + full clang    = 1:46:31
> stable/11 r350252 world + minimal clang =   56:52
> stable/11 r350252 world + full clang    = 1:14:30
>
> I cannot even begin to tell you how big of an impact this has on my
> low-end dual-core VPS box (world takes hours upon hours).
>
> We've been down this road before, many many times, since the
> introduction of clang/LLVM.  Here's just a few that went no where.  I
> couldn't find the more-useful one that had some concrete numbers in it,
> dating back to pre-2016 (maybe sometime in 2014 or 2015?):
>
>
https://lists.freebsd.org/pipermail/freebsd-current/2017-January/thread.html#64431
>
https://lists.freebsd.org/pipermail/freebsd-stable/2017-January/thread.html#86646
>
https://lists.freebsd.org/pipermail/freebsd-questions/2016-November/thread.html#274684
>
> Does anyone have a good/recent write-up on how to switch to gcc?  :-)
>
>
> System
> =====> * Intel Core 2 Quad Q9550 @ 2.83GHz
> * 8GB ECC RAM
> * Samsung SSD 840 EVO 250GB filesystem (UFS2 + SU (not SUJ) + TRIM) + 32GB
swap
> * Running stable/11 r349226
> * Misc notes
>   - r350330 happened to be what was "master" at the time of my
test
>   - r350252 was the commit on stable/11 immediately before r350259
>   - Switching to r350252 accomplished via: cd /usr/src && svnlite
up -r350252
>   - System uses kern.maxvnodes=856944, last tuned 2018/06/07
>
>
> Test #1, building r350330 minimal clang
> ======================================> # cat /etc/src.conf
> WITHOUT_ATM=true
> WITHOUT_BLUETOOTH=true
> WITHOUT_DEBUG_FILES=true
> WITHOUT_FLOPPY=true
> WITHOUT_FREEBSD_UPDATE=true
> WITHOUT_IPFILTER=true
> WITHOUT_IPX=true
> WITHOUT_LIB32=true
> WITHOUT_NDIS=true
> WITHOUT_NETGRAPH=true
> WITHOUT_PPP=true
> WITHOUT_SENDMAIL=true
> WITHOUT_TESTS=true
> WITHOUT_WIRELESS=true
> WITH_OPENSSH_NONE_CIPHER=true
> WITHOUT_CLANG_EXTRAS=true
> WITHOUT_CLANG_FULL=true
> WITHOUT_LLDB=true
> WITHOUT_LLVM_TARGET_AARCH64=true
> WITHOUT_LLVM_TARGET_ARM=true
> WITHOUT_LLVM_TARGET_MIPS=true
> WITHOUT_LLVM_TARGET_POWERPC=true
> WITHOUT_LLVM_TARGET_SPARC=true
> WITHOUT_REPRODUCIBLE_BUILD=true
> # cat /etc/make.conf
> KERNCONF=X7SBA_RELENG_11_amd64
> CPUTYPE?=core2
> SVN_UPDATE=yes
> STRIP> CFLAGS+=	-fno-omit-frame-pointer
>
> Result:
> # rm -fr /usr/obj/*
> # cd /usr/src
> # time make -j4 buildworld
> 19906.874u 1280.928s 1:29:33.51 394.3%  57966+778k 23504+14200io 13867pf+0w
> # time make -j4 buildkernel
> 1592.460u 196.047s 7:36.61 391.6%       48704+614k 6627+18158io 7361pf+0w
>
>
> Test #2, building r350330 full clang
> ===================================> "full clang" means same
as Test #1 but with these 3 src.conf lines
> commented out, i.e. CLANG_EXTRAS, CLANG_FULL, and LLDB are ENABLED:
>
> WITHOUT_CLANG_EXTRAS=true
> WITHOUT_CLANG_FULL=true
> WITHOUT_LLDB=true
>
> Result:
> # rm -fr /usr/obj/*
> # cd /usr/src
> # time make -j4 buildworld
> 23779.674u 1463.156s 1:46:30.75 394.9%  57621+783k 20093+15423io 7283pf+0w
> # time make -j4 buildkernel
> 1594.079u 194.345s 7:36.48 391.7%       48707+614k 5301+18013io 5342pf+0w
>
>
> Test #3, building r350252 minimal clang
> ======================================> Same configs as Test #1
>
> Result:
> # rm -fr /usr/obj/*
> # cd /usr/src
> # time make -j4 buildworld
> 12582.693u 882.543s 56:52.35 394.6%     62698+760k 21432+9694io 6923pf+0w
> # time make -j4 buildkernel
> 1649.559u 184.934s 7:48.01 391.9%       57053+622k 7566+18291io 5402pf+0w
>
>
> Test #4, building r350252 full clang
> ===================================> Same configs as Test #2
>
> # rm -fr /usr/obj/*
> # cd /usr/src
> # time make -j4 buildworld
> 16600.975u 1068.754s 1:14:29.53 395.3%  63271+774k 8683+10876io 4707pf+0w
> # time make -j4 buildkernel
> 1650.654u 183.966s 7:47.47 392.4%       57117+623k 2829+17951io 1926pf+0w
>