freebsd stable - Jan 2017 - Building Kernel and World with -j

Walter Parker <walterp at gmail.com> writes:
> For decades there has always been a warning not to do parallel builds of
> the kernel or the world (Linux kernel builds also suggest not to do this).
>
> Every once in a while, I see people post about 5 minutes. This only way I
> can see this happening is by doing a parallel build (-j 16 on a Xeon
> Monster box).
>
> Are parallel builds safe? If not, what are actual risk factors and can they
> be mitigated?
As a general rule, it's safe. But don't report failures from a
parallel build.

This is not so much an issue of parallel builds being unsupported
as of the logs being harder to read.

On Mon, Jan 23, 2017 at 6:10 AM, Lowell Gilbert <
freebsd-stable-local at be-well.ilk.org> wrote:
> Walter Parker <walterp at gmail.com> writes:
>
> > For decades there has always been a warning not to do parallel builds
of
> > the kernel or the world (Linux kernel builds also suggest not to do
> this).
> >
> > Every once in a while, I see people post about 5 minutes. This only
way I
> > can see this happening is by doing a parallel build (-j 16 on a Xeon
> > Monster box).
> >
> > Are parallel builds safe? If not, what are actual risk factors and can
> they
> > be mitigated?
>
> As a general rule, it's safe. But don't report failures from a
> parallel build.
>
> This is not so much an issue of parallel builds being unsupported
> as of the logs being harder to read.

Use of parallel builds of world and kernel are and have been supported
since at least 10.0. If a parallel build fails, the first step is usually
to do a single-job build. If it succeeds, there is a bug in the make
scripts that should be reported. If the single-job build also fails, a bug
should be reported with any errors included from the non-parallel build as
the parallel build makes the error context very difficult or even
impossible to determine from the log.

Generally, I think the number of jobs should be slightly greater than the
number of available CPU threads. Back in 10.0 days I ran some tests that
showed the for 4 and 8 thread systems the improvements for large numbers of
jobs peaked at about CPU thread count + 2 and significantly larger numbers
of jobs caused a slight deterioration in performance. This was not true on
early hyper-threaded CPUs which did not effectively optimize
hyper-threading and things may have changed in the 6 or 7 years since I
tested and may be very different for systems with large numbers of threads
seen in servers today.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkoberman at gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683