CentOS - Oct 2018 - What are the differences between systemd and non-systemd Linux distros?

On Oct 17, 2018, at 10:03 AM, Mark Rousell <mark.rousell at signal100.com>
wrote:
> 
> launchd is not being forced on them as systemd is in practice
Try doing without launchd on macOS.

If you think that?s irrelevant, count the number of MacBooks at the next FreeBSD
conference you attend.

For an init system to gain sufficient momentum, it must be the default, with no
easy way to avoid it.  Without that, you get things like:

1. TrueOS, where major non-core services still have no OpenRC script despite
OpenRC being the default for about a year.  There were no Samba or NUT OpenRC
scripts the last time I tried TrueOS, for example.  Even if that?s changed, it?s
still a reflection of the fundamental barrier to adoption that I?m talking about
here.

2. Lazy third-party Linux packages that still use SysVInit scripts, because
they?re just forward-porting old packages with minimal effort.
> I should add that the speaker also massively over-simplifies opposition
> to systemd on the basis that he incorrectly perceives it to be
> opposition to change. He seems to ignore the fact that, as above, there
> are substantive objections to the specific architecture and quality of
> systemd, not merely objections to change with no deeper reason.
While there certainly are objective problems with systemd?s design and
implementation, it is basic human psychology that many people will not move to a
newer system despite piles of advantages.

The major BSDs are fundamentally conservative at the project management level,
so I believe this tendency is stronger in the BSD user population than elsewhere
in the IT world.  It?s a form of self-selection bias: the BSDs are run
conservatively, so they attract a user base that is also technologically
conservative, from which come the next generation of core developers, who
therefore continue to run the project conservatively.  Consequently, the major
BSDs are even more conservative than the Enterprise Linuxes.

If it were otherwise, TrueOS would have long since taken over the FreeBSD world,
and nvi wouldn?t still be missing proper UTF-8 support.
> many people objecting to systemd
> would nevertheless favour more modern system/service management.
I?d love to see that quantified.

Alternatives to the BSD rc init system are readily available, yet I think if you
were to survey actual use, you?d find that over 99% of BSD boxes use the stock
init system.

Change has to be forced from the center out on this kind of thing.  Diffusion
from the outside in takes too long.

The question in my mind is how long it?s going to take for the major BSDs to
make such a change at the center, so that the majority of new installs will use
a modern init system.  The systemd project started in 2005, and wasn?t widely
deployed as the default until about 4 years ago.  If past is prologue, I think
this won?t happen on the BSDs for another decade or so, if ever.

Example: FreeBSD is just now moving to pkg-in-base in earnest, giving it
features I first saw in the default install of Debian in about 1995.

On 17/10/2018 20:03, Warren Young wrote:
> On Oct 17, 2018, at 10:03 AM, Mark Rousell <mark.rousell at
signal100.com> wrote:
>> launchd is not being forced on them as systemd is in practice
> Try doing without launchd on macOS.
>
> If you think that?s irrelevant, count the number of MacBooks at the next
FreeBSD conference you attend.
That's Mac. It's not Linux. And that's the point. Mac does not have
Linux's very particular culture and history.

Apple and oranges, and all that. Sure, launchd is an init system but
it's not on Linux. If launchd was on Linux and it had systemd's cultural
issues and, in many people's views, technical issues then the opposition
to it would be identical to the opposition to systemd.

When people go to Mac they accept what it is (mostly). That's a
fundamentally different culture to Linux (or BSD for that matter).
> For an init system to gain sufficient momentum, it must be the default,
with no easy way to avoid it.
That's an argument for authoritarianism, which seems to me to be
anathema to the overriding culture of Linux. Can you see why many people
might dislike the personalities involved with systemd, yet, when such an
argument is used in favour of systemd? :-)
>> I should add that the speaker also massively over-simplifies opposition
>> to systemd on the basis that he incorrectly perceives it to be
>> opposition to change. He seems to ignore the fact that, as above, there
>> are substantive objections to the specific architecture and quality of
>> systemd, not merely objections to change with no deeper reason.
> While there certainly are objective problems with systemd?s design and
implementation, it is basic human psychology that many people will not move to a
newer system despite piles of advantages.
Quite true but the fact that some people do dislike change (a) does not
make the substantive and objective problems, in many people's views,
with systemd any less real or important and (b) does not mean that the
speaker did not massively over-simplify the opposition to systemd , i.e.
he effectively claimed that it was all to do with fear of change when,
as you agree, there in fact are substantive, real and objective issues
which are widely recognised.
> The major BSDs are fundamentally conservative at the project management
level, so I believe this tendency is stronger in the BSD user population than
elsewhere in the IT world.  It?s a form of self-selection bias: the BSDs are run
conservatively, so they attract a user base that is also technologically
conservative, from which come the next generation of core developers, who
therefore continue to run the project conservatively.  Consequently, the major
BSDs are even more conservative than the Enterprise Linuxes.
An interesting observation. It seems to me that there are aspects of the
Linux culture that are at least as conservative as the BSDs in this
context (are perhaps shared with BSD). One of these aspects is the "do
one job and do it well" expectation of componentisation. In many
people's views, systemd wilfully and unnecessarily tramples all over
this cultural/technical requirement. If this is the case in many
people's views, then it makes a lot of sense that hey are unhappy with it.
>> many people objecting to systemd
>> would nevertheless favour more modern system/service management.
> I?d love to see that quantified.
None of these comments (neither mine nor those of the speaker of the
presentation) are easy to quantify. I can only say that I base my
comments mainly on the contents of technical mail lists and blogs and
similar and I have very frequently observed that (a) a common question
is how users can change init system to something other than either
systemd or SysVinit (depending on whether they are starting with a Linux
that is normally with or without systemd), and (b) there does seem to be
a very common thread that the time had come that something needed to be
done to update SysVinit but that systemd definitely should not be it
(and that the solution, whatever it was, should not have been introduced
in the way that systemd was).
> Alternatives to the BSD rc init system are readily available, yet I think
if you were to survey actual use, you?d find that over 99% of BSD boxes use the
stock init system.
That's a different metric. People may well stick with the stock init
system but that doesn't mean that they like it or really want it.
> Change has to be forced from the center out on this kind of thing.
Again, an appeal to authoritarianism. Excuse me if I don't wish to join
you on that. Authoritarianism, in all its forms, is dangerous and, in my
view, a form of vandalism.

Also one might ask: What centre? This is the world of Linux. Many people
don't recognise any centre, and quite sensibly so. Indeed, they use
Linux explicitly to avoid the centrism of the likes of Linux or Apple.

As I observed above, this authoritarian centrism in in part why systemd
is so despised. It was effectively forced on a great many users in in a
centrist, choice-limiting manner. A great many people regard this as a
bug, not a feature. I think they might be right.
> The question in my mind is how long it?s going to take for the major BSDs
to make such a change at the center, so that the majority of new installs will
use a modern init system.  The systemd project started in 2005, and wasn?t
widely deployed as the default until about 4 years ago.  If past is prologue, I
think this won?t happen on the BSDs for another decade or so, if ever.
>
> Example: FreeBSD is just now moving to pkg-in-base in earnest, giving it
features I first saw in the default install of Debian in about 1995.
That's up to them and their users, isn't it. Just because it's a
good
idea doesn't mean it has to be done in a hurry. This was and is one of
the problems with systemd on Linux.

-- 
Mark Rousell

On Oct 17, 2018, at 3:28 PM, Mark Rousell <mark.rousell at signal100.com>
wrote:
> 
> On 17/10/2018 20:03, Warren Young wrote:
>> On Oct 17, 2018, at 10:03 AM, Mark Rousell <mark.rousell at
signal100.com> wrote:
>>> launchd is not being forced on them as systemd is in practice
>> Try doing without launchd on macOS.
> 
> If launchd was on Linux and it had systemd's cultural
> issues and, in many people's views, technical issues then the
opposition
> to it would be identical to the opposition to systemd.
Try this gedankenexperiment instead: what if RHEL 7 shipped based on launchd
instead of systemd, with no differences relative to the version shipped in the
contemporaneous version of Mac OS X?

I?m uncertain as to whether the opposition would have been as great, but I?m
dead certain the opposition would have been vociferous and strident, because
Linux, though less conservative than the BSDs, is much more conservative than
macOS.

The systemd vs launchd vs sysvinit vs whatever-else arguments are more about
human factors than about technology, even though they?re usually couched as
technical battles.
> When people go to Mac they accept what it is (mostly).
I doubt most Mac people even know launchd exists, much less have an informed
technical opinion about it.  And of the small minority that do have such an
opinion, it almost certainly doesn?t drive buying decisions.  Maybe that?s what
you mean by accepting macOS as it is.

The thing I don?t get is, why is it different in the Linux world?  Why did we in
the Linux community spend so much time arguing about systemd over the past
several years?  Why is it still an active topic five years after the key events?
And why is the BSD community continuing to stir the pot?

Here?s what I want to see: I want one of the BSDs to clean Linux?s clock with a
thoroughly awesome modern init system that makes us Linux fans so jealous we
start noisily advocating to replace systemd with it, much as ZFS is starting to
replace the horrid lash-up that is ext4/xfs+md+LVM+DM.

What I *don?t* want is more of this retrenchment to SysVInit.  I liked it well
enough within its limitations, but we can do better in 2018.

(It?s a related tragedy that a slightly modified ksh88 remains the most powerful
general purpose scripting language mandated by POSIX three decades after it was
released by AT&T.  We?ve got better alternatives here, too.)
>> For an init system to gain sufficient momentum, it must be the default,
with no easy way to avoid it.
> 
> That's an argument for authoritarianism
I call it leadership.  Working code argues best.

Benno Rice is right: Lennart Poettering gets stuff done.

In the BSD world, they call the opposite tendency bikeshedding.  You can find a
thousand people willing to argue about why something shouldn?t be done, or why
it shouldn?t be done *that* way for every person capable, willing, and available
to write a given piece of software.

For all the complaints about systemd over the past several years, I note that
there is still no fork of CentOS 6 or CentOS 5, keeping the prior init system(s)
but updating their package set to recent versions.  Many would rather gripe
about change than put in the work it takes to maintain the status quo.

Then you get the crowd that tries to argue that we should just stay with what
works, apparently under the misapprehension that staying in place is a
zero-effort choice, when in reality it is at best an accrual of technical debt;
the bill will come due eventually, with interest.

I suspect these two groups overlap quite a lot, inconsistent though the combined
position is.
> the fact that some people do dislike change (a) does not
> make the substantive and objective problems, in many people's views,
> with systemd any less real or important
Of course not, but I also don?t see a lot of effort going into replacing systemd
with something better.  Most of the effort opposing systemd seems to be going
into anti-systemd advocacy campaigns, plus a tiny slice off to the side going
into retrenchment to SysVInit.  That?s conservatism, plain and simple.
> he effectively claimed that it was all to do with fear of change when,
> as you agree, there in fact are substantive, real and objective issues
> which are widely recognised.
I suspect that for many people, those are rationalizations rather than reasons.

I saw much the same sort of logic in the Unix vs Linux wars, roughly spanning
the decade surrounding 1996.

The Big Iron Unix and SCO Unix fans had all kinds of myopically rational reasons
why Linux wasn?t going to replace their OS of choice: journaled filesystems,
better SMP, STREAMS, hot-swap hardware, real ksh93 instead of that cheesy
nonstandard imitation Bash?

The analogy I used at the time is that the Unix fans saw Linux surfing behind
their big yacht and laughed at the tiny, flimsy, cheap little surfboard, not
realizing that it would take an awfully big wave to allow someone to be surfing
out in the middle of the ocean.  We?ve been skimming the splinters of that yacht
off the surface of the sea for decades now.

Like the Linux wave, the systemd wave took years to build beneath the surface. 
If you perceive that it broke fast, it?s only because you weren?t looking in the
right places, where you could see and even help direct that long build-up.

You can spend your time rationalizing against systemd while it sheds enough of
its weaknesses and gains enough features to sweep the planet.  You?re years late
if you want to stop that from happening, but I believe it can be done, just as
systemd replaced rc files, SysVInit, and Upstart.

It?ll take a lot of hard work, but it can be done.
> One of these aspects is the "do
> one job and do it well" expectation of componentisation.
That?s never been an especially hard and fast rule.  Witness Emacs, Perl, GNOME,
Firefox, ZFS?

My DVCS of choice is Fossil, which is about as far from the ?small pieces
loosely joined? philosophy as you can get, from which comes much of its power
and simplicity.  Fossil?s primary opposition is Git, which is famously difficult
to understand and use properly, even though it adheres perfectly to that
philosophy.

(Incidentally, I?ve spent a fair bit of time in recent months making Fossil work
better, so my calls to action in this thread are not just me proposing work for
other people to do.)

I *want* my init system to build a dependency graph and maintain that graph at
run time.  I *want* it to start services in parallel, consistent with that
dependency graph.  I *want* it to restart services that die unexpectedly, with
exponential back-offs.  I *want* it to have sensible answers to a ?status?
command without having to write the shell code to dig that that status out of
service-specific files.  I *want* my service definition scripts to be 15 lines,
rather than 150.  I *want* to be able to start services as a normal user, with
all of the power of a real system service, limited only by OS permissions,
without needing sudo.

That puts an awful lot of code into a single place.  It is not small pieces
loosely joined.  What it *is* is powerful.
> In many
> people's views, systemd wilfully and unnecessarily tramples all over
> this cultural/technical requirement. If this is the case in many
> people's views, then it makes a lot of sense that hey are unhappy with
it.
I?m not a particularly big fan of systemd.  I agree with much of what its
opponents dislike it for.

The only reason I enter these arguments is that I feel the need to fight against
the complainers, because I?d rather see opposition deflected toward a
replacement.

Give me a better option, and I?ll happily switch to it.

Tricky bit: ?better option? is evaluated on a systems basis, not on a
per-feature basis.  If the combined system doesn?t provide most of CentOS? key
virtues, as I see them, it isn?t a better option.

If you ask why I don?t just write something better myself, it?s because I don?t
care enough about the Linux init system to do that, so I?d rather spend time
advocating for and taking advantage of systemd?s features than complaining about
its weaknesses.

(Automatic service restarts saved me a lot of work just a few weeks ago!)

I?m also unwilling to stick with increasingly poorly supported technology.  If
the cheese moves, I?m following the cheese.
>> Alternatives to the BSD rc init system are readily available, yet I
think if you were to survey actual use, you?d find that over 99% of BSD boxes
use the stock init system.
> 
> That's a different metric. People may well stick with the stock init
> system but that doesn't mean that they like it or really want it.
That?s a large part of my point: if you are unable or unwilling to take the time
to replace a thing, why bother spending time griping about its weaknesses?

Talk about a first world problem: Oh, waaah, my automated software management
service isn?t ideally designed.  Woe is meeee!

It?s much like my macOS argument above: CentOS sucks in many ways, but I have no
good answer to the ?What?s better?? question for the applications where I use
it.  I?m not going to spend a bunch of time listing all the ways CentOS sucks:
either I?ll fix the problems myself, or I?ll work around them, or I?ll ignore
them.

Notice that putting up with systemd is only one of your several choices.  The
FOSS ecosystem advances every time someone scratches an itch.  My objection is
that we?ve got all these people claiming to have itches, but very, very few are
scratching.
>> Change has to be forced from the center out on this kind of thing.
> 
> Again, an appeal to authoritarianism. Excuse me if I don't wish to join
> you on that. Authoritarianism, in all its forms, is dangerous and, in my
> view, a form of vandalism.
In what useful sense do you believe you own the parts of CentOS affected by the
move to systemd?  It can?t be vandalism otherwise.

Intellectual property is no answer in the FOSS world, since that implies that
you?ve relinquished the relevant rights already, so you have little to no say in
how the wall gets reshaped, if in fact you built it in the first place.

I suspect it?s more that you found this free wall just sitting around online,
and no one was guarding it, and in fact there were people offering you free
copies of the wall, as many as you want, so you started making buildings out of
these walls.  Then someone changed the wall generator to produce
differently-shaped walls, and now you?re upset that the walls aren?t the same
any more.

I get it: you?re annoyed.  What I don?t get is why you believe your annoyance
should, of itself, effect a change in the world.

FOSS is a do-ocracy: he who does the work makes the rules.  If you want the wall
to be shaped differently, and no one is making new walls with that shape any
more, go make your own wall, or maintain an old wall that was already
correctly-shaped, by your lights.

In this case, I think the lowest friction path is to fork CentOS 5 or 6, then
backport as much of the feature set as is possible without dragging in systemd. 
No one?s done this, despite having over 7 years to react, dating from systemd?s
deployment as the default init system in Fedora.  Why not?

The closest thing I?m aware of in the Linux world is Devuan, which was low in
popularity a year ago and has been dropping ever since:

    https://distrowatch.com/dwres.php?resource=popularity

According to the same stats, CentOS is pretty stable, which I find appropriate.
:)
> Also one might ask: What centre?
In this case, the one currently headquartered in Raleigh, NC, USA.

If you want a different centre, pick one, or be your own.  That?s why we have
several major Linux flavors and hundreds of Linux distros.

Centralized control is clearly not the key problem here, since so much of this
anti-systemd sentiment is coming from the much more centralized BSD camps.

Centralized control didn?t do the Big Iron Unix vendors much good.  A better
thing came along and wiped them out.  The thing is, a whole lot of programmers
went to a whole lot of effort to make that happen.  Are you going to do more
than post to mailing lists in an effort to wipe out systemd?
> As I observed above, this authoritarian centrism in in part why systemd
> is so despised. It was effectively forced on a great many users in in a
> centrist, choice-limiting manner.
You also didn?t get a choice over SysVInit vs BSD style init, short of switching
from Red Hat to Slackware.  So what?

Someone has to make a decision.  We call those who do this often and well
leaders.

Leadership is not central control or authoritarianism: we get to choose which
leaders we wish to follow.

Don?t like where Poettering is taking his slices of the Linux world?  Fine;
follow someone else who is taking equivalent slices in a direction you like
better.  Or, be that someone!
> That's up to them and their users, isn't it. Just because it's
a good
> idea doesn't mean it has to be done in a hurry. This was and is one of
> the problems with systemd on Linux.
The first version of systemd was released in 2010, and it wasn?t deployed ?for
real? until RHEL 7 came out in 2014, as far as I?m concerned.  I don?t call that
?in a hurry.?

This in a world where Linux?s major competition all had systemd-like service
management by around 2005:

- Solaris 10 came out in early 2005 with the Service Management Facility (SMF),
which is very similar to systemd?s initial feature set.

- Several months later, Apple shipped Mac OS X ?Tiger? with launchd replacing
init(8).

- Microsoft, ever late to the party, shipped the last major piece missing
relative to the above about 18 months after Tiger came out, adding delayed-start
services to Windows Vista.

(This is probably where the ?2005? in my prior post came from, by the way.  I
thought systemd got started earlier.)

So yeah, systemd finally let Linux catch up to Vista in the area of background
service handling, an area where Linux is supposed to be superior, but it?s all
happening too fast?