CentOS - Oct 2018 - What are the differences between systemd and non-systemd Linux distros?

Valeri Galtsev wrote:
> On 10/17/18 7:55 PM, Warren Young wrote:
<snip>
>> Benno Rice is right: Lennart Poettering gets stuff done.
Because he's funded. And I strongly suspect that a lot of that funding
comes from M$'s interest in Upstream.
<snip>
>
> With all due respect, many people just stopped offering any argument
> about systemd, and simply fled elsewhere which in _their_ opinion
> (and I am one of them) lies better in what they with their education
> and life experience is more reasonably resembling system suitable
> for servers.
>
> Servers are key word for me. You can see me using macintosh laptop in
> variety of places, that doens't mean MacOS will be my choice for
server, so
> don't count laptopls into any statistics. The same is true about a
bunch
> of other sysadmins I know, who mostly use Apple laptops, whereas run
> Linux, or UNIX-like, or [truly] UNIX servers.
Actually, I've got CentOS on my 9 yr old Netbook, that I use while
traveling. Otherwise, my home workstation is CentOS 6, and I am NOT
looking forward to EOL.

But Valeri's correct: people are tired of screaming and yelling about
systemd, because we've had years now of the response being "tough,
it's
the Wave of the Future", and Poettering is like upper management: they
know, I mean, Everything, so why should they need to talk to end users (or
working sysadmins)?

Lack of screaming and yelling filling this venue is more because
"what's
the point?", and we have to get work done.

      mark

On Oct 18, 2018, at 9:41 AM, mark <m.roth at 5-cent.us>
wrote:
> On 10/17/18 7:55 PM, Warren Young wrote:
>>> Benno Rice is right: Lennart Poettering gets stuff done.
> 
> Because he's funded.
There are plenty of people with jobs that don?t get stuff done.
> I strongly suspect that a lot of that funding
> comes from M$'s interest in Upstream.

Soooo?systemd is a Microsoft conspiracy against Linux?
> my home workstation is CentOS 6, and I am NOT
> looking forward to EOL.
That?s what I meant with my comment about the technical debt bill coming due. 
You can?t ignore the changes in the external world forever.

The OpenSSL issue brought up in a prior post is another example of the same
basic problem.
> people are tired of screaming and yelling about
> systemd, because we've had years now of the response being "tough,
it's
> the Wave of the Future"
We covered that back when RHEL 7 was still in beta: the time is far too late to
change the init system of RHEL 7.  The fact that you?re tired of being ignored
doesn?t enter into it: you could still be yelling about it, and it still
wouldn?t change.  Red Has simply isn?t going to swap out its Enterprise Linux
init system within a major release cycle.

I believe it?s certain that RHEL 8 (and thus CentOS 8) will also be
systemd-based, since we?d be hearing about the change by now via Fedora if it
were otherwise.

Those of you who want a systemd-free CentOS-like OS to be available before
CentOS 6 hits EOL are going to have to see to that yourselves.  You cannot
expect it to just drop from the sky.
> Poettering is like upper management: they
> know, I mean, Everything, so why should they need to talk to end users (or
> working sysadmins)?
The suggestion that Red Hat is not listening to working system administrators
beggars belief.  That?s pretty much the basis of their company?s major income
stream.

What Red Hat is not doing is filling every demand from all working system
administrators.  They?re choosing which demands to address, as any software
project management must.

Red Hat has certainly heard the screams of the reactionaries.  Since that hasn?t
changed anything, I believe you have your demand?s answer.  So, what are you
going to do about it?

On 10/18/2018 4:41 PM, Warren Young wrote:
> On Oct 18, 2018, at 9:41 AM, mark <m.roth at 5-cent.us> wrote:
>> people are tired of screaming and yelling about
>> systemd, because we've had years now of the response being
"tough, it's
>> the Wave of the Future"
> We covered that back when RHEL 7 was still in beta: the time is far too
late to change the init system of RHEL 7.  The fact that you?re tired of being
ignored doesn?t enter into it: you could still be yelling about it, and it still
wouldn?t change.  Red Has simply isn?t going to swap out its Enterprise Linux
init system within a major release cycle.
>
> I believe it?s certain that RHEL 8 (and thus CentOS 8) will also be
systemd-based, since we?d be hearing about the change by now via Fedora if it
were otherwise.
>
This brings to mind a video I was pointed to not long ago of Brendan 
Conoboy's talk at a Dojo recently:

https://www.youtube.com/watch?v=HQsUdLPJW20

For quite a long time, many (perhaps most) folks had assumed that Fedora 
functioned more or less directly as the internal alpha for RHEL, with a 
branch at some point occurring, followed by pruning of packages, 
hardening, vendor testing, and release. Subsequently, CentOS (even after 
the RH integration) functioned *strictly* as a clean-room downstream 
rebuild, with the ability to do unsupported things, like alternate 
architectures, or heavier kernels, restricted to what could be done 
while maintaining a 100% binary compatible rebuild. Any contributions 
back up where taken to be incidental, from CentOS users reporting bugs 
that could be verified against RHEL.

Conoboy, on the other hand, takes great pains during the speech to 
describe a much more fluid and complex interaction between CentOS and 
its upstream, and puts forth CentOS as a mechanism (perhaps the best 
mechanism) for the winder EL community to contribute (something?) back 
into RHEL's future. He also gives clear signals that various Fedora 
steps have been in directions that Red Hat did not want EL necessarily 
going, and that the simplistic assumptions we've commonly been making 
aren't really correct.

Obviously, there seems to be a bit of a discrepancy there.

The wider EL community is trapped between a rock and a hard place 
somewhat. If you try to direct Fedora into the needs of EL users, you 
stand a good chance of getting told to pound stand, and that EL is 
getting in the way of bleeding-edge progress. Traditionally, CentOS has 
had its hands tied since it aims to be 100% compatible with upstream. 
Red Hat (and Red-Hat-as-a-sponsor-of-CentOS) might do well to clarify 
just what type of back-and-forth it wants out of the wider EL-using 
community. Does it want direct feedback in the form of tickets? Should 
people form SIGs? Obviously RHEL7 is not changing init systems, but 
where should one talk about the future?

>> Poettering is like upper management: they
>> know, I mean, Everything, so why should they need to talk to end users
(or
>> working sysadmins)?
> The suggestion that Red Hat is not listening to working system
administrators beggars belief.  That?s pretty much the basis of their company?s
major income stream.
>
> What Red Hat is not doing is filling every demand from all working system
administrators.  They?re choosing which demands to address, as any software
project management must.
This seems a bit specious. How many working SA's and Engineers at paid 
shops call up their Red Hat rep for something like this? This isn't the 
type of thing you demand a strategy conference call from them for unless 
you're absolutely huge, or you have a very bored manager. People just 
complained (heavily) about it internally, went back to fixing the latest 
crisis, and hoped the adults working on RHEL would do the right thing 
when it came to reliability. I'm sure Red Hat understands that looking 
at the financials of dropped licenses and counting up the total of any 
vague, complaining support tickets are not the whole picture.

On the other point (while keeping personalities out of it)... I think EL 
users are likely to have more experience in large, enterprise 
organizations -- the kind of orgs where technical decisions sometimes 
take a back seat to politics. Everyone's seen a land grab in person, and 
everyone's seen, and probably done themselves -- I know I have, 
techniques for getting a toe-hold, leveraging it into a larger area of 
control, and ensuring your project becomes pretty much indispensable. 
The suggestion that this has occasionally happened at Red Hat and that 
questionable technical situations might have resulted doesn't seem 
unreasonable, even if it is indeed out of scope for this list.


-jc

*** This response is my personal opinion and may not reflect that of my
employer. ***

>> people are tired of screaming and yelling about systemd, because
we've
>> had years now of the response being "tough, it's the Wave of
the
>> Future"
>
>We covered that back when RHEL 7 was still in beta: the time is far too late
to change the init system of RHEL 7.  The fact that you?re tired of being
ignored doesn?t enter into it: you could still be yelling about it, and it still
wouldn?t change.  Red Has simply isn?t going to >swap out its Enterprise
Linux init system within a major release cycle.
>
>I believe it?s certain that RHEL 8 (and thus CentOS 8) will also be
systemd-based, since we?d be hearing about the change by now via Fedora if it
were otherwise.
>
>Those of you who want a systemd-free CentOS-like OS to be available before
CentOS 6 hits EOL are going to have to see to that yourselves.  You cannot
expect it to just drop from the sky.
>
After spending the last year and a half preparing a major CentOS 6 based
appliance for the upgrade to CentOS 7, I can say when I started the project, I
was very much in the "I hate systemd" boat. It was new, different and
a drastic change from what I had become accustomed to. I have spent way too many
hours cursing systemd, converting init scripts, and handling the different way
it does things (like Java app daemons retuning non-zero exit codes for clean
shutdowns). Now that I have spent the time getting very intimate with systemd,
making it do what I need it to, and learning some of the neat tricks it has up
it's sleeves (like the xxx.mount definition files), I actually have come to
appreciate it, and the power it contains.

Is the conversion from sysVinit/Upstart services simple and easy? Not in the
least, particularly if you are used to the simplicity involved with dropping a
launch script in the /etc/init.d/ folder. Does CentOS 7 make allowances for some
of this pain? Yes, it still processes the /etc/init.d/ folder in order to allow
legacy services to launch as pseudo systemd services. Is it a perfect
workaround? Not at all, otherwise I would have had no reason to convert all our
services to systemd, and my project would have been done a year ago. Will there
ever be a way to automate upgrading a CentOS 6 system and services to CentOS 7
or 8 (like was asked in another thread this week)? It might be do-able for a
very basic file server, or possibly even a web server, but with the wide variety
of services run on top of CentOS, there would be no foolproof way of automating
the process. If someone was to spend the time to create an automated tool to
convert init scripts to systemd services, I have a feeling their life would
become an unmitigated hell trying to accommodate all the corner cases out there
where a simple conversion won't work (and we all know how people love to
complain that free software doesn't do what they need it to do for corner
case #65,535, and therefore the developer who spent their own time writing it to
fill the need their project had, should donate their own time to make it work
for corner case #65,535).

If CentOS 8 were to switch back from systemd, I think you would be able to see
the explosions from Alpha Centauri as all the developers out there lost their
minds after spending all this time converting their apps to work with systemd.
If you don't like change, you are more than welcome to go back to using
Windows XP (as too many businesses still do because they don't want to spend
the time and money updating their LOB software), I'll guarantee you the
script kiddies and crypto-criminals will love you.

Greg

On Friday 19 October 2018 00:41:12 Warren Young wrote:
>
> Soooo?systemd is a Microsoft conspiracy against Linux?
>
I looooove this

Now SystemD finally makes sense

> Valeri Galtsev wrote:
>> On 10/17/18 7:55 PM, Warren Young wrote:
> <snip>
>>> Benno Rice is right: Lennart Poettering gets stuff done.
>
> Because he's funded. And I strongly suspect that a lot of that funding
> comes from M$'s interest in Upstream.
> <snip>
>>
>> With all due respect, many people just stopped offering any argument
>> about systemd, and simply fled elsewhere which in _their_ opinion
>> (and I am one of them) lies better in what they with their education
>> and life experience is more reasonably resembling system suitable
>> for servers.
>>
>> Servers are key word for me. You can see me using macintosh laptop in
>> variety of places, that doens't mean MacOS will be my choice for
server,
>> so
>> don't count laptopls into any statistics. The same is true about a
bunch
>> of other sysadmins I know, who mostly use Apple laptops, whereas run
>> Linux, or UNIX-like, or [truly] UNIX servers.
>
> Actually, I've got CentOS on my 9 yr old Netbook, that I use while
> traveling. Otherwise, my home workstation is CentOS 6, and I am NOT
> looking forward to EOL.
>
> But Valeri's correct: people are tired of screaming and yelling about
> systemd, because we've had years now of the response being "tough,
it's
> the Wave of the Future", and Poettering is like upper management: they
> know, I mean, Everything, so why should they need to talk to end users (or
> working sysadmins)?
>
> Lack of screaming and yelling filling this venue is more because
"what's
> the point?", and we have to get work done.
Hi,

A lot was already said but let me underline a few things from my personal
point of view:

- the upgrade path from EL6 to EL7 is completely broken.

That's certainly a good thing for upstream, because they can sell even
more support and training. I don't blame them for trying to make money, I
just say from the technical point of view it's not the best solution.

For home users it doesn't hurt too much but for the enterprise market
it's
bad. Migrating complex systems is a huge amount of work and takes a lot of
time and manpower. In the end it means higher costs.

- the migration to systemd is not really finished carefully in EL7.

Just look into upstream's Bugzilla and see how many issues still exist and
will probably not be fixed.

I show you a simple example: we happen not mount some NFS filesystems on
servers like this in /etc/fstab:

ftp:/var/ftp/pub     /mnt/nfs          nfs   bg,soft   0 0

Now, with every Linux since the last millennium one could simply bring
down the system into maintenance mode with 'telinit 1', and all worked
fine.
Now try the same with EL7, do a 'systemctl rescue' or 'systemctl
emergency' and see what happens. With lightning speed it does the wrong
thing, brings down networked services, brings down the network, and
doesn't unmount the NFS filesystems. Then try a 'df' or
'lsof' in rescue
mode, it all hangs.

Of course I found a solution, mount it with the option
'x-systemd.requires=network-online.target' and it behaves correctly. But
really, it's broken, because it's always clear that NFS mounts always
only
work WITH network!

That's just a single small example how things don't work as expected.

- migrating from EL6 --> FreeBSD seems easier than migrating from EL6 -->
EL7, IMHO.

That's really an important point, because those who started using Linux
with Linux/systemd will be bound to Linux/systemd with their knowledge,
switching to a *BSD or other Unix will be difficult. For me, I don't like
to be limited in such ways.

In other words, systemd is a new operating system which still lacks a
kernel :-)

One thing I know for sure: if the *BSD folks were ever going to invent
something like systemd, they will do it in a way which hurts less.

Regards,
Simon

On Fri, Oct 19, 2018 at 01:07:46PM +0200, Simon Matter
wrote:
> That's really an important point, because those who started using Linux
> with Linux/systemd will be bound to Linux/systemd with their knowledge,
> switching to a *BSD or other Unix will be difficult. For me, I don't
like
> to be limited in such ways.
Having worked with the init systems in a bunch of different distros, I
really *loved* having to write a different SysV init script for debian
and RHEL, using different functions and different styles.  Also, don't
forget to actually package the Red Hat init scripts as
/etc/rc.d/init.d/, because /etc/init.d is a symlink, while on debian
it is the actual location, and if you weren't careful, your package
would create an /etc/init.d directory and suddenly it's not even found
by the init system.

Oh, and as for 'grokkable' shell scripts used by init, they bear only
a passing resemblance between distros, they even differed between
releases of the same distro, making it so you had to learn a different
weird init system for each distro.  Heck, there was even an argument
about which shell they're run with.  And it was always fun when shell
bugs cropped up in init scripts.  A vendor writes an init script using
bash features that aren't in another distro, but it still uses the
#!/bin/sh shebang so you get really weird and difficult-to-diagnose
startup errors.

And heaven forbid you actually want to *SEE* any shell errors.
Nothing is ever captured in any logs, you have to be physically
looking at a console (be it a glass terminal or serial line) to
capture the error.

So, yes, people starting to use systemd won't know about having to do
that.  They're also not custom-crafting Modelines in their XF86Config
file for a monitor that uses weird undocumented, non-VESA parameters,
nor are they trying to track down the right interrupt to run their
network card so it doesn't interfere with their sound card.  I'm sure
we could create a whole book of all the annoyances with older Linuxes
that have been largely solved.

I don't see systemd as the end-all, be-all init system, I just think
it's heading in a good direction.  Its important to provide feedback
like people have on this list, although people in the CentOS community
really ought to provide feedback to the upstream communities.

Here's a good example for me:  In other systemd-based distros, they've
got the systemd --user enabled (RHEL/CentOS have it patched out).
This breaks a lot of our use case because the systemd developers don't
think that different sessions of the same user are distinct, so they
want to use systemd --user to manage user processes.  This breaks if
you use session-based authentication services like Kerberos.  systemd
--user tries to start up processes outside of your PAM session, so it
won't have access to your kerberos tickets.  And of course, Gnome
Terminal now uses a gnome-terminal-server process to be the parent of
all terminal sessions, started by systemd (as your user, on behalf of
PID1).  So you log in, start up a terminal, and it doesn't have any
Kerberos tickets.  Now, what happens if you happen to use an NFS v4
volume for $HOME, which uses Kerberos 5 for authentication?  Now not
only does your terminal not have tickets, but IT CAN'T EVEN REACH
$HOME.  And of course, systemd --user wants to read and write files in
$HOME, so the whole thing is broken.

What do the systemd developers say?  They want it so anyone who
becomes your $USER should just automatically have access to your
Kerberos ticket cache, so systemd can work.  This is actually breaking
from the way Kerberos has worked for decades.  And it seems that the
systemd developers have just decided that their way is better.  But
I'm going to keep pushing back.



-- 
Jonathan Billings <billings at negate.org>

On Oct 19, 2018, at 5:07 AM, Simon Matter <simon.matter at invoca.ch>
wrote:
> 
> - the upgrade path from EL6 to EL7 is completely broken.
Under what conditions would you actually use it?

As we can see from the repeated attempts to get a reliable in-place upgrade
process working, the community doesn?t seem to have much interest in the idea:

   https://lists.centos.org/pipermail/centos/2018-October/170379.html

I believe this is because in-place upgrade is antithetical to the idea of a
?stable? Linux distro in the first place.  Once something?s configured and
running, you just want it to keep doing so.

In my world, OS upgrades are generally paired with new hardware or VMs.

I did just this on an Ubuntu VM recently, which does have an in-place upgrade
system.  I?d been ignoring its motd offers of upgrade for years, on purpose, and
only upgraded it from 14.04 LTS to 18.04 LTS when I needed to rebuild the VM
anyway.

That?s why I was on an LTS release in the first place: to give me the years of
stability that let me batch the changes up into a single big-bang upgrade. 
CentOS is even better in this regard, with version lifetimes up around 10 years,
rather than 5 for Ubuntu LTS.  One of the reasons I chose to upgrade it recently
was because Ubuntu 14.04 is about to fall out of support, so it was time to
move.

I believe a lot of the antipathy toward systemd is that people want ?LTS? to be
forever.  That?s not going to happen until the rest of the world stops changing.
That would be a very sad thing: it?s basically a wish for stagnation.

If upgrading via separate hardware or a new VM is difficult, it calls into
question the usefulness of your backup and restore strategy.

Another advantage of this style of upgrade is that you have the prior box online
and ready to fall back to if the manual upgrade fails.  If an in-place upgrade
fails, you?ve just lost the primary.