CentOS - Oct 2018 - What are the differences between systemd and non-systemd Linux distros?

On Fri, Oct 19, 2018 at 01:07:46PM +0200, Simon Matter
wrote:
> That's really an important point, because those who started using Linux
> with Linux/systemd will be bound to Linux/systemd with their knowledge,
> switching to a *BSD or other Unix will be difficult. For me, I don't
like
> to be limited in such ways.
Having worked with the init systems in a bunch of different distros, I
really *loved* having to write a different SysV init script for debian
and RHEL, using different functions and different styles.  Also, don't
forget to actually package the Red Hat init scripts as
/etc/rc.d/init.d/, because /etc/init.d is a symlink, while on debian
it is the actual location, and if you weren't careful, your package
would create an /etc/init.d directory and suddenly it's not even found
by the init system.

Oh, and as for 'grokkable' shell scripts used by init, they bear only
a passing resemblance between distros, they even differed between
releases of the same distro, making it so you had to learn a different
weird init system for each distro.  Heck, there was even an argument
about which shell they're run with.  And it was always fun when shell
bugs cropped up in init scripts.  A vendor writes an init script using
bash features that aren't in another distro, but it still uses the
#!/bin/sh shebang so you get really weird and difficult-to-diagnose
startup errors.

And heaven forbid you actually want to *SEE* any shell errors.
Nothing is ever captured in any logs, you have to be physically
looking at a console (be it a glass terminal or serial line) to
capture the error.

So, yes, people starting to use systemd won't know about having to do
that.  They're also not custom-crafting Modelines in their XF86Config
file for a monitor that uses weird undocumented, non-VESA parameters,
nor are they trying to track down the right interrupt to run their
network card so it doesn't interfere with their sound card.  I'm sure
we could create a whole book of all the annoyances with older Linuxes
that have been largely solved.

I don't see systemd as the end-all, be-all init system, I just think
it's heading in a good direction.  Its important to provide feedback
like people have on this list, although people in the CentOS community
really ought to provide feedback to the upstream communities.

Here's a good example for me:  In other systemd-based distros, they've
got the systemd --user enabled (RHEL/CentOS have it patched out).
This breaks a lot of our use case because the systemd developers don't
think that different sessions of the same user are distinct, so they
want to use systemd --user to manage user processes.  This breaks if
you use session-based authentication services like Kerberos.  systemd
--user tries to start up processes outside of your PAM session, so it
won't have access to your kerberos tickets.  And of course, Gnome
Terminal now uses a gnome-terminal-server process to be the parent of
all terminal sessions, started by systemd (as your user, on behalf of
PID1).  So you log in, start up a terminal, and it doesn't have any
Kerberos tickets.  Now, what happens if you happen to use an NFS v4
volume for $HOME, which uses Kerberos 5 for authentication?  Now not
only does your terminal not have tickets, but IT CAN'T EVEN REACH
$HOME.  And of course, systemd --user wants to read and write files in
$HOME, so the whole thing is broken.

What do the systemd developers say?  They want it so anyone who
becomes your $USER should just automatically have access to your
Kerberos ticket cache, so systemd can work.  This is actually breaking
from the way Kerberos has worked for decades.  And it seems that the
systemd developers have just decided that their way is better.  But
I'm going to keep pushing back.



-- 
Jonathan Billings <billings at negate.org>

On 10/19/2018 5:09 AM, Jonathan Billings wrote:
> On Fri, Oct 19, 2018 at 01:07:46PM +0200, Simon Matter wrote:
>> That's really an important point, because those who started using
Linux
>> with Linux/systemd will be bound to Linux/systemd with their knowledge,
>> switching to a *BSD or other Unix will be difficult. For me, I
don't like
>> to be limited in such ways.
> Having worked with the init systems in a bunch of different distros, I
> really *loved* having to write a different SysV init script for debian
> and RHEL, using different functions and different styles.  Also, don't
> forget to actually package the Red Hat init scripts as
> /etc/rc.d/init.d/, because /etc/init.d is a symlink, while on debian
> it is the actual location, and if you weren't careful, your package
> would create an /etc/init.d directory and suddenly it's not even found
> by the init system.
The first time I had to look at SysV init scripts on a Debian/Ubuntu box 
my eyes bled; if systemd had begun from that ecosystem I definitely 
would have understood its formation a bit more. But on Red Hat-derived 
distros, an initscript for a basic daemon is pretty simple and mostly 
boilerplate: copy/paste the sample file, maybe decide what you want to 
make tweakable in /etc/sysconfig/, then (if desired) build an RPM 
according to best practices.

Virtually everything you might need that isn't provided by the 
'functions' file is going to be your own custom logic for your own 
daemon, and it turns out that that usually doesn't change in a systemd 
landscape, resulting in a lot of workarounds, wrappers, and shell bits 
in unit files which would probably be more predictably understood in a 
single shell script to begin with.

Building a single init script that works across ALL Linux distributions 
(and other unices) is indeed painful and ugly, so if a vendor wanted to 
make a declarative config file and wash their hands of, that's 
understandable. But the same goes for an xinetd.conf snippet, or any 
other service manager of the same ilk. And making a boilerplate /Red 
Hat-specific /init script is trivial.


> Heck, there was even an argument
> about which shell they're run with.  And it was always fun when shell
> bugs cropped up in init scripts.  A vendor writes an init script using
> bash features that aren't in another distro, but it still uses the
> #!/bin/sh shebang so you get really weird and difficult-to-diagnose
> startup errors.
That's a larger *nix issue. As a proponent of dash on EL systems, I 
definitely think ensuring bashisms are called out and that /bin/sh means 
/bin/sh is probably a Good Thing.

> And heaven forbid you actually want to *SEE* any shell errors.
> Nothing is ever captured in any logs, you have to be physically
> looking at a console (be it a glass terminal or serial line) to
> capture the error.
The /sbin/service command is just a shell script. I'd suggest a patch to 
send stderr/out to logger as well if I thought it would be accepted. 
(And *manually executing* an init script with direct call was something 
we were already supposed to be deprecating; the service command was the 
standard environmental interface.)

Frankly, I've had a lot more problems debugging mysterious systemd-based 
startup failures than I ever had in a properly-written Red Hat init 
script. (Again, vendor-agnostic init scripts can be hot messes, but 
that's them...)


-jc

Japheth Cleaver wrote:
> On 10/19/2018 5:09 AM, Jonathan Billings wrote:
>> On Fri, Oct 19, 2018 at 01:07:46PM +0200, Simon Matter wrote:
<snip>
> The /sbin/service command is just a shell script. I'd suggest a patch
to
> send stderr/out to logger as well if I thought it would be accepted. (And
> *manually executing* an init script with direct call was something
> we were already supposed to be deprecating; the service command was the
> standard environmental interface.)
>
> Frankly, I've had a lot more problems debugging mysterious
systemd-based
> startup failures than I ever had in a properly-written Red Hat init
> script. (Again, vendor-agnostic init scripts can be hot messes, but
> that's them...)
Yeah. I have trouble finding the actual startup configs -
/etc/systemd/system? /var/lib? whereeverthehell they are, do a locate....
as opposed to /etc/init.d to find the damn name (nfs? nfsd? idmapd?
nfs-idmapd? rpc-idmapd?)

         mark