CentOS - Apr 2017 - OT: systemd Poll - So Long, and Thanks for All the fish.

On Wed, April 19, 2017 16:22, Chris Murphy wrote:
>
> Apple has had massively disruptive changes on OS X and iOS. Windows
> has had a fairly disruptive set of changes in Windows 10. About the
> only things that don't change are industrial OS's.
>
I have no idea how this reference applies to my earlier post.  We do
not use Apple or Windows servers and the desktop environment is
stabilised at Win7pro.  There will be be no Windows 10 here ever.  OSX
/ iOS employment is limited to personal devices, none of which are
permitted on premise in any case.
> When it comes to breaking user space, there's explicit rules against
> that in Linux kernel development. And internally consistent API/ABI
> stability is something you're getting in CentOS/RHEL kernels, it's
one
> of the points the distributions exist. But the idea that Windows and
> OS X have better overall API stability I think is untrue, having
> spoken to a very wide assortment of developers who build primarily
> user space apps.
This may be true.  It is likely important to software developers.  It
is also totally irrelevant to a business.

Businesses, other than software development houses and consultants,
are software users.  When a vendor massively rearranges things in
their software, deprecates scripting syntax that has existed for years
if not decades, and fundamentally changes the way the administration
of an operating system is presented it really matter not a wit to a
business that the internal kernel level api remains unchanged.  It is
the accumulated administrative experience that is lost in consequence
that concerns a business given that replacing that loss will cost
either directly in retraining or indirectly in error and resultant
disruption; or both.
>
> What does happen, in kernel ABI changes can break your driver, as
> there's no upstream promise for ABI compatibility within the kernel
> itself. The effect of this is very real on say, Android, and might be
> one of the reasons for Google's Fuscia project which puts most of the
> drivers, including video drivers, into user space. And Microsoft also
> rarely changes things in their kernel, so again drivers tend to not
> break.
>
>
And this illustrates the point that I attempting to make.  A business
owner assumes that whatever OS is used it will deal with the various
devices that make up its hardware environment. For if it does not then
they seek an OS that does.  However, vanishingly few firms in my
experience (i.e.NONE) have ever had operational programming staff
write or even modify a device driver.  A business is in existence to
make money for its owners not dick around with esoteric computer
theory and practice.

Red Hat, again in my sole opinion, increasingly appears to me to be
emulating another company notorious for shuffling the user interface
to little evident purpose other than profit.  That is good business
for them.  It is not good for us.

Bear in mind that we have been RedHat/Whitebox/CA-OS/CentOS users
since 1998 so it is not like we are moving away from Linux with
anything like enthusiasm. But this upgrade treadmill that has
developed within RH is simply too costly for us to bear any longer. 
The idea that one has to rebuild from scratch entire host systems and
then laboriously port over data and customised portions to a new host
simply to upgrade the underlying OS is absolutely ludicrous. Consider
the tremendous labour costs regularly incurred in accomplishing what
amounts to maintaining the status quo.

We just upgraded a FreeBSD host from 10.3 to 11.0 in situ without
problem; and with very little downtime (three reboots in the space of
30 minutes).  This was no standalone device either.  It was the OS
running the metal for multiple BHyve virtual machines, themselves
running various operating systems (but mainly FreeBSD-11).  One of
said vms being our Samba-4 AD-DC.  And, had it all gone south then,
given we use ZFS in FreeBSD, and that we snapshot regularly, getting
back to 10.3 would have been, and still could be, nearly
instantaneous.

Think about what that would take in terms of man hours to accomplish
moving from EL6 to 7.  And moving from 5 to 6 was not much better. 
This is just too expensive to repeat every three years.

And allow me to forestall any claims that the chimera that is 'cloud
computing' is the answer. All that does is make creating the requisite
new platforms marginally less tedious. And that small advantage is
purchased at the cost of handling over control of all your data to
entities who are thoroughly discredited with respect to security and
privacy.

I am not anti or pro systemd, upstart, or etc/rc (or any other
software although I admit to holding a generally dim view of things
from Redmond).  I do not really care what is used so long as it works
and that introducing it does not greatly diminish the value of
existing user skills and knowledge.  However, I am past the point of
patience with gratuitous changes that offer no appreciable benefit to
the parties tasked with dealing them.  Systemd is not the problem.  It
is a symptom of a deeper malaise, indifference.


-- 
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3

> 
> Think about what that would take in terms of man hours to accomplish
> moving from EL6 to 7.  And moving from 5 to 6 was not much better. 
> This is just too expensive to repeat every three years.
So why do it? There is absolutely nothing wrong with sticking with EL6
for a long time, certainly for the lifetime of the hardware - EL5 has
only just gone EoL, and if you pay RH you can still have it on support.
 Just because EL7 exists, it doesn't mean that you have to upgrade to
it. I've only just started to seriously roll out CentOS7, and then
mostly only on new machines.

P.

On Thu, Apr 20, 2017 at 09:33:30AM -0400, James B. Byrne
wrote:
> Red Hat, again in my sole opinion, increasingly appears to me to be
> emulating another company notorious for shuffling the user interface
> to little evident purpose other than profit.  That is good business
> for them.  It is not good for us.
>From my perspective as a Red Hat customer who supports hundreds of
RHEL7 Workstation systems, Red Hat really doesn't seem to care or test
their Workstation product.  Their support doesn't seem to have much
training when it comes to problems with the GUI.  Since GNOME itself
moves along at a much faster pace than RHEL, I always end up looking
for archives of documentation, and trawling through GNOME's bugzilla.

Red Hat makes its business on the Server side.  They don't really care
about graphical user interfaces apart from the installer.

-- 
Jonathan Billings <billings at negate.org>

On Apr 20, 2017, at 7:33 AM, James B. Byrne <byrnejb at harte-lyne.ca>
wrote:
> 
> When a vendor ... fundamentally changes the way the administration
> of an operating system is presented
I?ve gotten the sense from this other part of the thread that the answer to my
question, ?What are you moving to?? is FreeBSD.

If you think FreeBSD system administration hasn?t changed over the past 10
years, you must not have been using it that long.  What makes you think it won?t
change again in the next 10 years, possibly in very large breaking ways?
> vanishingly few firms in my
> experience (i.e.NONE) have ever had operational programming staff
> write or even modify a device driver.
My company is very small.  I?ve modified device drivers to make them work
properly on Linux, purely in a ?scratch my own itch? kind of way.

I assure, you, many larger organizations also do this or something similar. 
Netflix is famous for using FreeBSD on their streaming servers and for tuning
the FreeBSD kernel heavily for that purpose.
> A business is in existence to
> make money for its owners not dick around with esoteric computer
> theory and practice.
I?m not glorifying change for its own sake.  I?m just saying it happens, and
however inessential it may be to your business? operations is really not
on-point.  The fact is that it happens everywhere in this industry, so your only
choice is in which bag of changes you want to deal with, not whether you get a
bag of changes.
> The idea that one has to rebuild from scratch entire host systems and
> then laboriously port over data and customised portions to a new host
> simply to upgrade the underlying OS is absolutely ludicrous.
I find that most hardware is ready to fall over by the time the CentOS that was
installed on it drops out of support anyway.

That is to say, I think the right way to use CentOS is to install one major
version on the hardware when it?s built, and then ride it for the 7-10 years
until that OS version drops out of support.  (7 being the worst case, when you
install a new system juuuust before the next major OS version comes out.)

Then there?s all the change that is outside the OS proper.  For example, there?s
all the current changes in the way encryption is handled, which would require
operational changes anyway.  You can?t keep running BIND 4 on your public-facing
DNS servers, for example, even if all the security problems were somehow fixed
without changing any user interface.

Ditto mail, HTTP, and many other critical services, since old versions often
don?t even speak today?s required protocols.  (TLS 1.1 minimum, DMARC, DKIM,
SPF, etc.)

FreeBSD, this supposed bastion of stability, now actively discourages you from
using BIND in the first place, for example.  Now they want you to migrate to NSD
+ Unbound.  Oh noes, more change!
> Consider
> the tremendous labour costs regularly incurred in accomplishing what
> amounts to maintaining the status quo.
If you only wanted the status quo ante, why upgrade at all?

Obvious answer: because you actually do want *some* change.
> We just upgraded a FreeBSD host from 10.3 to 11.0 in situ without
> problem
Lucky you.  I?ve had such upgrades take a system out for a day, working around
all the breakages.

Upgrading FreeBSD is historically one of the most painful things about it.  It?s
getting better, but only by changing how everything about packaging was done. 
Holy ChangeLogs, Batman!

Don?t get the wrong idea that I don?t like FreeBSD, by the way.  I know these
things about it because I use it regularly.  This is one of those ?bags of
changes? I referred to above.  Sometimes I want the Linux bag, and sometimes I
want the FreeBSD bag, and I know going into the decision that each bag implies a
future bag of changes I?ll have to deal with.
> It was the OS running the metal for multiple BHyve virtual machines
Ah, more change.  Bhyve only goes back to FreeBSD 10, so if you were using
FreeBSD prior to that, you?d have had to either drag forward whatever VM manager
you were using or migrate to bhyve.
> given we use ZFS in FreeBSD, and that we snapshot regularly, getting
> back to 10.3 would have been, and still could be, nearly
> instantaneous.
That?s a great reason to pick FreeBSD.  Just don?t fool yourself that by
switching that you?ve somehow gotten off the upgrade treadmill.  You?ve only
switched bags.
> Systemd is not the problem.  It
> is a symptom of a deeper malaise, indifference.
systemd offers benefits to certain classes of end users which could not have
been achieved without *some* kind of change.

We can argue about how well systemd did its job ? I share many of the negative
opinions about it ? but I think you?ll have a very tough time convincing me that
we could have gotten all the benefits without changing the user interface.

Again it comes back to the bag of features: if you didn?t want any of the
features systemd brought, then you may be right to abandon Linux.  (?May?
because it feels like being a one-issue voter, to me.)  It is good that we still
have substantially different OSes to choose from.

And that?s why I use *all* the major OSes and several weird ones besides.  None
of it is perfect, yet it all has its place.

On 04/20/2017 05:55 PM, Warren Young wrote:
> ... I find that most hardware is ready to fall over by the time the 
> CentOS that was installed on it drops out of support anyway.
> ...
James' point isn't the hardware cost, it's the people cost for 
retraining.  In many ways the Fedora treadmill is easier, being that 
there are many more smaller jumps than the huge leap from C6 to C7. For 
the most part, however, I agree with most of your post.  I strongly 
disagree with the paragraph above, though.

I have worked for non-profits for most of my career thus far, which 
spans almost 30 years.  Non-profits by their very nature live on the 
slimmest of margins, and donations of hardware by individuals and 
companies have been in my experience the bread and butter for obtaining 
server-quality hardware.  The typical donation will be at least one or 
two generations old before the non-profit gets it; my current employer 
is just putting in production some IBM BladeCenters with the dual-socket 
Opteron LS20 blades (10+ years old).  Given the spiky workload, these 
blades are suitable for the targeted use, and the electrical 
requirements aren't a problem (I've done the math; it would take ten 
years or more to justify the purchase price of a new blade based on 
power savings alone, and our power is quite inexpensive here).  At least 
I can use very recent blades, and the eBay prices for 5-year-old blades 
are pretty good, so when I need that much more power I can get it.

Oh, and the LS20 blades are built like tanks.  We have a couple hundred 
of them that were donated, and we're going to use them.

For what it's worth, CentOS 7, once installed, works great as long as 
the lack of a GUI console isn't a problem (something with the 
BladeCenter's KVM switch and C7's kernel keeps the keyboard from working
properly).

And don't even get me started on networking equipment, where I still 
have Catalyst 5500-series hardware in production.  (going on 20 years 
old and still trucking!)

And having said that, I just pulled out of service a server for another 
non-profit that had a power supply fan seize.  I posted about moving its 
application Friday.  It is an AMD K6-2/400 with a Western Digital 6GB 
boot drive and a Maxtor 30GB data drive, running Red Hat Linux 5.2.  The 
Antec power supply was put into service in 1999.  It stopped working 
Friday, and could have probably been put back into operation with a new 
power supply without a huge amount of work, but I decided it was time.  
Heh, it was time ten years ago!

The 6GB WD drive was only 19 years old; while I honestly wanted to see 
it turn 20, it was time (power supply glitches caused by overheating of 
the power supply; worst-case for hard disk death in my experience).  
Yeah, 24x7 operation for 19 years with minimal downtime.  I'm going to 
personally put it back into service for hysterical raisins, since the 
VA-503+ board doesn't need re-cap and it runs very well for what it is.  
I'm not sure what I'm going to run on it yet.  (It will be in service 
for the same reasons I'm going to put a Reh CPU280 running UZI280 into 
service.....).

> And that?s why I use *all* the major OSes and several weird ones besides. 
None of it is perfect, yet it all has its place.
I couldn't agree more.