CentOS - Dec 2020 - I'm looking forward to the future of CentOS Stream

Personally, I think that changing focus on CentOS Stream is going to 
make CentOS (and maybe even RHEL) better in the same way and for the 
same reasons that Fedora is a better distribution than Red Hat Linux 
was.  CentOS Stream should fix the biggest problems that CentOS has had 
in the past, providing a reliable, free LTS distribution with community 
participation.

Having read the announcement, along with hundreds of reactions in blogs, 
forums, and mailing lists, I am of the opinion that all (or a reasonable 
approximation thereof) of the vocal concern is the result of the 
overloaded term "stable" as it applies to software distributions.  If
we
imagine a spectrum of individuals in which one end of the spectrum is 
individuals whose primary occupation is in release engineering for 
software distributions and the other end is individuals who primarily 
consume software distributions, I would expect that individuals on one 
end to mostly use the term "stable" in the sense of compatibility and 
semantic versioning, and the other end to use the same term in the sense 
of having fewer bugs.  The use of that word causes people at one end of 
the spectrum to infer a completely different message than people at the 
other end intend to communicate.

If we never use the ambiguous term "stable" and instead use the terms 
compatibility and reliability (the two common meanings of "stable" at 
different ends of the spectrum), I think that various aspects of CentOS 
Stream will be better than CentOS, or the same as CentOS.

With respect to compatibility:

I think most developers are familiar with semantic versioning.  Semantic 
versioning is used by some applications and libraries to indicate the 
nature and extent of changes.  The version is presented numerically as 
Major.Minor.Revision.  When new interfaces are added, the minor number 
is increased.  Those changes don?t affect backward compatibility.  When 
individual interfaces are changed or removed, the major number is 
increased.  Those changes aren?t backward compatible.  That allows 
consumers to infer that if an application is compatible with 8.1, then 
it is also compatible with 8.2 and later, but might not be compatible 
with 9.0.

Red Hat Enterprise Linux applies that concept to the entire software 
distribution, providing independent software vendors a convenient means 
of communicating their compatibility.  If they?ve tested their 
application on RHEL 8.2, then any RHEL 8 host patched to at least that 
release is expected to run the application.  Moreover, Red Hat will 
continue to publish security patches to each given minor release?s 
channel, allowing consumers to "pin" a host to a minor release.  Those
hosts will not receive feature updates, but will mitigate vulnerabilities.

CentOS Stream isn?t going to feature minor releases, and isn?t going to 
provide semantic versioning of the distribution.  The same application 
that the vendor has validated on RHEL 8.2 will run on a fully patched 
CentOS Stream 8 host, but might not run on a host that isn?t fully 
patched.  On the surface, it appears that CentOS users will lose the 
convenience provided by semantic versioning.  I would simply point out 
that the CentOS developers have never supported running CentOS in any 
state other than fully patched.  They don?t publish security information 
in the package repositories, and they don?t support any means of pinning 
a host to a minor release.

For practical purposes, CentOS Stream will need to be fully patched for 
compatibility purposes, just like CentOS is, and will be equally suited 
for production purposes.

To put a really find point on that: Semantic versioning is only 
meaningful for hosts that are not fully patched.  A fully patched host 
is expected to be compatible with any application validated for that 
major release.

With respect to reliability:

Many of the people concerned about the change in focus refer to CentOS 
Stream as a "beta" for RHEL.  That is not how Red Hat or the CentOS 
maintainers describe CentOS Stream( anywhere that I've seen), and I 
think it ignores most of the development, testing, and distribution 
pipeline.

At the risk of oversimplifying that pipeline a whole lot, in the future 
Free Software will pass through several stages on the way to RHEL:

     Stage 1: (Software Development) The majority of development and 
testing is done in individual upstream projects, outside of Red Hat.

     Stage 2: (Release Development, aka Rawhide) The initial work to 
build and integrate individual packages with the rest of the software 
distribution is done in what is essentially a development branch of the 
software distribution.

     Stage 3: (Stable[1], aka Fedora) Packages that have passed through 
review and QA are published for general use.  There is no minor release, 
as major releases occur every 6 months and are supported for only 13 
months, anyway.  Compatibility is maintained by prohibiting significant 
version changes for applications and libraries "whenever possible." 
Users expect no new features during a release, but no breaking changes 
either.

     Stage 4: (Long Term Support, aka CentOS Stream)  Packages that 
CentOS Stream includes from Fedora have proven reliable in a variety of 
workloads managed by many users of Fedora.  These packages are expected 
to be very reliable as a result of testing by their developers, by 
package maintainers, and by real-world users.  They are included in 
CentOS Stream when they are ready.

     Stage 5: (Long Term Support with Semantic Versioning of the OS, aka 
Red Hat Enterprise Linux)  Packages that RHEL includes from CentOS 
Stream have a similar level of QA, but package updates that introduce 
new features and interfaces are accepted only once every six months when 
Red Hat publishes a minor release.

There?s a lot of concern that CentOS Stream will offer less reliability 
than RHEL, but there?s currently no reason to believe that will be true. 
  There is no evidence that the minor releases that are part of the RHEL 
lifecycle improve reliability, and as far as I know that's not the 
reason they're used.  Their function is to offer semantic versioning of 
the OS.

CentOS Stream will probably offer the same level of compatibility and 
reliability that CentOS does today, and should be equally appropriate or 
inappropriate for production use in the future as CentOS is now in that 
respect.  And that brings me to the one aspect where I think CentOS 
Stream will resolve the one major, glaring problem that CentOS has 
today, that most users ignore: Security.

With respect to security:

Today, CentOS is a release stage after Stage 5 described above.  The 
CentOS maintainers begin work on a minor release after that release is 
available to RHEL consumers, and the process of rebuilding those 
packages is often very time consuming.  CentOS maintainers have to 
reverse-engineer the exact order in which packages are built, with the 
exact set of installed and available packages in the build environment 
in order to ensure that the resulting package actually uses the same 
interfaces that RHEL?s packages do.  All packages require that ordering 
and build environment matching, but most packages are published in small 
sets and ordering is much easier to identify than it is when they are 
published in a large batch.

As a result, security updates can?t be published for CentOS while the 
maintainers are rebuilding the minor release, because the build 
dependencies aren?t available yet.  Those windows occur every six 
months, and are typically a month or more in length. [2]

Today, CentOS users accept the risk that for roughly two months out of 
the year, their systems may have known vulnerabilities with no patch to 
remediate the problem.  Personally, I think that?s a huge risk that 
needs to be weighed against the costs of RHEL licenses whenever CentOS 
is used in production.

The good news is that CentOS Stream looks like it won't have that 
problem.  CentOS Stream updates still won?t be prepared early, while 
vulnerability details are embargoed, but there aren?t any windows in 
which CentOS Stream can?t immediately begin work on preparing updates 
once the embargo ends.  That means that CentOS Stream will be as secure 
as CentOS is today in between minor updates, and significantly more 
secure than CentOS is today while its maintainers prepare minor releases.

The trade-off of significantly improving security in exchange for giving 
up semantic versioning of the OS is a huge improvement for production 
purposes.

In addition, the announcement of the change in focus indicates that 
Fedora maintainers should have much better visibility into work going on 
within the RHEL release engineering process, and more opportunity to 
participate in that work, and I look forward to that.  CentOS?s big 
security problem has always been compounded by the lack of any external 
visibility into the process.

Based on the information available today, I expect CentOS to be a very 
reliable, reasonably secure distribution of GNU/Linux with Long Term 
Support.  And judging by Red Hat?s mention that Facebook?s internal 
groups either are already using an internally curated OS built from 
CentOS Stream, or will be using it soon, I think I?m not alone in 
believing that.


1: I?m using the term "stable" here because I expect it to have both 
common meanings: compatibility isn?t broken within a release, and the 
software is expected to be reliable.

2: https://en.wikipedia.org/wiki/CentOS#CentOS_version_8

On 11.12.2020 08:25, Gordon Messmer wrote:
[...]
> For practical purposes, CentOS Stream will need to be fully patched for 
> compatibility purposes, just like CentOS is, and will be equally suited 
> for production purposes.
Allow me to disagree. We both trust Chris Wright's words, don't we? CTO 
won't lie. Citing him:

"To be exact, CentOS Stream is an upstream development platform for 
ecosystem developers. It will be updated several times a day. This is 
not a production operating system."

[...]
> Based on the information available today, I expect CentOS to be a very 
> reliable, reasonably secure distribution of GNU/Linux with Long Term 
> Support.? And judging by Red Hat?s mention that Facebook?s internal 
> groups either are already using an internally curated OS built from 
> CentOS Stream, or will be using it soon, I think I?m not alone in 
> believing that.
I do not wish to argue with all your statements. Mostly they look 
reasonable. However, there's an unpredictable variable in this equation, 
namely RH.

The major problem here is the breach of trust. A year ago RH's CTO is 
singing charming songs that CentOS won't go, now we see an abrupt 
direction change. This time, CTO keeps silent (I wonder why).

Also, there's change in patterns. With CentOS, I reduce updates to 
minimal ones. That's significant: the management doesn't like the idea 
that updates can be applied daily, and glitches may happen at any 
moment. The management prefers the known devil.

With current CentOS life cycle the number of upgrades is typically 
small. And even if I reduce the number of CentOS Stream upgrades to 
minimal one, the base advantage of CentOS is lost: predictability. At 
any given moment I could be sure that it has the same quirks and bugs 
the matching RHEL has.

CentOS Stream has its advantages and use cases. The problem is, no one 
cared to estimate what use cases of majority of current CentOS users are.

Damn, RH could at least bring formal apologies for changing the promised 
lifecycle. Instead we see the typical marketing blah-blah-blah of how 
that would benefit everyone. Nothing shows better the actual RH attitude 
towards the CentOS community.

-- 
Sincerely,

Konstantin Boyandin
system administrator (ProWide Labs Ltd. - IPHost Network Monitor)

Le 11/12/2020 ? 02:25, Gordon Messmer a ?crit?:
> Personally, I think that changing focus on CentOS Stream is going to make
> CentOS (and maybe even RHEL) better in the same way and for the same
reasons
> that Fedora is a better distribution than Red Hat Linux was.
Using Fedora on production servers is like climbing without a rope.

It's possible. I've even seen some folks do it.

:o)

Cheers from a climber

-- 
Microlinux - Solutions informatiques durables
7, place de l'?glise - 30730 Montpezat
Site : https://www.microlinux.fr
Blog : https://blog.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32
Mob. : 06 51 80 12 12

On Thu, Dec 10, 2020 at 05:25:16PM -0800, Gordon Messmer
wrote:
> ... snip ...
> Today, CentOS is a release stage after Stage 5 described above.  The CentOS
> maintainers begin work on a minor release after that release is available
to
> RHEL consumers, and the process of rebuilding those packages is often very
> time consuming.  CentOS maintainers have to reverse-engineer the exact
order
> in which packages are built, with the exact set of installed and available
> packages in the build environment in order to ensure that the resulting
> package actually uses the same interfaces that RHEL???s packages do.  All
> packages require that ordering and build environment matching, but most
> packages are published in small sets and ordering is much easier to
identify
> than it is when they are published in a large batch.
> 
> As a result, security updates can???t be published for CentOS while the
> maintainers are rebuilding the minor release, because the build
dependencies
> aren???t available yet.  Those windows occur every six months, and are
> typically a month or more in length. [2]
> 
> Today, CentOS users accept the risk that for roughly two months out of the
> year, their systems may have known vulnerabilities with no patch to
> remediate the problem.  Personally, I think that???s a huge risk that needs
> to be weighed against the costs of RHEL licenses whenever CentOS is used in
> production.
> 
> The good news is that CentOS Stream looks like it won't have that
problem.
> CentOS Stream updates still won???t be prepared early, while vulnerability
> details are embargoed, but there aren???t any windows in which CentOS
Stream
> can???t immediately begin work on preparing updates once the embargo ends.
> That means that CentOS Stream will be as secure as CentOS is today in
> between minor updates, and significantly more secure than CentOS is today
> while its maintainers prepare minor releases.
While I agree with your entire post, Gordon, this specific point I think is
the most critical. In our environment, we already need to look to the
Continuous Release repos to get critical security updates during this
embargo period. I'm betting Stream will be no less well vetted than the CR
repos, and likely will be better. In any case, the burden for tracking down
the updates will be much less with Stream: we'll just get the packages
through our normal channels, rather than going on a hunt through CVEs and
Bugzilla, then temporarily enabling the CR repos for just the period of
time when we need to get the updates before disabling them again.
> ... snip ...
-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department (UW Medicine), System Administrator
-- Foege Building S046, (206)-685-7354
-- Pronouns: He/Him/His