CentOS - Dec 2020 - I'm looking forward to the future of CentOS Stream

On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote:
> And I will repeat that millions of CentOS users found free clone of RHEL
> trustworthy enough to use it for production, even without "official
> endorsement".

Exactly.? That's why it's so weird that those people, today, think that 
CentOS Stream won't be usable, based on their interpretation of the 
official statements from Red Hat.? Red Hat's statements weren't taken 
into consideration before, but now they're a sign of doom?

> If they ... even allowed ANYONE ELSE that was not employed by Red Hat in
> 2014 to even come close to learning the secrets of rebuild, no backlash
> would have happened

I'm going to stop you there, because the CentOS maintainers kept that 
process out of public visibility long before Red Hat was ever involved.? 
If you think users should know more about the process, then you are 
pointing fingers at the *wrong* people.

I don't want this to become a flame war.? So rather than pointing 
fingers, let's focus on the fact that CentOS Stream promises to be 
developed in the open, resolving the problem that you're describing.

Red Hat is fixing the thing you're complaining about.

Red Hat is giving us the thing that has been requested more often, by 
more people, than any other change in CentOS, and the result is that the 
press is full of stories about users being angry, because five people on 
the mailing lists sent a lot of messages.? (About half of the traffic in 
the threads on centos and centos-devel comes from five people, and 
various people replying to them.)

> But no, as soon as Oracle started rebuilding RHEL source code Red Hat
> first made things difficult for everyone to create kernels (source code
> was not srpms anymore but tar?)

You're misinformed.? Kernels are still built from SRPM, but the archive 
used is no longer an upstream release with a series of patches.

The reason for the change is not insidious.? It's unfortunate that the 
pristine source + patches can't be maintained, I agree, but speaking as 
a developer: maintaining hundreds of patches that touch intersecting 
files and rebasing them all when earlier patches are updated is an 
incredibly difficult and time consuming task.? And, if I remember 
correctly, applying all of those patches took almost as long as building 
the kernel.? If it takes that long to just prepare the source code, 
that's a major hit to productivity when a developer needs to work on the 
code or build the SRPM to validate changes.

And, ultimately, there's very little value in shipping those patches 
when the vast majority of them are already in the current version of the 
upstream kernel, and they're merely backported to the older release that 
Red Hat supports.? It's an entirely different story when distributions 
are shipping patches that they don't push upstream, but that's not 
generally what you see with the kernel package.

On 13.12.2020 11:48, Gordon Messmer wrote:
> On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote:
>> And I will repeat that millions of CentOS users found free clone of
RHEL
>> trustworthy enough to use it for production, even without
"official
>> endorsement".
> 
> 
> Exactly.? That's why it's so weird that those people, today, think
that
> CentOS Stream won't be usable, based on their interpretation of the 
> official statements from Red Hat.? Red Hat's statements weren't
taken
> into consideration before, but now they're a sign of doom?
Who exactly said "doom"?

CentOS Stream won't match corresponding stable RHEL version, that's all.
While CentOS was matching it, it was stable enough to use it safely on 
production.

Now that it becomes constant beta, it might be considered less stable, 
all the compatibility arguments have been uttered many a time.

Doom or no doom, everyone decides for oneself.

The primary problem is breach of trust. All the other consequences are 
mostly technical.

-- 
Sincerely,

Konstantin Boyandin
system administrator (ProWide Labs Ltd. - IPHost Network Monitor)

On 13/12/2020 6:48 ?.?., Gordon Messmer wrote:
> Red Hat is giving us the thing that has been requested more often, by 
> more people, than any other change in CentOS, and the result is that 
> the press is full of stories about users being angry, because five 
> people on the mailing lists sent a lot of messages.? (About half of 
> the traffic in the threads on centos and centos-devel comes from five 
> people, and various people replying to them.)
Not really. I am afraid you are missing the point. Also see: 7500 
sysadmins and growing are already explicitly rejecting the change.
> ...but speaking as a developer...
That's the problem: you are speaking as a developer, not as a sysadmin. 
CentOS is clearly for sysadmins, not for developers.

On 13/12/2020 10:22 ?.?., Nicolas Kovacs wrote:
> For the last 16 years, the explicit scope of the CentOS project has 
> been to
> rebuild RHEL "bug by bug". No more no less. A fact that has been
stressed
> repeatedly by the maintainers on this list. So admins all over the world
> trusted this.
>
> Words do have a meaning. 
+1

And what is worse: RH are pushing their users to their competitors 
(read: OL).

RH are pulling their own eyes out... It's a shame.

And all that because they decided to stop supporting a quite extensive 
worldwide amount of orgs and sysadmins who need a safe and dependable 
production OS that does not cost a fortune, although many of those at 
some point in time might become RH support customers!

Now RH earned discontent and distrust from a large part of their FOSS 
community.

Such a sad end for CentOS... (In fact it is an end indeed.)

OL, Rocky Linux (when fully established) and other such projects 
(mentioned in various threads) will gain large parts of this extensive 
group. Some many end up in using CentOS Stream, but the core part of 
this vibrant community will probably lost by RH and the good old CentOS 
group (now in RH). Time will show.

That's a pity, because a large and conscious part of this community 
indeed has some affinity to Karanbir et al, greatly respecting their 
history and efforts...

RH (& CentOS) still has some small window of opportunity to announce 
full support of CentOS 8 to its EOL.

Cheers,
Nick

On 12/13/20 5:48 AM, Gordon Messmer wrote:
> On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote:
>> And I will repeat that millions of CentOS users found free clone of
RHEL
>> trustworthy enough to use it for production, even without
"official
>> endorsement".
> 
> 
> Exactly.? That's why it's so weird that those people, today, think
that
> CentOS Stream won't be usable, based on their interpretation of the
> official statements from Red Hat.? Red Hat's statements weren't
taken
> into consideration before, but now they're a sign of doom?
Do not turn this upside down. Many obviously smarter then me saw this
coming. Us more gullible believed when corporation told us nothing will
change, while they took control of entire project. Death by thousand
cuts is always more preferable by corporations, less bad PR.
And they are not sign of doom but death, but of only this particular
clone, others will take it's mantle. The reason I an agitated is I
believed and with all my hearth supported this project, and now is owned
by heartless profit-driven corporation....
> 
> 
>> If they ... even allowed ANYONE ELSE that was not employed by Red Hat
in
>> 2014 to even come close to learning the secrets of rebuild, no backlash
>> would have happened
> 
> 
> I'm going to stop you there, because the CentOS maintainers kept that
> process out of public visibility long before Red Hat was ever involved.?
> If you think users should know more about the process, then you are
> pointing fingers at the *wrong* people.
> 
> I don't want this to become a flame war.? So rather than pointing
> fingers, let's focus on the fact that CentOS Stream promises to be
> developed in the open, resolving the problem that you're describing.
> 
> Red Hat is fixing the thing you're complaining about.
Don't be silly. They wanted to control the process, and to prevent
anyone else from cutting into their process.
You should read their manifesto for stream, only RH employees will be
able to change that code, others will only be able to report bugs/issues
and to sit and watch what RH does with them.
Only real difference with stream is for those few hundred developers
that are developing software that runs on RHEL so their code can be
deployed as soon as new point release is launched.

And even that RH does more for it's own gain, so that (opensurce?)
projects/software important to RH can be deployed without delay, lest
users ditch RH and go elsewhere.

> 
> Red Hat is giving us the thing that has been requested more often, by
> more people, than any other change in CentOS, and the result is that the
> press is full of stories about users being angry, because five people on
> the mailing lists sent a lot of messages.? (About half of the traffic in
> the threads on centos and centos-devel comes from five people, and
> various people replying to them.)
When people are happy with something they do not voice their content on
the mailing list, mailing list is only to voice your discontent. You
heard about "silent majority", right? Ever though why it is called
that?

> 
> 
>> But no, as soon as Oracle started rebuilding RHEL source code Red Hat
>> first made things difficult for everyone to create kernels (source code
>> was not srpms anymore but tar?)
> 
> 
> You're misinformed.? Kernels are still built from SRPM, but the archive
> used is no longer an upstream release with a series of patches.
Not misinformed, I was part of that discussion 9 years ago, but my
memory was little rusty, but I was part of that discussion and It did
not bother be to track down one of the comments about kernel tarballs
being published as monolith code vs trackable patches, to make problems
for builders of RHEL clones:

"There is the kernel tarball issue. That's understandable, but real.
Red Hat is publishing their kernels as tarballs, rather than as the
vanilla tarball with a list of ordered patches against it. This makes
layering in, or out, specific patches much more awkward. Was Oracle
being any better about this? Does anyone have actual pointers to their
SRPM's?"
(https://lists.centos.org/pipermail/centos-devel/2011-March/063442.html)
> 
> The reason for the change is not insidious.? It's unfortunate that the
> pristine source + patches can't be maintained, I agree, but speaking as
> a developer: maintaining hundreds of patches that touch intersecting
> files and rebasing them all when earlier patches are updated is an
> incredibly difficult and time consuming task.? And, if I remember
> correctly, applying all of those patches took almost as long as building
> the kernel.? If it takes that long to just prepare the source code,
> that's a major hit to productivity when a developer needs to work on
the
> code or build the SRPM to validate changes.
> 
> And, ultimately, there's very little value in shipping those patches
> when the vast majority of them are already in the current version of the
> upstream kernel, and they're merely backported to the older release
that
> Red Hat supports.? It's an entirely different story when distributions
> are shipping patches that they don't push upstream, but that's not
> generally what you see with the kernel package.
Point is that RH DID slow down build of clones due to this change, and
for several months. I can not say about intent, but in hindsight it fits
the pattern, I even thought the same back then (mail in link was direct
reply to my mail).

> 
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos

-- 
Ljubomir Ljubojevic
(Love is in the Air)
PL Computers
Serbia, Europe

StarOS, Mikrotik and CentOS/RHEL/Linux consultant

On Sat, Dec 12, 2020 at 11:48 PM Gordon Messmer <gordon.messmer at
gmail.com>
wrote:
> On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote:
> > And I will repeat that millions of CentOS users found free clone of
RHEL
> > trustworthy enough to use it for production, even without
"official
> > endorsement".
>
>
> Exactly.  That's why it's so weird that those people, today, think
that
> CentOS Stream won't be usable, based on their interpretation of the
> official statements from Red Hat.  Red Hat's statements weren't
taken
> into consideration before, but now they're a sign of doom?
>
>
> > If they ... even allowed ANYONE ELSE that was not employed by Red Hat
in
> > 2014 to even come close to learning the secrets of rebuild, no
backlash
> > would have happened
>
>
> I'm going to stop you there, because the CentOS maintainers kept that
> process out of public visibility long before Red Hat was ever involved.
> If you think users should know more about the process, then you are
> pointing fingers at the *wrong* people.
>
> I don't want this to become a flame war.  So rather than pointing
> fingers, let's focus on the fact that CentOS Stream promises to be
> developed in the open, resolving the problem that you're describing.
>
> Red Hat is fixing the thing you're complaining about.
>
> Red Hat is giving us the thing that has been requested more often, by
> more people, than any other change in CentOS, and the result is that the
> press is full of stories about users being angry, because five people on
> the mailing lists sent a lot of messages.  (About half of the traffic in
> the threads on centos and centos-devel comes from five people, and
> various people replying to them.)
>
>
As one of those "five people" I assure you, this is not just a few
angry
voices. If you, or anyone at Red Hat believe this is the case, you are very
sadly mistaken.

Here is the problem: When IBM took over Red Hat, and hence CentOS, these
words were posted on the CentOS Blog:


"What does this mean for Red Hat?s contributions to the CentOS project?

In short, nothing.

Red Hat always has and will continue to be a champion for open source and
projects like CentOS. IBM is committed to Red Hat?s independence and role
in open source software communities so that we can continue this work
without interruption or changes.

Our mission, governance, and objectives remain the same. We will continue
to execute the existing project roadmap."



This was *last year*. (CF
https://blog.centos.org/2019/07/ibm-red-hat-and-centos/) Note the last
sentence. The roadmap then had CentOS 8 supported through May 2029.

The simple fact is Red Hat reneged on a promise that hordes of us believed
and made a lot of plans on. It is now going to be very expensive, and
stress inducing to have to completely rethink everything we have done, and
are doing.

You damn right we are angry.


And there's *a lot* more than five of us.


> > But no, as soon as Oracle started rebuilding RHEL source code Red Hat
> > first made things difficult for everyone to create kernels (source
code
> > was not srpms anymore but tar?)
>
>
> You're misinformed.  Kernels are still built from SRPM, but the archive
> used is no longer an upstream release with a series of patches.
>
> The reason for the change is not insidious.  It's unfortunate that the
> pristine source + patches can't be maintained, I agree, but speaking as
> a developer: maintaining hundreds of patches that touch intersecting
> files and rebasing them all when earlier patches are updated is an
> incredibly difficult and time consuming task.  And, if I remember
> correctly, applying all of those patches took almost as long as building
> the kernel.  If it takes that long to just prepare the source code,
> that's a major hit to productivity when a developer needs to work on
the
> code or build the SRPM to validate changes.
>
> And, ultimately, there's very little value in shipping those patches
> when the vast majority of them are already in the current version of the
> upstream kernel, and they're merely backported to the older release
that
> Red Hat supports.  It's an entirely different story when distributions
> are shipping patches that they don't push upstream, but that's not
> generally what you see with the kernel package.
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

-- 

*Matt Phelps*

*Information Technology Specialist, Systems Administrator*

(Computation Facility, Smithsonian Astrophysical Observatory)

Center for Astrophysics | Harvard & Smithsonian


60 Garden Street | MS 39 | Cambridge, MA 02138
email: mphelps at cfa.harvard.edu


cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
<http://cfa.harvard.edu/twitter> | YouTube
<http://cfa.harvard.edu/youtube>
| Newsletter <http://cfa.harvard.edu/newsletter>