CentOS - Jun 2020 - Blog article about the state of CentOS

On 6/17/20 8:06 AM, Simon Matter via CentOS wrote:
>> Hi,
>>
>> I just read this blog article from austrian Linux expert Michael
Kofler.
>> For
>> those among you who don't know the guy, he's my home
country's number one
>> Linux
>> expert (known as "der Kofler") and most notably the author of
a series of
>> excellent books about Linux over the last 25 years.
>>
>> https://kofler.info/centos-8-wertlose-langzeitunterstuetzung/
>>
>> Disclaimer : I've been a CentOS user (and fan) since 4.x, I'm
using it on
>> all
>> my servers, and yes, I know the difference between upstream RHEL and
>> CentOS.
>>
>> The article is in german, but the statistics graph is eloquent enough
for
>> the
>> non-german-speaking users. It focuses on updates for CentOS 8, and more
>> exactly
>> the extended periods of time where there have been no updates
available.
>>
>> The author's theory ("unspoken truth"): while it's a
positive thing that
>> Red
>> Hat is sponsoring CentOS, the amount of sponsoring is just insufficient
>> enough
>> so that the product is "starved to death" by Red Hat (e. g.
IBM) to
>> encourage
>> users to move to RHEL.
> 
> I think if Red Hat really wanted to improve the situation, they could
> integrate the building of CentOS into the EL build system to produce both
> versions, RHEL and CentOS, at the same time. In the end 99% of the bits
> are the same anyway. If the delay of CentOS builds is really wanted by Red
> Hat, it would be nice of them to speak it out - and change the name to
> COS, because the ent is not true anymore :-)
> 
> Up to now I thought the big delay with 8 is more an accident than wanted.
> Would be nice to hear what Red Hat says about it. Maybe the problem is not
> known well enough in the Red Hat universe.
>

No one is trynig to make anything slower.

And CentOS Stream 'is' going to be how RHEL is developed in the future.
 So, all this is happening.

But modules introduced in RHEL 8 requires a who new build system (as
koji set up) and a whole new module build back end (MBS).

If you would rather use Oracle for your open source needs .. well, that
is a decision you can make and be responsible for.

If you would instead have feedback directly into the RHEL development
process as a community .. then CentOS is where you want to be.

This stuff is open source, and you are all gown ups who can make your
own decisions.

I can assure you .. I am working my butt off everyday to make CentOS
Linux the best it can be.  If you want to compare what the CentOS team
(a small team) can do compared to Oracle (a tmulti billin dollar
corporation who bought Sun Microsystems .. took over Java and Open
Office, etc)  .. well, we can not provide the resources they can provide.

But Red hat heard the community .. that the community and Red Hat
customers want more direct input into RHEL development.  And Red Hat is
taking action to open up RHEL development in CentOS Stream.

You get to decide who you trust.

Thanks,
Johnny hUghes



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20200617/0a228579/attachment.sig>

> On Jun 17, 2020, at 9:15 AM, Johnny Hughes <johnny at centos.org>
wrote:
> 
> On 6/17/20 8:06 AM, Simon Matter via CentOS wrote:
>>> Hi,
>>> 
>>> I just read this blog article from austrian Linux expert Michael
Kofler.
>>> For
>>> those among you who don't know the guy, he's my home
country's number one
>>> Linux
>>> expert (known as "der Kofler") and most notably the
author of a series of
>>> excellent books about Linux over the last 25 years.
>>> 
>>> https://kofler.info/centos-8-wertlose-langzeitunterstuetzung/
>>> 
>>> Disclaimer : I've been a CentOS user (and fan) since 4.x,
I'm using it on
>>> all
>>> my servers, and yes, I know the difference between upstream RHEL
and
>>> CentOS.
>>> 
>>> The article is in german, but the statistics graph is eloquent
enough for
>>> the
>>> non-german-speaking users. It focuses on updates for CentOS 8, and
more
>>> exactly
>>> the extended periods of time where there have been no updates
available.
>>> 
>>> The author's theory ("unspoken truth"): while
it's a positive thing that
>>> Red
>>> Hat is sponsoring CentOS, the amount of sponsoring is just
insufficient
>>> enough
>>> so that the product is "starved to death" by Red Hat (e.
g. IBM) to
>>> encourage
>>> users to move to RHEL.
>> 
>> I think if Red Hat really wanted to improve the situation, they could
>> integrate the building of CentOS into the EL build system to produce
both
>> versions, RHEL and CentOS, at the same time. In the end 99% of the bits
>> are the same anyway. If the delay of CentOS builds is really wanted by
Red
>> Hat, it would be nice of them to speak it out - and change the name to
>> COS, because the ent is not true anymore :-)
>> 
>> Up to now I thought the big delay with 8 is more an accident than
wanted.
>> Would be nice to hear what Red Hat says about it. Maybe the problem is
not
>> known well enough in the Red Hat universe.
>> 
> 
> 
> No one is trynig to make anything slower.
> 
> And CentOS Stream 'is' going to be how RHEL is developed in the
future.
> So, all this is happening.
> 
> But modules introduced in RHEL 8 requires a who new build system (as
> koji set up) and a whole new module build back end (MBS).
> 
> If you would rather use Oracle for your open source needs .. well, that
> is a decision you can make and be responsible for.
> 
> If you would instead have feedback directly into the RHEL development
> process as a community .. then CentOS is where you want to be.
> 
> This stuff is open source, and you are all gown ups who can make your
> own decisions.
> 
> I can assure you .. I am working my butt off everyday to make CentOS
> Linux the best it can be.  If you want to compare what the CentOS team
> (a small team) can do compared to Oracle (a tmulti billin dollar
> corporation who bought Sun Microsystems .. took over Java and Open
> Office, etc)  .. well, we can not provide the resources they can provide.
> 
> But Red hat heard the community .. that the community and Red Hat
> customers want more direct input into RHEL development.  And Red Hat is
> taking action to open up RHEL development in CentOS Stream.
> 
> You get to decide who you trust.
> 
Thank you, Johnny!

I was going to reply to original post that: Nobody ever promised to pay CentOS
team to do what they do. That was my understanding always.

Changes in the OS, new components (do not mean systemd explicitly), and how new
release is built may or may not be part of longer/bigger plan of RHEL owner.

Without regard to the above, RedHat always was following meticulously the letter
of GNU license. And the reputation IBM has (in my book), does not raise red flag
after they bought out RedHat. Time will show, but I?m sure with acquisition of
RedHat by IBM ?nothing changed? for open source community in general, and CentOS
existence in particular. Not dramatically at least. And someone mentioned
Oracle, and I agree, Oracle has quite opposite reputation in my boot to that of
IBM.

Thanks, Johnny, very much again!!

Note: I?m just humble outsider, not CentOS project member.

Valeri
> Thanks,
> Johnny hUghes
> 
> 
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos

Hi Johnny,
thank you for your and all centos team works.

Many of us know how much work is needed for building new releases and
maintaining C6 and C7, plus CentOS Stream and modules (Appstream). This is
a huge work for a small team. Again thank you.

For me OL is not an alternative.

As reported in my previous message I'm not worried about how much time is
required to build the new (major/minor) release, it will be ready when it
will be. My major concern is about the "security update blackout" that
take
long as the build process.

I would ask to you:

1. Why all security fix are stopped when a new release building process is
started? There is a way or possibility to run the two process in parallel?

2. When a build process is started and a security fix released there is a
way for your team to "suspend" the building process, release security
updates (for 6/7.x or 8.1) and resume the builing process? I think that
many users (included me) will have less disappointment having security
updates instead of receiving a  "signal lost" when building process
takes
its way.

Thank you for your time.

Alessandro.

Il Mer 17 Giu 2020, 16:15 Johnny Hughes <johnny at centos.org> ha scritto:
> On 6/17/20 8:06 AM, Simon Matter via CentOS wrote:
> >> Hi,
> >>
> >> I just read this blog article from austrian Linux expert Michael
Kofler.
> >> For
> >> those among you who don't know the guy, he's my home
country's number
> one
> >> Linux
> >> expert (known as "der Kofler") and most notably the
author of a series
> of
> >> excellent books about Linux over the last 25 years.
> >>
> >> https://kofler.info/centos-8-wertlose-langzeitunterstuetzung/
> >>
> >> Disclaimer : I've been a CentOS user (and fan) since 4.x,
I'm using it
> on
> >> all
> >> my servers, and yes, I know the difference between upstream RHEL
and
> >> CentOS.
> >>
> >> The article is in german, but the statistics graph is eloquent
enough
> for
> >> the
> >> non-german-speaking users. It focuses on updates for CentOS 8, and
more
> >> exactly
> >> the extended periods of time where there have been no updates
available.
> >>
> >> The author's theory ("unspoken truth"): while
it's a positive thing that
> >> Red
> >> Hat is sponsoring CentOS, the amount of sponsoring is just
insufficient
> >> enough
> >> so that the product is "starved to death" by Red Hat (e.
g. IBM) to
> >> encourage
> >> users to move to RHEL.
> >
> > I think if Red Hat really wanted to improve the situation, they could
> > integrate the building of CentOS into the EL build system to produce
both
> > versions, RHEL and CentOS, at the same time. In the end 99% of the
bits
> > are the same anyway. If the delay of CentOS builds is really wanted by
> Red
> > Hat, it would be nice of them to speak it out - and change the name to
> > COS, because the ent is not true anymore :-)
> >
> > Up to now I thought the big delay with 8 is more an accident than
wanted.
> > Would be nice to hear what Red Hat says about it. Maybe the problem is
> not
> > known well enough in the Red Hat universe.
> >
>
>
> No one is trynig to make anything slower.
>
> And CentOS Stream 'is' going to be how RHEL is developed in the
future.
>  So, all this is happening.
>
> But modules introduced in RHEL 8 requires a who new build system (as
> koji set up) and a whole new module build back end (MBS).
>
> If you would rather use Oracle for your open source needs .. well, that
> is a decision you can make and be responsible for.
>
> If you would instead have feedback directly into the RHEL development
> process as a community .. then CentOS is where you want to be.
>
> This stuff is open source, and you are all gown ups who can make your
> own decisions.
>
> I can assure you .. I am working my butt off everyday to make CentOS
> Linux the best it can be.  If you want to compare what the CentOS team
> (a small team) can do compared to Oracle (a tmulti billin dollar
> corporation who bought Sun Microsystems .. took over Java and Open
> Office, etc)  .. well, we can not provide the resources they can provide.
>
> But Red hat heard the community .. that the community and Red Hat
> customers want more direct input into RHEL development.  And Red Hat is
> taking action to open up RHEL development in CentOS Stream.
>
> You get to decide who you trust.
>
> Thanks,
> Johnny hUghes
>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

Once upon a time, Alessandro Baggi <alessandro.baggi at gmail.com>
said:
> As reported in my previous message I'm not worried about how much time
is
> required to build the new (major/minor) release, it will be ready when it
> will be. My major concern is about the "security update blackout"
that take
> long as the build process.
I'm not involved in building CentOS, but the issue is that it is a
rebuild of upstream.  When RHEL 8.2 is released, there are no more
upstream updates released for RHEL 8.1; they are all on top of the RHEL
8.2 release.  So, until the time that CentOS can rebuild RHEL 8.2 and
make a new CentOS release, there can't be any updates for CentOS 8.1.

RHEL 8 introduced modules, which complicated the build system and
required new tooling, so CentOS has had a bunch of "under the hood"
work
to catch up.  Hopefully, once that's ironed out, the gap between a RHEL
8.x release and the corresponding CentOS release will drop.

-- 
Chris Adams <linux at cmadams.net>

On Wed, 17 Jun 2020 at 13:11, Alessandro Baggi
<alessandro.baggi at gmail.com> wrote:
>
> Hi Johnny,
> thank you for your and all centos team works.
>
> Many of us know how much work is needed for building new releases and
> maintaining C6 and C7, plus CentOS Stream and modules (Appstream). This is
> a huge work for a small team. Again thank you.
>
> For me OL is not an alternative.
>
> As reported in my previous message I'm not worried about how much time
is
> required to build the new (major/minor) release, it will be ready when it
> will be. My major concern is about the "security update blackout"
that take
> long as the build process.
>
> I would ask to you:
>
> 1. Why all security fix are stopped when a new release building process is
> started? There is a way or possibility to run the two process in parallel?
>
Most of the security fixes after the minor release are based off being
built with the code from that minor release. Sometimes the code may
even not compile without the rest of the packages involved. Deciding
which packages can and can not be affected by the minor release can be
a full time job.

> 2. When a build process is started and a security fix released there is a
> way for your team to "suspend" the building process, release
security
> updates (for 6/7.x or 8.1) and resume the builing process? I think that
> many users (included me) will have less disappointment having security
> updates instead of receiving a  "signal lost" when building
process takes
> its way.
>
It would require a second build system which built against the older
set of packages, it would also require someone to edit those packages
so they can be replaced by later released ones after the main release
is done. It also requires work on making sure conflicts and broken
packages are dealt with.

A lot of this is sausage making.. We all want the end product of a
cooked sausage.. but there is a lot of messy work which adding more
people to doesn't help unless you can add a lot of other things which
are supposed to appear ex-nihilo and fully formed.  You can have a lot
of people volunteer to stand around but there are only so many places
to push meat on one side, turn cranks in the middle and one tube which
the mess gets stuffed into.  Every extra thing requires months and
months of work and preparation to set up while still trying to make
the other items. However the only time people seem to get bothered
enough to even try to set up their own build system and find out how
much mess it is.. is after the current release is delayed.

> Thank you for your time.
>
> Alessandro.
>
> Il Mer 17 Giu 2020, 16:15 Johnny Hughes <johnny at centos.org> ha
scritto:
>
> > On 6/17/20 8:06 AM, Simon Matter via CentOS wrote:
> > >> Hi,
> > >>
> > >> I just read this blog article from austrian Linux expert
Michael Kofler.
> > >> For
> > >> those among you who don't know the guy, he's my home
country's number
> > one
> > >> Linux
> > >> expert (known as "der Kofler") and most notably the
author of a series
> > of
> > >> excellent books about Linux over the last 25 years.
> > >>
> > >> https://kofler.info/centos-8-wertlose-langzeitunterstuetzung/
> > >>
> > >> Disclaimer : I've been a CentOS user (and fan) since 4.x,
I'm using it
> > on
> > >> all
> > >> my servers, and yes, I know the difference between upstream
RHEL and
> > >> CentOS.
> > >>
> > >> The article is in german, but the statistics graph is
eloquent enough
> > for
> > >> the
> > >> non-german-speaking users. It focuses on updates for CentOS
8, and more
> > >> exactly
> > >> the extended periods of time where there have been no updates
available.
> > >>
> > >> The author's theory ("unspoken truth"): while
it's a positive thing that
> > >> Red
> > >> Hat is sponsoring CentOS, the amount of sponsoring is just
insufficient
> > >> enough
> > >> so that the product is "starved to death" by Red
Hat (e. g. IBM) to
> > >> encourage
> > >> users to move to RHEL.
> > >
> > > I think if Red Hat really wanted to improve the situation, they
could
> > > integrate the building of CentOS into the EL build system to
produce both
> > > versions, RHEL and CentOS, at the same time. In the end 99% of
the bits
> > > are the same anyway. If the delay of CentOS builds is really
wanted by
> > Red
> > > Hat, it would be nice of them to speak it out - and change the
name to
> > > COS, because the ent is not true anymore :-)
> > >
> > > Up to now I thought the big delay with 8 is more an accident than
wanted.
> > > Would be nice to hear what Red Hat says about it. Maybe the
problem is
> > not
> > > known well enough in the Red Hat universe.
> > >
> >
> >
> > No one is trynig to make anything slower.
> >
> > And CentOS Stream 'is' going to be how RHEL is developed in
the future.
> >  So, all this is happening.
> >
> > But modules introduced in RHEL 8 requires a who new build system (as
> > koji set up) and a whole new module build back end (MBS).
> >
> > If you would rather use Oracle for your open source needs .. well,
that
> > is a decision you can make and be responsible for.
> >
> > If you would instead have feedback directly into the RHEL development
> > process as a community .. then CentOS is where you want to be.
> >
> > This stuff is open source, and you are all gown ups who can make your
> > own decisions.
> >
> > I can assure you .. I am working my butt off everyday to make CentOS
> > Linux the best it can be.  If you want to compare what the CentOS team
> > (a small team) can do compared to Oracle (a tmulti billin dollar
> > corporation who bought Sun Microsystems .. took over Java and Open
> > Office, etc)  .. well, we can not provide the resources they can
provide.
> >
> > But Red hat heard the community .. that the community and Red Hat
> > customers want more direct input into RHEL development.  And Red Hat
is
> > taking action to open up RHEL development in CentOS Stream.
> >
> > You get to decide who you trust.
> >
> > Thanks,
> > Johnny hUghes
> >
> >
> >
> > _______________________________________________
> > CentOS mailing list
> > CentOS at centos.org
> > https://lists.centos.org/mailman/listinfo/centos
> >
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos


-- 
Stephen J Smoogen.

Hi Johnny,

Am Mi., 17. Juni 2020 um 16:16 Uhr schrieb Johnny Hughes <johnny at
centos.org
>:
> [...]
> No one is trynig to make anything slower.
>
This is good to hear but ...
> [...]
> I can assure you .. I am working my butt off everyday to make CentOS
> Linux the best it can be.  If you want to compare what the CentOS team
> (a small team) can do compared to Oracle (a tmulti billin dollar
> corporation who bought Sun Microsystems .. took over Java and Open
> Office, etc)  .. well, we can not provide the resources they can provide.
> [...]

... then I'm missing some kind of strategy on how to improve things (or
maybe I simply don't know they exist). Without having a detailed background
about the project setup and the way the small team works, I would at a
first glance assume that there are two options to improve. One would be to
make the team bigger (as you said, it's open and I'm pretty sure there
are
enough volunteers) or a second one, by increasing the automation level for
the CentOS production.

What do you think and are there discussions in the core CentOS community to
go either one of these routes?

Regards Thomas
-- 
Linux ... enjoy the ride!

On 6/17/20 12:11 PM, Alessandro Baggi wrote:
> Hi Johnny,
> thank you for your and all centos team works.
> 
> Many of us know how much work is needed for building new releases and
> maintaining C6 and C7, plus CentOS Stream and modules (Appstream). This is
> a huge work for a small team. Again thank you.
> 
> For me OL is not an alternative.
> 
> As reported in my previous message I'm not worried about how much time
is
> required to build the new (major/minor) release, it will be ready when it
> will be. My major concern is about the "security update blackout"
that take
> long as the build process.
> 
> I would ask to you:
> 
> 1. Why all security fix are stopped when a new release building process is
> started? There is a way or possibility to run the two process in parallel?
So .. when a point release happens .. say 7.8 to 7.9 (just an example ..
could be 6.10 to 6.11 or 8.1 to 8.2, etc)

Those packages are built against EACH other, one at a time.  Once we
build the new gcc, new kernel, and new glibc (if they are reqruies) ..
then all the OTHER updated packages are built against those new
libraries.. they therefore need those NEW shared libraries to run.  So
the new files have to be released as a set, not individually.
> 
> 2. When a build process is started and a security fix released there is a
> way for your team to "suspend" the building process, release
security
> updates (for 6/7.x or 8.1) and resume the builing process? I think that
> many users (included me) will have less disappointment having security
> updates instead of receiving a  "signal lost" when building
process takes
> its way.
It makes no difference if the update is a bugfix update or a security
update.  If 500 packages get released at the same time, they have to be
built in a specific order in order to match how they were built in RHEL.

We have to build them, one at a time, then individually test them to
make sure they LINK against the proper new libraries and not older
libraries.

Also any UPDATES released to the new version , after RHEL does the point
release (so updates FOR 7.9 after the 7.9 release) need to wait until
the 7.9 release is done and tested to be built .. as they were built
against RHEL 7.9 and not RHEL 7.8

So, you can't just build items out of order at point release time.


We have to build the 500 packages , in a specific order. We then have to
test the packages, and usually rebuild several of them again for bad
links, etc.

This is the process that takes time .. testing and getting the proper
links to the proper shared libraries.

If we quickly release bad files .. then we have to rebuild them and
re-release them with different versions that RHEL has (because they have
to replace our previuosly BAD release).  That is not good for anyone.

Hopefully this answers your question.
<snip>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20200619/12dff85f/attachment.sig>

On 6/18/20 11:12 AM, Thomas Bendler wrote:
> Hi Johnny,
> 
> Am Mi., 17. Juni 2020 um 16:16 Uhr schrieb Johnny Hughes <johnny at
centos.org
>> :
> 
>> [...]
>> No one is trynig to make anything slower.
>>
> 
> This is good to hear but ...
> 
>> [...]
>> I can assure you .. I am working my butt off everyday to make CentOS
>> Linux the best it can be.  If you want to compare what the CentOS team
>> (a small team) can do compared to Oracle (a tmulti billin dollar
>> corporation who bought Sun Microsystems .. took over Java and Open
>> Office, etc)  .. well, we can not provide the resources they can
provide.
>> [...]
> 
> 
> ... then I'm missing some kind of strategy on how to improve things (or
> maybe I simply don't know they exist). Without having a detailed
background
> about the project setup and the way the small team works, I would at a
> first glance assume that there are two options to improve. One would be to
> make the team bigger (as you said, it's open and I'm pretty sure
there are
> enough volunteers) or a second one, by increasing the automation level for
> the CentOS production.
> 
> What do you think and are there discussions in the core CentOS community to
> go either one of these routes?
> 
While both of those would be helpful .. neither will really SOLVE the
issue.  This is an iterative process.  You build something, test it,
rebuild all the bad parts, test it again .. rinse, repeat

We usually have the first build 3 or so days after they release.

Many things build out of order.  They get tested and rebuilt.

> Regards Thomas
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20200619/0cfbbc1e/attachment.sig>