CentOS - Dec 2020 - [EXT] Re: https://blog.centos.org/2020/12/future-is-centos-stream/

On Wed, Dec 9, 2020 at 7:21 AM Phil Perry <pperry at elrepo.org> wrote:
> On 09/12/2020 03:26, Brendan Conoboy wrote:
> > On Tue, Dec 8, 2020 at 4:19 PM Pete Biggs <pete at biggs.org.uk>
wrote:
> >
> >> On Tue, 2020-12-08 at 17:54 -0500, Matthew Miller wrote:
> >>> On Tue, Dec 08, 2020 at 03:15:17PM +0000, Pete Biggs wrote:
> >>
> >>>> "CentOS will become the developer playground"
> >>>
> >>> This one is categorically not the case. Even Fedora isn't
a developer
> >>> playground. Everything landing in CentOS Stream is actually
*planned*
> >> (with
> >>> emphasis intentional) to go in a future RHEL release.
> >>
> >> It's all the talk of SIGs and developing and testing and that
Stream
> >> will be the centerpiece of that. That's what I meant.
> >>
> >
> > I don't know if I'd call SIGs a playground, but they're
certainly an
> > important venue for communities to explore variations.
> >
> >
> >>> Previously, all the development around RHEL releases was done
in
> secret,
> >> in
> >>> the Red Hat black box. Now it's out of the box and can be
watched.
> There
> >> may
> >>> be some launch pains, but I expect the average quality of an
update
> >> hitting
> >>> CentOS Stream to be very high.
> >>
> >> I don't get that from the documents released today.  If Stream
is *not*
> >> a test-bed, then surely the code that appears in Stream must be
fully
> >> formed in secret behind the scenes first. Yes, it will appear
piecemeal
> >> rather than in one big chunk, but it has been categorically denied
that
> >> Stream is not a RHEL 8.n+1 beta and is more a RHEL 8.n+1
RC/rolling
> >> release.
> >>
> >
> > I think maybe some of the nervousness about CentOS Stream comes from
RHEL
> > seeming opacity on its development model.  As one of the architects of
> our
> > development process I'd be happy to field questions.  I'll
start by
> making
> > a two points in case they're in any way unclear:
> >
> > 1. Everything that goes into RHEL lands upstream first, then the
patches
> > are backported into the RHEL releases.
> > 2. Most of the work we do or plan on doing is in bugzilla.redhat.com.
> If
> > you go into the RHEL8 product queue today and file a bug you'll
see
> "CentOS
> > Stream" as a "Version" where an issue is encountered.
> >
> > I think what a lot of people are concerned about is the
rolling-release
> >> aspect of this. There will be no definitive versioning of CentOS
in the
> >> future - all you will be able to say is "fully updated"
and it won't be
> >> possible to slot a CentOS system in to exactly match a RHEL
version.
> >> Will third party RPMs built against RHEL 8.x be installable on a
CentOS
> >> 8 Stream system? The answer is surely "it depends", but
there are a lot
> >> of hardware vendors that target drivers to RHEL releases, which
may
> >> well make CentOS non-viable for hardware that doesn't have
drivers
> >> built in to the kernel.
> >>
> >
> > Generally if they follow the ABI guidelines I would expect it to work.
> > Those are here:
> https://access.redhat.com/articles/rhel8-abi-compatibility
> >
> > For loadable kernel modules there's a kernel ABI.
> >
> >
>
> Hi Brendan,
>
> This point is *critical*, so I apologise in advance for the lengthy
> post, *you* are breaking the kernel ABI between RHEL8 and Stream.
>
> One of the main unique selling points of RHEL is the stability of it's
> kernel ABI. No other distro provides this. The very nature of rolling
> kernel updates in Stream breaks the kernel ABI and breaks compatibility
> between RHEL8 and Stream. What works on RHEL8 may not work on Stream. At
> the kernel level, the two products diverge in fundamental compatibility
> and are not compatible, are no longer the same.
>
> How bad is this divergence / breakage? Well, we know the kernel ABI will
> change from time to time, almost exclusively at new point releases due
> to the massive backporting work that goes into the RHEL kernel. And this
> is fine, we know it's coming, we know when it's coming, and we can
plan
> for it's impact. It's a price well worth paying.
>
> To put this in context, at elrepo I currently help maintain around 50
> 3rd party kernel driver packages for RHEL8. When RH released RHEL8.3,
> every single package in our repository broke due to changes in the
> kernel ABI in the 6 month period between RHEL8.2 and RHEL8.3. It's not
> ideal, but we accept it as a price we pay for the otherwise excellent
> stability of the kernel ABI during the proceeding 6 months. As I said
> above, we know it's coming, we know when it's coming, and we can
plan
> for it.
>
> Now contrast that with Stream. Every kernel update in Stream has the
> potential to break the kernel ABI causing packages built for RHEL to
> break. We don't know when that may happen (only that it will), we
don't
> know how often it will happen, we have no idea which packages it will
> break. and we have no way to fix it. Consequently, elrepo has been
> unable to support Stream kernels.
>
> It is not just elrepo's users that the Stream kernels will affect. All
> OEM hardware manufacturers releasing 3rd party driver rpms as part of
> Red Hat's Driver Update Programme or otherwise will be similarly
> affected, and their driver updates will not be applicable to or
> compatible with CentOS Stream. In fact, RHEL's own driver update
> packages will likely need rebuilding against each Stream kernel update,
> although presumably you are in the unique position of being able to
> integrate any changes directly into the Stream kernel negating the need
> for driver updates and to mitigating the impact to yourselves.
>
> Luckily the solution is simple - do not include rolling beta kernel
> updates as part of Stream - rather just ship regular RHEL8 kernel
> updates that retain kernel ABI compatibility rather than breaking
> compatibility of the Stream product. I see no compelling reason to ship
> rolling kernel updates to Stream - it's not like I will ever be able to
> commit a patch and have you accept/merge it, and it's totally useless
> for me as a developer as I can't develop anything against a constantly
> moving target, and anyone who does require access has always had that
> available through their Red Hat OEM Partner/Developer accounts. Shipping
> rolling kernel updates in Stream adds no value and simply breaks
> compatibility between RHEL8 and Stream at the most fundamental level. If
> you are able to retain kernel ABI compatibility between RHEL8 and Stream
> kernels, then we (and other OEMs) will be able to continue to support
> Stream users, otherwise Stream users will have to look to alternative
> solutions.

Hi Phil,

Thank you for these details, I really appreciate you taking the time to
share this.  It sounds like for your use-cases interim kernel updates are
somewhere between not-useful and harmful, but I know for others it's
actually a benefit.  As a temporary workaround, elsewhere in this thread
somebody mentioned they're already cherry-picking which updates to use and
which to skip.  The only trouble is when a kernel update comes out with
both things you want and things you don't want.  So a better approach is
called for.  While I'm not sure how we'll get there, it seems like the
mutually satisfying end result would be one where third party binary
drivers work with CentOS Stream kernels.  Let's see what we can do.

-- 
Brendan Conoboy / Linux Project Lead / Red Hat, Inc.

On 09/12/2020 18.10, Brendan Conoboy wrote:
> On Wed, Dec 9, 2020 at 7:21 AM Phil Perry <pperry at elrepo.org>
wrote:
> 
>> On 09/12/2020 03:26, Brendan Conoboy wrote:
>>> On Tue, Dec 8, 2020 at 4:19 PM Pete Biggs <pete at
biggs.org.uk> wrote:
>>>
>>>> On Tue, 2020-12-08 at 17:54 -0500, Matthew Miller wrote:
>>>>> On Tue, Dec 08, 2020 at 03:15:17PM +0000, Pete Biggs wrote:
>>>>
>>>>>> "CentOS will become the developer playground"
>>>>>
>>>>> This one is categorically not the case. Even Fedora
isn't a developer
>>>>> playground. Everything landing in CentOS Stream is actually
*planned*
>>>> (with
>>>>> emphasis intentional) to go in a future RHEL release.
>>>>
>>>> It's all the talk of SIGs and developing and testing and
that Stream
>>>> will be the centerpiece of that. That's what I meant.
>>>>
>>>
>>> I don't know if I'd call SIGs a playground, but they're
certainly an
>>> important venue for communities to explore variations.
>>>
>>>
>>>>> Previously, all the development around RHEL releases was
done in
>> secret,
>>>> in
>>>>> the Red Hat black box. Now it's out of the box and can
be watched.
>> There
>>>> may
>>>>> be some launch pains, but I expect the average quality of
an update
>>>> hitting
>>>>> CentOS Stream to be very high.
>>>>
>>>> I don't get that from the documents released today.  If
Stream is *not*
>>>> a test-bed, then surely the code that appears in Stream must be
fully
>>>> formed in secret behind the scenes first. Yes, it will appear
piecemeal
>>>> rather than in one big chunk, but it has been categorically
denied that
>>>> Stream is not a RHEL 8.n+1 beta and is more a RHEL 8.n+1
RC/rolling
>>>> release.
>>>>
>>>
>>> I think maybe some of the nervousness about CentOS Stream comes
from RHEL
>>> seeming opacity on its development model.  As one of the architects
of
>> our
>>> development process I'd be happy to field questions.  I'll
start by
>> making
>>> a two points in case they're in any way unclear:
>>>
>>> 1. Everything that goes into RHEL lands upstream first, then the
patches
>>> are backported into the RHEL releases.
>>> 2. Most of the work we do or plan on doing is in
bugzilla.redhat.com.
>> If
>>> you go into the RHEL8 product queue today and file a bug you'll
see
>> "CentOS
>>> Stream" as a "Version" where an issue is
encountered.
>>>
>>> I think what a lot of people are concerned about is the
rolling-release
>>>> aspect of this. There will be no definitive versioning of
CentOS in the
>>>> future - all you will be able to say is "fully
updated" and it won't be
>>>> possible to slot a CentOS system in to exactly match a RHEL
version.
>>>> Will third party RPMs built against RHEL 8.x be installable on
a CentOS
>>>> 8 Stream system? The answer is surely "it depends",
but there are a lot
>>>> of hardware vendors that target drivers to RHEL releases, which
may
>>>> well make CentOS non-viable for hardware that doesn't have
drivers
>>>> built in to the kernel.
>>>>
>>>
>>> Generally if they follow the ABI guidelines I would expect it to
work.
>>> Those are here:
>> https://access.redhat.com/articles/rhel8-abi-compatibility
>>>
>>> For loadable kernel modules there's a kernel ABI.
>>>
>>>
>>
>> Hi Brendan,
>>
>> This point is *critical*, so I apologise in advance for the lengthy
>> post, *you* are breaking the kernel ABI between RHEL8 and Stream.
>>
>> One of the main unique selling points of RHEL is the stability of
it's
>> kernel ABI. No other distro provides this. The very nature of rolling
>> kernel updates in Stream breaks the kernel ABI and breaks compatibility
>> between RHEL8 and Stream. What works on RHEL8 may not work on Stream.
At
>> the kernel level, the two products diverge in fundamental compatibility
>> and are not compatible, are no longer the same.
>>
>> How bad is this divergence / breakage? Well, we know the kernel ABI
will
>> change from time to time, almost exclusively at new point releases due
>> to the massive backporting work that goes into the RHEL kernel. And
this
>> is fine, we know it's coming, we know when it's coming, and we
can plan
>> for it's impact. It's a price well worth paying.
>>
>> To put this in context, at elrepo I currently help maintain around 50
>> 3rd party kernel driver packages for RHEL8. When RH released RHEL8.3,
>> every single package in our repository broke due to changes in the
>> kernel ABI in the 6 month period between RHEL8.2 and RHEL8.3. It's
not
>> ideal, but we accept it as a price we pay for the otherwise excellent
>> stability of the kernel ABI during the proceeding 6 months. As I said
>> above, we know it's coming, we know when it's coming, and we
can plan
>> for it.
>>
>> Now contrast that with Stream. Every kernel update in Stream has the
>> potential to break the kernel ABI causing packages built for RHEL to
>> break. We don't know when that may happen (only that it will), we
don't
>> know how often it will happen, we have no idea which packages it will
>> break. and we have no way to fix it. Consequently, elrepo has been
>> unable to support Stream kernels.
>>
>> It is not just elrepo's users that the Stream kernels will affect.
All
>> OEM hardware manufacturers releasing 3rd party driver rpms as part of
>> Red Hat's Driver Update Programme or otherwise will be similarly
>> affected, and their driver updates will not be applicable to or
>> compatible with CentOS Stream. In fact, RHEL's own driver update
>> packages will likely need rebuilding against each Stream kernel update,
>> although presumably you are in the unique position of being able to
>> integrate any changes directly into the Stream kernel negating the need
>> for driver updates and to mitigating the impact to yourselves.
>>
>> Luckily the solution is simple - do not include rolling beta kernel
>> updates as part of Stream - rather just ship regular RHEL8 kernel
>> updates that retain kernel ABI compatibility rather than breaking
>> compatibility of the Stream product. I see no compelling reason to ship
>> rolling kernel updates to Stream - it's not like I will ever be
able to
>> commit a patch and have you accept/merge it, and it's totally
useless
>> for me as a developer as I can't develop anything against a
constantly
>> moving target, and anyone who does require access has always had that
>> available through their Red Hat OEM Partner/Developer accounts.
Shipping
>> rolling kernel updates in Stream adds no value and simply breaks
>> compatibility between RHEL8 and Stream at the most fundamental level.
If
>> you are able to retain kernel ABI compatibility between RHEL8 and
Stream
>> kernels, then we (and other OEMs) will be able to continue to support
>> Stream users, otherwise Stream users will have to look to alternative
>> solutions.
> 
> 
> Hi Phil,
> 
> Thank you for these details, I really appreciate you taking the time to
> share this.  It sounds like for your use-cases interim kernel updates are
> somewhere between not-useful and harmful, but I know for others it's
> actually a benefit.  As a temporary workaround, elsewhere in this thread
> somebody mentioned they're already cherry-picking which updates to use
and
> which to skip.  The only trouble is when a kernel update comes out with
> both things you want and things you don't want.  So a better approach
is
> called for.  While I'm not sure how we'll get there, it seems like
the
> mutually satisfying end result would be one where third party binary
> drivers work with CentOS Stream kernels.  Let's see what we can do.
> 
Hi Brendan,

I support what Phil said. Guaranteed kernel ABI compability between 
CentOS Stream and current RHEL minor release would resolve most of my 
concerns (i.e. supporting 3rd party driver rpms).

However this would disallow updating the CentOS Stream kernel as it is 
currently done. Maybe offering both, i.e. allow to optionally stay on 
the current minor release kernel version (including its updates) is 
feasible?

On 12/9/20 12:10 PM, Brendan Conoboy wrote:
> While I'm not sure how we'll get there, it seems like the
> mutually satisfying end result would be one where third party binary
> drivers work with CentOS Stream kernels.  Let's see what we can do.
>
So, I want to address this part a bit.? In MANY cases, it's not a 
third-party driver that ELrepo packages; it's an in-kernel driver that 
Red Hat has decided to disable.? Such as the megaraid_sas driver I need 
for my servers.