CentOS - Oct 2019 - kpatch (live kernel patching) in CentOS 7.7?

On 10/3/19 9:35 PM, Stephen John Smoogen wrote:
> On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew <mphelps at
cfa.harvard.edu> wrote:
>>
>> On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin <jperrin at centos.org>
wrote:
>>
>>>
>>>
>>> On 10/3/19 1:32 PM, Phelps, Matthew wrote:
>>>> Forgive me if this has been answered before and I've missed
it.
>>>>
>>>> This https://access.redhat.com/solutions/2206511 says live
kernel
>>> patches
>>>> will be available via yum updates as of RHEL 7.7. Is this
carried over to
>>>> CentOS 7.7.1908?
>>>>
>>>
>>> The functionality should be available, but we don't provide
patches in
>>> this way, no.
> 
>>
>> What would it take to make this happen? This would be a huge help to
those
>> of us running servers. Not to mention it would make the world a more
secure
>> place :)
>>
The short answer is "a team of kernel engineers, which we don't
have".
Smooge's overview which I've left below is great at explaining some of
this:
>> Is it an upstream issue? No SRPMS available? Etc?
>>
It's quite a bit more work than just SRPM (re) building. This is one of 
those things where if your workflow requires this functionality rather 
than the occasional reboot you should really just pay for RHEL. They put 
far more people and testing behind this feature than the team building 
CentOS is able to.

(DISCLAIMER: I work for RH, so that may not sound as true as it is)

>> Just trying to understand. I don't follow the centos-devel list.
Has this
>> been discussed there, or elsewhere?
>>
> 
> There is a lot to go into making a correct kpatch. You have to
> determine that you have a working kpatch (you can have one which works
> on 1% and corrupts 80% and crashes 19%), you have to determine that
> the patch fixes the problem (you can build patches which should do the
> right thing but don't), and you have to determine that it doesn't
add
> in some sort of long term corruption of memory/disk/etc. That takes
> specialized kernel expertise, a large amount of varied hardware to
> test the patch on, some amount of time, and a very large test suite.
> 
> You can also only live patch a system so many times and in only
> certain places. There are just some parts of the kernel which have to
> be rebooted and others you can put in a patch which works but your
> performance is going to be 25% of what it was before. There are other
> places that if you patch.. that is it.. try another and you hardlock.
> As much as some sites like to call it some sort of panacea for never
> having to reboot again.. it is really meant to be a tourniquet to air
> chopter the crash victim to a hospital. They may still not make it...
> you are just giving them a chance.
> 
> 
> 
-- 
Jim Perrin
The CentOS Project | http://www.centos.org
twitter: @BitIntegrity | GPG Key: FA09AD77

On Fri, Oct 4, 2019 at 6:33 AM Jim Perrin <jperrin at centos.org> wrote:
>
>
> On 10/3/19 9:35 PM, Stephen John Smoogen wrote:
> > On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew <mphelps at
cfa.harvard.edu>
> wrote:
> >>
> >> On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin <jperrin at
centos.org> wrote:
> >>
> >>>
> >>>
> >>> On 10/3/19 1:32 PM, Phelps, Matthew wrote:
> >>>> Forgive me if this has been answered before and I've
missed it.
> >>>>
> >>>> This https://access.redhat.com/solutions/2206511 says live
kernel
> >>> patches
> >>>> will be available via yum updates as of RHEL 7.7. Is this
carried
> over to
> >>>> CentOS 7.7.1908?
> >>>>
> >>>
> >>> The functionality should be available, but we don't
provide patches in
> >>> this way, no.
> >
> >>
> >> What would it take to make this happen? This would be a huge help
to
> those
> >> of us running servers. Not to mention it would make the world a
more
> secure
> >> place :)
> >>
>
> The short answer is "a team of kernel engineers, which we don't
have".
> Smooge's overview which I've left below is great at explaining some
of
> this:
>
>
I don't understand. If RHEL is putting out patches, and CentOS is a
recompile of RHEL, hasn't that "team of kernel engineers " already
done the
work?

I fully realize this is not a panacea for never rebooting again, but if we
can patch a critical kernel bug immediately, then schedule less disruptive
reboots in a week or three, this would help tremendously.
>> Is it an upstream issue? No SRPMS available? Etc?
> >>
>
> It's quite a bit more work than just SRPM (re) building. This is one of
> those things where if your workflow requires this functionality rather
> than the occasional reboot you should really just pay for RHEL. They put
> far more people and testing behind this feature than the team building
> CentOS is able to.
>
> (DISCLAIMER: I work for RH, so that may not sound as true as it is)
>
>
I knew someone was going to say that. :) In our case, as I'm sure is the
case for many other environments, we are a noncommercial CentOS shop that
can't afford the resources to have a mixed environment, not to mention the
RHEL licenses. Not all of the machines I'm thinking of are critical
infrastructure. We have many researchers running simulations that take
weeks, sometimes months, to finish, and avoiding the occasional forced
immediate reboot for a critical kernel bug would help expand Human
Knowledge :).

Anyway, I saw the functionality for live kernel patching in the RHEL 7.7
release notes, which the CentOS 7.7.1908 release notes pointed to, and
assumed (hoped?) that it would be available for us as well. If it won't
ever be provided, then I suggest the CentOS documentation be updated to
explicitly state so.

>
> >> Just trying to understand. I don't follow the centos-devel
list. Has
> this
> >> been discussed there, or elsewhere?
> >>
> >
> > There is a lot to go into making a correct kpatch. You have to
> > determine that you have a working kpatch (you can have one which works
> > on 1% and corrupts 80% and crashes 19%), you have to determine that
> > the patch fixes the problem (you can build patches which should do the
> > right thing but don't), and you have to determine that it
doesn't add
> > in some sort of long term corruption of memory/disk/etc. That takes
> > specialized kernel expertise, a large amount of varied hardware to
> > test the patch on, some amount of time, and a very large test suite.
> >
> > You can also only live patch a system so many times and in only
> > certain places. There are just some parts of the kernel which have to
> > be rebooted and others you can put in a patch which works but your
> > performance is going to be 25% of what it was before. There are other
> > places that if you patch.. that is it.. try another and you hardlock.
> > As much as some sites like to call it some sort of panacea for never
> > having to reboot again.. it is really meant to be a tourniquet to air
> > chopter the crash victim to a hospital. They may still not make it...
> > you are just giving them a chance.
> >
> >
> >
>
> --
> Jim Perrin
> The CentOS Project | http://www.centos.org
> twitter: @BitIntegrity | GPG Key: FA09AD77
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

-- 

*Matt Phelps*

*Information Technology Specialist, Systems Administrator*

(Computation Facility, Smithsonian Astrophysical Observatory)

Center for Astrophysics | Harvard & Smithsonian


60 Garden Street | MS 39 | Cambridge, MA 02138
email: mphelps at cfa.harvard.edu


cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
<http://cfa.harvard.edu/twitter> | YouTube
<http://cfa.harvard.edu/youtube>
| Newsletter <http://cfa.harvard.edu/newsletter>

On Fri, 4 Oct 2019 at 08:18, Phelps, Matthew <mphelps at cfa.harvard.edu>
wrote:
>
> On Fri, Oct 4, 2019 at 6:33 AM Jim Perrin <jperrin at centos.org>
wrote:
>
> >
> >
> > On 10/3/19 9:35 PM, Stephen John Smoogen wrote:
> > > On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew <mphelps at
cfa.harvard.edu>
> > wrote:
> > >>
> > >> On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin <jperrin at
centos.org> wrote:
> > >>
> > >>>
> > >>>
> > >>> On 10/3/19 1:32 PM, Phelps, Matthew wrote:
> > >>>> Forgive me if this has been answered before and
I've missed it.
> > >>>>
> > >>>> This https://access.redhat.com/solutions/2206511 says
live kernel
> > >>> patches
> > >>>> will be available via yum updates as of RHEL 7.7. Is
this carried
> > over to
> > >>>> CentOS 7.7.1908?
> > >>>>
> > >>>
> > >>> The functionality should be available, but we don't
provide patches in
> > >>> this way, no.
> > >
> > >>
> > >> What would it take to make this happen? This would be a huge
help to
> > those
> > >> of us running servers. Not to mention it would make the world
a more
> > secure
> > >> place :)
> > >>
> >
> > The short answer is "a team of kernel engineers, which we
don't have".
> > Smooge's overview which I've left below is great at explaining
some of
> > this:
> >
> >
> I don't understand. If RHEL is putting out patches, and CentOS is a
> recompile of RHEL, hasn't that "team of kernel engineers "
already done the
> work?
>
No. because most of the work on making a patch is after the kernel is
compiled and working. Thus even though you have the same source code,
similar compilers etc.. there are going to be differences which have
to be looked at to make sure it is really working. A CentOS kernel is
not exactly the same as a RHEL kernel is not the same as a Oracle
kernel is not the same as the one you recompiled locally. From most
operational points they seem the same, but kernel patching is where
those differences really show up.

Yes it would be easy to set up some automated tool which 'made'
kpatches.. and I expect they may 'work' for most systems. But I also
expect that they would also eat babies more times than people would
like. If sites really need them, they can set up the tooling
themselves and make them work when they know they want it. Trying to
make it a general purpose answer for something which may corrupt data
5 or 20% or 40% of the time.. is just waiting to be on Slashdot daily
(wait do we do Slashdot anymore.. Reddit? nope the kids aren't there
anymore either.. ok someplace daily) in a bad way.





-- 
Stephen J Smoogen.