CentOS - Oct 2019 - kpatch (live kernel patching) in CentOS 7.7?

On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin <jperrin at centos.org> wrote:
>
>
> On 10/3/19 1:32 PM, Phelps, Matthew wrote:
> > Forgive me if this has been answered before and I've missed it.
> >
> > This https://access.redhat.com/solutions/2206511 says live kernel
> patches
> > will be available via yum updates as of RHEL 7.7. Is this carried over
to
> > CentOS 7.7.1908?
> >
>
> The functionality should be available, but we don't provide patches in
> this way, no.
>
>
>
> --
> Jim Perrin
> The CentOS Project | http://www.centos.org
> twitter: @BitIntegrity | GPG Key: FA09AD77
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

What would it take to make this happen? This would be a huge help to those
of us running servers. Not to mention it would make the world a more secure
place :)

Is it an upstream issue? No SRPMS available? Etc?

Just trying to understand. I don't follow the centos-devel list. Has this
been discussed there, or elsewhere?

-- 

*Matt Phelps*

*Information Technology Specialist, Systems Administrator*

(Computation Facility, Smithsonian Astrophysical Observatory)

Center for Astrophysics | Harvard & Smithsonian


60 Garden Street | MS 39 | Cambridge, MA 02138
email: mphelps at cfa.harvard.edu


cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
<http://cfa.harvard.edu/twitter> | YouTube
<http://cfa.harvard.edu/youtube>
| Newsletter <http://cfa.harvard.edu/newsletter>

On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew <mphelps at cfa.harvard.edu>
wrote:
>
> On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin <jperrin at centos.org>
wrote:
>
> >
> >
> > On 10/3/19 1:32 PM, Phelps, Matthew wrote:
> > > Forgive me if this has been answered before and I've missed
it.
> > >
> > > This https://access.redhat.com/solutions/2206511 says live kernel
> > patches
> > > will be available via yum updates as of RHEL 7.7. Is this carried
over to
> > > CentOS 7.7.1908?
> > >
> >
> > The functionality should be available, but we don't provide
patches in
> > this way, no.
>
> What would it take to make this happen? This would be a huge help to those
> of us running servers. Not to mention it would make the world a more secure
> place :)
>
> Is it an upstream issue? No SRPMS available? Etc?
>
> Just trying to understand. I don't follow the centos-devel list. Has
this
> been discussed there, or elsewhere?
>
There is a lot to go into making a correct kpatch. You have to
determine that you have a working kpatch (you can have one which works
on 1% and corrupts 80% and crashes 19%), you have to determine that
the patch fixes the problem (you can build patches which should do the
right thing but don't), and you have to determine that it doesn't add
in some sort of long term corruption of memory/disk/etc. That takes
specialized kernel expertise, a large amount of varied hardware to
test the patch on, some amount of time, and a very large test suite.

You can also only live patch a system so many times and in only
certain places. There are just some parts of the kernel which have to
be rebooted and others you can put in a patch which works but your
performance is going to be 25% of what it was before. There are other
places that if you patch.. that is it.. try another and you hardlock.
As much as some sites like to call it some sort of panacea for never
having to reboot again.. it is really meant to be a tourniquet to air
chopter the crash victim to a hospital. They may still not make it...
you are just giving them a chance.



-- 
Stephen J Smoogen.

On Thu, 3 Oct 2019 21:35:15 -0400
Stephen John Smoogen wrote:
> There is a lot to go into making a correct kpatch. 
Thank you ever so much for your very clear description of what's needed to
accomplish something like this.

I, for one, had no idea.  And I feel much smarter now than before!

-- 
MELVILLE THEATRE ~ Real D 3D Digital Cinema ~ www.melvilletheatre.com

On 10/3/19 9:35 PM, Stephen John Smoogen wrote:
> On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew <mphelps at
cfa.harvard.edu> wrote:
>>
>> On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin <jperrin at centos.org>
wrote:
>>
>>>
>>>
>>> On 10/3/19 1:32 PM, Phelps, Matthew wrote:
>>>> Forgive me if this has been answered before and I've missed
it.
>>>>
>>>> This https://access.redhat.com/solutions/2206511 says live
kernel
>>> patches
>>>> will be available via yum updates as of RHEL 7.7. Is this
carried over to
>>>> CentOS 7.7.1908?
>>>>
>>>
>>> The functionality should be available, but we don't provide
patches in
>>> this way, no.
> 
>>
>> What would it take to make this happen? This would be a huge help to
those
>> of us running servers. Not to mention it would make the world a more
secure
>> place :)
>>
The short answer is "a team of kernel engineers, which we don't
have".
Smooge's overview which I've left below is great at explaining some of
this:
>> Is it an upstream issue? No SRPMS available? Etc?
>>
It's quite a bit more work than just SRPM (re) building. This is one of 
those things where if your workflow requires this functionality rather 
than the occasional reboot you should really just pay for RHEL. They put 
far more people and testing behind this feature than the team building 
CentOS is able to.

(DISCLAIMER: I work for RH, so that may not sound as true as it is)

>> Just trying to understand. I don't follow the centos-devel list.
Has this
>> been discussed there, or elsewhere?
>>
> 
> There is a lot to go into making a correct kpatch. You have to
> determine that you have a working kpatch (you can have one which works
> on 1% and corrupts 80% and crashes 19%), you have to determine that
> the patch fixes the problem (you can build patches which should do the
> right thing but don't), and you have to determine that it doesn't
add
> in some sort of long term corruption of memory/disk/etc. That takes
> specialized kernel expertise, a large amount of varied hardware to
> test the patch on, some amount of time, and a very large test suite.
> 
> You can also only live patch a system so many times and in only
> certain places. There are just some parts of the kernel which have to
> be rebooted and others you can put in a patch which works but your
> performance is going to be 25% of what it was before. There are other
> places that if you patch.. that is it.. try another and you hardlock.
> As much as some sites like to call it some sort of panacea for never
> having to reboot again.. it is really meant to be a tourniquet to air
> chopter the crash victim to a hospital. They may still not make it...
> you are just giving them a chance.
> 
> 
> 
-- 
Jim Perrin
The CentOS Project | http://www.centos.org
twitter: @BitIntegrity | GPG Key: FA09AD77