Stephen John Smoogen
2019-Oct-04 13:23 UTC
[CentOS] kpatch (live kernel patching) in CentOS 7.7?
On Fri, 4 Oct 2019 at 08:18, Phelps, Matthew <mphelps at cfa.harvard.edu> wrote:> > On Fri, Oct 4, 2019 at 6:33 AM Jim Perrin <jperrin at centos.org> wrote: > > > > > > > On 10/3/19 9:35 PM, Stephen John Smoogen wrote: > > > On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew <mphelps at cfa.harvard.edu> > > wrote: > > >> > > >> On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin <jperrin at centos.org> wrote: > > >> > > >>> > > >>> > > >>> On 10/3/19 1:32 PM, Phelps, Matthew wrote: > > >>>> Forgive me if this has been answered before and I've missed it. > > >>>> > > >>>> This https://access.redhat.com/solutions/2206511 says live kernel > > >>> patches > > >>>> will be available via yum updates as of RHEL 7.7. Is this carried > > over to > > >>>> CentOS 7.7.1908? > > >>>> > > >>> > > >>> The functionality should be available, but we don't provide patches in > > >>> this way, no. > > > > > >> > > >> What would it take to make this happen? This would be a huge help to > > those > > >> of us running servers. Not to mention it would make the world a more > > secure > > >> place :) > > >> > > > > The short answer is "a team of kernel engineers, which we don't have". > > Smooge's overview which I've left below is great at explaining some of > > this: > > > > > I don't understand. If RHEL is putting out patches, and CentOS is a > recompile of RHEL, hasn't that "team of kernel engineers " already done the > work? >No. because most of the work on making a patch is after the kernel is compiled and working. Thus even though you have the same source code, similar compilers etc.. there are going to be differences which have to be looked at to make sure it is really working. A CentOS kernel is not exactly the same as a RHEL kernel is not the same as a Oracle kernel is not the same as the one you recompiled locally. From most operational points they seem the same, but kernel patching is where those differences really show up. Yes it would be easy to set up some automated tool which 'made' kpatches.. and I expect they may 'work' for most systems. But I also expect that they would also eat babies more times than people would like. If sites really need them, they can set up the tooling themselves and make them work when they know they want it. Trying to make it a general purpose answer for something which may corrupt data 5 or 20% or 40% of the time.. is just waiting to be on Slashdot daily (wait do we do Slashdot anymore.. Reddit? nope the kids aren't there anymore either.. ok someplace daily) in a bad way. -- Stephen J Smoogen.
Phelps, Matthew
2019-Oct-04 13:35 UTC
[CentOS] kpatch (live kernel patching) in CentOS 7.7?
On Fri, Oct 4, 2019 at 9:24 AM Stephen John Smoogen <smooge at gmail.com> wrote:> On Fri, 4 Oct 2019 at 08:18, Phelps, Matthew <mphelps at cfa.harvard.edu> > wrote: > > > > On Fri, Oct 4, 2019 at 6:33 AM Jim Perrin <jperrin at centos.org> wrote: > > > > > > > > > > > On 10/3/19 9:35 PM, Stephen John Smoogen wrote: > > > > On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew < > mphelps at cfa.harvard.edu> > > > wrote: > > > >> > > > >> On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin <jperrin at centos.org> > wrote: > > > >> > > > >>> > > > >>> > > > >>> On 10/3/19 1:32 PM, Phelps, Matthew wrote: > > > >>>> Forgive me if this has been answered before and I've missed it. > > > >>>> > > > >>>> This https://access.redhat.com/solutions/2206511 says live kernel > > > >>> patches > > > >>>> will be available via yum updates as of RHEL 7.7. Is this carried > > > over to > > > >>>> CentOS 7.7.1908? > > > >>>> > > > >>> > > > >>> The functionality should be available, but we don't provide > patches in > > > >>> this way, no. > > > > > > > >> > > > >> What would it take to make this happen? This would be a huge help to > > > those > > > >> of us running servers. Not to mention it would make the world a more > > > secure > > > >> place :) > > > >> > > > > > > The short answer is "a team of kernel engineers, which we don't have". > > > Smooge's overview which I've left below is great at explaining some of > > > this: > > > > > > > > I don't understand. If RHEL is putting out patches, and CentOS is a > > recompile of RHEL, hasn't that "team of kernel engineers " already done > the > > work? > > > > No. because most of the work on making a patch is after the kernel is > compiled and working. Thus even though you have the same source code, > similar compilers etc.. there are going to be differences which have > to be looked at to make sure it is really working. A CentOS kernel is > not exactly the same as a RHEL kernel is not the same as a Oracle > kernel is not the same as the one you recompiled locally. From most > operational points they seem the same, but kernel patching is where > those differences really show up. > > Yes it would be easy to set up some automated tool which 'made' > kpatches.. and I expect they may 'work' for most systems. But I also > expect that they would also eat babies more times than people would > like. If sites really need them, they can set up the tooling > themselves and make them work when they know they want it. Trying to > make it a general purpose answer for something which may corrupt data > 5 or 20% or 40% of the time.. is just waiting to be on Slashdot daily > (wait do we do Slashdot anymore.. Reddit? nope the kids aren't there > anymore either.. ok someplace daily) in a bad way. > > >Thanks for the explanation(s). I'm still puzzled why RedHat is doing it then, and making it more generally available (to paying customers even), if it's so dire a proposition that it will fail so badly, so often. That seems counter-intuitive to me. Anyway, I again point out that the CentOS documentation should be made clear that this functionality won't ever be coming to CentOS. -Matt -- *Matt Phelps* *Information Technology Specialist, Systems Administrator* (Computation Facility, Smithsonian Astrophysical Observatory) Center for Astrophysics | Harvard & Smithsonian 60 Garden Street | MS 39 | Cambridge, MA 02138 email: mphelps at cfa.harvard.edu cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube> | Newsletter <http://cfa.harvard.edu/newsletter>
Am 04.10.19 um 15:35 schrieb Phelps, Matthew:> I'm still puzzled why RedHat is doing it then, and making it more generally > available (to paying customers even), if it's so dire a proposition that it > will fail so badly, so often. That seems counter-intuitive to me.I've been using kernel live patching on an Ubuntu Machine for several years now without any problems (Ubuntu offers them for registered users for free (three machines per account)) and haven't noticed any downsides so far.
Stephen John Smoogen
2019-Oct-04 13:56 UTC
[CentOS] kpatch (live kernel patching) in CentOS 7.7?
On Fri, 4 Oct 2019 at 09:36, Phelps, Matthew <mphelps at cfa.harvard.edu> wrote:> > On Fri, Oct 4, 2019 at 9:24 AM Stephen John Smoogen <smooge at gmail.com> > wrote: >> Thanks for the explanation(s). > > I'm still puzzled why RedHat is doing it then, and making it more generally > available (to paying customers even), if it's so dire a proposition that it > will fail so badly, so often. That seems counter-intuitive to me. >Because they have kernel developers, qa, and other staff dedicated to making that kpatch work? They have a large set of servers to test different workloads? They have some time before the kernel is built internally and when it is made available externally to do all this and hand tune any problems found? Because big companies are paying a large amount of money to make it work and so the extra labour is profitable? In the past, all of this would be a challenge for people to come together and show that they can also do it themselves... or improve on something to make it so less labour intensive at parts. If that happens, I am happy to have laid out the challenge :).> Anyway, I again point out that the CentOS documentation should be made > clear that this functionality won't ever be coming to CentOS. > > -Matt > > > -- > > *Matt Phelps* > > *Information Technology Specialist, Systems Administrator* > > (Computation Facility, Smithsonian Astrophysical Observatory) > > Center for Astrophysics | Harvard & Smithsonian > > > 60 Garden Street | MS 39 | Cambridge, MA 02138 > email: mphelps at cfa.harvard.edu > > > cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter > <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube> > | Newsletter <http://cfa.harvard.edu/newsletter> > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos-- Stephen J Smoogen.
On 10/4/19 9:35 AM, Phelps, Matthew wrote:> ... > I'm still puzzled why RedHat is doing it then, and making it more > generally available (to paying customers even), if it's so dire a > proposition that it will fail so badly, so often. That seems > counter-intuitive to me.It would likely boil down to a risk-benefit analysis; for RHEL RH is willing to take the risks associated with it due to the added benefits of offering it.? And, well, the elephant in the room is that it is one of the things that make an RHEL subscription more attractive, whether that's an intended effect or not. Ubuntu/Canonical apparently made a different analysis, per another poster in-thread. Of course, I'm in a similar situation to you in that we're a non-profit and don't have the budget for RHEL subscriptions.? So what I've done here is to stay on top of what the kernel issues are, and schedule reboots accordingly, and take those long-running analysis job machines and temporarily suspend general Internet accessibility until a reboot is possible if the kernel issue warrants that.? I likely don't have anywhere near as many of those jobs running as you, but I still can sympathize!> Anyway, I again point out that the CentOS documentation should be made > clear that this functionality won't ever be coming to CentOS.I would suggest the team, rather than a blanket statement that it's 'never' coming to CentOS would articulate (Smooge's? posts are a great start!) what it would take from the community to make it happen, thus leaving the question open-ended.