On 07/08/2020 10:01, Johnny Hughes wrote:> On 8/7/20 3:46 AM, Nicolas Kovacs wrote: >> Le 07/08/2020 ? 09:40, Alessandro Baggi a ?crit?: >>> Probably many users have not updated their machines between the bug release and >>> the resolution (thanks to your fast apply in the weekend, thank you) and many >>> update their centos machines on a 2 months base (if not worst). I think also >>> that many users of CentOS user base have not proclamed their >>> disappointement/the issue on this list or in other channels. For example I >>> simply updated in the wrong time. >> >> I'm using yum-cron to keep all my server updated on a daily basis. >> >> And my question "How could this have passed Q & A" was obviously directed at >> Red Hat... and *not* at Johnny Hughes and the CentOS team who do their best to >> deliver the best possible downstream system. I raise my morning coffee mug to >> your health, guys. >> >> Cheers, >> >> Niki >> > I can assure you .. a BUNCH of testing was done. Because of the scope > of this udpate, the CentOS team was looped in during the embargo stage > (we normally are not .. Red Hat Engineering got permission to make this > happen for this issue). Normally we see things that are open source only > .. not embargoed content. Once the embargo gets lifted, the items > become open source. Kudos to the RH team for making this happen. > > The CentOS team worked with the RHEL team on this update for several > days (more than a week, for sure, maybe 2 weeks) > > I gained MUCH respect for all those guys .. especially Peter Jones. He > is Mr.Secure Boot. > > I personally tested both the c8 and c7 solutions on several machines > (All i have access to actually, including several personal machines that > have secureboot). I saw some of the testing that happened on the RHEL > side. It was extensive. >I'll just add to Johnny's already comprehensive reply. As a member of the CentOS QA team, I personally tested the update on 3 physical machines and all worked fine. Moreover, the QA team was not able to replicate the issue on a single physical machine available to them - the first indication of a problem came from public reports. We give up a huge amount of our personal time and resources to ensure CentOS (and RHEL) are the very best products they can be. I'm unsure what more could have been done.> Microsoft, Debian, Ubuntu and others also had issues with this .. so if > you are losing trust, you are losing it with all OS vendors WRT this issue. > > All I can say is .. this issue was the hardest thing I have been > involved with since starting with the CentOS Project 17 years ago. > > Obviously, everyone involved in this build would have prevented this > from happening if they could have. Secureboot is complicated. > >
On 8/7/20 5:30 AM, Phil Perry wrote:> On 07/08/2020 10:01, Johnny Hughes wrote: >> On 8/7/20 3:46 AM, Nicolas Kovacs wrote: >>> Le 07/08/2020 ? 09:40, Alessandro Baggi a ?crit?: >>>> Probably many users have not updated their machines between the bug >>>> release and >>>> the resolution (thanks to your fast apply in the weekend, thank you) >>>> and many >>>> update their centos machines on a 2 months base (if not worst). I >>>> think also >>>> that many users of CentOS user base have not proclamed their >>>> disappointement/the issue on this list or in other channels. For >>>> example I >>>> simply updated in the wrong time. >>> >>> I'm using yum-cron to keep all my server updated on a daily basis. >>> >>> And my question "How could this have passed Q & A" was obviously >>> directed at >>> Red Hat... and *not* at Johnny Hughes and the CentOS team who do >>> their best to >>> deliver the best possible downstream system. I raise my morning >>> coffee mug to >>> your health, guys. >>> >>> Cheers, >>> >>> Niki >>> >> I can assure you .. a BUNCH of testing was done.? Because of the scope >> of this udpate, the CentOS team was looped in during the embargo stage >> (we normally are not .. Red Hat Engineering got permission to make this >> happen for this issue). Normally we see things that are open source only >> .. not embargoed content.? Once the embargo gets lifted, the items >> become open source.? Kudos to the RH team for making this happen. >> >> The CentOS team worked with the RHEL team on this update for several >> days (more than a week, for sure, maybe 2 weeks) >> >> I gained MUCH respect for all those guys .. especially? Peter Jones.? He >> is Mr.Secure Boot. >> >> I personally tested both the c8 and c7 solutions on several machines >> (All i have access to actually, including several personal machines that >> have secureboot).? I saw some of the testing that happened on the RHEL >> side.? It was extensive. >> > > I'll just add to Johnny's already comprehensive reply. As a member of > the CentOS QA team, I personally tested the update on 3 physical > machines and all worked fine. Moreover, the QA team was not able to > replicate the issue on a single physical machine available to them - the > first indication of a problem came from public reports. We give up a > huge amount of our personal time and resources to ensure CentOS (and > RHEL) are the very best products they can be. I'm unsure what more could > have been done.Thanks Phil, I very much appreciate all you and the rest of the QA team do. I know it is a knee jerk reaction to say .. how did that not get caught. I actually said it MYSELF for this very issue. But looking back, I am not sure how we could have caught it. "Stuff Happens" :) There are just a huge number of possible combinations.> >> Microsoft, Debian, Ubuntu and others also had issues with this .. so if >> you are losing trust, you are losing it with all OS vendors WRT this >> issue. >> >> All I can say is .. this issue was the hardest thing I have been >> involved with since starting with the CentOS Project 17 years ago. >> >> Obviously, everyone involved in this build would have prevented this >> from happening if they could have.? Secureboot is complicated.-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://lists.centos.org/pipermail/centos/attachments/20200807/f848033a/attachment.sig>
> On 8/7/20 5:30 AM, Phil Perry wrote: >> On 07/08/2020 10:01, Johnny Hughes wrote: >>> On 8/7/20 3:46 AM, Nicolas Kovacs wrote: >>>> Le 07/08/2020 ? 09:40, Alessandro Baggi a ?crit?: >>>>> Probably many users have not updated their machines between the bug >>>>> release and >>>>> the resolution (thanks to your fast apply in the weekend, thank you) >>>>> and many >>>>> update their centos machines on a 2 months base (if not worst). I >>>>> think also >>>>> that many users of CentOS user base have not proclamed their >>>>> disappointement/the issue on this list or in other channels. For >>>>> example I >>>>> simply updated in the wrong time. >>>> >>>> I'm using yum-cron to keep all my server updated on a daily basis. >>>> >>>> And my question "How could this have passed Q & A" was obviously >>>> directed at >>>> Red Hat... and *not* at Johnny Hughes and the CentOS team who do >>>> their best to >>>> deliver the best possible downstream system. I raise my morning >>>> coffee mug to >>>> your health, guys. >>>> >>>> Cheers, >>>> >>>> Niki >>>> >>> I can assure you .. a BUNCH of testing was done.? Because of the scope >>> of this udpate, the CentOS team was looped in during the embargo stage >>> (we normally are not .. Red Hat Engineering got permission to make this >>> happen for this issue). Normally we see things that are open source >>> only >>> .. not embargoed content.? Once the embargo gets lifted, the items >>> become open source.? Kudos to the RH team for making this happen. >>> >>> The CentOS team worked with the RHEL team on this update for several >>> days (more than a week, for sure, maybe 2 weeks) >>> >>> I gained MUCH respect for all those guys .. especially? Peter Jones.? >>> He >>> is Mr.Secure Boot. >>> >>> I personally tested both the c8 and c7 solutions on several machines >>> (All i have access to actually, including several personal machines >>> that >>> have secureboot).? I saw some of the testing that happened on the RHEL >>> side.? It was extensive. >>> >> >> I'll just add to Johnny's already comprehensive reply. As a member of >> the CentOS QA team, I personally tested the update on 3 physical >> machines and all worked fine. Moreover, the QA team was not able to >> replicate the issue on a single physical machine available to them - the >> first indication of a problem came from public reports. We give up a >> huge amount of our personal time and resources to ensure CentOS (and >> RHEL) are the very best products they can be. I'm unsure what more could >> have been done. > > Thanks Phil, > > I very much appreciate all you and the rest of the QA team do. > > I know it is a knee jerk reaction to say .. how did that not get caught. > I actually said it MYSELF for this very issue. But looking back, I am > not sure how we could have caught it. > > "Stuff Happens" :) >Crowd testing? Feed the green bananas to the crowd and let them ripe. It works well for some of the biggest software companies :-) At least it could make sense for directly hardware related stuff like kernel, boot loader, firmware/microcode and similar. Regards, Simon
Il 07/08/20 14:53, Johnny Hughes ha scritto:> On 8/7/20 5:30 AM, Phil Perry wrote: >> On 07/08/2020 10:01, Johnny Hughes wrote: >>> On 8/7/20 3:46 AM, Nicolas Kovacs wrote: >>>> Le 07/08/2020 ? 09:40, Alessandro Baggi a ?crit?: >>>>> Probably many users have not updated their machines between the bug >>>>> release and >>>>> the resolution (thanks to your fast apply in the weekend, thank you) >>>>> and many >>>>> update their centos machines on a 2 months base (if not worst). I >>>>> think also >>>>> that many users of CentOS user base have not proclamed their >>>>> disappointement/the issue on this list or in other channels. For >>>>> example I >>>>> simply updated in the wrong time. >>>> I'm using yum-cron to keep all my server updated on a daily basis. >>>> >>>> And my question "How could this have passed Q & A" was obviously >>>> directed at >>>> Red Hat... and *not* at Johnny Hughes and the CentOS team who do >>>> their best to >>>> deliver the best possible downstream system. I raise my morning >>>> coffee mug to >>>> your health, guys. >>>> >>>> Cheers, >>>> >>>> Niki >>>> >>> I can assure you .. a BUNCH of testing was done.? Because of the scope >>> of this udpate, the CentOS team was looped in during the embargo stage >>> (we normally are not .. Red Hat Engineering got permission to make this >>> happen for this issue). Normally we see things that are open source only >>> .. not embargoed content.? Once the embargo gets lifted, the items >>> become open source.? Kudos to the RH team for making this happen. >>> >>> The CentOS team worked with the RHEL team on this update for several >>> days (more than a week, for sure, maybe 2 weeks) >>> >>> I gained MUCH respect for all those guys .. especially? Peter Jones.? He >>> is Mr.Secure Boot. >>> >>> I personally tested both the c8 and c7 solutions on several machines >>> (All i have access to actually, including several personal machines that >>> have secureboot).? I saw some of the testing that happened on the RHEL >>> side.? It was extensive. >>> >> I'll just add to Johnny's already comprehensive reply. As a member of >> the CentOS QA team, I personally tested the update on 3 physical >> machines and all worked fine. Moreover, the QA team was not able to >> replicate the issue on a single physical machine available to them - the >> first indication of a problem came from public reports. We give up a >> huge amount of our personal time and resources to ensure CentOS (and >> RHEL) are the very best products they can be. I'm unsure what more could >> have been done. > Thanks Phil, > > I very much appreciate all you and the rest of the QA team do. > > I know it is a knee jerk reaction to say .. how did that not get caught. > I actually said it MYSELF for this very issue. But looking back, I am > not sure how we could have caught it. > > "Stuff Happens" :) > > There are just a huge number of possible combinations.Hi Johnny, what is the current status of the notification tool for security updates on C8? There are possibilities to get soon announces on ML for EL8? Would be great have the tool working. Thank you.