Karanbir Singh
2016-Apr-16 09:09 UTC
[CentOS-announce] Notification of outage : mirrorlist.centos.org (ipv4) 2016-05-16 08:58 UTC
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, At 05:00 UTC 2016-05-16 an automated component update caused the mirrorlist service for CentOS to go down for all IPv4 based services. The IPv6 service for mirrorlist.centos.org was unaffected. By 08:11 UTC 2016-05-16 I rolled back the impacted components, disabled the update mechanics, and restarted services. By 08:15 UTC 2016-05-16 Services were returning to normal. By 08:23 UTC 2016-05-16 we had multiple confirmations from around the world that services were restored. - --------------- system wide followup: - - We will work to extend and add where needed, better testing around each component involved in such roles. - - I will work with Fabian and make sure that all our automated component and system changes that impact a production service are only run during regular working hours for the team. - --------------- root cause: all the components involved in the downtime are backed up and we will start looking at the root cause of why services went down, for now the immediate focus was to restore services which was done by a rollback. - --------------- reporting issues: Note that for real time, time sensitive issues always drop into #centos-devel on irc.freenode.net and let us know, along with filing a bug report at bugs.centos.org - For any non time critical issues, please report them at bugs.centos.org against the 'Infrastructure' project, and we will aim to address them as soon as possible. - -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJXEgFAAAoJEI3Oi2Mx7xbtHQoIAKBsD6xIlBCfuqRcx0AHto2g dwB7Pl5ijwVkqsIMpDNq4a3aws72P1PTwqi19PbnkwACh0kPxx8iT8/8RL4LLvmC dahoMjYrIpZYULij6u0QEy9RM9HZ7Xyo6lGe41+Rz1yFMv+QgqjhYBtt+EuvBrO1 ILM1PsQypXsxQaGKRMnhzjcIWXhpNNBinRAVEOmzLiHU2WYDefUwlB1gtHx5JmZl l/vWLSlMWRiC5HaDtKGBV5rPickqBrRkFfvGdYezrj3KHsoj8NCUhdir75gd6RXN b++FRAuRZZGcvCYquvwbrG2V+Ix1GG8GQBs2ej3p4wvaw9G5gBCR2hEa6ZINrQ8=ZdAB -----END PGP SIGNATURE-----
Karanbir Singh
2016-Apr-16 22:16 UTC
[CentOS-announce] *correction* Notification of outage : mirrorlist.centos.org (ipv4) 2016-04-16 08:58 UTC
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Apologies for this, the correct date should be : 2016-04-16 Sat. The timestamps remain accurate. On 16/04/16 10:09, Karanbir Singh wrote:> Hi, > > At 05:00 UTC 2016-05-16 an automated component update caused the > mirrorlist service for CentOS to go down for all IPv4 based > services. The IPv6 service for mirrorlist.centos.org was > unaffected. > > By 08:11 UTC 2016-05-16 I rolled back the impacted components, > disabled the update mechanics, and restarted services. > > By 08:15 UTC 2016-05-16 Services were returning to normal. > > By 08:23 UTC 2016-05-16 we had multiple confirmations from around > the world that services were restored. > > --------------- system wide followup: > > - We will work to extend and add where needed, better testing > around each component involved in such roles. > > - I will work with Fabian and make sure that all our automated > component and system changes that impact a production service are > only run during regular working hours for the team. > > --------------- root cause: > > all the components involved in the downtime are backed up and we > will start looking at the root cause of why services went down, for > now the immediate focus was to restore services which was done by a > rollback. > > --------------- reporting issues: > > Note that for real time, time sensitive issues always drop into > #centos-devel on irc.freenode.net and let us know, along with > filing a bug report at bugs.centos.org - For any non time critical > issues, please report them at bugs.centos.org against the > 'Infrastructure' project, and we will aim to address them as soon > as possible. >- -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJXErmsAAoJEI3Oi2Mx7xbtJdgIAI1+4KTYCdI4O+kFlrJ5IlMy TwFGYt7O1WyGDfXC27VLehDxNP0lFiZOd1TMMSWy0/P64TYcBr5GLa4Vc7WLpLef ywgjvgNE9VY69z4D1Lkgdxlgnhg31tPtW8kA1Py24nKrEFMYYi7/2tUGZn43kE8D SRn0CywFizCFIScB0xI7uSJRE2PzFFwixIG4LaEdqppxcId6hsKb80hzpIP35pPU eXlagl5CMad2XD0aVqKrqoU5W9vS9swKv9rc6ASsUHB28W7KcT0mq0YsIojhS81L w8hiSsTpW3b62uBb55MIvkg1Y11H0qlkW306ZaZAXjTpRsvNaz82+qRmgcVVJvc=Uvpc -----END PGP SIGNATURE-----