Asle Ommundsen
2020-Jan-17 12:36 UTC
[CentOS] After upgrade to CentOS 8.1 default gateway missing
On Thu, 16 Jan 2020 15:34:43 +0100, Stephen John Smoogen <smooge at gmail.com> wrote:> On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen <aommundsen at gmail.com> > wrote: >> >> Hi, >> >> Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a >> reboot of the first server the network was unavailable. In IPMI console >> everything except the network was looking good. Network was unreachable. >> No errors in NetworkManager. I also restarted NetworkManager, but it did >> not help. Then I discovered that the default gateway suddenly was >> missing. >> >> Then I rebooted the server one more time, but network was still down. >> >> Then both myself and a technician in my datcenter was debugging this (I >> had to wake him up in the middle of the night, costing me a lot of >> money), >> without finding any reason for why the default gateway was missing after >> reboot. >> >> Then we rebooted the server a third time, and all of a sudden the >> problem >> was gone and the default gateway was back. [...cut...]> In order to determine what is going on you need to give a lot more > information. > > 1. How do these boxes get their network information? DHCP or static > 2. If they are static, what controls the setting of ips: > NetworkManager or network-scripts > 3. If they are static, how are they set in > /etc/sysconfig/network-scripts/ > 4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY> 5. If you are using network-manager, what does nmtui or the graphical > tool say the gateway or default route is?Here is answers to your list. I have anonymized the some of the data: 1) Static ip configuration 2) This should be NetworkManager. nmcli output: eno1: connected to eno1 inet4 1.1.1.234/29 route4 1.1.1.232/29 route4 0.0.0.0/0 eno2: connected to eno2 inet4 192.168.0.5/24 route4 192.168.0.0/24 [root at server ~]# nmcli d show | grep IP4.GATEWA IP4.GATEWAY: 1.1.1.1.233 3) TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=eno1 UUID=1f9ec889-3c64-470a-894b-05543ee44c29 DEVICE=eno1 ONBOOT=yes IPADDR=1.1.1..234 PREFIX=29 GATEWAY=1.1.1.233 IPV6_PRIVACY=no 4) Yes 5) [root at server ~]# nmcli d show | grep IP4.GATEWA IP4.GATEWAY: 1.1.1.233 nmtui shows the same gateway. Kind regards, Asle Ommundsen
Simon Matter
2020-Jan-17 13:39 UTC
[CentOS] After upgrade to CentOS 8.1 default gateway missing
> On Thu, 16 Jan 2020 15:34:43 +0100, Stephen John Smoogen > <smooge at gmail.com> wrote: > >> On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen <aommundsen at gmail.com> >> wrote: >>> >>> Hi, >>> >>> Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after >>> a >>> reboot of the first server the network was unavailable. In IPMI console >>> everything except the network was looking good. Network was >>> unreachable. >>> No errors in NetworkManager. I also restarted NetworkManager, but it >>> did >>> not help. Then I discovered that the default gateway suddenly was >>> missing. >>> >>> Then I rebooted the server one more time, but network was still down. >>> >>> Then both myself and a technician in my datcenter was debugging this (I >>> had to wake him up in the middle of the night, costing me a lot of >>> money), >>> without finding any reason for why the default gateway was missing >>> after >>> reboot. >>> >>> Then we rebooted the server a third time, and all of a sudden the >>> problem >>> was gone and the default gateway was back. [...cut...] > >> In order to determine what is going on you need to give a lot more >> information. >> >> 1. How do these boxes get their network information? DHCP or static >> 2. If they are static, what controls the setting of ips: >> NetworkManager or network-scripts >> 3. If they are static, how are they set in >> /etc/sysconfig/network-scripts/ >> 4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY>> 5. If you are using network-manager, what does nmtui or the graphical >> tool say the gateway or default route is? > > Here is answers to your list. I have anonymized the some of the data: > > 1) Static ip configuration > > 2) This should be NetworkManager. > nmcli output: > > eno1: connected to eno1 > inet4 1.1.1.234/29 > route4 1.1.1.232/29 > route4 0.0.0.0/0 > > eno2: connected to eno2 > inet4 192.168.0.5/24 > route4 192.168.0.0/24 > > [root at server ~]# nmcli d show | grep IP4.GATEWA > IP4.GATEWAY: 1.1.1.1.233 > > 3) > TYPE=Ethernet > PROXY_METHOD=none > BROWSER_ONLY=no > BOOTPROTO=none > DEFROUTE=yes > IPV4_FAILURE_FATAL=yes > IPV6INIT=yes > IPV6_AUTOCONF=yes > IPV6_DEFROUTE=yes > IPV6_FAILURE_FATAL=no > IPV6_ADDR_GEN_MODE=stable-privacy > NAME=eno1 > UUID=1f9ec889-3c64-470a-894b-05543ee44c29 > DEVICE=eno1 > ONBOOT=yes > IPADDR=1.1.1..234 > PREFIX=29 > GATEWAY=1.1.1.233 > IPV6_PRIVACY=noAnything in the logs about what was going on? If you reboot this server again and again, does the problem show up again? Regards, Simon
Asle Ommundsen
2020-Jan-17 14:12 UTC
[CentOS] After upgrade to CentOS 8.1 default gateway missing
On Fri, 17 Jan 2020 14:39:19 +0100, Simon Matter via CentOS <centos at centos.org> wrote:> Anything in the logs about what was going on? If you reboot this server > again and again, does the problem show up again?This is shared hosting servers that is in production with customers live sites. So I don't want to reboot again if not absolutely needed. I guess I will find out if the problem continues the next time there is a new kernel. I would like to avoid doing reboots before a new kernel is released. I have not had time to go over the logs yet. I was hoping that somone else experienced the same, but it does not seems so. I can't believe I would be the only one in the world this would happen to, and also that it happened on both my CentOS 8 servers. Kind regards, Asle Ommundsen
Stephen John Smoogen
2020-Jan-17 17:42 UTC
[CentOS] After upgrade to CentOS 8.1 default gateway missing
On Fri, 17 Jan 2020 at 07:37, Asle Ommundsen <aommundsen at gmail.com> wrote:> > On Thu, 16 Jan 2020 15:34:43 +0100, Stephen John Smoogen > <smooge at gmail.com> wrote: > > > On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen <aommundsen at gmail.com> > > wrote: > >> > >> Hi, > >> > >> Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a > >> reboot of the first server the network was unavailable. In IPMI console > >> everything except the network was looking good. Network was unreachable. > >> No errors in NetworkManager. I also restarted NetworkManager, but it did > >> not help. Then I discovered that the default gateway suddenly was > >> missing. > >> > >> Then I rebooted the server one more time, but network was still down. > >> > >> Then both myself and a technician in my datcenter was debugging this (I > >> had to wake him up in the middle of the night, costing me a lot of > >> money), > >> without finding any reason for why the default gateway was missing after > >> reboot. > >> > >> Then we rebooted the server a third time, and all of a sudden the > >> problem > >> was gone and the default gateway was back. [...cut...] > > > In order to determine what is going on you need to give a lot more > > information. > > > > 1. How do these boxes get their network information? DHCP or static > > 2. If they are static, what controls the setting of ips: > > NetworkManager or network-scripts > > 3. If they are static, how are they set in > > /etc/sysconfig/network-scripts/ > > 4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY> > 5. If you are using network-manager, what does nmtui or the graphical > > tool say the gateway or default route is? > > Here is answers to your list. I have anonymized the some of the data: >Thank you for the detailed answers.. they eliminated most of the 'easy-to-fix' problems. If you do not have ipv6 you might want to turn that off as I have seen some problems where a router gives enough info to cause routing issues but shouldn't have. If you do have ipv6 and it works.. then never mind. The only other confused one i have seen is where eno1 and eno2 both have DEFROUTE=yes defined.. and you can't do that. Otherwise.. I am not sure and it will take going through the logs or repeating it happen to diagnose better.> 1) Static ip configuration > > 2) This should be NetworkManager. > nmcli output: > > eno1: connected to eno1 > inet4 1.1.1.234/29 > route4 1.1.1.232/29 > route4 0.0.0.0/0 > > eno2: connected to eno2 > inet4 192.168.0.5/24 > route4 192.168.0.0/24 > > [root at server ~]# nmcli d show | grep IP4.GATEWA > IP4.GATEWAY: 1.1.1.1.233 > > 3) > TYPE=Ethernet > PROXY_METHOD=none > BROWSER_ONLY=no > BOOTPROTO=none > DEFROUTE=yes > IPV4_FAILURE_FATAL=yes > IPV6INIT=yes > IPV6_AUTOCONF=yes > IPV6_DEFROUTE=yes > IPV6_FAILURE_FATAL=no > IPV6_ADDR_GEN_MODE=stable-privacy > NAME=eno1 > UUID=1f9ec889-3c64-470a-894b-05543ee44c29 > DEVICE=eno1 > ONBOOT=yes > IPADDR=1.1.1..234 > PREFIX=29 > GATEWAY=1.1.1.233 > IPV6_PRIVACY=no > > 4) Yes > > 5) > [root at server ~]# nmcli d show | grep IP4.GATEWA > IP4.GATEWAY: 1.1.1.233 > > nmtui shows the same gateway. > > Kind regards, > Asle Ommundsen > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos-- Stephen J Smoogen.