CentOS - Jan 2020 - After upgrade to CentOS 8.1 default gateway missing

Hi,

Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a  
reboot of the first server the network was unavailable. In IPMI console  
everything except the network was looking good. Network was unreachable.  
No errors in NetworkManager. I also restarted NetworkManager, but it did  
not help. Then I discovered that the default gateway suddenly was missing.

Then I rebooted the server one more time, but network was still down.

Then both myself and a technician in my datcenter was debugging this (I  
had to wake him up in the middle of the night, costing me a lot of money),  
without finding any reason for why the default gateway was missing after  
reboot.

Then we rebooted the server a third time, and all of a sudden the problem  
was gone and the default gateway was back.

Then, after this, I also upgraded my second CentOS 8 server to CentOS 8.1  
and did a reboot. And the very same thing happened to this server after  
reboot! The default gateway was missing after reboot, and network was  
down. Then I did a extra reboot of this server also, and when it came back  
up everything was working correct and the default gateway was back.

So the first server I needed to reboot two extra times to have the default  
gateway back and network working. And the second server I needed to reboot  
only one extra time for the problem to be solved.

The two boxes is not VPSs or anything, but bare metal dedicated servers.

Also prior to upgrading to CentOS 8.1, these two boxes has been rebooted  
serveral times previously, without any problems at all. Only after  
upgrading to CentOS 8.1 this happened on both of them for the first time.

I can't believe I am the only one that experience this? My guess is that  
this is a unknown random intermittent bug in CentOS 8.1 that kicked in. I  
just hope this does not continue to happen in the future. If you  
experienced the same, please share it with a reply. Thank you!

Kind regards,
Asle Ommundsen

On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen <aommundsen at gmail.com>
wrote:
>
> Hi,
>
> Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a
> reboot of the first server the network was unavailable. In IPMI console
> everything except the network was looking good. Network was unreachable.
> No errors in NetworkManager. I also restarted NetworkManager, but it did
> not help. Then I discovered that the default gateway suddenly was missing.
>
> Then I rebooted the server one more time, but network was still down.
>
> Then both myself and a technician in my datcenter was debugging this (I
> had to wake him up in the middle of the night, costing me a lot of money),
> without finding any reason for why the default gateway was missing after
> reboot.
>
> Then we rebooted the server a third time, and all of a sudden the problem
> was gone and the default gateway was back.
>
> Then, after this, I also upgraded my second CentOS 8 server to CentOS 8.1
> and did a reboot. And the very same thing happened to this server after
> reboot! The default gateway was missing after reboot, and network was
> down. Then I did a extra reboot of this server also, and when it came back
> up everything was working correct and the default gateway was back.
>
> So the first server I needed to reboot two extra times to have the default
> gateway back and network working. And the second server I needed to reboot
> only one extra time for the problem to be solved.
>
> The two boxes is not VPSs or anything, but bare metal dedicated servers.
>
> Also prior to upgrading to CentOS 8.1, these two boxes has been rebooted
> serveral times previously, without any problems at all. Only after
> upgrading to CentOS 8.1 this happened on both of them for the first time.
>
> I can't believe I am the only one that experience this? My guess is
that
> this is a unknown random intermittent bug in CentOS 8.1 that kicked in. I
> just hope this does not continue to happen in the future. If you
> experienced the same, please share it with a reply. Thank you!
In order to determine what is going on you need to give a lot more information.

1. How do these boxes get their network information? DHCP or static
2. If they are static, what controls the setting of ips:
NetworkManager or network-scripts
3. If they are static, how are they set in /etc/sysconfig/network-scripts/
4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY5. If you are
using network-manager, what does nmtui or the graphical
tool say the gateway or default route is?

> Kind regards,
> Asle Ommundsen
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos


-- 
Stephen J Smoogen.

On Thu, 16 Jan 2020 15:34:43 +0100, Stephen John Smoogen  
<smooge at gmail.com> wrote:
> On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen <aommundsen at
gmail.com>
> wrote:
>>
>> Hi,
>>
>> Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after
a
>> reboot of the first server the network was unavailable. In IPMI console
>> everything except the network was looking good. Network was
unreachable.
>> No errors in NetworkManager. I also restarted NetworkManager, but it
did
>> not help. Then I discovered that the default gateway suddenly was  
>> missing.
>>
>> Then I rebooted the server one more time, but network was still down.
>>
>> Then both myself and a technician in my datcenter was debugging this (I
>> had to wake him up in the middle of the night, costing me a lot of  
>> money),
>> without finding any reason for why the default gateway was missing
after
>> reboot.
>>
>> Then we rebooted the server a third time, and all of a sudden the  
>> problem
>> was gone and the default gateway was back. [...cut...]
> In order to determine what is going on you need to give a lot more  
> information.
>
> 1. How do these boxes get their network information? DHCP or static
> 2. If they are static, what controls the setting of ips:
> NetworkManager or network-scripts
> 3. If they are static, how are they set in  
> /etc/sysconfig/network-scripts/
> 4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY> 5. If
you are using network-manager, what does nmtui or the graphical
> tool say the gateway or default route is?
Here is answers to your list. I have anonymized the some of the data:

1) Static ip configuration

2) This should be NetworkManager.
nmcli output:

eno1: connected to eno1
         inet4 1.1.1.234/29
         route4 1.1.1.232/29
         route4 0.0.0.0/0

eno2: connected to eno2
         inet4 192.168.0.5/24
         route4 192.168.0.0/24

[root at server ~]# nmcli d show | grep IP4.GATEWA
IP4.GATEWAY: 1.1.1.1.233

3)
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno1
UUID=1f9ec889-3c64-470a-894b-05543ee44c29
DEVICE=eno1
ONBOOT=yes
IPADDR=1.1.1..234
PREFIX=29
GATEWAY=1.1.1.233
IPV6_PRIVACY=no

4) Yes

5)
[root at server ~]# nmcli d show | grep IP4.GATEWA
IP4.GATEWAY: 1.1.1.233

nmtui shows the same gateway.

Kind regards,
Asle Ommundsen