CentOS - Dec 2015 - Network services start before network is up since migrating to 7.2

Hello James,
> Well it looks like you are using the network service rather than the
> recommended NetworkManager ...
Yes. That's the way our security experts made the models I use to setup my
servers.
I'll test a migration to NetworkManager, and take their advice on it.
> 
> The network service is not blocking  the flow so it executes and systemd
> carries on ...
> 
> From the point of view of the system as soon as /etc/init.d/network start
> has been called the service is running as a state... as you can see from
> your logs lots of other services also start before the network interface
> itself is up.
I understand this, but why only on one of my servers ? Is the order the services
start only a question of latencies ?
> 
> There's a few of different ways of accomplishing what you want ...
> 
> Keep in mind that you must not edit files in /usr/lib/systemd/ if you want
> to maintain your sanity for future updates... use overrides in
> /etc/systemd/system/foo.service.d
Ok. Thank you for the tip. I'm trying to avoid this workaround, anyway.
> 
> The real reason httpd/sshd/snmpd failed there is that unlike the default
> configuration of these you aren't listening on all addresses (:: or
> 0.0.0.0) but on a specific 172.X address ... which isn't present until
the
> network adaptor is up and configured.
It is by design, for security considerations. So I can't make the services
listen on all interfaces.
> 3) Provide overrides for each service to order it after
> network-online.target (which is effectively when the non-local IP address
> can be found on the interface) as per the systemd.special man page
> documenting this.
> 
> Look at man systemd.special for more detail on this ...
I'll take a look on this.
> 
> Incidentally I just tried a quick test in a VM and it would appear
> NetworkManager.service completed with an IP on the network interface before
> network.target was considered reached ... you may want to test this on your
> system to see if it's a race condition or it actually works out that
way
> for you as a systemctl cat NetworkManager indicates it should be before
> network and it looks like it may block progress until it's on dbus ...
Ok, I'll try, and see if that solves my problem. Thank you.

Sylvain CANOINE.

Pensez ENVIRONNEMENT : n'imprimer que si ncessaire

On 12/17/2015 07:59 AM, Sylvain CANOINE wrote:
>> Well it looks like you are using the network service rather than the
>> recommended NetworkManager ...
> Yes. That's the way our security experts made the models I use to setup
my servers.
> I'll test a migration to NetworkManager, and take their advice on it.
Note that the behavior you're seeing is documented:
http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

If you're using NetworkManager, you can "systemctl enable 
NetworkManager-wait-online.service" and you won't have to override any 
of the individual services.
>> The network service is not blocking  the flow so it executes and
systemd
>> carries on ...
>>
>>  From the point of view of the system as soon as /etc/init.d/network
start
>> has been called the service is running as a state... as you can see
from
>> your logs lots of other services also start before the network
interface
>> itself is up.
> I understand this, but why only on one of my servers ? Is the order the
services start only a question of latencies ?
Basically, yes.
>
>> Incidentally I just tried a quick test in a VM and it would appear
>> NetworkManager.service completed with an IP on the network interface
before
>> network.target was considered reached ... you may want to test this on
your
>> system to see if it's a race condition
It is, as documented above.

> If you're using NetworkManager, you can "systemctl enable
> NetworkManager-wait-online.service" and you won't have to override
any
> of the individual services.
Our security experts don't want me to use NetworkManager... It's even
uninstalled on the models, so I understand better why all the required files are
not here :

# systemctl status NetworkManager-wait-online.service
? NetworkManager-wait-online.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

So I made a crappy but easy-to-deploy script to make the services start after
network is online :

for fic in $(grep -rl "After=.*network.target" /lib/systemd/system |
cut -d/ -f5 | grep -v "network-online.target")
do
   [ ! -d "/etc/systemd/system/${fic}.d" ] && mkdir -v
"/etc/systemd/system/${fic}.d"
   echo -e "[Unit]\nAfter=network-online.target" >
"/etc/systemd/system/${fic}.d/local-network-online.conf" &&
echo "/etc/systemd/system/${fic}.d/local-network-online.conf"
done
systemctl daemon-reload

That's working as is, so I'll keep this workaround for now.

Sylvain.
Pensez ENVIRONNEMENT : n'imprimer que si ncessaire

----- Mail original -----
> De: "Gordon Messmer" <gordon.messmer at gmail.com>
> ?: "centos" <centos at centos.org>
> Envoy?: Vendredi 18 D?cembre 2015 12:06:26
> Objet: Re: [CentOS] Network services start before network is up since
migrating to 7.2
>>> The network service is not blocking  the flow so it executes and
systemd
>>> carries on ...
>>>
>>>  From the point of view of the system as soon as
/etc/init.d/network start
>>> has been called the service is running as a state... as you can see
from
>>> your logs lots of other services also start before the network
interface
>>> itself is up.
>> I understand this, but why only on one of my servers ? Is the order the
services
>> start only a question of latencies ?
I'm confused. I updated two more servers this afternoon, and... all is
working well. The services start in correct order. Even after three reboots.
So only one of the (now) five updated servers doesn't start properly.

Then what is the difference ? All I see for now is the network.target unit seems
not active on the failing server.

(failing) # systemctl list-units|grep network
network.service                                                                 
loaded active exited    LSB: Bring up/down networking
rhel-import-state.service                                                       
loaded active exited    Import network configuration from initramfs
network-online.target                                                           
loaded active active    Network is Online
(failing) # systemctl status network
? network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network)
   Active: active (exited) since lun. 2015-12-21 12:49:31 CET; 1 day 5h ago
     Docs: man:systemd-sysv-generator(8)

d?c. 21 12:49:35 (failing) systemd[1]: Starting LSB: Bring up/down networking...
d?c. 21 12:49:26 (failing) network[747]: Activation de l'interface loopback
:  [  OK  ]
d?c. 21 12:49:28 (failing) network[747]: Activation de l'interface ens160 : 
[  OK  ]
d?c. 21 12:49:31 (failing) network[747]: Activation de l'interface ens192 : 
[  OK  ]
d?c. 21 12:49:31 (failing) systemd[1]: Started LSB: Bring up/down networking.


(correct) # systemctl list-units|grep network
network.service                                                                 
loaded active exited    LSB: Bring up/down networking
rhel-import-state.service                                                       
loaded active exited    Import network configuration from initramfs
network-online.target                                                           
loaded active active    Network is Online
network.target                                                                  
loaded active active    Network
(correct) # systemctl status network
? network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network)
   Active: active (exited) since mar. 2015-12-22 17:42:15 CET; 33min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 753 ExecStart=/etc/rc.d/init.d/network start (code=exited,
status=0/SUCCESS)

d?c. 22 17:42:07 (correct) systemd[1]: Starting LSB: Bring up/down networking...
d?c. 22 17:42:10 (correct) network[753]: Activation de l'interface loopback
:  [  OK  ]
d?c. 22 17:42:13 (correct) NET[935]: /etc/sysconfig/network-scripts/ifup-post :
updated /etc/resolv.conf
d?c. 22 17:42:13 (correct) network[753]: Activation de l'interface ens160 : 
[  OK  ]
d?c. 22 17:42:15 (correct) network[753]: Activation de l'interface ens192 : 
[  OK  ]
d?c. 22 17:42:15 (correct) systemd[1]: Started LSB: Bring up/down networking.


To be continued...

Sylvain.

Pensez ENVIRONNEMENT : n'imprimer que si ncessaire