CentOS - Jan 2017 - disable/mask NetworkManager leads to unit startup fails

Hello!

 

For me it was best practice to disable "NetworkManager" on headless
installations.

Now suddenly I ran into an problem with several programs not starting
correctly upon boot anymore. 

The problem seems to be that their unit files contain
"After=network.target"
but network.target wont wait till network is up and working, just waits for
some low level network stuff.

 

Now I read that it would be good practice to edit this and use
"After=network-online.target" instead. But I don't want to touch
several
systemd unit files shipped via rpm packages. 

 

Software like unbound, postfix and so on which needs to bind to ip+port on
startup fails on my systems now since I disabled NetworkManager.

 

What solved the problem for me was to install "systemd-networkd"
instead and
enable it upon boot.

It's system unit file pulls in "network.target" as
"Wants" this way software
that says "After=network.target" will start after network cards are
configured. Why there is this behavior I haven't found out yet. 

 

Whats your best practice to archieve stable networking upon reboot? Do you
use NetworkManager or systemd-networkd or maybe something exotic anyway?

 

PS: I have already filed an Bugzilla report here:
https://bugzilla.redhat.com/show_bug.cgi?id=1412570

 

Thanks!

On 16 January 2017 at 15:24, Oberdorfer Patrick
<patrick.oberdorfer at a1telekom.at> wrote:
> Hello!
>
>
>
> For me it was best practice to disable "NetworkManager" on
headless
> installations.
>
> Now suddenly I ran into an problem with several programs not starting
> correctly upon boot anymore.
>
> The problem seems to be that their unit files contain
"After=network.target"
> but network.target wont wait till network is up and working, just waits for
> some low level network stuff.
>
>
>
> Now I read that it would be good practice to edit this and use
> "After=network-online.target" instead. But I don't want to
touch several
> systemd unit files shipped via rpm packages.
>
>
>
> Software like unbound, postfix and so on which needs to bind to ip+port on
> startup fails on my systems now since I disabled NetworkManager.
>
>
>
> What solved the problem for me was to install "systemd-networkd"
instead and
> enable it upon boot.
>
> It's system unit file pulls in "network.target" as
"Wants" this way software
> that says "After=network.target" will start after network cards
are
> configured. Why there is this behavior I haven't found out yet.
>
>
>
> Whats your best practice to archieve stable networking upon reboot? Do you
> use NetworkManager or systemd-networkd or maybe something exotic anyway?
>
>
>

Use NetworkManager ... it's the supported RH configuration tool.

This article is a little old (7.3 had a NM rebase and I've not
revisited it yet ... the prime gain in 7.3 though is arbitrary
layering of vlan/bridge/bond/etc negating the caveat at the end for
edge case).

https://www.hogarthuk.com/?q=node/8

As for the targets ... man systemd.special to get a better understanding:

https://www.freedesktop.org/software/systemd/man/systemd.special.html

By default services you install will listen on all interfaces, usually
a configuration with something like ListenAddress :: to indicate this
(there is an internal ipv6->ipv4 translation going on so you only
should listen on :: and not 0.0.0.0 as well) ... when a service binds
to a port on all addresses then the address does not need to exist yet
- the kernel just handles it.

If you configure your service to bind on a specific IP though then
they way the binding system call works it already needs to exist (by
default) which is why with the default configuration if the service
tries to start before the IPs are applied (quite likely given parallel
startup) you have a nice race condition and a fairly high likelihood
of failure.

There's two ways to handle this - and it depends on the service.

You can order it via after (and potential requires if you want no
attempt to start the service if the target fails to activate)
network-online.target (see referenced man page) which won't be
complete until at least one IP is on any interface.

If it's your own application and you can set the flags on the actual
bind then you can use so_freebind on the socket opened, some
applications may have this as a compile time or configurable option.

Another option is to enable net.ipv4.ip_nonlocal_bind and
net.ipv6.ip_nonlocal_bind which allows the kernel to bind a socket on
an IP it doesn't have (do note the caveat it may break some things).

https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

If it's a systemd socket being used by the service then you can enable
the freebind option via FreeBind=true

https://www.freedesktop.org/software/systemd/man/systemd.socket.html

Overall you may be interested in this:

https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

I really wouldn't use systemd-network on CentOS as a general thing as
it's basically undocumented/unsupported by Red Hat at this time, with
NM being their actual supported solution.

https://access.redhat.com/solutions/783533 <-- network Vs NetworkManager

https://access.redhat.com/solutions/876183 <-- no systemd alternative
for NM in EL7


Hope that helps and explains things!

Thank you very much James for your deep explaining answer!
That cleared everything up for me and yes that helped a lot.

Have a nice day :)

-----Urspr?ngliche Nachricht-----
Von: CentOS [mailto:centos-bounces at centos.org] Im Auftrag von James Hogarth
Gesendet: Montag, 16. J?nner 2017 19:37
An: CentOS mailing list <centos at centos.org>
Betreff: Re: [CentOS] disable/mask NetworkManager leads to unit startup
fails

On 16 January 2017 at 15:24, Oberdorfer Patrick
<patrick.oberdorfer at a1telekom.at> wrote:
> Hello!
>
>
>
> For me it was best practice to disable "NetworkManager" on
headless
> installations.
>
> Now suddenly I ran into an problem with several programs not starting 
> correctly upon boot anymore.
>
> The problem seems to be that their unit files contain
"After=network.target"
> but network.target wont wait till network is up and working, just 
> waits for some low level network stuff.
>
>
>
> Now I read that it would be good practice to edit this and use 
> "After=network-online.target" instead. But I don't want to
touch
> several systemd unit files shipped via rpm packages.
>
>
>
> Software like unbound, postfix and so on which needs to bind to 
> ip+port on startup fails on my systems now since I disabled
NetworkManager.
>
>
>
> What solved the problem for me was to install "systemd-networkd" 
> instead and enable it upon boot.
>
> It's system unit file pulls in "network.target" as
"Wants" this way
> software that says "After=network.target" will start after
network
> cards are configured. Why there is this behavior I haven't found out
yet.
>
>
>
> Whats your best practice to archieve stable networking upon reboot? Do 
> you use NetworkManager or systemd-networkd or maybe something exotic
anyway?
>
>
>

Use NetworkManager ... it's the supported RH configuration tool.

This article is a little old (7.3 had a NM rebase and I've not revisited it
yet ... the prime gain in 7.3 though is arbitrary layering of
vlan/bridge/bond/etc negating the caveat at the end for edge case).

https://www.hogarthuk.com/?q=node/8

As for the targets ... man systemd.special to get a better understanding:

https://www.freedesktop.org/software/systemd/man/systemd.special.html

By default services you install will listen on all interfaces, usually a
configuration with something like ListenAddress :: to indicate this (there
is an internal ipv6->ipv4 translation going on so you only should listen on
:: and not 0.0.0.0 as well) ... when a service binds to a port on all
addresses then the address does not need to exist yet
- the kernel just handles it.

If you configure your service to bind on a specific IP though then they way
the binding system call works it already needs to exist (by
default) which is why with the default configuration if the service tries to
start before the IPs are applied (quite likely given parallel
startup) you have a nice race condition and a fairly high likelihood of
failure.

There's two ways to handle this - and it depends on the service.

You can order it via after (and potential requires if you want no attempt to
start the service if the target fails to activate) network-online.target
(see referenced man page) which won't be complete until at least one IP is
on any interface.

If it's your own application and you can set the flags on the actual bind
then you can use so_freebind on the socket opened, some applications may
have this as a compile time or configurable option.

Another option is to enable net.ipv4.ip_nonlocal_bind and
net.ipv6.ip_nonlocal_bind which allows the kernel to bind a socket on an IP
it doesn't have (do note the caveat it may break some things).

https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

If it's a systemd socket being used by the service then you can enable the
freebind option via FreeBind=true

https://www.freedesktop.org/software/systemd/man/systemd.socket.html

Overall you may be interested in this:

https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

I really wouldn't use systemd-network on CentOS as a general thing as
it's
basically undocumented/unsupported by Red Hat at this time, with NM being
their actual supported solution.

https://access.redhat.com/solutions/783533 <-- network Vs NetworkManager

https://access.redhat.com/solutions/876183 <-- no systemd alternative for NM
in EL7


Hope that helps and explains things!
_______________________________________________
CentOS mailing list
CentOS at centos.org
https://lists.centos.org/mailman/listinfo/centos