CentOS - Jul 2021 - CentOS 8: two external network adapters, two ISPs - routing problems

Hello!

Given: a CentOS 8-powered computer with three network adapters.

eth0, eth2: external, connected to two different ISPs
eth1: faces home network (intranet)

The task: allow accessing certain internal services from either ISP. 
There are several services, I only mention SSH below.

In the configs below:
IP1: external IP at first ISP (ISP1), assigned to eth0
Gateway1: IP of gateway provided by ISP1
Network1,Netmask1: related to IP1
IP2: external IP at second ISP (ISP2), assigned to eth2
Gateway2: IP of gateway provided by ISP2
Network2,Netmask2: related to IP2
LocalSSHIP: IP in intranet (eth1) where SSH server is running

Current configs follow. Routing tables:

echo "200 isp1" >> /etc/iproute2/rt_tables
echo "201 isp2" >> /etc/iproute2/rt_tables

Routing policies:
/etc/sysconfig/network-scripts/route-eth0

Network1 dev eth0 src IP1 table isp1
default via Gateway1 dev eth0 table isp1

/etc/sysconfig/network-scripts/route-eth2

Network2 dev eth2 src IP2 table isp2
default via Gateway2 dev eth2 table isp2

Routing rules:
/etc/sysconfig/network-scripts/rule-eth0

from IP1/32 table isp1

/etc/sysconfig/network-scripts/rule-eth2

from IP2/32 table isp2

iptables snippets. External traffic forwarded to local SSH server from 
both interfaces:

iptables -A PREROUTING -t nat -i eth0 -p tcp -d IP1 --dport 22 -j DNAT 
--to LocalSSHIP:22
iptables -A PREROUTING -t nat -i eth2 -p tcp -d IP2 --dport 22 -j DNAT 
--to LocalSSHIP:22
iptables -A FORWARD -p tcp -d LocalSSHIP --dport 22 -j ACCEPT

eth0 is default gateway:
$ ip route

default via Gateway1 dev eth0 proto static metric 100
default via Gateway2 dev eth2 proto static metric 101
...

$ ip rule

0:  from all lookup local
32764:  from IP2 lookup isp2
32765:  from IP1 lookup isp1
32766:  from all lookup main
32767:  from all lookup default

SNAT is applied for the traffic originating from eth1:

iptables -t nat -A POSTROUTING -i eth1 -o eth0 -j SNAT --to-source IP1

Current situation:

- All services forwarded from eth0 are working normally.
- All traffic originating from intranet passes out and back normally.
- All the attempts to access services from eth2 time out.

There are no obvious hints in /var/log/messages (such as complaints 
about "martian IPs").

I am somewhat at a loss here, all the pieces of advice would be very 
welcome.

-- 
Sincerely,

Konstantin Boyandin

On 7/30/21 4:27 PM, Konstantin Boyandin via CentOS wrote:
> Current situation:
> 
> - All services forwarded from eth0 are working normally.
> - All traffic originating from intranet passes out and back normally.
> - All the attempts to access services from eth2 time out.
> 
> There are no obvious hints in /var/log/messages (such as complaints
> about "martian IPs").
> 
> I am somewhat at a loss here, all the pieces of advice would be very
> welcome.
> 
The setup looks reasonable at first sight, and it is similar to what I have (on
a C7).
Have you tried some tcpdump on the interfaces?
If the src-based routing rules are not working well, it happens that
your answers go out the wrong ethernet and nothing works (especially if your
provider doesn't want to forward what looks like spoofed packets).

Regards.

-- 
    Roberto Ragusa    mail at robertoragusa.it

You need to SNAT the outbound traffic on eth0 and eth2 to use the 
interface address that the packets leave from.

On 31/07/2021 02:27, Konstantin Boyandin via CentOS
wrote:
> Hello!
>
> Given: a CentOS 8-powered computer with three network adapters.
>
> eth0, eth2: external, connected to two different ISPs
> eth1: faces home network (intranet)
>
> The task: allow accessing certain internal services from either ISP.
> There are several services, I only mention SSH below.
>
> In the configs below:
> IP1: external IP at first ISP (ISP1), assigned to eth0
> Gateway1: IP of gateway provided by ISP1
> Network1,Netmask1: related to IP1
> IP2: external IP at second ISP (ISP2), assigned to eth2
> Gateway2: IP of gateway provided by ISP2
> Network2,Netmask2: related to IP2
> LocalSSHIP: IP in intranet (eth1) where SSH server is running
>
> Current configs follow. Routing tables:
>
> echo "200 isp1" >> /etc/iproute2/rt_tables
> echo "201 isp2" >> /etc/iproute2/rt_tables
>
> Routing policies:
> /etc/sysconfig/network-scripts/route-eth0
>
> Network1 dev eth0 src IP1 table isp1
> default via Gateway1 dev eth0 table isp1
>
> /etc/sysconfig/network-scripts/route-eth2
>
> Network2 dev eth2 src IP2 table isp2
> default via Gateway2 dev eth2 table isp2
>
> Routing rules:
> /etc/sysconfig/network-scripts/rule-eth0
>
> from IP1/32 table isp1
>
> /etc/sysconfig/network-scripts/rule-eth2
>
> from IP2/32 table isp2
>
> iptables snippets. External traffic forwarded to local SSH server from
> both interfaces:
>
> iptables -A PREROUTING -t nat -i eth0 -p tcp -d IP1 --dport 22 -j DNAT
> --to LocalSSHIP:22
> iptables -A PREROUTING -t nat -i eth2 -p tcp -d IP2 --dport 22 -j DNAT
> --to LocalSSHIP:22
> iptables -A FORWARD -p tcp -d LocalSSHIP --dport 22 -j ACCEPT
>
> eth0 is default gateway:
> $ ip route
>
> default via Gateway1 dev eth0 proto static metric 100
> default via Gateway2 dev eth2 proto static metric 101
> ...
>
> $ ip rule
>
> 0:  from all lookup local
> 32764:  from IP2 lookup isp2
> 32765:  from IP1 lookup isp1
> 32766:  from all lookup main
> 32767:  from all lookup default
>
> SNAT is applied for the traffic originating from eth1:
>
> iptables -t nat -A POSTROUTING -i eth1 -o eth0 -j SNAT --to-source IP1
>
> Current situation:
>
> - All services forwarded from eth0 are working normally.
> - All traffic originating from intranet passes out and back normally.
> - All the attempts to access services from eth2 time out.
>
> There are no obvious hints in /var/log/messages (such as complaints
> about "martian IPs").
>
> I am somewhat at a loss here, all the pieces of advice would be very
> welcome.
>