Hello,
This is on Centos 6 and not something I think is wrong with Centos 6
but I am looking to see if anybody else has experienced this and
if there is solution. So thanks up front for indulging me.
Because Linux makes routing decisions before SNAT it is causing
problems when trying to use FTP with two upstream providers in
a load balanced setup.
Other than ftp, things seem to work OK. Below is my setup and tcpdump
output that shows ftp packets trying to go out the wrong interface.
ip ru sh
0: from all lookup local
200: from y.y.y.174 lookup t1
201: from x.x.x.217 lookup t2
32766: from all lookup main
32767: from all lookup default
ip r s
y.y.y.129 dev eth1 scope link
172.16.0.0/29 dev gre1 proto kernel scope link src 172.16.0.1
y.y.y.128/25 dev eth1 proto kernel scope link src y.y.y.174
10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.90
192.168.198.0/24 dev eth0 proto kernel scope link src 192.168.198.92
x.x.x.0/24 dev eth2 proto kernel scope link src x.x.x.217
10.0.128.0/17 dev eth0 proto kernel scope link src 10.0.129.88
default
nexthop via y.y.y.129 dev eth1 weight 1
nexthop via x.x.x.1 dev eth2 weight 1
ip r s tab t1
default via y.y.y.129 dev eth1 src y.y.y.174
ip r s tab t2
default via x.x.x.1 dev eth2 src x.x.x.217
Chain PREROUTING (policy ACCEPT 1050K packets, 128M bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 423K packets, 35M bytes)
pkts bytes target prot opt in out source
destination
0 0 ACCEPT all -- * eth1 10.0.1.0/24 10.0.0.0/8
0 0 ACCEPT all -- * eth1 10.0.1.0/24
172.16.0.0/12
0 0 ACCEPT all -- * eth1 10.0.1.0/24
192.168.0.0/16
58 3480 SNAT all -- * eth1 10.0.1.0/24 0.0.0.0/0
to:y.y.y.174
4 240 SNAT all -- * eth2 10.0.1.0/24 0.0.0.0/0
to:x.x.x.217
lsmod | grep nf_
nf_conntrack_ipv6 7207 3
nf_defrag_ipv6 9873 1 nf_conntrack_ipv6
nf_nat_ftp 2602 0
nf_nat 18580 2 iptable_nat,nf_nat_ftp
nf_conntrack_ipv4 7694 6 iptable_nat,nf_nat
nf_defrag_ipv4 1039 1 nf_conntrack_ipv4
nf_conntrack_ftp 10475 1 nf_nat_ftp
nf_conntrack 65524 7
iptable_nat,nf_conntrack_ipv6,xt_state,nf_nat_ftp,nf_nat,nf_conntrack_ipv4,nf_conntrack_ftp
ipv6 264769 41
nf_conntrack_ipv6,nf_defrag_ipv6,ip6table_mangle,ip6_tunnel,tunnel6
connection starts out eth2 - but then subsequent packets that should be
routed out eth2 are being sent out eth1 see below.
eth2 x.x.x.217
tcpdump -nli eth2 host 131.247.254.5
16:23:06.062451 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [S], seq
1482565198, win 5840, options [mss 1460,sackOK,TS val 423546705 ecr 0,nop,wscale
6], length 0
16:23:06.076788 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [S.], seq
740341460, ack 1482565199, win 5792, options [mss 1460,sackOK,TS val 2565444838
ecr 423546705,nop,wscale 7], length 0
16:23:06.077224 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423546720 ecr 2565444838], length 0
16:23:06.096900 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565444858 ecr 423546720], length 96
16:23:06.316866 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565445077 ecr 423546720], length 96
16:23:06.764410 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565445515 ecr 423546720], length 96
16:23:07.634411 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565446391 ecr 423546720], length 96
16:23:09.394482 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565448143 ecr 423546720], length 96
16:23:12.886997 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565451647 ecr 423546720], length 96
16:23:19.892082 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565458655 ecr 423546720], length 96
16:23:33.907303 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565472671 ecr 423546720], length 96
16:24:01.935273 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565500703 ecr 423546720], length 96
16:24:57.993631 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565556767 ecr 423546720], length 96
16:26:50.107951 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97,
ack 1, win 46, options [nop,nop,TS val 2565668895 ecr 423546720], length 96
16:28:06.104031 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [FP.], seq
97:111,
ack 1, win 46, options [nop,nop,TS val 2565744900 ecr 423546720], length 14
These packets should be going out eth2 not eth1
eth1 y.y.y.174
tcpdump -pnli eth1 host 131.247.254.5
16:23:06.097415 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack
740341557, win 92, options [nop,nop,TS val 423546741 ecr 2565444858], length 0
16:23:06.317381 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423546961 ecr 2565445077,nop,nop,sack 1
{4294967201:1}], length 0
16:23:06.764908 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423547408 ecr 2565445515,nop,nop,sack 1
{4294967201:1}], length 0
16:23:07.634894 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423548278 ecr 2565446391,nop,nop,sack 1
{4294967201:1}], length 0
16:23:09.394972 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423550038 ecr 2565448143,nop,nop,sack 1
{4294967201:1}], length 0
16:23:12.887529 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423553531 ecr 2565451647,nop,nop,sack 1
{4294967201:1}], length 0
16:23:19.892616 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423560536 ecr 2565458655,nop,nop,sack 1
{4294967201:1}], length 0
16:23:33.907736 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423574551 ecr 2565472671,nop,nop,sack 1
{4294967201:1}], length 0
16:23:40.173991 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423580817 ecr 2565472671], length 13
16:23:40.388692 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423581032 ecr 2565472671], length 13
16:23:40.819714 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423581463 ecr 2565472671], length 13
16:23:41.680729 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423582324 ecr 2565472671], length 13
16:23:43.404732 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423584048 ecr 2565472671], length 13
16:23:46.852787 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423587496 ecr 2565472671], length 13
16:23:53.756879 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423594400 ecr 2565472671], length 13
16:24:01.935822 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423602578 ecr 2565500703,nop,nop,sack 1
{4294967201:1}], length 0
16:24:07.549037 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423608192 ecr 2565500703], length 13
16:24:35.133346 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423635776 ecr 2565500703], length 13
16:24:57.994150 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423658636 ecr 2565556767,nop,nop,sack 1
{4294967201:1}], length 0
16:25:30.365963 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423691008 ecr 2565556767], length 13
16:26:50.108488 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win
92, options [nop,nop,TS val 423770749 ecr 2565668895,nop,nop,sack 1
{4294967201:1}], length 0
16:27:20.703243 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13,
ack 1, win 92, options [nop,nop,TS val 423801344 ecr 2565668895], length 13
16:28:06.104578 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [F.], seq 13,
ack
16, win 92, options [nop,nop,TS val 423846744 ecr 2565744900], length 0
Is there a way to make this work correctly?
Thanks,
Steve
On 09/26/2012 09:15 AM, Steve Clark wrote:> Is there a way to make this work correctly?Shorewall will generate a proper configuration if you specify the "track" option in the "providers" file. It might be a good idea to use that to generate your configs rather than building them by hand. I believe that you need to mark your connections and use the marks to select the routing table, in addition to using the "from" rules that you posted. Otherwise, nothing binds the connection to a fixed route/interface in a load balanced configuration.
On Thu, Sep 27, 2012 at 7:04 PM, Steve Clark <sclark at netwolves.com> wrote:> On 09/26/2012 11:57 PM, Manish Kathuria wrote: > > On Thu, Sep 27, 2012 at 7:46 AM, Gordon Messmer <yinyang at eburg.com> wrote: > > On 09/26/2012 09:15 AM, Steve Clark wrote: > > Is there a way to make this work correctly? > > In addition, you should ideally applying the following patches for > Static, Alternative Routes, Dead Gateway Detection & NAT and recompile > the kernel: > > http://www.ssi.bg/~ja/#routes > > > Hmmm... not being a kernel guru, correct me if I am wrong but isn't the > route patch used to detect dead nexthops? > I am already doing that from userspace. > > The second set looks like is calls the routing logic after the SNAT, is that > correct? This could solve the problem. > Why aren't these patches in the kernel? > > Thanks, >The routes-x.y-z.diff is a unified patch containing different parts which include support for Dead Gateway Detection as well. However, since that is limited to the first hop, it is preferable to have a userspace script as you are doing. I also use a script to check the accessibility of a remote popular site from each of the ISPs and based upon the response the links are treated alive or dead and the default gateway is changed. However, the routing problem as described by you will only be solved after applying this patch (routes-x.y-z.diff). As for marking the incoming packets to ensure that they go out from the same interface they came from, you could do something like the following: Using iptables mark the incoming traffic from external interfaces /sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark /sbin/iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1 /sbin/iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark /sbin/iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2 /sbin/iptables -t mangle -A PREROUTING -i eth2 -j CONNMARK --save-mark Add the following rules to your existing ones for policy routing /sbin/ip rule add fwmark 1 table T1 /sbin/ip rule add fwmark 2 table T2 Thanks, -- Manish Kathuria
On 09/27/2012 06:36 AM, Steve Clark wrote:> I was trying to figure out what criteria to use to mark the connection. > FTP is such a > braindead application, using to channels and active and passive mode. > What really > needs to happen is someway to tell the kernel to recheck the routing > after SNAT.I'm mostly sure that if you mark the *connection* to the FTP server, the related data will follow its path. Again, multipath routing is complex, and Shorewall will do it properly. At the very least, I recommend building a working configuration with Shorewall and then reading the rules that it compiles to understand why it handles routing the way that it does.
On Thu, Sep 27, 2012 at 8:55 PM, Steve Clark <sclark at netwolves.com> wrote:> On 09/27/2012 11:01 AM, Manish Kathuria wrote: > > On Thu, Sep 27, 2012 at 7:04 PM, Steve Clark <sclark at netwolves.com> wrote: > > On 09/26/2012 11:57 PM, Manish Kathuria wrote: > > On Thu, Sep 27, 2012 at 7:46 AM, Gordon Messmer <yinyang at eburg.com> wrote: > > On 09/26/2012 09:15 AM, Steve Clark wrote: > > The routes-x.y-z.diff is a unified patch containing different parts > which include support for Dead Gateway Detection as well. However, > since that is limited to the first hop, it is preferable to have a > userspace script as you are doing. I also use a script to check the > accessibility of a remote popular site from each of the ISPs and based > upon the response the links are treated alive or dead and the default > gateway is changed. However, the routing problem as described by you > will only be solved after applying this patch (routes-x.y-z.diff). > > As for marking the incoming packets to ensure that they go out from > the same interface they came from, you could do something like the > following: > > Using iptables mark the incoming traffic from external interfaces > > /sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark > /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark > /sbin/iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1 > /sbin/iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark > /sbin/iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2 > /sbin/iptables -t mangle -A PREROUTING -i eth2 -j CONNMARK --save-mark > > Add the following rules to your existing ones for policy routing > > /sbin/ip rule add fwmark 1 table T1 > /sbin/ip rule add fwmark 2 table T2 > > Hi Manish, > > Thanks for the info. The one question I have is about > /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark > > I thought the OUTPUT chain was only for packets originating locally. I am > only concerned > with clients behind my Linux router, do I still need this?Yes you are right but in case if you have any services running on the linux router itself (for example sshd) and accessible from the internet, it would help.> > Again, thanks much for responding. > > -- > Stephen Clark > NetWolves > Director of Technology > Phone: 813-579-3200 > Fax: 813-882-0209 > Email: steve.clark at netwolves.com > http://www.netwolves.com
> > The routes-x.y-z.diff is a unified patch containing different parts > which include support for Dead Gateway Detection as well. However, > since that is limited to the first hop, it is preferable to have a > userspace script as you are doing. I also use a script to check the > accessibility of a remote popular site from each of the ISPs and based > upon the response the links are treated alive or dead and the default > gateway is changed. However, the routing problem as described by you > will only be solved after applying this patch (routes-x.y-z.diff). > > As for marking the incoming packets to ensure that they go out from > the same interface they came from, you could do something like the > following: > > Using iptables mark the incoming traffic from external interfaces > > /sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark > /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark > /sbin/iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1 > /sbin/iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark > /sbin/iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2 > /sbin/iptables -t mangle -A PREROUTING -i eth2 -j CONNMARK --save-mark > > Add the following rules to your existing ones for policy routing > > /sbin/ip rule add fwmark 1 table T1 > /sbin/ip rule add fwmark 2 table T2 > > Hi Manish, > > Thanks for the info. The one question I have is about > /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark > > I thought the OUTPUT chain was only for packets originating locally. I am > only concerned > with clients behind my Linux router, do I still need this? > > Yes you are right but in case if you have any services running on the > linux router itself (for example sshd) and accessible from the > internet, it would help. > > > Hi Manish, > > The above rules appear to be for clients coming into the router from > external. They > don't solve the problem for clients inside the router going out thru the > load balanced > interfaces. > > I have done much googling and testing without much luck. At this point in > time I would > be satisfied with just being able to have a client inside the router do FTP > over just one > of the outbound interfaces without any load balancing for FTP. I have this > working but only > for active mode FTP by using the following: > > /sbin/iptables -t mangle -A PREROUTING -p tcp --dport 20:21 -j MARK > --set-mark 1 > > But it doesn't work for passive because you don't know what ports are going > to be used. > > > Regards, > > -- > Stephen ClarkI was under the impression that you are running a FTP server inside and were facing problems with the incoming traffic for the same. If you are primarily concerned with the outgoing traffic through two ISP links, please follow the following steps: 1. Refer to http://www.ssi.bg/~ja/nano.txt for creating your rules. 2. Recompile the kernel after applying Julian Anistov's routes patch (the URL is there in the earlier messages). 3. Make a script to check the status of the links and change the default gateway accordingly. Let me know if you need a script. 4. Make sure that your firewall (iptables) is stateful and allows related and established connections and the NAT and connection tracking modules (nf_conntrack, nf_conntrack_ftp, nf_nat and nf_nat_ftp) are loaded. I have followed this approach at a number of places without any problems related to FTP or other protocols. The only issue I faced was that the patch failed for all the CentOS 5.x kernels I tried (perhaps due to some conflict with an existing patch). But its working perfectly for the kernels in CentOS 6 and 6.1. Thanks, -- Manish
Sounds like an issue similar to what I experienced when trying to force all outgoing ssh traffic on a NAT'ed network to go through a particular interface. I've forgot the details, but running the following on the firewall helped for f in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 > $f done I'm no expert in advanced routing, so if it breaks, you get to keep the pieces. Christian. On 09/26/2012 06:15 PM, Steve Clark wrote:> Hello, > > This is on Centos 6 and not something I think is wrong with Centos 6 > but I am looking to see if anybody else has experienced this and > if there is solution. So thanks up front for indulging me. > > Because Linux makes routing decisions before SNAT it is causing > problems when trying to use FTP with two upstream providers in > a load balanced setup. > > Other than ftp, things seem to work OK. Below is my setup and tcpdump > output that shows ftp packets trying to go out the wrong interface. > > ip ru sh > 0: from all lookup local > 200: from y.y.y.174 lookup t1 > 201: from x.x.x.217 lookup t2 > 32766: from all lookup main > 32767: from all lookup default > > ip r s > y.y.y.129 dev eth1 scope link > 172.16.0.0/29 dev gre1 proto kernel scope link src 172.16.0.1 > y.y.y.128/25 dev eth1 proto kernel scope link src y.y.y.174 > 10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.90 > 192.168.198.0/24 dev eth0 proto kernel scope link src 192.168.198.92 > x.x.x.0/24 dev eth2 proto kernel scope link src x.x.x.217 > 10.0.128.0/17 dev eth0 proto kernel scope link src 10.0.129.88 > default > nexthop via y.y.y.129 dev eth1 weight 1 > nexthop via x.x.x.1 dev eth2 weight 1 > > ip r s tab t1 > default via y.y.y.129 dev eth1 src y.y.y.174 > > ip r s tab t2 > default via x.x.x.1 dev eth2 src x.x.x.217 > > Chain PREROUTING (policy ACCEPT 1050K packets, 128M bytes) > pkts bytes target prot opt in out source destination > > Chain POSTROUTING (policy ACCEPT 423K packets, 35M bytes) > pkts bytes target prot opt in out source destination > 0 0 ACCEPT all -- * eth1 10.0.1.0/24 10.0.0.0/8 > 0 0 ACCEPT all -- * eth1 10.0.1.0/24 172.16.0.0/12 > 0 0 ACCEPT all -- * eth1 10.0.1.0/24 192.168.0.0/16 > 58 3480 SNAT all -- * eth1 10.0.1.0/24 0.0.0.0/0 > to:y.y.y.174 > 4 240 SNAT all -- * eth2 10.0.1.0/24 0.0.0.0/0 > to:x.x.x.217 > > lsmod | grep nf_ > nf_conntrack_ipv6 7207 3 > nf_defrag_ipv6 9873 1 nf_conntrack_ipv6 > nf_nat_ftp 2602 0 > nf_nat 18580 2 iptable_nat,nf_nat_ftp > nf_conntrack_ipv4 7694 6 iptable_nat,nf_nat > nf_defrag_ipv4 1039 1 nf_conntrack_ipv4 > nf_conntrack_ftp 10475 1 nf_nat_ftp > nf_conntrack 65524 7 > iptable_nat,nf_conntrack_ipv6,xt_state,nf_nat_ftp,nf_nat,nf_conntrack_ipv4,nf_conntrack_ftp > ipv6 264769 41 > nf_conntrack_ipv6,nf_defrag_ipv6,ip6table_mangle,ip6_tunnel,tunnel6 > > connection starts out eth2 - but then subsequent packets that should be > routed out eth2 are being sent out eth1 see below. > eth2 x.x.x.217 > tcpdump -nli eth2 host 131.247.254.5 > > 16:23:06.062451 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [S], seq > 1482565198, win 5840, options [mss 1460,sackOK,TS val 423546705 ecr 0,nop,wscale > 6], length 0 > 16:23:06.076788 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [S.], seq > 740341460, ack 1482565199, win 5792, options [mss 1460,sackOK,TS val 2565444838 > ecr 423546705,nop,wscale 7], length 0 > 16:23:06.077224 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423546720 ecr 2565444838], length 0 > 16:23:06.096900 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565444858 ecr 423546720], length 96 > 16:23:06.316866 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565445077 ecr 423546720], length 96 > 16:23:06.764410 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565445515 ecr 423546720], length 96 > 16:23:07.634411 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565446391 ecr 423546720], length 96 > 16:23:09.394482 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565448143 ecr 423546720], length 96 > 16:23:12.886997 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565451647 ecr 423546720], length 96 > 16:23:19.892082 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565458655 ecr 423546720], length 96 > 16:23:33.907303 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565472671 ecr 423546720], length 96 > 16:24:01.935273 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565500703 ecr 423546720], length 96 > 16:24:57.993631 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565556767 ecr 423546720], length 96 > 16:26:50.107951 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, > ack 1, win 46, options [nop,nop,TS val 2565668895 ecr 423546720], length 96 > 16:28:06.104031 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [FP.], seq 97:111, > ack 1, win 46, options [nop,nop,TS val 2565744900 ecr 423546720], length 14 > > > These packets should be going out eth2 not eth1 > eth1 y.y.y.174 > tcpdump -pnli eth1 host 131.247.254.5 > > 16:23:06.097415 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack > 740341557, win 92, options [nop,nop,TS val 423546741 ecr 2565444858], length 0 > 16:23:06.317381 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423546961 ecr 2565445077,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:23:06.764908 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423547408 ecr 2565445515,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:23:07.634894 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423548278 ecr 2565446391,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:23:09.394972 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423550038 ecr 2565448143,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:23:12.887529 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423553531 ecr 2565451647,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:23:19.892616 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423560536 ecr 2565458655,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:23:33.907736 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423574551 ecr 2565472671,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:23:40.173991 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423580817 ecr 2565472671], length 13 > 16:23:40.388692 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423581032 ecr 2565472671], length 13 > 16:23:40.819714 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423581463 ecr 2565472671], length 13 > 16:23:41.680729 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423582324 ecr 2565472671], length 13 > 16:23:43.404732 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423584048 ecr 2565472671], length 13 > 16:23:46.852787 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423587496 ecr 2565472671], length 13 > 16:23:53.756879 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423594400 ecr 2565472671], length 13 > 16:24:01.935822 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423602578 ecr 2565500703,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:24:07.549037 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423608192 ecr 2565500703], length 13 > 16:24:35.133346 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423635776 ecr 2565500703], length 13 > 16:24:57.994150 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423658636 ecr 2565556767,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:25:30.365963 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423691008 ecr 2565556767], length 13 > 16:26:50.108488 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win > 92, options [nop,nop,TS val 423770749 ecr 2565668895,nop,nop,sack 1 > {4294967201:1}], length 0 > 16:27:20.703243 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, > ack 1, win 92, options [nop,nop,TS val 423801344 ecr 2565668895], length 13 > 16:28:06.104578 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [F.], seq 13, ack > 16, win 92, options [nop,nop,TS val 423846744 ecr 2565744900], length 0 > > Is there a way to make this work correctly? > > Thanks, > Steve > > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >
On Wed, Oct 3, 2012 at 7:00 PM, Steve Clark <sclark at netwolves.com> wrote:> On 10/03/2012 08:46 AM, Manish Kathuria wrote: > > I was under the impression that you are running a FTP server inside > and were facing problems with the incoming traffic for the same. If > you are primarily concerned with the outgoing traffic through two ISP > links, please follow the following steps: > > 1. Refer to http://www.ssi.bg/~ja/nano.txt for creating your rules. > 2. Recompile the kernel after applying Julian Anistov's routes patch > (the URL is there in the earlier messages). > 3. Make a script to check the status of the links and change the > default gateway accordingly. Let me know if you need a script. > 4. Make sure that your firewall (iptables) is stateful and allows > related and established connections and the NAT and connection > tracking modules (nf_conntrack, nf_conntrack_ftp, nf_nat and > nf_nat_ftp) are loaded. > > I have followed this approach at a number of places without any > problems related to FTP or other protocols. The only issue I faced was > that the patch failed for all the CentOS 5.x kernels I tried (perhaps > due to some conflict with an existing patch). But its working > perfectly for the kernels in CentOS 6 and 6.1. > > Thanks, > -- > Manish > > Hi Manish, > > Thanks for the response. > It is good to know there is a general solution. It is too bad that > the referenced patches were never merged into to main kernel tree, forcing > people > to have to build and maintain their own kernel. > > > -- > Stephen ClarkIn case you want to avoid compiling the kernel and are comfortable with FreeBSD, try pfSense, it also offers outbound load balancing and failover for multiple WAN links. -- Manish Kathuria