LARTC - Mar 2001 - Balancing ip traffic over two or more internet (adsl) connections

Hi.

 I''ve got here the following configuration:
(turn on fixed pitch font to be able to see the scheme in a more
accurately way)


 Client machines <-----> ADSL Router 1 <-----> INTERNET
 (192.168.0.0/24)        (192.168.0.229)  NAT

 Client machines <-----> ADSL Router 1 <-----> INTERNET
 (192.168.1.0/24)        (192.168.1.229)  NAT


 [if doesn''t really matter if it''s NAT or PAT]

 Currently each group of client machines belong to different
departments.

 This is not efficient at all because you can have a satured ADSL
connection and a load free ADSL connection at the same time, i.e,
there isn''t any kind of load sharing. In addition filtering and
management issues are limited to ADSL router''s capacity. These kind of
routers don''t use to have too much functionality. If I want to do some
kind of filtering (firewall) I should repeat the work on both routers
(enter same rules twice, in a per-router basis).

 I want to switch to the following scheme:


 Client machines <-> Linux IP Load-Balancer <-> ADSL Router X
<-> INET
 (192.168.0.0/24)                               (192.168.*.229) NAT

 Client machines <-> Linux IP Load-Balancer <-> ADSL Router X
<-> INET
 (192.168.1.0/24)                               (192.168.*.229) NAT


 In this new scheme we have a machine which will act as an IP
load-balancer. Advantages:
- load sharing. Optimize our internet connections use.
- I can set up the firewall at this machine instead of using the ADSL
routers -> Easier management.
- More powerful management. I can make internet traffic stats, etc. In
summary, I can run any network analyzer tool which runs on Linux (I''m
not limited to ADSL router''s tools).
- Easier scalability: I can add as many new ADSL routers as I want.

 Please note that with this new scheme we get load balancing in *BOTH*
directions (outgoing and incoming) due to the NAT performing! I mean:
- outgoing balancing is assured by our Linux Ip Load Balancer
- incoming packets would return via the same router they were sent
across (this wouldn''t be in that way if our clients machine had real
IP numbers [no NAT performed]).


 I also want to set up a transparent proxy (squid) over the Linux
IP-Load Balance, mainly for www-traffic. In this way, if any user
performs a http query, it will be conducted by squid, which is more
efficient (it has got cache, eg) and "loggeable/verbose" (I could have
www access stats: which user accesses some web pages, etc). And this
is trasparent to our end-users (client machines).

 If the requested service (icq, eg) is not handled by squid it should
be performed by direct routing.

 I''m currently using 2.2.18 kernel (although if would be no problem to
switch to 2.4 if proved as necessary!!)

 Main issue to solve:
- "session handling": *ALL* packets belonging to a same
"session"
should be conducted by *ONE and only one* ADSL router. Different
sessions could go across different ADSL routers but a particular
session should use the *same router for all of its packets*. This way
we can assure that our outgoing packets don''t arrive out of order to
the dst host (different ADSL connections can have different delays,
etc). One "session" could be *at least*:
1) TCP connection (from SYN to RST packets. All included)
2) UDP packets belonging to a same "transaction" (for those apps which
use udp based protocols).

 I have set up a test machine to act as the "Linux IP Load balancer"
and activated the "CONFIG_IP_ROUTE_MULTIPATH". Then I''ve set
up two
default routes with "ip route" and
"equalize"-"nexthop" options. But
it isn''t working correctly. I think this method distribute packets in
a round robin fashion but doesn''t have the "session handling"
problem
into account, I fear.

 How could I achieve what I want??? Can I set up a "balanced routing"
in a per-session basis? Please, any ideas are welcome. Comments from
people running the described (or similar) configuration are also
welcome.

 Thanks in advance!

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-    ** RoMaN SoFt / LLFB **  
       roman@madrid.com
   http://pagina.de/romansoft
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Thu, 15 Mar 2001, RoMaN SoFt / LLFB!! wrote:
>  I have set up a test machine to act as the "Linux IP Load
balancer"
> and activated the "CONFIG_IP_ROUTE_MULTIPATH". Then I''ve
set up two
> default routes with "ip route" and
"equalize"-"nexthop" options. But
> it isn''t working correctly. I think this method distribute packets
in
> a round robin fashion but doesn''t have the "session
handling" problem
> into account, I fear.
Actually, the kernel does not necessarily send packets that already have
a source address out the interface associated with that address. That might
very well be your problem. The session problem does not really exist, as the
IP multipath code only multiplexes at the route-level. Once a route has been
set up it is entered into the route cache and will stay there, over the same
interfaces, for as long as packets are exchanged over it. Note that the
route-cache is keyed on the triple (src-address,dst-address,type-of-service).
The last part of that key may bite you with your ''session
handling'', as at
least one notable piece of client software, OpenSSH, changes the
type-of-service somewhere during the session. Fortunately, the TOS field
can be thwacked using ipchains or iptables so you can work around that...

Doei, Arthur. (Who hopes this is clearer than it looks...)

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

On Thu, 15 Mar 2001 12:48:54 +0100 (MET), you wrote:
>> and activated the "CONFIG_IP_ROUTE_MULTIPATH". Then
I''ve set up two
>
>interfaces, for as long as packets are exchanged over it. Note that the
>route-cache is keyed on the triple
(src-address,dst-address,type-of-service).
 Yes, I knew that. That was the reason I decided to test it. Note that
there is also a kernel patch to avoid this route cache (supposed to be
valid for "non-session handling" balancing) which is NOT valid for my
purposes.

 But when I tested it (some time ago) I got some troubles. I''m
re-testing it.

 These are the lines I added to my /etc/rc.d/route script:

/sbin/route del default 2> /dev/null
/usr/sbin/ip route add default equalize \
    nexthop dev eth0 via 192.168.0.229 onlink \
    nexthop dev eth0 via 192.168.0.230 onlink

 My routing table stays as follow:

goliat:~ # ip route
192.168.3.0/24 via 192.168.0.230 dev eth0 
192.168.2.0/24 via 192.168.0.230 dev eth0 
192.168.1.0/24 via 192.168.0.230 dev eth0 
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.200 
127.0.0.0/8 dev lo  scope link 
default equalize 
        nexthop via 192.168.0.229  dev eth0 weight 1 onlink
        nexthop via 192.168.0.230  dev eth0 weight 1 onlink


 Let''s suppose:
- 192.168.0.230 = Router 1
- 192.168.0.229 = Router 2
- 192.168.0.200 = the Linux Balancer machine (the one these logs
belong to)

 If I understood all correctly with the above lines:
- machines at 192.168.x.0/24 -where x:=1,2,3- are reached using router
1 (I''ve done so because the above machines are only reachable via
router 1; router 2 is not valid here).
- machines at 192.168.0.0/24 (my local subnet) are reachable directly
via eth0.
- default gateways (the ones that will carry internet traffic) are
equally balanced between router 1 and router 2.
- it should maintain "sessions" (except the ones with changing TOS in
its IP packets).

 Right?
>The last part of that key may bite you with your ''session
handling'', as at
>least one notable piece of client software, OpenSSH, changes the
>type-of-service somewhere during the session. Fortunately, the TOS field
>can be thwacked using ipchains or iptables so you can work around that...
1) Could you exemplify this TOS field "hacking"?
2) Which services are currently affected with TOS changes?

 See (*)

3) Any other alternative apart from Multipath?
4) Anyone who is currently using the Multipath solution I''ve talked
about?


(*) At least FTP is affected!

goliat:~ # ftp 62.22.78.68
Connected to sniff.batmap.com.
220 Sniff FTP-Server ready
Name (62.22.78.68:roman): 
421 Service not available, remote server has closed connection.
ftp: Login failed.
ftp: No control connection for command.
ftp>

 If I sniff at 62.22.78.68 (ftp server):

16:31:57.564330 62.174.129.161.5665 > 62.22.78.68.ftp: S
2788746458:2788746458(0) win 32767 <mss 1460,sackOK,timestamp 343505
0,nop,wscale 0> (DF)
16:31:57.564363 62.22.78.68.ftp > 62.174.129.161.5665: S
2785652370:2785652370(0) ack 2788746459 win 32120 <mss
1460,sackOK,timestamp 805012719 343505,nop,wscale 0> (DF)
16:31:58.153962 62.174.129.161.5665 > 62.22.78.68.ftp: . 1:1(0) ack 1
win 65160 <nop,nop,timestamp 343564 805012719> (DF)
16:31:58.157026 62.22.78.68.dbreporter > 62.174.129.161.ident: S
2786506672:2786506672(0) win 32120 <mss 1460,sackOK,timestamp
805012778 0,nop,wscale 0> (DF)
16:31:58.572871 62.174.129.161.ident > 62.22.78.68.dbreporter: R
0:0(0) ack 2786506673 win 0
16:31:58.579242 62.22.78.68.neod2 > 194.98.65.65.domain: 15249+ PTR?
161.129.174.62.in-addr.arpa. (45)
16:31:58.613088 194.98.65.65.domain > 62.22.78.68.neod2: 15249 1/2/2
PTR 161-MAD2-X2.red.retevision.es. (185) (DF)
16:31:58.613986 62.22.78.68.telesis-licman > 62.174.129.161.ident: S
2784376363:2784376363(0) win 32120 <mss 1460,sackOK,timestamp
805012824 0,nop,wscale 0> (DF)
16:31:58.739687 0:2:b9:7a:87:4a > 1:0:0:0:0:0 sap aa ui/C
16:31:59.092067 62.174.129.161.ident > 62.22.78.68.telesis-licman: R
0:0(0) ack 2784376364 win 0
16:31:59.092245 62.22.78.68.ftp > 62.174.129.161.5665: P 1:29(28) ack
1 win 32120 <nop,nop,timestamp 805012872 343564> (DF)
16:31:59.133590 62.83.23.69.7167 > 62.22.78.68.ftp: .
2788746459:2788746459(0) ack 2785652399 win 65160 <nop,nop,timestamp
343662 805012872> (DF) [tos 0x10] 
16:31:59.133617 62.22.78.68.ftp > 62.83.23.69.7167: R
2785652399:2785652399(0) win 0 [tos 0x10] 
16:31:59.297491 0:2:b9:7a:87:4a > 1:0:0:0:0:0 802.1d ui/C
16:31:59.552596 62.83.23.69.7167 > 62.22.78.68.ftp: P 0:12(12) ack 1
win 65160 <nop,nop,timestamp 343704 805012872> (DF) [tos 0x10] 
16:31:59.552613 62.22.78.68.ftp > 62.83.23.69.7167: R
2785652399:2785652399(0) win 0 [tos 0x10] 
16:32:01.297944 0:2:b9:7a:87:4a > 1:0:0:0:0:0 802.1d ui/C
16:32:02.090026 62.22.78.68.ftp > 62.174.129.161.5665: P 1:29(28) ack
1 win 32120 <nop,nop,timestamp 805013172 343564> (DF)
16:32:02.126023 62.174.129.161.5665 > 62.22.78.68.ftp: R
2788746459:2788746459(0) win 0
16:32:02.560028 arp who-has 62.22.78.65 tell 62.22.78.68
(0:c0:26:a0:a4:63)
16:32:02.560946 arp reply 62.22.78.65 is-at 0:2:fd:8a:c7:e0
(0:c0:26:a0:a4:63)
16:32:03.300793 0:2:b9:7a:87:4a > 1:0:0:0:0:0 802.1d ui/C

 Having a look at tcpdump''s log, you can easily notice that:
1) 62.174.129.161 starts ftp session (control connection) (client
port: 5665)
2) 62.83.23.69 starts the ftp data session (because linux ftp client
starts with passive mode enable by default). Note the "[tos 0x10]"
mark. Effectively this connection has a different TOS!

 These IP''s belong to my two different outgoing routers. This is an
example of TOS problem.

 What''s the better way to achieve TOS uniformity? Is this a good idea
to use same TOS for *ALL* outgoing packets!????

 Moreover, is this possible to make the "TOS translation" with
ipchains?? (remember I''m using 2.2 kernel!).

 Well, enough for today, isn''t it? :-))

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-    ** RoMaN SoFt / LLFB **  
       roman@madrid.com
   http://pagina.de/romansoft
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Thu, 15 Mar 2001, RoMaN SoFt / LLFB !! wrote:
> On Thu, 15 Mar 2001 12:48:54 +0100 (MET), you wrote:
>
> >> and activated the "CONFIG_IP_ROUTE_MULTIPATH". Then
I''ve set up two
> >
> >interfaces, for as long as packets are exchanged over it. Note that the
> >route-cache is keyed on the triple
(src-address,dst-address,type-of-service).
>
>  Yes, I knew that. That was the reason I decided to test it. Note that
> there is also a kernel patch to avoid this route cache (supposed to be
> valid for "non-session handling" balancing) which is NOT valid
for my
> purposes.
>
>  But when I tested it (some time ago) I got some troubles. I''m
> re-testing it.
>
>  These are the lines I added to my /etc/rc.d/route script:
>
> /sbin/route del default 2> /dev/null
> /usr/sbin/ip route add default equalize \
>     nexthop dev eth0 via 192.168.0.229 onlink \
>     nexthop dev eth0 via 192.168.0.230 onlink
>
>  My routing table stays as follow:
>
> goliat:~ # ip route
> 192.168.3.0/24 via 192.168.0.230 dev eth0
> 192.168.2.0/24 via 192.168.0.230 dev eth0
> 192.168.1.0/24 via 192.168.0.230 dev eth0
> 192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.200
> 127.0.0.0/8 dev lo  scope link
> default equalize
>         nexthop via 192.168.0.229  dev eth0 weight 1 onlink
>         nexthop via 192.168.0.230  dev eth0 weight 1 onlink
>
>
>  Let''s suppose:
> - 192.168.0.230 = Router 1
> - 192.168.0.229 = Router 2
> - 192.168.0.200 = the Linux Balancer machine (the one these logs
> belong to)
>
>  If I understood all correctly with the above lines:
> - machines at 192.168.x.0/24 -where x:=1,2,3- are reached using router
> 1 (I''ve done so because the above machines are only reachable via
> router 1; router 2 is not valid here).
> - machines at 192.168.0.0/24 (my local subnet) are reachable directly
> via eth0.
> - default gateways (the ones that will carry internet traffic) are
> equally balanced between router 1 and router 2.
> - it should maintain "sessions" (except the ones with changing
TOS in
> its IP packets).
>
>  Right?
Right. This is assuming no NAT and a single IP address for the linux
balancer.
> >The last part of that key may bite you with your ''session
handling'', as at
> >least one notable piece of client software, OpenSSH, changes the
> >type-of-service somewhere during the session. Fortunately, the TOS
field
> >can be thwacked using ipchains or iptables so you can work around
that...
>
> 1) Could you exemplify this TOS field "hacking"?
ipchains <yourmatchfields> -t 0x01 0x00

This simply sets the TOS to be equal on all packets that match the rest of
the rule.
> 2) Which services are currently affected with TOS changes?
I have no idea. I know OpenSSH does, and is very weird in that, as it
changes the TOS field on an *existing* TCP connection.
> 3) Any other alternative apart from Multipath?
Probably, but none that I have personal experience with.
> 4) Anyone who is currently using the Multipath solution I''ve
talked
> about?
One of X/OS''s customers (X/OS, http://www.xos.nl/, is my employer) has
a
setup like you''ve described working with 3 ADSL modems and a cable
modem.
And yes, it seems to work without glitches.
> (*) At least FTP is affected!
[snip]
>  Having a look at tcpdump''s log, you can easily notice that:
> 1) 62.174.129.161 starts ftp session (control connection) (client
> port: 5665)
> 2) 62.83.23.69 starts the ftp data session (because linux ftp client
> starts with passive mode enable by default). Note the "[tos
0x10]"
> mark. Effectively this connection has a different TOS!
>  These IP''s belong to my two different outgoing routers. This is
an
> example of TOS problem.
This is not a very good example, as a new TCP connection is started. FTP is
one of the services that is very tricky to deal with in such situations,
precisely because of the separate control and data connections. But yes, the
basic problem is the same as with OpenSSH: suddenly an extra entry appears
in the route-cache, going out over a different gateway, breaking
functionality.
>  What''s the better way to achieve TOS uniformity? Is this a good
idea
> to use same TOS for *ALL* outgoing packets!????
Clearing the TOS should pose no problems whatsoever.
>  Moreover, is this possible to make the "TOS translation" with
> ipchains?? (remember I''m using 2.2 kernel!).
Yes. The -t option to ipchains does it.
>  Well, enough for today, isn''t it? :-))
Quite. Back to coding...

Doei, Arthur.

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

On Thu, 15 Mar 2001 17:56:37 +0100 (MET), you wrote:
>> 1) Could you exemplify this TOS field "hacking"?
>
>ipchains <yourmatchfields> -t 0x01 0x00
 Ummm. I don''t get it to work... I''ve created the following
test
ipchains rule (see log):

goliat:~ # ipchains -F                                                
goliat:~ # ipchains -A output -p tcp --source-port 20:21 -b -t 0x01
0x00 -j ACCEPT -l     
goliat:~ # ipchains -L
Chain input (policy ACCEPT):
Chain forward (policy ACCEPT):
Chain output (policy ACCEPT):
target     prot opt     source                destination
ports
ACCEPT     tcp  ----l-  anywhere             anywhere
ftp-data:ftp ->   any
ACCEPT     tcp  ----l-  anywhere             anywhere              any
->   ftp-data:ftp
goliat:~ # ftp 62.22.78.68
Connected to sniff.batmap.com.
220 Sniff FTP-Server ready
Name (62.22.78.68:roman): 
421 Service not available, remote server has closed connection.
ftp: Login failed.
ftp: No control connection for command.
ftp> bye
goliat:~ #


Now, we have a look to the ftp server:
sniff:~ # tcpdump -ni eth0
User level filter, protocol ALL, datagram packet socket
tcpdump: listening on eth0
11:06:16.821899 62.174.128.49.7772 > 62.22.78.68.ftp: S
395928648:395928648(0) win 32767 <mss 1460,sackOK,timestamp 7029574
0,nop,wscale 0> (DF)
11:06:16.821934 62.22.78.68.ftp > 62.174.128.49.7772: S
384708931:384708931(0) ack 395928649 win 32120 <mss
1460,sackOK,timestamp 811698645 7029574,nop,wscale 0> (DF)
11:06:16.863172 62.174.128.49.7772 > 62.22.78.68.ftp: . 1:1(0) ack 1
win 65160 <nop,nop,timestamp 7029631 811698645> (DF)
11:06:16.866108 62.22.78.68.ampr-inter > 62.174.128.49.ident: S
390274631:390274631(0) win 32120 <mss 1460,sackOK,timestamp 811698649
0,nop,wscale 0> (DF)
11:06:16.900480 62.174.128.49.ident > 62.22.78.68.ampr-inter: R 0:0(0)
ack 390274632 win 0
11:06:16.906885 62.22.78.68.1053 > 194.98.65.65.domain: 3187+ PTR?
49.128.174.62.in-addr.arpa. (44)
11:06:16.940612 194.98.65.65.domain > 62.22.78.68.1053: 3187 1/2/2 PTR
49-MAD2-X1.red.retevision.es. (183) (DF)
11:06:16.941507 62.22.78.68.sdsc-lm > 62.174.128.49.ident: S
384359382:384359382(0) win 32120 <mss 1460,sackOK,timestamp 811698657
0,nop,wscale 0> (DF)
11:06:16.980458 62.174.128.49.ident > 62.22.78.68.sdsc-lm: R 0:0(0)
ack 384359383 win 0
11:06:16.980641 62.22.78.68.ftp > 62.174.128.49.7772: P 1:29(28) ack 1
win 32120 <nop,nop,timestamp 811698661 7029631> (DF)
11:06:17.038691 62.175.108.41.7857 > 62.22.78.68.ftp: .
395928649:395928649(0) ack 384708960 win 65160 <nop,nop,timestamp
7029648 811698661> (DF)
11:06:17.038720 62.22.78.68.ftp > 62.175.108.41.7857: R
384708960:384708960(0) win 0
11:06:17.229273 0:2:b9:7a:87:4a > 1:0:0:0:0:0 802.1d ui/C
11:06:19.980029 62.22.78.68.ftp > 62.174.128.49.7772: P 1:29(28) ack 1
win 32120 <nop,nop,timestamp 811698961 7029631> (DF)
11:06:20.026685 62.174.128.49.7772 > 62.22.78.68.ftp: R
395928649:395928649(0) win 0
11:06:21.235019 0:2:b9:7a:87:4a > 1:0:0:0:0:0 802.1d ui/C
11:06:21.820027 arp who-has 62.22.78.65 tell 62.22.78.68
(0:c0:26:a0:a4:63)
11:06:21.821221 arp reply 62.22.78.65 is-at 0:2:fd:8a:c7:e0
(0:c0:26:a0:a4:63)

23 packets received by filter
sniff:~ #


 It looks like the TOS translation has been produced ok. Despite
packets having same TOS, the machine doing multipath has chosen
DIFFERENT paths, which isn''t the expected behaviour. :-????

 Am I doing something wrong?

 I only could imagine that TOS translation is being doing AFTER
multipath has acted. Is it possible? In this case, how to avoid it?

 Could you paste some ipchains rules and/or other useful config files
for your (working) configuration? Perhaps this may helps.

 Any help would be *highly* appreciated.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-    ** RoMaN SoFt / LLFB **  
       roman@madrid.com
   http://pagina.de/romansoft
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Fri, 16 Mar 2001, RoMaN SoFt / LLFB!! wrote:
> On Thu, 15 Mar 2001 17:56:37 +0100 (MET), you wrote:
>
> >> 1) Could you exemplify this TOS field "hacking"?
> >
> >ipchains <yourmatchfields> -t 0x01 0x00
>
>  Ummm. I don''t get it to work... I''ve created the
following test
> ipchains rule (see log):
>
> goliat:~ # ipchains -F
> goliat:~ # ipchains -A output -p tcp --source-port 20:21 -b -t 0x01
> 0x00 -j ACCEPT -l
Looks okay. Note that passive ftp return data is *not* necessarily on
port 20 or 21...

For testing I would try clearing the TOS field on *all* outgoing packets.
>  I only could imagine that TOS translation is being doing AFTER
> multipath has acted. Is it possible? In this case, how to avoid it?
Yes, that is theoretically possible, if you are ftp''ing directly from
the
machine that does the multipath routing.
>  Could you paste some ipchains rules and/or other useful config files
> for your (working) configuration? Perhaps this may helps.
I''ve already paraphrased most of the complete config. What you may
still
need is rules and corresponding routing tables to do static non-multipath
routing if you already have a source address for your packets. This would
probably fix the ftp problem. So, you would make tables 1 and 2 for the two
uplink gateways, and add the following routes (assuming 10.1.1.1 and
10.2.2.1 as the gateways and 10.1.1.2 and 10.2.2.2 as the local ip
addresses):

ip route add 10.1.1.1 dev eth1 table 1
ip route add default via 10.1.1.1 table 1
ip route add 10.2.2.1 dev eth1 table 2
ip route add default via 10.2.2.1 table 2

ip rule add from 10.1.1.2 table 1 prio 100
ip rule add from 10.2.2.2 table 2 prio 200
>  Any help would be *highly* appreciated.
Hope this helps.

Doei, Arthur.

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

On Fri, 16 Mar 2001 11:41:10 +0100 (MET), you wrote:
>On Fri, 16 Mar 2001, RoMaN SoFt / LLFB!! wrote:
>
>> On Thu, 15 Mar 2001 17:56:37 +0100 (MET), you wrote:
>>
>> >> 1) Could you exemplify this TOS field "hacking"?
>> >
>> >ipchains <yourmatchfields> -t 0x01 0x00
>>
>>  Ummm. I don''t get it to work... I''ve created the
following test
>> ipchains rule (see log):
>>
>> goliat:~ # ipchains -F
>> goliat:~ # ipchains -A output -p tcp --source-port 20:21 -b -t 0x01
>> 0x00 -j ACCEPT -l
>
>Looks okay. Note that passive ftp return data is *not* necessarily on
>port 20 or 21...
>
>For testing I would try clearing the TOS field on *all* outgoing packets.
 Another test:
- backup: another local machine routed via goliat (the multipath
gateway)
- goliat (the multipath gateway) configured as follows:

goliat:~ # ipchains -F
goliat:~ # ipchains -A output -t 0x01 0x00 -j ACCEPT -l
goliat:~ # ipchains -A forward -t 0x01 0x00 -j ACCEPT -l
goliat:~ # ipchains -A input -t 0x01 0x00 -j ACCEPT -l

(not all rules are really necessary but...)

 Now:

backup:/usr/local/scripts # ftp 62.22.78.68
Connected to sniff.batmap.com.
220 Sniff FTP-Server ready
Name (62.22.78.68:roman): 
421 Service not available, remote server has closed connection.
ftp: Login failed.
ftp: No control connection for command.
ftp> bye
backup:/usr/local/scripts #

 Same error.
>>  I only could imagine that TOS translation is being doing AFTER
>> multipath has acted. Is it possible? In this case, how to avoid it?
>
>Yes, that is theoretically possible, if you are ftp''ing directly
from the
>machine that does the multipath routing.
 I''ve demostrated the error persists although I use another machine.
>I''ve already paraphrased most of the complete config. What you may
still
>need is rules and corresponding routing tables to do static non-multipath
>routing if you already have a source address for your packets. This would
 I don''t understand. What do you mean with "you have the source
address"? An normal IP packet always have a src address.

 I think I''m going to write a new post detailing the problem with more
accurate logs and I''ll post it to linux-kernel mailing list.

 Ummm, I have another idea: as I did the TOS hacking upon launching
the previous ftp probes, perhaps "evil" routes keep on cached and
that''s the reason TOS hack activation doesn''t care. Could it
be? I''ll
have to re-look the adv.routing-howto, I don''t remember how to clear
cached routes.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-    ** RoMaN SoFt / LLFB **  
       roman@madrid.com
   http://pagina.de/romansoft
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Fri, 16 Mar 2001, RoMaN SoFt / LLFB !! wrote:

[snip]
>  Ummm, I have another idea: as I did the TOS hacking upon launching
> the previous ftp probes, perhaps "evil" routes keep on cached and
> that''s the reason TOS hack activation doesn''t care. Could
it be? I''ll
> have to re-look the adv.routing-howto, I don''t remember how to
clear
> cached routes.
Aaargh... I was assuming you cleared the route cache prior to every test.
It''s very easy:  ip route flush table cache  (or, shorter: ip r f t
cache)

Doei, Arthur.

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

Hello:

 Some comments / ideas:

- "Jorge Boncompte [DTI2]" <jorge@dti2.net> suggested me to
remove the
"equalize" option. I tested it. It does NOT work at all.
- I''ve done some tests with TOS translation. Logs are included at the
bottom of this post. Summarizing:
1) If I make TOS translation on OUTPUT or FORWARD chain it does NOT
solve the problem at all.
2) If I make TOS translation on INPUT chain it does PARTIALLY solve
the problem. "Partially" means that it works ok for local machines
accesing our gateway, but it does not work for connections originated
in the gateway itself.

Explanation: this is a rude scheme of chains bypassing:

(incoming) packet ---> INPUT --> Routing decision --> FORWARD -->
OUTPUT -> (outgoing) packet

(this is valid for a packet coming from another local machine into our
gateway machine)

 As you can see the routing decision is performed AFTER input check
and BEFORE forward&output check. Therefore if we make TOS translation
on forward or output chains it will NOT affect the multipath decision
(which is taken before the translation is performed) and multipath
algorith will BREAK our "session" (assuming it uses changing TOS
values, as FTP does). See logs "Case 1" and "Case 3".

 The TOS translation is only valid is performed on INPUT chain because
multipath decision is performed after it. It partially solves the
problem. At least it works for local machines (not our gateway
itself).

 Nevertheless if the connection is originated from the gateway itself,
then packets do NOT traverse the INPUT chain. So our TOS filter is
useless :-(. I haven''t found any solution for that.

 I''m not a kernel hacker. But I suppose the "good" solution
would be
to patch kernel for removing TOS checking. This way our problem
wouldn''t exist.

 Arthur van Leeuwen <arthurvl@sci.kun.nl> kindly suggested a trick
with static routes involved, to solve this last problem. But I want to
set up a transparent proxy on the gateway machine. Therefore almost
all traffic will (virtually) originate on the gateway machine (where
the proxy will be located) and I need this traffic to be balanced too
(as the other local traffic does).

 If I''m missing something or I''m wrong, please, correct me.
Also I''d
like to receive more (working if possible) ideas.

 Thanks all (specially to Arthur).

 Román.-


-=LOGS=-

======= Case 1 ======
goliat:~ # ipchains -A output -t 0x01 0x00

backup:~ # arp -a
roman (192.168.0.247) at 00:C0:26:F0:40:9A [ether] on eth0
backup:~ # ftp 62.22.78.68
Connected to sniff.batmap.com.
220 Sniff FTP-Server ready
Name (62.22.78.68:roman): 
421 Service not available, remote server has closed connection.
ftp: Login failed.
ftp: No control connection for command.
ftp> bye
backup:~ # arp -na
? (192.168.0.247) at 00:C0:26:F0:40:9A [ether] on eth0
? (192.168.0.200) at 00:50:04:0D:6A:12 [ether] on eth0
? (192.168.0.230) at 00:80:2D:DC:26:1E [ether] on eth0
? (192.168.0.229) at 00:80:2D:DC:28:27 [ether] on eth0
backup:~ # arp -n 
Address                 HWtype  HWaddress           Flags Mask
Iface
192.168.0.247           ether   00:C0:26:F0:40:9A   C
eth0
192.168.0.200           ether   00:50:04:0D:6A:12   C
eth0
192.168.0.230           ether   00:80:2D:DC:26:1E   C
eth0
192.168.0.229           ether   00:80:2D:DC:28:27   C
eth0
backup:~ # 

17:57:47.006607 62.174.128.49.6551 > 62.22.78.68.ftp: S
357098797:357098797(0) win 32767 <mss 1460,sackOK,timestamp 6991
0,nop,wscale 0> (DF)
17:57:47.006643 62.22.78.68.ftp > 62.174.128.49.6551: S
673396295:673396295(0) ack 357098798 win 32120 <mss
1460,sackOK,timestamp 814167663 6991,nop,wscale 0> (DF)
17:57:47.043320 62.174.128.49.6551 > 62.22.78.68.ftp: . 1:1(0) ack 1
win 65160 <nop,nop,timestamp 6995 814167663> (DF)
17:57:47.149299 62.22.78.68.ftp > 62.174.128.49.6551: P 1:29(28) ack 1
win 32120 <nop,nop,timestamp 814167677 6995> (DF)
17:57:47.207938 62.175.108.41.4259 > 62.22.78.68.ftp: .
357098798:357098798(0) ack 673396324 win 65160 <nop,nop,timestamp 7010
814167677> (DF)
17:57:47.207966 62.22.78.68.ftp > 62.175.108.41.4259: R
673396324:673396324(0) win 0
17:57:47.898878 62.175.108.41.4259 > 62.22.78.68.ftp: P 0:12(12) ack 1
win 65160 <nop,nop,timestamp 7078 814167677> (DF)
17:57:47.898892 62.22.78.68.ftp > 62.175.108.41.4259: R
673396324:673396324(0) win 0
17:57:50.140028 62.22.78.68.ftp > 62.174.128.49.6551: P 1:29(28) ack 1
win 32120 <nop,nop,timestamp 814167977 6995> (DF)
17:57:50.182538 62.174.128.49.6551 > 62.22.78.68.ftp: R
357098798:357098798(0) win 0


======= Case 2 ======
goliat:~ # ipchains -A input -t 0x01 0x00

backup:~ # ftp 62.22.78.68
Connected to sniff.batmap.com.
220 Sniff FTP-Server ready
Name (62.22.78.68:roman): 
331 Password required for roman.
Password:
230 User roman logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> bye
221 Goodbye.
backup:~ #

sniff:~ # tcpdump -ni eth0 tcp port 21 or 20
User level filter, protocol ALL, datagram packet socket
tcpdump: listening on eth0
18:06:13.818618 62.174.128.49.6893 > 62.22.78.68.ftp: S
893209756:893209756(0) win 32767 <mss 1460,sackOK,timestamp 30163
0,nop,wscale 0> (DF)
18:06:13.818762 62.22.78.68.ftp > 62.174.128.49.6893: S
1223828639:1223828639(0) ack 893209757 win 32120 <mss
1460,sackOK,timestamp 814218344 30163,nop,wscale 0> (DF)
18:06:13.855538 62.174.128.49.6893 > 62.22.78.68.ftp: . 1:1(0) ack 1
win 65160 <nop,nop,timestamp 30175 814218344> (DF)
18:06:13.959701 62.22.78.68.ftp > 62.174.128.49.6893: P 1:29(28) ack 1
win 32120 <nop,nop,timestamp 814218358 30175> (DF)
18:06:13.999058 62.174.128.49.6893 > 62.22.78.68.ftp: . 1:1(0) ack 29
win 65160 <nop,nop,timestamp 30190 814218358> (DF)
18:06:15.175588 62.174.128.49.6893 > 62.22.78.68.ftp: P 1:13(12) ack
29 win 65160 <nop,nop,timestamp 30292 814218358> (DF)
18:06:15.175608 62.22.78.68.ftp > 62.174.128.49.6893: . 29:29(0) ack
13 win 32120 <nop,nop,timestamp 814218480 30292> (DF)
18:06:15.176848 62.22.78.68.ftp > 62.174.128.49.6893: P 29:63(34) ack
13 win 32120 <nop,nop,timestamp 814218480 30292> (DF)
18:06:15.246166 62.174.128.49.6893 > 62.22.78.68.ftp: . 13:13(0) ack
63 win 65160 <nop,nop,timestamp 30315 814218480> (DF)
18:06:17.902895 62.174.128.49.6893 > 62.22.78.68.ftp: P 13:28(15) ack
63 win 65160 <nop,nop,timestamp 30499 814218480> (DF)
18:06:17.912514 62.22.78.68.ftp > 62.174.128.49.6893: P 63:90(27) ack
28 win 32120 <nop,nop,timestamp 814218754 30499> (DF)
18:06:17.953806 62.174.128.49.6893 > 62.22.78.68.ftp: P 28:33(5) ack
90 win 65160 <nop,nop,timestamp 30585 814218754> (DF)
18:06:17.953989 62.22.78.68.ftp > 62.174.128.49.6893: P 90:131(41) ack
33 win 32120 <nop,nop,timestamp 814218758 30585> (DF)
18:06:18.011966 62.174.128.49.6893 > 62.22.78.68.ftp: P 33:39(6) ack
131 win 65160 <nop,nop,timestamp 30591 814218758> (DF)
18:06:18.012101 62.22.78.68.ftp > 62.174.128.49.6893: P 131:150(19)
ack 39 win 32120 <nop,nop,timestamp 814218764 30591> (DF)
18:06:18.066255 62.174.128.49.6893 > 62.22.78.68.ftp: . 39:39(0) ack
150 win 65160 <nop,nop,timestamp 30597 814218764> (DF)
18:06:19.272030 62.174.128.49.6893 > 62.22.78.68.ftp: P 39:45(6) ack
150 win 65160 <nop,nop,timestamp 30700 814218764> (DF)
18:06:19.272125 62.22.78.68.ftp > 62.174.128.49.6893: P 150:164(14)
ack 45 win 32120 <nop,nop,timestamp 814218890 30700> (DF)
18:06:19.272789 62.22.78.68.ftp > 62.174.128.49.6893: F 164:164(0) ack
45 win 32120 <nop,nop,timestamp 814218890 30700> (DF)
18:06:19.312433 62.174.128.49.6893 > 62.22.78.68.ftp: F 45:45(0) ack
164 win 65160 <nop,nop,timestamp 30721 814218890> (DF)
18:06:19.312465 62.22.78.68.ftp > 62.174.128.49.6893: . 165:165(0) ack
46 win 32120 <nop,nop,timestamp 814218894 30721> (DF)
18:06:19.320287 62.174.128.49.6893 > 62.22.78.68.ftp: . 46:46(0) ack
165 win 65160 <nop,nop,timestamp 30722 814218890> (DF)

36 packets received by filter
sniff:~ # 


======= Case 3 ======
ipchains -A forward -t 0x01 0x00

backup:~ # ftp 62.22.78.68
Connected to sniff.batmap.com.
220 Sniff FTP-Server ready
Name (62.22.78.68:roman): 
421 Service not available, remote server has closed connection.
ftp: Login failed.
ftp: No control connection for command.

18:19:39.609405 62.174.128.49.7379 > 62.22.78.68.ftp: S
1736918725:1736918725(0) win 32767 <mss 1460,sackOK,timestamp 9345
0,nop,wscale 0> (DF)
18:19:39.609518 62.22.78.68.ftp > 62.174.128.49.7379: S
2068636790:2068636790(0) ack 1736918726 win 32120 <mss
1460,sackOK,timestamp 814298923 9345,nop,wscale 0> (DF)
18:19:39.904265 62.174.128.49.7379 > 62.22.78.68.ftp: . 1:1(0) ack 1
win 65160 <nop,nop,timestamp 9370 814298923> (DF)
18:19:42.574780 62.22.78.68.ftp > 62.174.128.49.7379: P 1:29(28) ack 1
win 32120 <nop,nop,timestamp 814299220 9370> (DF)
18:19:44.204673 62.175.108.41.4776 > 62.22.78.68.ftp: .
1736918726:1736918726(0) ack 2068636819 win 65160 <nop,nop,timestamp
9805 814299220> (DF)
18:19:44.204717 62.22.78.68.ftp > 62.175.108.41.4776: R
2068636819:2068636819(0) win 0
18:19:45.570027 62.22.78.68.ftp > 62.174.128.49.7379: P 1:29(28) ack 1
win 32120 <nop,nop,timestamp 814299520 9370> (DF)
18:19:46.405868 62.174.128.49.7379 > 62.22.78.68.ftp: R
1736918726:1736918726(0) win 0


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-    ** RoMaN SoFt / LLFB **  
       roman@madrid.com
   http://pagina.de/romansoft
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Fri, Mar 16, 2001 at 07:25:14PM +0100, RoMaN SoFt / LLFB!! wrote:
>  I''m not a kernel hacker. But I suppose the "good"
solution would be
> to patch kernel for removing TOS checking. This way our problem
> wouldn''t exist.
> 
>  Arthur van Leeuwen <arthurvl@sci.kun.nl> kindly suggested a trick
> with static routes involved, to solve this last problem. But I want to
> set up a transparent proxy on the gateway machine. Therefore almost
> all traffic will (virtually) originate on the gateway machine (where
> the proxy will be located) and I need this traffic to be balanced too
> (as the other local traffic does).
> 
>  If I''m missing something or I''m wrong, please, correct
me. Also I''d
> like to receive more (working if possible) ideas.
> 
>  Thanks all (specially to Arthur).
> 
>  Román.-
> 
> 
I would find an old pci machine, and use that as the "equalizing"
machine.
You can have the proxy behind that, and have the eq box send the connections
needing proxying to the squid box.

Are you using the eq box as a firewall too?  Under best conditions, the only
open service on the firewall would be ssh or none.

Mike

On Fri, 16 Mar 2001 10:32:52 -0800, you wrote:
>I would find an old pci machine, and use that as the "equalizing"
machine.
>You can have the proxy behind that, and have the eq box send the connections
>needing proxying to the squid box.
>
>Are you using the eq box as a firewall too?  Under best conditions, the only
>open service on the firewall would be ssh or none.
 This is a bit paranoid for a little LAN: almost all our employers
have few computer skills and the ones with some "computers''
control"
are totally trusted. The untrusted world is out of the LAN, where
proxy service will be hidden (filtered). In addition both outgoing
routers are performing NAT so inherently we''ve got some extra
protection (it is virtually impossible to establish a connection from
out of the LAN into it).

 Summarizing: although running a proxy, it should not be seen from the
outside.

 By the way, your statement is very well known and generally it should
be taken into account if possible.

 Regards.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-    ** RoMaN SoFt / LLFB **  
       roman@madrid.com
   http://pagina.de/romansoft
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Fri, Mar 16, 2001 at 08:10:08PM +0100, RoMaN SoFt / LLFB !!
wrote:
> On Fri, 16 Mar 2001 10:32:52 -0800, you wrote:
> 
> >I would find an old pci machine, and use that as the
"equalizing" machine.
> >You can have the proxy behind that, and have the eq box send the
connections
> >needing proxying to the squid box.
> >
> >Are you using the eq box as a firewall too?  Under best conditions, the
only
> >open service on the firewall would be ssh or none.
> 
>  This is a bit paranoid for a little LAN: almost all our employers
> have few computer skills and the ones with some "computers''
control"
> are totally trusted. The untrusted world is out of the LAN, where
> proxy service will be hidden (filtered). In addition both outgoing
> routers are performing NAT so inherently we''ve got some extra
> protection (it is virtually impossible to establish a connection from
> out of the LAN into it).
> 
>  Summarizing: although running a proxy, it should not be seen from the
> outside.
> 
>  By the way, your statement is very well known and generally it should
> be taken into account if possible.
> 
I understand everything you''re saying, I''ve setup a
firewall/mailserver/file
server/monitoring station/trans proxy/masq in several places, but you will
have a LOT fewer headaches if you keep your firewall seperate from
everything else.

I''m working on creating a DMZ perimiter network and putting the actual
network behind two firewalls.  Anything where you have anything as critical
as needing multiple links to the internet should be setup this way.

I will have a lot fewer layers of complication once I seperate my firewall
from the rest... not to say it''s impossible.

You also have fewer places that need constant updating, since I can''t
take
my file server down as often as I''d like to be able to install the
latest
kernel for firewalling...

Mike

On Fri, 16 Mar 2001, RoMaN SoFt / LLFB !! wrote:
>  Arthur van Leeuwen <arthurvl@sci.kun.nl> kindly suggested a trick
> with static routes involved, to solve this last problem. But I want to
> set up a transparent proxy on the gateway machine. Therefore almost
> all traffic will (virtually) originate on the gateway machine (where
> the proxy will be located) and I need this traffic to be balanced too
> (as the other local traffic does).
It actually will be balanced, if you use squid. The setup is *exactly* the
way our customer''s setup is made. The beauty of those static routes is
that
the ftp client will bind on a specific source address to connect the data
connection. That source address is the source address of the control
connection.

Doei, Arthur. (And yes, you want to do input firewalling... for more reasons
               than just the one mentioned. :))

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching