Hi All,
I have a triple ISP, double DMZ, single LAN firewall that isn''t routing
traffic over the secondary and tertiary ISP''s. All traffic is heading
out
over the primary link :( Here''s a basic diagram of our firewall, the
modems
on the triplet of Internet connections simply hold up the PPPoA/E links and
NAT the NET1/2/3 addresses to the respective outside IP:
+---------{*INTERNET*}--------+
| | |
(modem1) (modem2) (modem3)
172.16.3.1 172.16.4.1 172.16.5.1
| | |
NET1 NET2 NET2
| | |
172.16.3.2 172.16.4.1 172.16.5.2
(eth3) (eth4) (eth5)
| | |
+-+--------------+--------------+-+
10.10.100.0/24 | | 10.10.101.0/24
DMZ1 - (eth1) --+ FIREWALL +-- (eth2) - DMZ2
| |
+----------------+----------------+
|
(eth0)
10.10.10.0/24
LAN
Attached is a recent shorewall dump. Below is the relevant files (let me know
if I missed any):
The physical interfaces are abstracted out in the "params" file:
LAN_IF=eth0
DMZ1_IF=eth1
DMZ2_IF=eth2
NET_IF1=eth3
NET_IF2=eth4
NET_IF3=eth5
interfaces:
#ZONE INTERFACE BROADCAST OPTIONS
lan $LAN_IF detect routeback
dmz1 $DMZ1_IF detect -
dmz2 $DMZ2_IF detect -
net $NET_IF1 detect $NET_OPTIONS
net $NET_IF2 detect $NET_OPTIONS
net $NET_IF3 detect $NET_OPTIONS
... where
NET_OPTIONS=arp_filter,nosmurfs,tcpflags,routefilter
providers:
#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
squid 1 202 - $LAN_IF $PROXYSVR loose -
$ISP1 2 1 main $NET_IF1 $NET_IF1_GW $PROVOPTS $COPY
$ISP2 3 2 main $NET_IF2 $NET_IF2_GW $PROVOPTS $COPY
$ISP3 4 3 main $NET_IF3 $NET_IF3_GW $PROVOPTS $COPY
...where:
COPY=$LAN_IF,$DMZ1_IF,$DMZ2_IF (ie. eth0,eth1,eth2)
PROVOPTS=track,loose
The NET_IFx_GW variables are the ADSL modems'' IP''s (again,
defined in params)
as follows:
NET_IF1_GW=172.16.3.1
NET_IF2_GW=172.16.4.1
NET_IF3_GW=172.16.5.1
tcdevices:
#INTERFACE IN-BANDWITH OUT-BANDWIDTH
$NET_IF1 $NET_IF1_IN $NET_IF1_OUT
$NET_IF2 $NET_IF2_IN $NET_IF2_OUT
$NET_IF3 $NET_IF3_IN $NET_IF3_OUT
tcclasses:
#INTERFACE MARK RATE CEIL PRIORITY OPTIONS
#
# Primary Interface (ADSL2+ 24M/1.5M)
$NET_IF1 10 full full 1 tcp-ack,tos-minimize-delay
$NET_IF1 20 9*full/10 9*full/10 2
$NET_IF1 30 6*full/10 6*full/10 3 default
#
# Secondary Interface (ADSL1 1.5M/256K)
$NET_IF2 40 full full 4 tcp-ack,tos-minimize-delay
$NET_IF2 50 6*full/10 6*full/10 5 default
#
# Tertiary Interface (ADSL1 512K/128K)
$NET_IF3 60 full full 6 default
tcrules:
#MARK SOURCE DEST PROTO DEST SOURCE USER TEST LENGTH TOS
# PORT PORT
10 $LAN_NETWORK $ANY_IP tcp ssh - - - 0:512
20 $LAN_NETWORK $ANY_IP tcp ssh - - - 513:
20 $LAN_NETWORK $ANY_IP tcp $WWW
20 $LAN_NETWORK $ANY_IP tcp $FTP
30 $LAN_NETWORK $ANY_IP tcp nntp
40 $LAN_NETWORK $ANY_IP tcp $STREAM
50 $LAN_NETWORK $ANY_IP tcp $ALLMAIL
50 $LAN_NETWORK $ANY_IP tcp $IM
50 $LAN_NETWORK $ANY_IP udp $IM
50 $LAN_NETWORK $ANY_IP tcp $P2P
50 $LAN_NETWORK $ANY_IP udp $P2P
50 $LAN_NETWORK $ANY_IP tcp $GAMES
50 $LAN_NETWORK $ANY_IP udp $GAMES
50 $LAN_NETWORK $ANY_IP all
I thought I followed all the docs but I feel like I''ve missed something
really
basic. Any insights?
Thanks in advance,
James
--
Test-tube babies shouldn''t throw stones.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
Tom Eastep
2007-Aug-14 02:53 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
James Gray wrote:> > I thought I followed all the docs but I feel like I''ve missed something really > basic.Like maybe Shorewall FAQ 57? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Tom Eastep
2007-Aug-14 03:01 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
Tom Eastep wrote:> James Gray wrote: > >> I thought I followed all the docs but I feel like I''ve missed something really >> basic. > > Like maybe Shorewall FAQ 57? >And if you follow that FAQ and still have problems, then please consider this from the MultiISP article: Important If you specify ''balance'' and still find that all traffic is going out through only one provider, you may need to install a kernel built with CONFIG_IP_ROUTE_MULTIPATH_CACHED=n. Several users have reported that this change has corrected similar problems. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Tom Eastep
2007-Aug-14 14:08 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
James Gray wrote:> The NET_IFx_GW variables are the ADSL modems'' IP''s (again, defined in params) > as follows: > NET_IF1_GW=172.16.3.1 > NET_IF2_GW=172.16.4.1 > NET_IF3_GW=172.16.5.1 > > tcdevices: > #INTERFACE IN-BANDWITH OUT-BANDWIDTH > $NET_IF1 $NET_IF1_IN $NET_IF1_OUT > $NET_IF2 $NET_IF2_IN $NET_IF2_OUT > $NET_IF3 $NET_IF3_IN $NET_IF3_OUT > > tcclasses: > #INTERFACE MARK RATE CEIL PRIORITY OPTIONS > # > # Primary Interface (ADSL2+ 24M/1.5M) > $NET_IF1 10 full full 1 tcp-ack,tos-minimize-delay > $NET_IF1 20 9*full/10 9*full/10 2 > $NET_IF1 30 6*full/10 6*full/10 3 default > # > # Secondary Interface (ADSL1 1.5M/256K) > $NET_IF2 40 full full 4 tcp-ack,tos-minimize-delay > $NET_IF2 50 6*full/10 6*full/10 5 default > # > # Tertiary Interface (ADSL1 512K/128K) > $NET_IF3 60 full full 6 default > > tcrules: > #MARK SOURCE DEST PROTO DEST SOURCE USER TEST LENGTH TOS > # PORT PORT > 10 $LAN_NETWORK $ANY_IP tcp ssh - - - 0:512 > 20 $LAN_NETWORK $ANY_IP tcp ssh - - - 513: > 20 $LAN_NETWORK $ANY_IP tcp $WWW > 20 $LAN_NETWORK $ANY_IP tcp $FTP > 30 $LAN_NETWORK $ANY_IP tcp nntp > 40 $LAN_NETWORK $ANY_IP tcp $STREAM > 50 $LAN_NETWORK $ANY_IP tcp $ALLMAIL > 50 $LAN_NETWORK $ANY_IP tcp $IM > 50 $LAN_NETWORK $ANY_IP udp $IM > 50 $LAN_NETWORK $ANY_IP tcp $P2P > 50 $LAN_NETWORK $ANY_IP udp $P2P > 50 $LAN_NETWORK $ANY_IP tcp $GAMES > 50 $LAN_NETWORK $ANY_IP udp $GAMES > 50 $LAN_NETWORK $ANY_IP all >In taking another look at your configuration this morning, I see a couple of more problems: a) You have MARK_IN_FORWARD_CHAIN=No in shorewall.conf so all of your marking rules are going in the PREROUTING chain. With multi-ISP routing, your traffic shaping marking must be done in the FORWARD chain. b) You have failed to grasp the notion that tcrules are ''last-match-wins'' not ''first-match''wins''. So all traffic through your router is being marked with fwmark = 50. These two blunders have an effect on your multi-ISP problem. After PREROUTING, all packets have a mark value of 50. That doesn''t match any of the fwmark values given for your Providers so traffic is routed according to the ''main'' routing table. Because you didn''t specify ''balance'' on your providers, your main routing table has a single default route via eth3 so all traffic is sent out of that interface. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
James Gray
2007-Aug-14 21:47 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
Tom Eastep wrote:> James Gray wrote: > >> I thought I followed all the docs but I feel like I''ve missed something really >> basic. > > Like maybe Shorewall FAQ 57? > > -TomThanks Tom. I really appreciate the fast response :) I''ve been doing most of the config offline using the 3.x PDF documentation, and it doesn''t lay it out as plainly as FAQ 57. My bad. I replaced "loose" with "balance" in the providers options. However, after restarting shorewall (sudo service shorewall restart) the routing totally wigged out. Traffic was going out on the two interfaces (ISP1/2) but if data was coming back, it wasn''t reaching the clients. I reverted to the old config and all was good (all traffic on one interface). I''ve simplified the config to ignore the 3rd ISP at this stage and I''ll see if the new (untried) 2 ISP config fairs any better. Has anyone else seen this sort of problem though? The traffic shaping rules are the same in the new config as my original post, but the providers file has had the 3rd ISP removed if anyone wants to look at the setup. *Sigh*...more fiddling. Again, thanks for your help Tom. -- James ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Tom Eastep
2007-Aug-14 23:55 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
James Gray wrote:> Tom Eastep wrote: >> James Gray wrote: >> >>> I thought I followed all the docs but I feel like I''ve missed something really >>> basic. >> Like maybe Shorewall FAQ 57? >> >> -Tom > > Thanks Tom. I really appreciate the fast response :) I''ve been doing > most of the config offline using the 3.x PDF documentation, and it > doesn''t lay it out as plainly as FAQ 57. My bad. > > I replaced "loose" with "balance" in the providers options. However, > after restarting shorewall (sudo service shorewall restart) the routing > totally wigged out. Traffic was going out on the two interfaces > (ISP1/2) but if data was coming back, it wasn''t reaching the clients. I > reverted to the old config and all was good (all traffic on one interface).Your tcrules are so completely broken (see my other post) that this isn''t surprising. I suggest that you totally forget traffic shaping for the time being and get multi-ISP working the way that you want it. Then *and only then* should you add traffic shaping. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
James Gray
2007-Aug-15 00:05 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
On Wed, 15 Aug 2007 09:55:06 am Tom Eastep wrote:> James Gray wrote: > > Tom Eastep wrote: > >> James Gray wrote: > >>> I thought I followed all the docs but I feel like I''ve missed something > >>> really basic. > >> > >> Like maybe Shorewall FAQ 57? > >> > >> -Tom > > > > Thanks Tom. I really appreciate the fast response :) I''ve been doing > > most of the config offline using the 3.x PDF documentation, and it > > doesn''t lay it out as plainly as FAQ 57. My bad. > > > > I replaced "loose" with "balance" in the providers options. However, > > after restarting shorewall (sudo service shorewall restart) the routing > > totally wigged out. Traffic was going out on the two interfaces > > (ISP1/2) but if data was coming back, it wasn''t reaching the clients. I > > reverted to the old config and all was good (all traffic on one > > interface). > > Your tcrules are so completely broken (see my other post) that this > isn''t surprising.:P Yep - I''ve reversed them (but am leaving them commented out for now) and re-read the documentation. I''ve also fixed the MARK_IN_FORWARD_CHAIN in shorewall.conf.> I suggest that you totally forget traffic shaping for the time being and > get multi-ISP working the way that you want it. Then *and only then* > should you add traffic shaping.Yup. I''m coming from a Cisco background and whilst I understand TCP/IP networking, they way all this shaping and routing stuff happens in Linux is new. I''m already on the road you have suggested - keep it simple, get the multi-ISP setup working, then worry about traffic shaping. I really appreciate your help. Do you have a paypal account for donations? (seriously) Cheers, James ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Tom Eastep
2007-Aug-15 00:12 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
James Gray wrote:> > I really appreciate your help. Do you have a paypal account for donations? > (seriously) >Please see the Shorewall home page (http://www.shorewall.net/shorewall_index.htm#Donations). I prefer that donations be made to one of my charities or to a charity of your choice. Thanks! -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
James Gray
2007-Aug-15 07:04 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
On Wed, 15 Aug 2007 09:55:06 am Tom Eastep wrote:> James Gray wrote: > > Tom Eastep wrote: > >> James Gray wrote: > >>> I thought I followed all the docs but I feel like I''ve missed something > >>> really basic. > >> > >> Like maybe Shorewall FAQ 57? > >> > >> -Tom > > > > Thanks Tom. I really appreciate the fast response :) I''ve been doing > > most of the config offline using the 3.x PDF documentation, and it > > doesn''t lay it out as plainly as FAQ 57. My bad. > > > > I replaced "loose" with "balance" in the providers options. However, > > after restarting shorewall (sudo service shorewall restart) the routing > > totally wigged out. Traffic was going out on the two interfaces > > (ISP1/2) but if data was coming back, it wasn''t reaching the clients. I > > reverted to the old config and all was good (all traffic on one > > interface). > > Your tcrules are so completely broken (see my other post) that this > isn''t surprising. > > I suggest that you totally forget traffic shaping for the time being and > get multi-ISP working the way that you want it. Then *and only then* > should you add traffic shaping.Ok. I got the multi-ISP stuff going without any traffic shaping but that''s not particularly useful for us. We must have certain traffic going out over specific links, otherwise the service will fail (tcpwrappers "paranoid" and certain services that must originate from one link or the other). But there''s traffic that should be going over specific links over both, and other traffic bound to an interface that should be on the other :( For instance, ALL ssh (tcp/22) traffic should be going out over NET_IF1 (eth3, via 172.16.3.1) with mark of 10 or 20. But here''s a tcp trace from the LAN: $tcptraceroute XX.XX.XX.XX 22 Selected device eth0, address 10.10.10.74, port 37321 for outgoing packets Tracing the path to XX.XX.XX.XX on TCP port 22 (ssh), 30 hops max 1 10.10.10.1 0.727 ms 0.142 ms 0.128 ms 2 172.16.4.1 0.859 ms 0.656 ms 0.643 ms <--- *** NO! *** 3 203.38.103.1 11.029 ms 10.983 ms 9.575 ms 4 TenGigabitEthernet8-1.ken17.Sydney.telstra.net (203.50.20.27) 10.486 ms 10.770 ms 11.849 ms 5 ge-2-1-0-25.bdr5.hay.connect.com.au (203.63.130.250) 11.091 ms 10.497 ms 12.283 ms 6 gigabitethernet0-1.cor10.hay.connect.com.au (203.63.217.3) 11.623 ms 11.821 ms 10.474 ms 7 * * * Hop #2 should be going out via 172.16.3.1. The router it''s going through is actually NET_IF2 (eth4). Consequently, the traffic is dropped because the destination will only accept connections from the first ISP (NET_IF1). I thought the config below would achieve the desired result...but apparently not. I''ve changed the providers OPTIONS for the two ISP''s to "track,balance". Which got things working...apart from this weird traffic/routing behaviour. Attached is another shorewall dump whilst running with the config below. tcrules: #MARK SOURCE DEST PROTO DEST SOURCE USER TEST LENGTH TOS # PORT(S) PORT(S) 50 $ANY_IP 50 $FW 50 $LAN_NETWORK $ANY_IP udp $GAMES 50 $LAN_NETWORK $ANY_IP tcp $GAMES 50 $LAN_NETWORK $ANY_IP udp $P2P 50 $LAN_NETWORK $ANY_IP tcp $P2P 50 $LAN_NETWORK $ANY_IP udp $IM 50 $LAN_NETWORK $ANY_IP tcp $IM 50 $LAN_NETWORK $ANY_IP tcp $ALLMAIL 40 $LAN_NETWORK $ANY_IP tcp $STREAM 30 $LAN_NETWORK $ANY_IP tcp nntp 30 $DMZ1_NETWORK $ANY_IP tcp $ALLMAIL 20 $LAN_NETWORK $ANY_IP tcp $FTP 20 $LAN_NETWORK $ANY_IP tcp $WWW 20 $LAN_NETWORK $ANY_IP tcp ssh - - - 513: 10 $LAN_NETWORK $ANY_IP tcp domain 10 $LAN_NETWORK $ANY_IP udp domain 10 $LAN_NETWORK $ANY_IP tcp ssh - - - 0:512 tcclasses: #INTERFACE MARK RATE CEIL PRIORITY OPTIONS $NET_IF1 10 full full 1 tcp-ack,tos-minimize-delay $NET_IF1 20 9*full/10 9*full/10 2 $NET_IF1 30 6*full/10 6*full/10 3 default $NET_IF2 40 full full 4 tcp-ack,tos-minimize-delay $NET_IF2 50 6*full/10 6*full/10 5 default providers: #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY squid 1 202 - $LAN_IF $PROXYSVR loose - $ISP1 2 1 main $NET_IF1 $NET_IF1_GW $PROVOPTS $COPY $ISP2 3 2 main $NET_IF2 $NET_IF2_GW $PROVOPTS $COPY (COPY=eth0,eth1,eth2) What is left to make this work....it feels close :-/ Cheers, James -- "All snakes who wish to remain in Ireland will please raise their right hands." -- Saint Patrick ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
Jerry Vonau
2007-Aug-15 21:37 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
James Gray wrote:> On Wed, 15 Aug 2007 09:55:06 am Tom Eastep wrote: >> James Gray wrote: >>> Tom Eastep wrote: >>>> James Gray wrote: >>>>> I thought I followed all the docs but I feel like I''ve missed something >>>>> really basic. >>>> Like maybe Shorewall FAQ 57? >>>> >>>> -Tom >>> Thanks Tom. I really appreciate the fast response :) I''ve been doing >>> most of the config offline using the 3.x PDF documentation, and it >>> doesn''t lay it out as plainly as FAQ 57. My bad. >>> >>> I replaced "loose" with "balance" in the providers options. However, >>> after restarting shorewall (sudo service shorewall restart) the routing >>> totally wigged out. Traffic was going out on the two interfaces >>> (ISP1/2) but if data was coming back, it wasn''t reaching the clients. I >>> reverted to the old config and all was good (all traffic on one >>> interface). >> Your tcrules are so completely broken (see my other post) that this >> isn''t surprising. >> >> I suggest that you totally forget traffic shaping for the time being and >> get multi-ISP working the way that you want it. Then *and only then* >> should you add traffic shaping. > > Ok. I got the multi-ISP stuff going without any traffic shaping but that''s > not particularly useful for us. We must have certain traffic going out over > specific links, otherwise the service will fail (tcpwrappers "paranoid" and > certain services that must originate from one link or the other). But > there''s traffic that should be going over specific links over both, and other > traffic bound to an interface that should be on the other :( > > For instance, ALL ssh (tcp/22) traffic should be going out over NET_IF1 (eth3, > via 172.16.3.1) with mark of 10 or 20. But here''s a tcp trace from the LAN: >That is only for traffic control, that is in the forward chain. The mark of 10 or 20 relates to routing and a provider how??> $tcptraceroute XX.XX.XX.XX 22 > Selected device eth0, address 10.10.10.74, port 37321 for outgoing packets > Tracing the path to XX.XX.XX.XX on TCP port 22 (ssh), 30 hops max > 1 10.10.10.1 0.727 ms 0.142 ms 0.128 ms > 2 172.16.4.1 0.859 ms 0.656 ms 0.643 ms <--- *** NO! *** > 3 203.38.103.1 11.029 ms 10.983 ms 9.575 ms > 4 TenGigabitEthernet8-1.ken17.Sydney.telstra.net (203.50.20.27) 10.486 ms > 10.770 ms 11.849 ms > 5 ge-2-1-0-25.bdr5.hay.connect.com.au (203.63.130.250) 11.091 ms 10.497 ms > 12.283 ms > 6 gigabitethernet0-1.cor10.hay.connect.com.au (203.63.217.3) 11.623 ms > 11.821 ms 10.474 ms > 7 * * * > > Hop #2 should be going out via 172.16.3.1. The router it''s going through is > actually NET_IF2 (eth4). Consequently, the traffic is dropped because the > destination will only accept connections from the first ISP (NET_IF1). I > thought the config below would achieve the desired result...but apparently > not. > > I''ve changed the providers OPTIONS for the two ISP''s to "track,balance". > Which got things working...apart from this weird traffic/routing behaviour. > Attached is another shorewall dump whilst running with the config below. > > tcrules: > #MARK SOURCE DEST PROTO DEST SOURCE USER TEST LENGTH TOS > # PORT(S) PORT(S)<snip>> 20 $LAN_NETWORK $ANY_IP tcp ssh - - - 513: > 10 $LAN_NETWORK $ANY_IP tcp ssh - - - 0:512 >the routing rules: 0: from all lookup local 10001: from all fwmark 0x1 lookup IINET 10002: from all fwmark 0x2 lookup TELSTRA 10202: from all fwmark 0xca lookup squid 20256: from 172.16.3.2 lookup IINET 20512: from 172.16.4.2 lookup TELSTRA 32766: from all lookup main 32767: from all lookup default When using the tcrules file to override balancing to use only one isp, you should be using the providers'' mark here (in the tcpre chain, that is part of the prerouting chain) to direct traffic into the providers'' routing table to pick your preferred isp. You''ll need to use something like: 1:P $LAN_NETWORK $ANY_IP tcp ssh 1 = mark of your "preferred" provider P = use mark in prerouting chain> > What is left to make this work....it feels close :-/ > > Cheers, > > James >Hope that is your fix.. Jerry ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
James Gray
2007-Aug-15 23:59 UTC
Re: Multi-ISP + Traffic Shaping Problem (Shorewall 3.4.5)
From: "Jerry Vonau">> For instance, ALL ssh (tcp/22) traffic should be going out over NET_IF1 (eth3, >> via 172.16.3.1) with mark of 10 or 20. But here''s a tcp trace from the LAN: > > That is only for traffic control, that is in the forward chain. The mark > of 10 or 20 relates to routing and a provider how??Thanks Jerry. I was of the understanding that as the marks 10/20 are associated with the interface for the required ISP, that traffic so marked will be routed out that ISP: tcclasses: #INTERFACE MARK RATE CEIL PRIORITY OPTIONS $NET_IF1 10 full full 1 tcp-ack,tos-minimize-delay $NET_IF1 20 9*full/10 9*full/10 2 $NET_IF1 30 6*full/10 6*full/10 3 default ...etc>> tcrules: >> #MARK SOURCE DEST PROTO DEST SOURCE USER TEST LENGTH TOS >> # PORT(S) PORT(S)<snip>>> 20 $LAN_NETWORK $ANY_IP tcp ssh - - - 513: >> 10 $LAN_NETWORK $ANY_IP tcp ssh - - - 0:512 > > the routing rules: > > 0: from all lookup local > 10001: from all fwmark 0x1 lookup IINET > 10002: from all fwmark 0x2 lookup TELSTRA > 10202: from all fwmark 0xca lookup squid > 20256: from 172.16.3.2 lookup IINET > 20512: from 172.16.4.2 lookup TELSTRA > 32766: from all lookup main > 32767: from all lookup default > > When using the tcrules file to override balancing to use only one isp, > you should be using the providers'' mark here (in the tcpre chain, that > is part of the prerouting chain) to direct traffic into the providers'' > routing table to pick your preferred isp. You''ll need to use something like: > > 1:P $LAN_NETWORK $ANY_IP tcp ssh > > 1 = mark of your "preferred" provider > P = use mark in prerouting chainWont that bypass the traffic shaping? That''s a show stopper for us. We rely heavily on SSH and it can''t really be delayed waiting for Joe User to finish downloading a CD ISO :( Maybe I should be using classes in the tcrules instead: providers: #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY $ISP1 1 1 main $NET_IF1 $NET_IF1_GW $PROVOPTS $COPY $ISP2 2 2 main $NET_IF2 $NET_IF2_GW $PROVOPTS $COPY squid 3 202 - $LAN_IF $PROXYSVR loose - tcrules: #MARK SOURCE DEST PROTO DEST SOURCE USER TEST LENGTH TOS # PORT(S) PORT(S) -- Snipped -- 1:120 $LAN_NETWORK $ANY_IP tcp ssh - - - 513: 1:110 $LAN_NETWORK $ANY_IP tcp ssh - - - 0:512 Or is this a two-step process? One rule in the prerouting chain to force a specific ISP, then another rule in the forward chain to mark the traffic for shaping?>> What is left to make this work....it feels close :-/ > > Hope that is your fix..Me too Jerry :) I''ll give it a shot, but like I said, even if the routing works we need the traffic shaping too. Cheers, James ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/