William R Sowerbutts
2005-Jul-16 16:32 UTC
Multiple providers connected over PPP appears to be broken
Hi,
I''m using Shorewall 2.4.1 on Debian "testing".
I have two ADSL connections. My intention is to use one of these connections
for HTTP/HTTPS traffic, and a second connection for all other traffic.
However, this is not quite working correctly.
Incoming connections to each ADSL connection work perfectly and traffic is
returned with the correct source address and via the correct interface. This
is a good start. Outgoing HTTP traffic is masqueraded to the correct source IP
address and sent out of the correct interface, however the reply traffic from
the remote web server is received but is apparently dropped somewhere inside
my firewall/router system.
The two ADSL connections are terminated at my end on Sangoma S518 PCI ADSL
cards. Both ISPs run PPP over ATM on their ADSL circuits. The Sangoma drivers
are running in "TTY mode", which means that the PPP frames are
presented on a
TTY device. I use the normal userspace "pppd" application, so this
setup is
not much different to using a pair of dialup modems.
One ADSL line (from Andrews & Arnold) is connected to ppp0. This line has a
couple of subnets routed to it, and the PPP link has a static IP address.
(A&A
are a great UK ISP, I wholeheartedly recommend them)
The second ADSL line (from UKOnline, whose customer support is stunningly
craptastic) is connected to ppp1. This line has an address dynamically
allocated each time the line comes up.
I''ve followed the instructions on the "Shorewall and Routing"
page. I had to
replace the referenes to "eth0" and "eth1" with
"ppp0" and "ppp1". My
/etc/shorewall/providers file looks like this:
#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS
aa 1 1 main ppp0 detect track
uko 2 2 main ppp1 detect track
TC_ENABLED is set to Yes in /etc/shorewall/shorewall.conf.
My /etc/shorewall/tcrules file looks like this:
#MARK SOURCE DEST PROTO PORT(S) CLIENT USER
TEST
# PORT(S)
2:P 192.168.0.0/16 0.0.0.0/0 tcp http,https
eth0 and eth1 are local networks. eth0.2 is my DMZ, which runs on a VLAN. My
/etc/shorewall/masq file therefore looks like this:
#INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC
# onto the DMZ
eth0.2 eth0 81.2.88.222
eth0.2 eth1 81.2.88.222
# Internet via A&A
ppp0 eth0 81.2.88.222
ppp0 eth1 81.2.88.222
# Internet via UKOnline
ppp1 eth0
ppp1 eth1
This all seems fine and dandy to me.
So, when I try to connect from a host on the local network to a remote HTTP
server (by doing "telnet roscoe.wn.com 80"), the connection eventually
times
out. Hmmm, let''s see where it''s going wrong ...
Using tcpdump, I see that the packets are sent correctly from my system on the
local network to the default gateway (which is the firewall):
17:13:04.808259 IP (tos 0x10, ttl 64, id 29186, offset 0, flags [DF], length:
60) 192.168.100.198.47562 > 195.149.84.15.80: S [tcp sum ok]
1686854051:1686854051(0) win 5840 <mss 1460,sackOK,timestamp 750463905
0,nop,wscale 2>
17:13:07.807271 IP (tos 0x10, ttl 64, id 29188, offset 0, flags [DF], length:
60) 192.168.100.198.47562 > 195.149.84.15.80: S [tcp sum ok]
1686854051:1686854051(0) win 5840 <mss 1460,sackOK,timestamp 750466905
0,nop,wscale 2>
So far so good. The firewall receives these on its local interface, eth0
(please bear in mind that there is a little clock skew between these two
systems):
17:13:04.844772 IP (tos 0x10, ttl 64, id 29186, offset 0, flags [DF], length:
60) 192.168.100.198.47562 > 195.149.84.15.80: S [tcp sum ok]
1686854051:1686854051(0) win 5840 <mss 1460,sackOK,timestamp 750463905
0,nop,wscale 2>
17:13:07.843676 IP (tos 0x10, ttl 64, id 29188, offset 0, flags [DF], length:
60) 192.168.100.198.47562 > 195.149.84.15.80: S [tcp sum ok]
1686854051:1686854051(0) win 5840 <mss 1460,sackOK,timestamp 750466905
0,nop,wscale 2>
The firewall masquerades them and sends them out on ppp1, and receives the
replies from the webserver:
17:13:04.844843 IP (tos 0x10, ttl 63, id 29186, offset 0, flags [DF], length:
60) 195.172.52.51.47562 > 195.149.84.15.80: S [tcp sum ok]
1686854051:1686854051(0) win 5840 <mss 1420,sackOK,timestamp 750463905
0,nop,wscale 2>
17:13:04.853423 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], length: 60)
195.149.84.15.80 > 195.172.52.51.47562: S [tcp sum ok]
4042245451:4042245451(0) ack 1686854052 win 5792 <mss 1460,sackOK,timestamp
218272019 750463905,nop,wscale 0>
17:13:06.326670 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], length: 60)
195.149.84.15.80 > 195.172.52.51.47561: S [tcp sum ok]
3972198297:3972198297(0) ack 1647498586 win 5792 <mss 1460,sackOK,timestamp
218272167 750418987,nop,wscale 0>
17:13:07.843705 IP (tos 0x10, ttl 63, id 29188, offset 0, flags [DF], length:
60) 195.172.52.51.47562 > 195.149.84.15.80: S [tcp sum ok]
1686854051:1686854051(0) win 5840 <mss 1420,sackOK,timestamp 750466905
0,nop,wscale 2>
17:13:07.853129 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], length: 60)
195.149.84.15.80 > 195.172.52.51.47562: S [tcp sum ok]
4042245451:4042245451(0) ack 1686854052 win 5792 <mss 1460,sackOK,timestamp
218272319 750463905,nop,wscale 0>
17:13:07.926063 IP (tos 0x0, ttl 53, id 0, offset 0, flags [DF], length: 60)
195.149.84.15.80 > 195.172.52.51.47562: S [tcp sum ok]
4042245451:4042245451(0) ack 1686854052 win 5792 <mss 1460,sackOK,timestamp
218272327 750463905,nop,wscale 0>
However, although replies are received from the webserver as shown above,
these are NOT sent back to the masqueraded client on the local network. The
client on the local network retries several times and eventually the
connection times out.
How can I fix this? The output of "/sbin/shorewall status" is
attached, as
requested. The command was run immediately after attempting to open a
connection to port 80 on 195.149.84.15 (note that the source port number is
different as this was a separate attempt to that shown above)
Any help would be greatly appreciated, I feel like I''m banging my head
against
the wall here.
Incidentally, if anyone wants to try out the S518 PCI ADSL cards, they are
pretty good *BUT* they don''t appear to play nicely with SMP systems --
the
connection randomly stops carrying traffic after a few hours. Caveat emptor.
Also, I hear that Sangoma will be bringing out a revised, design in Low
Profile PCI format that will apparently be cheaper.
Many thanks,
Will
_________________________________________________________________________
William R Sowerbutts will@sowerbutts.com
"Carpe post meridiem"
http://sowerbutts.com
main(){char*s=">#=0>
^#X@#@^7=",c=0,m;for(;c<15;c++)for
(m=-1;m<7;putchar(m++/6&c%3/2?10:s[c]-31&1<<m?42:32));}
Jerry Vonau
2005-Jul-16 18:01 UTC
Re: Multiple providers connected over PPP appears to be broken
> I''ve followed the instructions on the "Shorewall and Routing" page. I hadto> replace the referenes to "eth0" and "eth1" with "ppp0" and "ppp1". My > /etc/shorewall/providers file looks like this: > > #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAYOPTIONS> aa 1 1 main ppp0 detecttrack> uko 2 2 main ppp1 detecttrack> > TC_ENABLED is set to Yes in /etc/shorewall/shorewall.conf. > My /etc/shorewall/tcrules file looks like this: > > #MARK SOURCE DEST PROTO PORT(S) CLIENTUSER TEST> # PORT(S) > 2:P 192.168.0.0/16 0.0.0.0/0 tcp http,https >Might want to try 2:PC for the mark. The providers file uses CONNMARK marking, while the above just mark packets and not the connection, not quite the same mark. Jerry ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
William R Sowerbutts
2005-Jul-16 18:26 UTC
Re: Multiple providers connected over PPP appears to be broken
On Sat, Jul 16, 2005 at 01:01:55PM -0500, Jerry Vonau wrote:>> I''ve followed the instructions on the "Shorewall and Routing" page. I had >> to replace the referenes to "eth0" and "eth1" with "ppp0" and "ppp1". My >> /etc/shorewall/providers file looks like this: >> >> #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS >> aa 1 1 main ppp0 detect track >> uko 2 2 main ppp1 detect track > >> >> TC_ENABLED is set to Yes in /etc/shorewall/shorewall.conf. >> My /etc/shorewall/tcrules file looks like this: >> >> #MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST >> # PORT(S) >> 2:P 192.168.0.0/16 0.0.0.0/0 tcp http,https >> > >Might want to try 2:PC for the mark. The providers file uses CONNMARK >marking, while the above just mark packets and not the connection, not quite >the same mark. > >JerryAhhhhhhh -- thanks for the tip. I had to use ":CP" instead of ":PC" or Shorewall would fail to start. This changes things, but now it''s broken in a different way. Previously: The packet would be masqueraded to the correct source IP address for ppp1, and the packet would be sent out of ppp1. The replies would come back in over ppp1, but would never reach the masquerading client -- apparently because I hadn''t marked the connection with the appropriate mark value, so Shorewall was dropping them. Now, after the changing the MARK column to "2:CP": The packet goes out on ppp1, which is correct, but the source address is set to an address reachable only via ppp0. The ISP on ppp1 doesn''t do source filtering, which is good, but of course the replies come back in over ppp0 since they have the wrong source address. I''ve tried specifying the source address by hand in the "masq" file; this doesn''t do anything. So I''m seeing this traffic going out on ppp1: 19:21:03.435799 IP (tos 0x10, ttl 63, id 59734, offset 0, flags [DF], length: 52) 81.2.88.222.34815 > 195.149.84.15.80: . [tcp sum ok] ack 3545182261 win 1460 <nop,nop,timestamp 758142261 219039915> The source address should be set to the address of ppp1, which is dynamically allocated (currently 195.172.52.51). How can I arrange for this to happen? I''ve attached the new output of "/sbin/shorewall status". Thanks, Will _________________________________________________________________________ William R Sowerbutts will@sowerbutts.com "Carpe post meridiem" http://sowerbutts.com main(){char*s=">#=0> ^#X@#@^7=",c=0,m;for(;c<15;c++)for (m=-1;m<7;putchar(m++/6&c%3/2?10:s[c]-31&1<<m?42:32));}
Jerry Vonau
2005-Jul-16 19:16 UTC
Re: Multiple providers connected over PPP appears to be broken
> >> #MARK SOURCE DEST PROTO PORT(S) CLIENTUSER TEST> >> #PORT(S)> >> 2:P 192.168.0.0/16 0.0.0.0/0 tcp http,https > >> > > > >Might want to try 2:PC for the mark. The providers file uses CONNMARK > >marking, while the above just mark packets and not the connection, notquite> >the same mark. > > > >Jerry > > Ahhhhhhh -- thanks for the tip. I had to use ":CP" instead of ":PC" or > Shorewall would fail to start.Opps typo sorry.> This changes things, but now it''s broken in a different way. > > Previously: The packet would be masqueraded to the correct source IPaddress> for ppp1, and the packet would be sent out of ppp1. The replies wouldcome> back in over ppp1, but would never reach the masquerading client --apparently> because I hadn''t marked the connection with the appropriate mark value,so> Shorewall was dropping them. > > Now, after the changing the MARK column to "2:CP": The packet goes out on > ppp1, which is correct, but the source address is set to an addressreachable> only via ppp0. The ISP on ppp1 doesn''t do source filtering, which isgood, but> of course the replies come back in over ppp0 since they have the wrongsource> address. > > I''ve tried specifying the source address by hand in the "masq" file; this > doesn''t do anything. > > So I''m seeing this traffic going out on ppp1: >In the providers file try replacing "track" with "track,balance"> 19:21:03.435799 IP (tos 0x10, ttl 63, id 59734, offset 0, flags [DF],length: 52) 81.2.88.222.34815 > 195.149.84.15.80: . [tcp sum ok] ack 3545182261 win 1460 <nop,nop,timestamp 758142261 219039915>> > The source address should be set to the address of ppp1, which isdynamically> allocated (currently 195.172.52.51). How can I arrange for this tohappen? From FAQ1d dynamic addresses: ETH0_IP=`find_first_interface_address eth0` In params try: ISP2IP=`find_first_interface_address pppX` replacing the X as required, then use $ISP2IP in the masq file for that provider. Ensure shorewall is restarted every time a provider is reconnected after being disconnected. Jerry ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click