thr3ads.net - Shorewall users - Alternate route for HTTP traffic? [Mar 2013]

If this information is useful, please help other people find it:
Share via:

2013-Mar-17 14:41 UTC

Alternate route for HTTP traffic?

I'm trying to figure out how to add selective routing of HTTP traffic over
an openvpn tunnel.

Before I post a dump to the mailing list I wanted to get confirmation that the
configuration I'm trying is the correct one, and if it isn't, what
specifically is needed to achieve the objective.

The gateway router and firewall ("Jeeves") OS is Ubuntu 12.04 Precise
(with Shorewall 4.4.26.1) connected to the Internet using a PPP connection over
VDSL (ppp0). The ppp0 interface connects via a
bridged external VDSL modem which is connected to a 100Mbps ethernet interface
(vdsl). An openvpn tunnel (tun0) connects to a data-centre dedicated server
("Pella") which is configured for
masquerading. The local area network is connected using a bridge (br0) that
combines eight ethernet ports (5 x 1000Mbps, 3 x 100Mbps) and a hostapd managed
wireless lan (wlan0). There is a second
link (eth9) to a management ethernet interface on the VDSL modem.

                               v---------mgmt--> eth9------¬
Internet <--> ISP <--PPP--> VDSL modem <--PPP-->
vdsl--Jeeves--br0 [eth1:8,wlan0] <-- LAN 10.254.251.0/24 -->
  ^                                                        |
  |                                                        |
eth0--pella--tun0 <-- 10.254.252.17 -- 10.254.252.18 --> tun0

I applied a fix mentioned for Ubuntu:

$ sed -i '1,1 s,/bin/sh,/bin/bash,' /sbin/shorewall

Squid3 on the firewall (Jeeves) is used in transparent proxy mode using
Shorewall rules to redirect requests for TCP port 80 to port 3128.
dnsmasq on Jeeves is configured to use upstream DNS servers via tun0.

/etc/shorewall/rules
# Redirect HTTP port 80 requests to the transparent squid proxy running on the
firewall
REDIRECT  loc        3128     tcp      www              -         
!10.0.0.0/8,192.168.1.0/24

For a time the default route on Jeeves was via tun0 so all traffic went via
Pella. That all worked fine.

Later it became desirable to route HTTPS and other traffic directly to the ISP
on ppp0 (better latency) so the default route became ppp0. DNS traffic is still
routed via tun0 (a combination of the
dnsmasq DNS server entries, and openvpn's client routes for specific DNS
hosts).

HTTP traffic from the LAN is being proxied by squid3 and using the ppp0
interface. I'd like to have just this HTTP traffic routed over tun0.

I couldn't find any specific documentation on this scenario but the
suggestion seems to be to combine packet marking with an alternate route.
Therefore I started off with:

/etc/shorewall/tcrules
# MARK SOURCE DEST PROTO DEST_PORTS SOURCE_PORTS USER TEST LENGTH TOS
  80   $FW    -    tcp   80         -            -    -    -      -

I confirmed via 'shorewall iptrace -p tcp --dport 80' that the mark is
being applied to the packets. Then I added a provider:

/etc/shorewall/providers
# NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
  VPN  1      80   main      tun0      detect  track,loose br0

I have tried various permutations of OPTIONS and COPY - these are the most
recent iteration.


This configuration fails to work correctly. Specifically, using tcpdump on both
ends of tun0 (Jeeves and Pella) and watching the iptrace output I can see that
packets are being intercepted by
squid3, proxied out via tun0, replies come back in on tun0 but do not reach the
original client. I can't be sure if they reach squid3 or not. Trying a
manual connection from Jeeves seems to indicate
the response packets aren't making it to the client process, e.g.:

$ wget -O - http://gb.archive.ubuntu.com/ubuntu/

tries to connect to each of the round-robin IPs associated with the hostname but
eventually fails even though the tcpdump and iptrace logging shows responses
returning via tun0 to Jeeves.

This affects LAN clients *and* direct connections from Jeeves whether or not the
Squid transparent proxy is active, which leads me to believe I'm missing
something in the Shorewall configuration.

Based on an analysis of the logs I thought maybe SNAT or MASQ was required for
the traffic being routed via tun0 so I tried two variations but neither seems to
solve the issue:

/etc/shorewall/masq
#INTERFACE		SOURCE		ADDRESS		PROTO	PORT(S)	IPSEC	MARK
ppp0	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16	82.71.24.87
eth9	br0
#tun0    10.0.0.0/8,127.0.0.0/8 10.254.252.18
#tun0    br0


------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar
_______________________________________________
Shorewall-users mailing list
Shorewall-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shorewall-users

Tom Eastep

2013-Mar-17 15:09 UTC

head link

Re: Alternate route for HTTP traffic?

On 03/17/2013 07:41 AM, TJ wrote:
> This configuration fails to work correctly. Specifically, using
> tcpdump on both ends of tun0 (Jeeves and Pella) and watching the
> iptrace output I can see that packets are being intercepted by 
> squid3, proxied out via tun0, replies come back in on tun0 but do not
> reach the original client. I can''t be sure if they reach squid3 or
> not. Trying a manual connection from Jeeves seems to indicate the
> response packets aren''t making it to the client process, e.g.:
> 
> $ wget -O - http://gb.archive.ubuntu.com/ubuntu/
> 
> tries to connect to each of the round-robin IPs associated with the
> hostname but eventually fails even though the tcpdump and iptrace
> logging shows responses returning via tun0 to Jeeves.
Your system log is likely full of ''Martian'' messages. Please
confirm.
> 
> This affects LAN clients *and* direct connections from Jeeves whether
> or not the Squid transparent proxy is active, which leads me to
> believe I''m missing something in the Shorewall configuration.
> 
Probably need to set both log_martians and routefilter to 0 for tun0 in
/etc/shorewall/interfaces.

-Tom
-- 
Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________



------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

2013-Mar-17 16:46 UTC

head link

Re: Alternate route for HTTP traffic?

On 17/03/13 15:09, Tom Eastep wrote:

I assume your focus on what iss happening is an indication that I do have the
correct basic configuration?
>>
>> $ wget -O - http://gb.archive.ubuntu.com/ubuntu/
>>
>> tries to connect to each of the round-robin IPs associated with the
>> hostname but eventually fails even though the tcpdump and iptrace
>> logging shows responses returning via tun0 to Jeeves.
> 
> Your system log is likely full of ''Martian'' messages.
Please confirm.
Messages containing "martian source" or "ll header" are
being directed to /var/log/network.log. Checking that file shows it was, some of
the times I was experimenting with this configuration, seeing
lots of "martian source" warnings, although not within the couple of
hours.
>>
>> This affects LAN clients *and* direct connections from Jeeves whether
>> or not the Squid transparent proxy is active, which leads me to
>> believe I''m missing something in the Shorewall configuration.
>>
> 
> Probably need to set both log_martians and routefilter to 0 for tun0 in
> /etc/shorewall/interfaces.
In the data-centre, the host "pella":

$ for if in eth0 eth0:0 tun0; do ifconfig $if | head -n 2; done
eth0      Link encap:Ethernet  HWaddr f2:3c:91:96:85:58
          inet addr:109.74.197.121  Bcast:109.74.197.255  Mask:255.255.255.0
eth0:0    Link encap:Ethernet  HWaddr f2:3c:91:96:85:58
          inet addr:109.74.197.122  Bcast:109.74.197.255  Mask:255.255.255.0
tun0      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.254.252.1  P-t-P:10.254.252.2  Mask:255.255.255.255

Back on host "Jeeves":

$ for if in ppp0 tun0 br0; do ifconfig $if | head -n 2; done
ppp0      Link encap:Point-to-Point Protocol
          inet addr:82.71.24.87  P-t-P:62.3.83.21  Mask:255.255.255.255
tun0      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.254.252.18  P-t-P:10.254.252.17  Mask:255.255.255.255
br0       Link encap:Ethernet  HWaddr 00:04:23:bd:f6:2c
          inet addr:10.254.251.1  Bcast:10.254.251.255  Mask:255.255.255.0

/etc/shorewall/interfaces:
#ZONE	INTERFACE	BROADCAST	OPTIONS
net	ppp0	-	routefilter,tcpflags,logmartians,nosmurfs,mss=1452
net	ppp1	-	routefilter,tcpflags,logmartians,nosmurfs,mss=1452
loc	dsl0	detect	routefilter,tcpflags,logmartians,nosmurfs
loc	br0	detect	dhcp,routefilter,tcpflags,logmartians,nosmurfs
loc	wlan0	detect	dhcp,routefilter,tcpflags,logmartians,nosmurfs
loc	eth9	detect	routefilter,tcpflags,logmartians,nosmurfs
dc	tun0	detect	routefilter=0,tcpflags,logmartians=0,nosmurfs

/etc/shorewall/masq:
#INTERFACE		SOURCE		ADDRESS		PROTO	PORT(S)	IPSEC	MARK
ppp0	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16	82.71.24.87
#ppp1	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16	82.71.24.87
eth9	br0
#tun0    10.0.0.0/8,127.0.0.0/8 10.254.252.18
#tun0    br0

Note: dsl0 and ppp1 are hold-overs from the former ADSL line before a recent
upgrade to VDSL. dsl0 is an ethernet interface
on an internal PCI ADSL2+ modem. ppp1 is not used but its configuration is
maintained in the system just-in-case.

$ ip rule ls
0:	from all lookup local
10000:	from all fwmark 0x50/0xff lookup VPN
32766:	from all lookup main
32767:	from all lookup default

$ ip route ls table VPN
default via 10.254.252.17 dev tun0  src 10.254.252.18
10.254.1.0 via 10.254.251.50 dev br0
10.254.251.0/24 dev br0  proto kernel  scope link  src 10.254.251.1
10.254.252.1 via 10.254.252.17 dev tun0
10.254.252.17 dev tun0  scope link  src 10.254.252.18
109.74.192.20 via 10.254.252.17 dev tun0
109.74.193.20 via 10.254.252.17 dev tun0

$ ip route ls
default via 62.3.83.21 dev ppp0
10.254.1.0 via 10.254.251.50 dev br0
10.254.248.0/30 dev eth9  proto kernel  scope link  src 10.254.248.2
10.254.251.0/24 dev br0  proto kernel  scope link  src 10.254.251.1
10.254.252.1 via 10.254.252.17 dev tun0
10.254.252.17 dev tun0  scope link  src 10.254.252.18
62.3.83.21 dev ppp0  proto kernel  scope link  src 82.71.24.87
109.74.192.20 via 10.254.252.17 dev tun0
109.74.193.20 via 10.254.252.17 dev tun0

Note: the last two entries are the DNS server routes added by openvpn.

Testing fails with:

$ wget -O - http://gb.archive.ubuntu.com/ubuntu/
--2013-03-17 16:25:11--  http://gb.archive.ubuntu.com/ubuntu/
Resolving gb.archive.ubuntu.com (gb.archive.ubuntu.com)... 91.189.92.201,
91.189.92.202, 194.169.254.10, ...
Connecting to gb.archive.ubuntu.com (gb.archive.ubuntu.com)|91.189.92.201|:80...
^C

$ sudo tcpdump -ni tun0 port 80 or tcp port 3128
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
16:25:12.024613 IP 82.71.24.87.39176 > 91.189.92.201.80: Flags [S], seq
1971087366, win 14520, options [mss 1452,sackOK,TS val 168165226 ecr
0,nop,wscale 4], length 0
16:25:13.023137 IP 82.71.24.87.39176 > 91.189.92.201.80: Flags [S], seq
1971087366, win 14520, options [mss 1452,sackOK,TS val 168165476 ecr
0,nop,wscale 4], length 0

Notice the source IP on tun0 is the IP of ppp0. This was why I added the MASQ
entry:

/etc/shorewall/masq
#INTERFACE		SOURCE		ADDRESS		PROTO	PORT(S)	IPSEC	MARK
ppp0	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16	82.71.24.87
#ppp1	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16	82.71.24.87
eth9	br0
tun0    10.0.0.0/8,127.0.0.0/8 10.254.252.18
#tun0    br0

$ sudo iptables -t nat -nL | awk  ''$0~/(POSTROUTING|(ppp0|tun0)_masq)/
{ show=1} $0~/^$/ {show=0} {if (show) print $0}''
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
ppp0_masq  all  --  0.0.0.0/0            0.0.0.0/0
eth9_masq  all  --  0.0.0.0/0            0.0.0.0/0
tun0_masq  all  --  0.0.0.0/0            0.0.0.0/0
Chain ppp0_masq (1 references)
target     prot opt source               destination
SNAT       all  --  10.0.0.0/8           0.0.0.0/0            to:82.71.24.87
SNAT       all  --  169.254.0.0/16       0.0.0.0/0            to:82.71.24.87
SNAT       all  --  172.16.0.0/12        0.0.0.0/0            to:82.71.24.87
SNAT       all  --  192.168.0.0/16       0.0.0.0/0            to:82.71.24.87
Chain tun0_masq (1 references)
target     prot opt source               destination
SNAT       all  --  10.0.0.0/8           0.0.0.0/0            to:10.254.252.18
SNAT       all  --  127.0.0.0/8          0.0.0.0/0            to:10.254.252.18

But the same result:

$ sudo tcpdump -ni tun0 port 80 or tcp port 3128
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
16:25:12.024613 IP 82.71.24.87.39176 > 91.189.92.201.80: Flags [S], seq
1971087366, win 14520, options [mss 1452,sackOK,TS val 168165226 ecr
0,nop,wscale 4], length 0


The ''iptrace'' seems to show that SNAT isn''t
happening:

$ clear && tail -n 0 -f /var/log/kern.log | tee /tmp/nf_Trace.log |
egrep ''(91\.189\.92).*(UID=13|(SPT|DPT)=80|0x50)''
Mar 17 16:36:45 jeeves kernel: [673654.416999] TRACE: raw:OUTPUT:policy:3 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000
Mar 17 16:36:45 jeeves kernel: [673654.417032] TRACE: mangle:OUTPUT:rule:2 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000
Mar 17 16:36:45 jeeves kernel: [673654.417054] TRACE: mangle:tcout:rule:1 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000
Mar 17 16:36:45 jeeves kernel: [673654.417075] TRACE: mangle:tcout:return:2 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417090] TRACE: mangle:OUTPUT:policy:3 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417126] TRACE: nat:OUTPUT:policy:1 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417150] TRACE: filter:OUTPUT:rule:1 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417191] TRACE: filter:fw2net:rule:6 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417204] TRACE: mangle:POSTROUTING:rule:1
IN= OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417224] TRACE: mangle:tcpost:return:1 IN=
OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417242] TRACE:
mangle:POSTROUTING:policy:2 IN= OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201
LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417254] TRACE: nat:POSTROUTING:rule:3 IN=
OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417269] TRACE: nat:tun0_masq:return:3 IN=
OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:45 jeeves kernel: [673654.417281] TRACE: nat:POSTROUTING:policy:4
IN= OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33201 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A4AC0000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:46 jeeves kernel: [673655.416051] TRACE: raw:OUTPUT:policy:3 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000
Mar 17 16:36:46 jeeves kernel: [673655.416074] TRACE: mangle:OUTPUT:rule:2 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000
Mar 17 16:36:46 jeeves kernel: [673655.416088] TRACE: mangle:tcout:rule:1 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000
Mar 17 16:36:46 jeeves kernel: [673655.416100] TRACE: mangle:tcout:return:2 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:46 jeeves kernel: [673655.416112] TRACE: mangle:OUTPUT:policy:3 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:46 jeeves kernel: [673655.416131] TRACE: filter:OUTPUT:rule:1 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:46 jeeves kernel: [673655.416165] TRACE: filter:fw2net:rule:6 IN=
OUT=ppp0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:46 jeeves kernel: [673655.416177] TRACE: mangle:POSTROUTING:rule:1
IN= OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:46 jeeves kernel: [673655.416190] TRACE: mangle:tcpost:return:1 IN=
OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000 MARK=0x50
Mar 17 16:36:46 jeeves kernel: [673655.416202] TRACE:
mangle:POSTROUTING:policy:2 IN= OUT=tun0 SRC=82.71.24.87 DST=91.189.92.201
LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=33202 DF PROTO=TCP SPT=39245 DPT=80
SEQ=974071069 ACK=0 WINDOW=14520 RES=0x00 SYN URGP=0 OPT
(020405AC0402080A0A08A5A60000000001030304) UID=1000 GID=1000 MARK=0x50
^C

Note: UID=13 is the ''proxy'' user that squid3 runs as.

At one point late yesterday I had a configuration that was returning replies via
tun0 although the processes on Jeeves were not seeing them, but I''ve
not been able to rediscover that config today.
That had the correct source IP (10.254.252.18). Tiredness and confusion
don''t help!

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

Tom Eastep

2013-Mar-17 18:33 UTC

head link

Re: Alternate route for HTTP traffic?

On 3/17/13 9:46 AM, "TJ" <shorewall@iam.tj> wrote:
>
>$ sudo tcpdump -ni tun0 port 80 or tcp port 3128
>listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
>16:25:12.024613 IP 82.71.24.87.39176 > 91.189.92.201.80: Flags [S], seq
>1971087366, win 14520, options [mss 1452,sackOK,TS val 168165226 ecr
>0,nop,wscale 4], length 0
>16:25:13.023137 IP 82.71.24.87.39176 > 91.189.92.201.80: Flags [S], seq
>1971087366, win 14520, options [mss 1452,sackOK,TS val 168165476 ecr
>0,nop,wscale 4], length 0
>
>Notice the source IP on tun0 is the IP of ppp0. This was why I added the
>MASQ entry:
>
>/etc/shorewall/masq
>#INTERFACE		SOURCE		ADDRESS		PROTO	PORT(S)	IPSEC	MARK
>ppp0	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16	82.71.24.87
>#ppp1	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16	82.71.24.87
>eth9	br0
>tun0    10.0.0.0/8,127.0.0.0/8 10.254.252.18
>#tun0    br0
>
That masq entry certainly won''t change the SOURCE ip from 82.71.24.87
to
10.254.252.18. It only works for 10.0.0.0/8 in the SOURCE (you can''t
NAT
127.0.0.0/8 so that part of the rule is just extra noise).
>-Tom
You do not need a parachute to skydive. You only need a parachute to
skydive twice.





------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

2013-Mar-17 21:15 UTC

head link

Re: Alternate route for HTTP traffic?

On 17/03/13 18:33, Tom Eastep wrote:>> /etc/shorewall/masq
>> #INTERFACE		SOURCE		ADDRESS		PROTO	PORT(S)	IPSEC	MARK
>> ppp0	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16	82.71.24.87
>> #ppp1	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16
82.71.24.87
>> eth9	br0
>> tun0    10.0.0.0/8,127.0.0.0/8 10.254.252.18
>> #tun0    br0
>>
> 
> That masq entry certainly won''t change the SOURCE ip from
82.71.24.87 to
> 10.254.252.18. It only works for 10.0.0.0/8 in the SOURCE (you
can''t NAT
> 127.0.0.0/8 so that part of the rule is just extra noise).
Of course! I think I added 127.0.0.0/8 in desperation when testing connections
from local processes on Jeeves.

The problem seems to be something to do with packet marking - I tried it
manually for a client on the LAN and it works fine:

$ ip route add default via 10.254.252.17 dev tun0 table VPN
$ ip rule add from 10.254.251.50 lookup VPN prio 10001
$ echo 0 > /proc/sys/net/ipv4/conf/tun0/rp_filter

But if I try marking HTTP packers it fails when using:

$ ip rule del from 10.254.251.50 lookup VPN prio 10001
$ ip rule add fwmark 0x50/0xffffffff lookup VPN
$ iptables -t mangle -A tcfor -p tcp --dport 80 -j MARK --set-xmark
0x50/0xffffffff

Which - apart from using table ''tcfor'' instead of
''tcout'' - is the same as the Shorewall rule. The Shorewall
rule remains, too:

$ sudo iptables -t mangle -S
[sudo] password for tj:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N tcfor
-N tcin
-N tcout
-N tcpost
-N tcpre
-A PREROUTING -j tcpre
-A INPUT -j tcin
-A FORWARD -j MARK --set-xmark 0x0/0xff
-A FORWARD -j tcfor
-A OUTPUT -j tcout
-A POSTROUTING -j tcpost
-A tcfor -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x50/0xffffffff
-A tcout -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x50/0xffffffff

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

Tom Eastep

2013-Mar-17 21:26 UTC

head link

Re: Alternate route for HTTP traffic?

On 3/17/13 2:15 PM, "TJ" <shorewall@iam.tj> wrote:
>On 17/03/13 18:33, Tom Eastep wrote:
>>> /etc/shorewall/masq
>>> #INTERFACE		SOURCE		ADDRESS		PROTO	PORT(S)	IPSEC	MARK
>>> ppp0	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16
82.71.24.87
>>> 
>>>#ppp1	10.0.0.0/8,169.254.0.0/16,172.16.0.0/12,192.168.0.0/16
82.71.24.87
>>> eth9	br0
>>> tun0    10.0.0.0/8,127.0.0.0/8 10.254.252.18
>>> #tun0    br0
>>>
>> 
>> That masq entry certainly won''t change the SOURCE ip from
82.71.24.87 to
>> 10.254.252.18. It only works for 10.0.0.0/8 in the SOURCE (you
can''t NAT
>> 127.0.0.0/8 so that part of the rule is just extra noise).
>
>Of course! I think I added 127.0.0.0/8 in desperation when testing
>connections from local processes on Jeeves.
>
>The problem seems to be something to do with packet marking - I tried it
>manually for a client on the LAN and it works fine:
>
>$ ip route add default via 10.254.252.17 dev tun0 table VPN
>$ ip rule add from 10.254.251.50 lookup VPN prio 10001
>$ echo 0 > /proc/sys/net/ipv4/conf/tun0/rp_filter
>
>But if I try marking HTTP packers it fails when using:
>
>$ ip rule del from 10.254.251.50 lookup VPN prio 10001
>$ ip rule add fwmark 0x50/0xffffffff lookup VPN
>$ iptables -t mangle -A tcfor -p tcp --dport 80 -j MARK --set-xmark
>0x50/0xffffffff
>
>Which - apart from using table ''tcfor'' instead of
''tcout'' - is the same
>as the Shorewall rule. The Shorewall rule remains, too:
>
>$ sudo iptables -t mangle -S
>[sudo] password for tj:
>-P PREROUTING ACCEPT
>-P INPUT ACCEPT
>-P FORWARD ACCEPT
>-P OUTPUT ACCEPT
>-P POSTROUTING ACCEPT
>-N tcfor
>-N tcin
>-N tcout
>-N tcpost
>-N tcpre
>-A PREROUTING -j tcpre
>-A INPUT -j tcin
>-A FORWARD -j MARK --set-xmark 0x0/0xff
>-A FORWARD -j tcfor
>-A OUTPUT -j tcout
>-A POSTROUTING -j tcpost
>-A tcfor -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x50/0xffffffff
>-A tcout -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x50/0xffffffff
You must do your marking in the PREROUTING chain.

-Tom
You do not need a parachute to skydive. You only need a parachute to
skydive twice.





------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_mar

Shorewall users - Mar 2013 - Alternate route for HTTP traffic?

Alternate route for HTTP traffic?

Re: Alternate route for HTTP traffic?

Re: Alternate route for HTTP traffic?

Re: Alternate route for HTTP traffic?

Re: Alternate route for HTTP traffic?

Re: Alternate route for HTTP traffic?