"A Tale of TTL Troubles"
I was hired to implement VPN for a subnet. The owner has a /27 at his
home site, and he wanted to have the machines there answering BOTH on
those IP addresses and some addresses at a remote colocation provider.
Make sense? Not to me either. :( I think he''s trying to fool his
customers into thinking he has a physical presence in the colocation
city. But it''s his money, and a very interesting issue.
I set up a simple openvpn peer-mode tunnel. It mostly seems to work,
except return packets back through the VPN are giving me trouble.
Sorry, I''m not allowed to post details, so IP addresses are munged. :(
HOME_SITE_NET=x.x.x.0/27 [ 0-31 ]
HOME_DEF_GATEWAY=x.x.x.1 (I have no access to this router.)
HOME_VPN_ENDPOINT=x.x.x.4, a/k/a y.y.y.38
REMOTE_SITE_NET=y.y.y.0/24
(out of this I was told to bind .38 on the VPN peer and route .39
through .60 via the VPN. Sigh ... CIDR masks would have been nice.)
REMOTE_DEF_GATEWAY=y.y.y.1 (I have no access to this router either.)
REMOTE_VPN_ENDPOINT=y.y.y.37, a/k/a 192.168.255.37
(The latter is only used for the tunnel traffic.)
Routing on HOME_VPN_ENDPOINT :
[root@localhost ~]# ip route list
y.y.y.39 dev eth0 scope link
192.168.255.37 dev tun0 proto kernel scope link src y.y.y.38
y.y.y.40/29 dev eth0 scope link
y.y.y.48/28 dev eth0 scope link
x.x.x.0/27 dev eth0 proto kernel scope link src x.x.x.4
169.254.0.0/16 dev eth0 scope link
default via x.x.x.1 dev eth0
[root@localhost ~]# ip route list table colo
default via 192.168.255.37 dev tun0 src y.y.y.38
[root@localhost ~]# ip rule list
0: from all lookup local
32763: from y.y.y.48/28 lookup colo
32764: from y.y.y.40/29 lookup colo
32765: from y.y.y.39 lookup colo
32766: from all lookup main
32767: from all lookup default
The 3 "dev eth0 scope link" routes are what was needed to cover .39
through .60. It goes over, to .63, oh well.
One of the machines is x.x.x.6 and y.y.y.39, .49, .59, and .60 on the
VPN. It''s my guinea pig, hereinafter called "www". CentOS
release 4.4
(Final) 2.6.9-42.ELsmp on each of www and HOME_VPN_ENDPOINT; I think
REMOTE_VPN_ENDPOINT is similar, but I don''t think it''s the
problem.
[root@www ~]# ip route list
y.y.y.0/27 dev eth0 proto kernel scope link src y.y.y.6
169.254.0.0/16 dev eth0 scope link
default via y.y.y.1 dev eth0
[root@www ~]# ip rule list
0: from all lookup local
32765: from x.x.x.32/27 lookup vpn
32766: from all lookup main
32767: from all lookup default
[root@www ~]# ip route list table vpn
default via y.y.y.4 dev eth0
I tested the routing by using my own IP in an iptables -j LOG rule,
followed by an nmap -sP of the range.
[root@localhost ~]# iptables-save
# Generated by iptables-save v1.2.11 on Mon Dec 11 09:10:27 2006
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:LogVpn - [0:0]
-A FORWARD -d y.y.y.32/255.255.255.224 -j LogVpn
-A FORWARD -s y.y.y.32/255.255.255.224 -j LogVpn
-A LogVpn -s my.IP.add.ress -j LOG --log-prefix "5Vpn IN: "
-A LogVpn -d my.IP.add.ress -j LOG --log-prefix "5Vpn OUT: "
COMMIT
# Completed on Mon Dec 11 09:10:27 2006
Results of the scan: all but y.y.y.39 were up. Logs: um, rather than ask
you to pick through all that, I''ll summarise:
[root@localhost ~]# dmesg | grep ^5Vpn
...
.39 worked perfectly. The ping and HTTP requests came IN=tun0 and
OUT=eth0; the replies came IN=eth0 and OUT=tun0. The inbound ICMP had
TTL=39. The inbound HTTP had TTL=33, a difference of 6.
.49 seemed to have a routing loop, and the "www" host apparently
didn''t
answer the ARP queries. We only got "5Vpn IN:" packets, none
"OUT:".
Each packet passed through 3 times, with ICMP TTL values of 34, 22 and
10, and HTTP TTL values of 26, 14, and 2. Interestingly, each time the
IN and OUT interface was eth0. And coincidentally (?) it''s 13 hops from
me to www''s VPN IP, and 11 hops from www to REMOTE_VPN_ENDPOINT going
through HOME_DEF_GATEWAY.
.59 and .60 both worked, but the return packets must have gone through
HOME_DEF_GATEWAY, because both IN=eth0 and OUT=eth0. for each of the
"IN:" and "OUT:" packets. Inbound TTL values were 38 and 47
for ICMP,
and 34 and 40 for HTTP, on each of .59 and .60 respectively.
Meanwhile, over on www I set up similar log rules, and the results for
this scan pretty well aligned with the logs on HOME_VPN_ENDPOINT. Same
TTLs on inbound as noted above. Nothing was logged for .49 in or out,
again leading me to think that www ignored the ARP queries.
I wait awhile, try again, and I get different results. But never does
everything work right.
This is nuts!! Looks like sometimes www won''t answer ARP for its
locally-bound IP addresses, and the rules on HOME_VPN_ENDPOINT are
being ignored. And how the VPN packets could possibly come in on eth0
is beyond me.
What am I missing here? What can I try?
One idea I had, was possibly to do proxy ARP on the HOME site too. Tell
the HOME machines to use 192.168.255.37 as the default route in their
vpn tables. That takes HOME_VPN_ENDPOINT routing rules out of the
equation.
Trying that now, similar intermittent results. ARRRRRRRRRGH!!
--
Offlist mail to this address is discarded unless
"/dev/rob0" or "not-spam" is in Subject: header
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net''s Techsay panel and you''ll get the chance
to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
/dev/rob0
2006-Dec-13 06:06 UTC
Re: OpenVPN, proxy ARP for an entire subnet (Linux endpoints)
On Monday 11 December 2006 19:16, I wrote:> "A Tale of TTL Troubles" > > I was hired to implement VPN for a subnet. The owner has a /27 at his > home site, and he wanted to have the machines there answering BOTH on > those IP addresses and some addresses at a remote colocation > provider.snip> Sorry, I''m not allowed to post details, so IP addresses are munged. > :(I have more information, and as it concerns my own IP addresses now, this doesn''t require munging.> HOME_SITE_NET=x.x.x.0/27 [ 0-31 ] > HOME_DEF_GATEWAY=x.x.x.1 (I have no access to this router.) > HOME_VPN_ENDPOINT=x.x.x.4, a/k/a y.y.y.38 > > REMOTE_SITE_NET=y.y.y.0/24 > (out of this I was told to bind .38 on the VPN peer and route .39 > through .60 via the VPN. Sigh ... CIDR masks would have been > nice.) > REMOTE_DEF_GATEWAY=y.y.y.1 (I have no access to this router either.) > REMOTE_VPN_ENDPOINT=y.y.y.37, a/k/a 192.168.255.37 > (The latter is only used for the tunnel traffic.)(I omitted OS and kernel for this: it''s CentOS 4.2 / 2.6.9-22.ELsmp.) MyHomeNet=192.168.6.0/24 # Yes, RFC 1918, but that is not significant MyHomeGateway=192.168.6.1 # Slamd64 10.2, 2.6.18.3 from kernel.org MyHomeVPNpeer=192.168.6.4, a/k/a 72.9.234.116 # Slackware 10.0, 2.4.31 # (with VMware patch, but I doubt that matters.) MyRemoteNet=72.9.234.112/29 # .112-.114 are in use, .118 reserved MyRemoteGateway=72.9.234.65 # no access to this machine MyRemoteVPNpeer=72.9.234.112/26, a/k/a 192.168.6.12 (The latter is only used for the tunnel traffic.) # Slackware 10.1, 2.6.10-skas3-v7 (the User-mode Linux patch) That is correct: we have a /29 out of the /26. So the network and broadcast addresses are not a factor.> Routing on HOME_VPN_ENDPOINT : > > [root@localhost ~]# ip route list > y.y.y.39 dev eth0 scope link > 192.168.255.37 dev tun0 proto kernel scope link src y.y.y.38 > y.y.y.40/29 dev eth0 scope link > y.y.y.48/28 dev eth0 scope link > x.x.x.0/27 dev eth0 proto kernel scope link src x.x.x.4 > 169.254.0.0/16 dev eth0 scope link > default via x.x.x.1 dev eth0 > [root@localhost ~]# ip route list table colo > default via 192.168.255.37 dev tun0 src y.y.y.38 > [root@localhost ~]# ip rule list > 0: from all lookup local > 32763: from y.y.y.48/28 lookup colo > 32764: from y.y.y.40/29 lookup colo > 32765: from y.y.y.39 lookup colo > 32766: from all lookup main > 32767: from all lookup default > > The 3 "dev eth0 scope link" routes are what was needed to cover .39 > through .60. It goes over, to .63, oh well.Routing on MyHomeVPNpeer: root@whn:~# ip route list 192.168.6.12 dev tun0 proto kernel scope link src 72.9.234.116 72.9.234.119 dev eth0 scope link 72.9.234.117 dev eth0 scope link 72.9.234.115 dev eth0 scope link 192.168.6.0/24 dev eth0 proto kernel scope link src 192.168.6.4 127.0.0.0/8 dev lo scope link default via 192.168.6.1 dev eth0 metric 1 root@whn:~# ip rule list 0: from all lookup local 32762: from 72.9.234.119 lookup thanks 32763: from 72.9.234.115 lookup thanks 32764: from 72.9.234.117 lookup thanks 32765: from 72.9.234.116 lookup thanks 32766: from all lookup main 32767: from all lookup default root@whn:~# ip route list table thanks default via 192.168.6.12 dev tun0 Of course /proc/sys/net/ipv4/ip_forward is "1". That''s also true on the ones featured in our last episode.> One of the machines is x.x.x.6 and y.y.y.39, .49, .59, and .60 on the > VPN. It''s my guinea pig, hereinafter called "www". CentOS release 4.4 > (Final) 2.6.9-42.ELsmp on each of www and HOME_VPN_ENDPOINT; I think > REMOTE_VPN_ENDPOINT is similar, but I don''t think it''s the problem.MyHomeGateway, the Slamd64 machine, was the volunteer for our sadistic experiments. Here I do not think the fact that it''s the default gateway for MyHomeVPNpeer is significant, because results prove that traffic moves in both directions through the VPN.> [root@www ~]# ip route list > y.y.y.0/27 dev eth0 proto kernel scope link src y.y.y.6 > 169.254.0.0/16 dev eth0 scope link > default via y.y.y.1 dev eth0 > [root@www ~]# ip rule list > 0: from all lookup local > 32765: from x.x.x.32/27 lookup vpn > 32766: from all lookup main > 32767: from all lookup default > [root@www ~]# ip route list table vpn > default via y.y.y.4 dev eth0If you don''t mind I''m again going to munge my real home IP address. That''s not a factor anyway. Routes and addresses on MyHomeGateway: root@miniluv:~# ip addr list 1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop link/ether ce:23:26:b7:ef:0c brd ff:ff:ff:ff:ff:ff 3: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:30:48:56:ed:12 brd ff:ff:ff:ff:ff:ff inet 192.168.6.1/24 brd 192.168.6.255 scope global eth0 inet 72.9.234.117/32 scope global eth0 inet 72.9.234.115/32 scope global eth0 inet 72.9.234.119/32 scope global eth0 valid_lft forever preferred_lft forever 4: eth1: <BROADCAST,MULTICAST,NOTRAILERS,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:80:c8:1b:72:4c brd ff:ff:ff:ff:ff:ff inet my.ip.add.ress/nm brd isp.brd.cast.ip scope global eth1 valid_lft forever preferred_lft forever root@miniluv:~# ip route list 192.168.6.0/24 dev eth0 proto kernel scope link src 192.168.6.1 isp.su.bn.et/nm dev eth1 proto kernel scope link src my.ip.add.ress 127.0.0.0/8 dev lo scope link default via isp.rou.ter.ip dev eth1 root@miniluv:~# ip rule list 0: from all lookup local 32763: from 72.9.234.119 lookup vpn 32764: from 72.9.234.115 lookup vpn 32765: from 72.9.234.117 lookup vpn 32766: from all lookup main 32767: from all lookup default root@miniluv:~# ip route list table vpn default via 192.168.6.4 dev eth0> I tested the routing by using my own IP in an iptables -j LOG rule, > followed by an nmap -sP of the range.This time I did it from a third site, but to get a better picture I added logging in raw/PREROUTING too. This time the logging was at the RemoteVPNpeer rather than the local one, but anyway, it shows everything moving as expected. (LocalVPNpeer, being a 2.4.31 host, doesn''t have iptable_raw.)> [root@localhost ~]# iptables-save > # Generated by iptables-save v1.2.11 on Mon Dec 11 09:10:27 2006 > *filter > > :INPUT ACCEPT [0:0] > :FORWARD ACCEPT [0:0] > :OUTPUT ACCEPT [0:0] > :LogVpn - [0:0] > > -A FORWARD -d y.y.y.32/255.255.255.224 -j LogVpn > -A FORWARD -s y.y.y.32/255.255.255.224 -j LogVpn > -A LogVpn -s my.IP.add.ress -j LOG --log-prefix "5Vpn IN: " > -A LogVpn -d my.IP.add.ress -j LOG --log-prefix "5Vpn OUT: " > COMMIT > # Completed on Mon Dec 11 09:10:27 2006Here I''ll only show the significant parts of iptables-save on RemoteVPNpeer: # Generated by iptables-save v1.2.11 on Wed Dec 13 04:17:29 2006 *raw :PREROUTING ACCEPT [149316:38901352] :OUTPUT ACCEPT [155833:112104141] -A PREROUTING -s 72.9.234.112/29 -d 3rd.par.ty.IP -j LOG --log-prefix "3-raw OUT: " -A PREROUTING -s 3rd.par.ty.IP -d 72.9.234.112/29 -j LOG --log-prefix "3-raw IN: " COMMIT [ ... ] # Generated by iptables-save v1.2.11 on Wed Dec 13 04:17:29 2006 *filter :INPUT DROP [0:0] :FORWARD DROP [0:0] [ ... ] -A INPUT -j Reject -A FORWARD -s 3rd.par.ty.IP -j LogVpn -A FORWARD -d 3rd.par.ty.IP -j LogVpn [ ... ] -A LogVpn -s 3rd.par.ty.IP -d 72.9.234.112/29 -j LOG --log-prefix "3-filter IN: " -A LogVpn -s 72.9.234.112/29 -d 3rd.par.ty.IP -j LOG --log-prefix "3-filter OUT: " -A LogVpn -j ACCEPT [ ... ] So, every packet is logged twice, once in raw and once in filter. And to ensure that all requests are replied, everything from 3rd.par.ty.IP is accepted before any other filter/FORWARD rules.> Results of the scan: all but y.y.y.39 were up. Logs: um, rather than > ask you to pick through all that, I''ll summarise:Results of the scan from 3rd.par.ty.IP of 72.9.234.115-119 showed all but .118 is up. And the logs were pristine: "raw IN:" and "filter IN:" for each ICMP echo and HTTP request in perfect numeric sequence. After those were similarly-arranged "raw OUT:" and "filter OUT:" logs for ICMP and HTTP replies. The IN: packets all came in eth0, the OUT: ones all went out tun0, and the filter ones showed both interfaces properly. What is the difference here? Mainly OS and kernel, I think. Did I miss anything? So, any input here at all? Could this be an obscure bug in the Centos (RHEL) kernels? How to proceed? -- Offlist mail to this address is discarded unless "/dev/rob0" or "not-spam" is in Subject: header