Stephen Clark wrote:> Robert Noland wrote: >> On Thu, 2008-11-13 at 07:48 -0500, Stephen Clark wrote: >>> Julian Elischer wrote: >>>> Stephen Clark wrote: >>>>> Julian Elischer wrote: >>>>>> you will need to define the setup and question better. >>>> thanks.. cleaning it up a bit more... >>>> >>>> 10.0.129.1 FreeBSD workstation >>>> ^ >>>> | >>>> | ethernet >>>> | >>>> v >>>> 10.0.128.1 Freebsd FW "A" >>>> ^ >>>> | >>>> | gre / ipsec >>>> | >>>> v >>>> 192.168.3.1 FreeBSD FW "B" >>>> ^ >>>> | >>>> | ethernet >>>> | >>>> v >>>> 192.168.3.86 linux workstation >>>> >>>>> $ sudo traceroute 192.168.3.86 >>>>> traceroute to 192.168.3.86 (192.168.3.86), 64 hops max, 40 byte >>>>> packets >>>>> 1 HQFirewallRS.com (10.0.128.1) 0.575 ms 0.423 ms 0.173 ms >>>>> 2 * * * >>>>> 3 192.168.3.86 (192.168.3.86) 47.972 ms 45.174 ms 49.968 ms >>>>> >>>>> No response from the FreeBSD "B" box. >>>>> >>>>> When I do a tcpdump on "B" of the gre interface I see UDP packets >>>>> with a TTL of 1 but no ICMP response packets being sent back. >>>>> If I do the traceroute from the linux workstation 192.168.3.86 I get >>>>> similar results - I don't see a response from the FreeBSD "A" box. >>>> could you try using just GRE encasulation? >>>> (i.e. turn off IPSEC for now) >>>> >>>> I think that is much more likely to be where the problem is.. >>>> >>>> >>> I'll have to set this up to test it. >> >> The ttl exceeded is triggered from one of two places. Either >> netinet/ip_fastfwd.c if fast_forwarding is enabled or in >> netinet/ip_input.c. Look for the code relating to IPTTLDEC. This isn't >> your problem though... If ttl were not being decremented, the packet >> would just be forwarded on to the next hop (IP_STEALTH), which would >> just make the firewalls invisible. The fact that you are seeing * * * >> indicates that you are not receiving the ttl exceeded message for the >> packet sent with that particular ttl. I still think that the issue you >> are seeing is that one way or another the generated ICMP response isn't >> making it back onto the tunnel. Either via security policy, firewall or >> routing. > Your right, when I do a tcpdump on the gre interface I see the udp > packet come > in with a ttl=1 but I don't see a response icmp packet. I have tested > this with > all the firewalls disabled to make sure the icmp packet was not being > blocked. > I just ran another test and did tcpdump on all the other interfaces to > make sure > the icmp's were not being misrouted, it seems they are not being > generated for some reason. Also just using gre's without the underlying > ipsec tunnels seems to > work properly. >> >> robert. >> >>> What code in the FreeBSD kernel is responsible for generating the >>> response ICMP dest unreachable message? >>> > >Another data point I had been using option FILTER_GIF I tried a kernel without that option and it behaved the same. Steve -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson)
Stephen Clark
2008-Nov-14 06:31 UTC
FreeBSD 6.3 ipsec and traceroute doesn't work as good as Linux -why?
10.0.129.1 FreeBSD workstation ^ | | ethernet | v 10.0.128.1 Freebsd FW "A" ^ | | ipsec | v 192.168.2.1 Linux FW "B" ^ | | ethernet | v 192.168.2.20 linux workstation from 192.168.2.20 Linux<->ipsec<->FreeBSD traceroute -I 10.0.129.1 traceroute to 10.0.129.1 (10.0.129.1), 30 hops max, 60 byte packets 1 192.168.2.1 (192.168.2.1) 0.434 ms 0.425 ms 0.423 ms 2 * * * 3 sclark (10.0.129.1) 42.418 ms 42.419 ms 42.727 ms traceroute -I 10.0.128.1 traceroute to 10.0.128.1 (10.0.128.1), 30 hops max, 60 byte packets 1 192.168.2.1 (192.168.2.1) 0.398 ms 0.504 ms 0.505 ms 2 10.0.128.1 (10.0.128.1) 36.066 ms 36.052 ms 37.800 ms traceroute 10.0.129.1 traceroute to 10.0.129.1 (10.0.129.1), 30 hops max, 60 byte packets 1 192.168.2.1 (192.168.2.1) 0.484 ms 0.464 ms 0.447 ms 2 * * * 3 sclark (10.0.129.1) 41.406 ms 41.391 ms 47.812 ms traceroute 10.0.128.1 traceroute to 10.0.128.1 (10.0.128.1), 30 hops max, 60 byte packets 1 (192.168.2.1) 0.473 ms 0.444 ms 0.427 ms 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * *^C from 10.0.129.1 FreeBSD<->ipsec<->Linux sudo traceroute 192.168.2.20 traceroute to 192.168.2.20 (192.168.2.20), 64 hops max, 40 byte packets 1 HQFirewallRS.com (10.0.128.1) 0.761 ms 2.551 ms 4.017 ms 2 * * * 3 192.168.2.20 (192.168.2.20) 19.956 ms 27.425 ms 27.487 ms sclark:~ $ sudo traceroute 192.168.2.1 traceroute to 192.168.2.1 (192.168.2.1), 64 hops max, 40 byte packets 1 HQFirewallRS.com (10.0.128.1) 8.069 ms 2.952 ms 4.050 ms 2 home (192.168.2.1) 26.338 ms 22.132 ms 24.233 ms sclark:~ $ sudo traceroute -I 192.168.2.20 traceroute to 192.168.2.20 (192.168.2.20), 64 hops max, 60 byte packets 1 HQFirewallRS.com (10.0.128.1) 0.714 ms 0.806 ms 0.221 ms 2 home (192.168.2.1) 25.260 ms 25.312 ms 25.868 ms 3 192.168.2.20 (192.168.2.20) 36.477 ms 24.828 ms 24.903 ms sclark:~ $ sudo traceroute -I 192.168.2.1 traceroute to 192.168.2.1 (192.168.2.1), 64 hops max, 60 byte packets 1 HQFirewallRS.com (10.0.128.1) 2.219 ms 1.889 ms 4.491 ms 2 home (192.168.2.1) 26.172 ms 25.706 ms 24.981 ms tracerouteing to Linux never just gives a * * *, * * *, * * *, etc -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson)
Julian Elischer wrote:> Stephen Clark wrote: >> Stephen Clark wrote: > >>>>>> >>>>>> 10.0.129.1 FreeBSD workstation >>>>>> ^ >>>>>> | >>>>>> | ethernet >>>>>> | >>>>>> v >>>>>> 10.0.128.1 Freebsd FW "A" >>>>>> ^ >>>>>> | >>>>>> | gre / ipsec >>>>>> | >>>>>> v >>>>>> 192.168.3.1 FreeBSD FW "B" >>>>>> ^ >>>>>> | >>>>>> | ethernet >>>>>> | >>>>>> v >>>>>> 192.168.3.86 linux workstation >>>>>> > >>> Also just using gre's without the underlying ipsec tunnels seems to >>> work properly. > > > This is the crux of the matter. > IPSEC happens INSIDE the IP stack. The IP stack is responsible for > the ICMP generation so it is much more likely that there is an > interaction there. > > Now is there an IPSEC rule to make sure that the ICMP packet can get > back? It could b ehtat in teh IP stack there is some confusion as to > whether the return packet should be encrypted or not and it might get > dropped. > > the code involved is in /sys/netinet and /sys/netipsec but you'll > probably regret looking in there ;-) > > > >>> >>> >> Another data point I had been using option FILTER_GIF I tried a kernel >> without that option and it behaved the same. >> >> Steve >> >I agree I put a diag in ip_input.c if (ip->ip_ttl <= IPTTLDEC) { icmp_error(m, ICMP_TIMXCEED, ICMP_TIMXCEED_INTRANS, 0, 0); return; and sure enough it is calling icmp_error, but I think it can't figure out how to route the packet back. I been looking at my SPD to see if I can make some adjustment to the policy that would help. -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson)
Stephen Clark wrote:> Stephen Clark wrote:>>>>> >>>>> 10.0.129.1 FreeBSD workstation >>>>> ^ >>>>> | >>>>> | ethernet >>>>> | >>>>> v >>>>> 10.0.128.1 Freebsd FW "A" >>>>> ^ >>>>> | >>>>> | gre / ipsec >>>>> | >>>>> v >>>>> 192.168.3.1 FreeBSD FW "B" >>>>> ^ >>>>> | >>>>> | ethernet >>>>> | >>>>> v >>>>> 192.168.3.86 linux workstation >>>>>>> Also just using gre's without the >> underlying ipsec tunnels seems to >> work properly.This is the crux of the matter. IPSEC happens INSIDE the IP stack. The IP stack is responsible for the ICMP generation so it is much more likely that there is an interaction there. Now is there an IPSEC rule to make sure that the ICMP packet can get back? It could b ehtat in teh IP stack there is some confusion as to whether the return packet should be encrypted or not and it might get dropped. the code involved is in /sys/netinet and /sys/netipsec but you'll probably regret looking in there ;-)>> >> > Another data point I had been using option FILTER_GIF I tried a kernel > without that option and it behaved the same. > > Steve >