chris@monsterserve.net
2008-Sep-12 09:53 UTC
Strange Multi-homed Traceroute/Ping failure for some IPs on some routes
Dear all, If I do cause offence by posting OT here I apologise in advance, I am however desperate for help and after posting on other forums without any ideas I know many networking experts will see this here and hope they can enlighten me. I will gladly donate some PayPal money to the person who can help. I have a leased line on 83.111.160.6 (/30 subnet, gw is 83.111.160.5), and they route an additional block 83.111.196.56/29 (83.111.196.57 to 83.111.196.62 useable) over the link. I have a Debian box, and the routed block IP?s are setup as aliases. I have setup the box to accept ssh and ping for each IP alias. /etc/network/interfaces auto eth3 iface eth3 inet static address 83.111.160.6 netmask 255.255.255.252 up ip addr add 83.111.196.57/29 brd 83.111.196.63 dev eth3 label eth3:0 up ip addr add 83.111.196.58/29 brd 83.111.196.63 dev eth3 label eth3:1 up ip addr add 83.111.196.59/29 brd 83.111.196.63 dev eth3 label eth3:2 up ip addr add 83.111.196.60/29 brd 83.111.196.63 dev eth3 label eth3:3 up ip addr add 83.111.196.61/29 brd 83.111.196.63 dev eth3 label eth3:4 up ip addr add 83.111.196.62/29 brd 83.111.196.63 dev eth3 label eth3:5 And here is a snippet from the Shorewall rules config: Ping/ACCEPT net $FW Ping/ACCEPT net $FW:83.111.196.57 Ping/ACCEPT net $FW:83.111.196.58 Ping/ACCEPT net $FW:83.111.196.59 Ping/ACCEPT net $FW:83.111.196.60 Ping/ACCEPT net $FW:83.111.196.61 Ping/ACCEPT net $FW:83.111.196.62 I can ping 83.111.160.6 fine everywhere from any host on the internet, but I can?t ping all of the routed IP addresses from external hosts. Some IPs work and some don?t. With Shorewall set to reject icmp and ssh, some of the connection attempts to IPs that work are listed as being dropped, but traffic doesn?t even seem to hit the others at all and no entries are made. This is a multi-ISP configuration with two providers, however I am 99.999% sure this isn''t a Shorewall issue at all for reasons I will explain below. Siteuptime.com shows some of its sites able to connect to IPs within the routed block and others unable (US sites ok, London failed). I also have a number of traceroutes from network-tools.com which I attach to this mail. Some of the IPs within the routed block don?t seem to be hitting the firewall at all and are routed off into space (from reject logs or lack activity on the ISP ethernet to fibre converter data transfer LEDs). This isn''t a ping issue either, SSH, SMTP etc do not work on the broken IPs. Now here is the strangest thing, I have a couple of servers in the UK and they have dual interfaces. On one of the boxes, ping fails from one interface, but works when ping is initiated on another, to the same destination host. **** TRACE FROM MY UK SERVERS **** [root@stripe ~]# ping 83.111.196.59 -I 85.234.115.64 PING 83.111.196.59 (83.111.196.59) from 85.234.115.64 : 56(84) bytes of data. --- 83.111.196.59 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3002ms [root@stripe ~]# ping 83.111.196.60 -I 85.234.115.64 PING 83.111.196.60 (83.111.196.60) from 85.234.115.64 : 56(84) bytes of data. 64 bytes from 83.111.196.60: icmp_seq=1 ttl=56 time=159 ms 64 bytes from 83.111.196.60: icmp_seq=2 ttl=56 time=159 ms --- 83.111.196.60 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 159.024/159.221/159.418/0.197 ms [root@stripe ~]# ping 83.111.196.61 -I 85.234.115.64 PING 83.111.196.61 (83.111.196.61) from 85.234.115.64 : 56(84) bytes of data. 64 bytes from 83.111.196.61: icmp_seq=1 ttl=54 time=148 ms 64 bytes from 83.111.196.61: icmp_seq=2 ttl=54 time=148 ms --- 83.111.196.61 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 148.549/148.615/148.681/0.066 ms [root@stripe ~]# ping 83.111.196.62 -I 85.234.115.64 PING 83.111.196.62 (83.111.196.62) from 85.234.115.64 : 56(84) bytes of data. --- 83.111.196.62 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2000ms [root@stripe ~]# ping 83.111.196.59 -I 85.234.115.115 PING 83.111.196.59 (83.111.196.59) from 85.234.115.115 : 56(84) bytes of data. 64 bytes from 83.111.196.59: icmp_seq=1 ttl=57 time=149 ms 64 bytes from 83.111.196.59: icmp_seq=2 ttl=57 time=158 ms --- 83.111.196.59 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 149.200/153.985/158.771/4.801 ms [root@stripe ~]# ping 83.111.196.60 -I 85.234.115.115 PING 83.111.196.60 (83.111.196.60) from 85.234.115.115 : 56(84) bytes of data. --- 83.111.196.60 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 2999ms [root@stripe ~]# ping 83.111.196.61 -I 85.234.115.115 PING 83.111.196.61 (83.111.196.61) from 85.234.115.115 : 56(84) bytes of data. --- 83.111.196.61 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2000ms [root@stripe ~]# ping 83.111.196.62 -I 85.234.115.115 PING 83.111.196.62 (83.111.196.62) from 85.234.115.115 : 56(84) bytes of data. 64 bytes from 83.111.196.62: icmp_seq=1 ttl=56 time=168 ms 64 bytes from 83.111.196.62: icmp_seq=2 ttl=56 time=178 ms --- 83.111.196.62 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 168.441/173.542/178.644/5.118 ms Sending from Stripe using interface 85.234.115.64, my IPs 83.111.196.60 and 83.111.196.61 are ok, but .59 and .62 fail. Strangely, sending from Stripe using interface 85.234.115.115 the opposite is true, .59 and .62 are ok but .60 and .61 fail! My other servers fail connecting to .59 and .62. I would greatly appreciate any pointers on this issue, I have already contacted my ISP and they fail to believe that something is wrong. It would be most appreciated if others could let me know if they can contact the above IP addresses. I will gladly donate some money via PayPal to get this resolved ASAP. Kind regards, Chris ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
Jerry Vonau
2008-Sep-12 23:49 UTC
Re: Strange Multi-homed Traceroute/Ping failure for some IPs on some routes
chris@monsterserve.net wrote:> Dear all, > > If I do cause offence by posting OT here I apologise in advance, I am > however desperate for help and after posting on other forums without any > ideas I know many networking experts will see this here and hope they > can enlighten me. I will gladly donate some PayPal money to the person > who can help. > > I have a leased line on 83.111.160.6 (/30 subnet, gw is 83.111.160.5), > and they route an additional block 83.111.196.56/29 (83.111.196.57 to > 83.111.196.62 useable) over the link. > > I have a Debian box, and the routed block IP?s are setup as aliases. I > have setup the box to accept ssh and ping for each IP alias. > > /etc/network/interfaces auto eth3 > iface eth3 inet static > address 83.111.160.6 > netmask 255.255.255.252 > up ip addr add 83.111.196.57/29 brd 83.111.196.63 dev eth3 label eth3:0 > up ip addr add 83.111.196.58/29 brd 83.111.196.63 dev eth3 label eth3:1 > up ip addr add 83.111.196.59/29 brd 83.111.196.63 dev eth3 label eth3:2 > up ip addr add 83.111.196.60/29 brd 83.111.196.63 dev eth3 label eth3:3 > up ip addr add 83.111.196.61/29 brd 83.111.196.63 dev eth3 label eth3:4 > up ip addr add 83.111.196.62/29 brd 83.111.196.63 dev eth3 label eth3:5 > > And here is a snippet from the Shorewall rules config: > > Ping/ACCEPT net $FW > Ping/ACCEPT net $FW:83.111.196.57 > Ping/ACCEPT net $FW:83.111.196.58 > Ping/ACCEPT net $FW:83.111.196.59 > Ping/ACCEPT net $FW:83.111.196.60 > Ping/ACCEPT net $FW:83.111.196.61 > Ping/ACCEPT net $FW:83.111.196.62 > > I can ping 83.111.160.6 fine everywhere from any host on the internet, > but I can?t ping all of the routed IP addresses from external hosts. > Some IPs work and some don?t. With Shorewall set to reject icmp and ssh, > some of the connection attempts to IPs that work are listed as being > dropped, but traffic doesn?t even seem to hit the others at all and no > entries are made. This is a multi-ISP configuration with two providers, > however I am 99.999% sure this isn''t a Shorewall issue at all for > reasons I will explain below.<snip>> Sending from Stripe using interface 85.234.115.64, my IPs 83.111.196.60 > and 83.111.196.61 are ok, but .59 and .62 fail. Strangely, sending from > Stripe using interface 85.234.115.115 the opposite is true, .59 and .62 > are ok but .60 and .61 fail! My other servers fail connecting to .59 and > .62. > > I would greatly appreciate any pointers on this issue, I have already > contacted my ISP and they fail to believe that something is wrong. It > would be most appreciated if others could let me know if they can > contact the above IP addresses. I will gladly donate some money via > PayPal to get this resolved ASAP. > > Kind regards, > > Chris >Quoting Tom " For connection problems, we need the output of ''shorewall dump'' collected as described at: http://www.shorewall.net/support.htm#Guidelines" You have an interesting problem, but from where I am without the dump, I have no clue on your setup. FWIW, 58, 59, 62 work from here while 60, 61 fail to respond to ping. Jerry ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
chris@monsterserve.net
2008-Sep-13 08:10 UTC
Re: Strange Multi-homed Traceroute/Ping failure for some IPs on some routes
> Quoting Tom " For connection problems, we need the output of ''shorewall > dump'' collected as described at: > http://www.shorewall.net/support.htm#Guidelines" > > You have an interesting problem, but from where I am without the dump, I > have no clue on your setup. FWIW, 58, 59, 62 work from here while 60, 61 > fail to respond to ping. > > JerryHi Jerry, thanks for the reply. I didnt include the dump as I am not pointing my finger at Shorewall! However you are right, I should have included and have done so now. Please find attatched. SSH *should* also work on those hosts, but I have found where one of my boxes fails connecting to some hosts, my second box is the *exact* opposite. Those hosts that don''t respond to box1, do to box2, and vice-versa. Please feel free to run tcptraceroutes/traceroutes to the IPs, SSH etc, the config is set to respond to ICMP and SSH on all IPs. As mentioned, from activity lights the packets dont even seem to be routed to me at all. Regards Chris ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
chris@monsterserve.net
2008-Sep-13 09:20 UTC
Re: Strange Multi-homed Traceroute/Ping failure for some IPs on some routes
> SSH *should* also work on those hosts, but I have found where one of > my boxes fails connecting to some hosts, my second box is the *exact* > opposite. Those hosts that don''t respond to box1, do to box2, and > vice-versa.Dear all, I''ve just finished a phone call to my ISP for the 3rd time today and got lucky as this person was able to see the fault, apparently their router 83.111.206.182 has two interfaces with the same IP address... not a good day out. Good news is they have finally confirmed the fault is with them and will now take 24/48 hours to fix. Sorry to have posted here regarding a non-shorewall issue (as i suspected), I did so as this the best list of networking experts that I know of and appreciate your support. Thank you. Many thanks and regards, Chris ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer''s challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/