asterisk users - Jun 2010 - Slightly OT: trying to mangle packets from Asterisk for a multiple ISP setup (reward)

Hi,

 

Reward offered: 50$ (paypal), and I am sure this is a ridiculous thing I
have missing.

 

My goal: On a 2 NIC Asterisk box, to send packets that came in Asterisk on
NIC1 back to NIC 1, and NIC2 back to NIC 2. (basically, send them back the
same way they came from).

 

I have been doing what was recommended to me and mangling packets left and
right.  I have reached a point where I am stuck, and can`t imagine why this
last little step isnt working.

 

As you know, Asterisk sends all packets "from" the default IP (in my
case,
NIC 1 IP).  So connections to NIC 1 work fine, to NIC 2 they don`t.  I
therefore put in some routing rules to help me.  Some example, a phone
(remote PBX setup) coming in from 65.77.77.77 works fine because of these
(slightly obfuscated by changing IPs shown) routing rules:

 

ip rule show: 

0:      from all lookup 255

32759:  from all fwmark 0x64 lookup ISP2 (<----- this is key to my issue)

32760:  from all to 65.77.77.77 lookup ISP2

32766:  from all lookup main

32767:  from all lookup default

 

ip route show table ISP2:

default via 22.22.22.22 dev eth1  src 22.22.22.21

 

BUT I can't reliably know where the phones come from (long story), or what
IP they use (ISP1 or ISP2) to connect to me. So instead, I have done this
with iptables:

 

MARK       all  --  anywhere             STRING match "22.22.22.21"
ALGO
name bm TO 65535MARK set 0x64

 

Basically saying to mark all packets that have the string
"22.22.22.21" in
it's SIP content (meaning they came in on NIC2 originally because the phone
registered to 22.22.22.21) with mark 0x64.  And that works fine, because
another rule that LOGs these marked packets is logging them correctly.

 

Because of my above routing rules, packets going out marked with 0x64 or
those going to 65.77.77.77 should go to the same ip route (route table
ISP2).  Mysteriously, I see that packets going to 65.77.77.77 (using
wireshark) are correctly mangled as coming from 22.22.22.21, but not those
marked with 0x64.  Those still go out through the default routing table.

 

What the heck am I missing? I believe I have done my homework, but there is
no more door left to bang my head on.

 

Mike

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.digium.com/pipermail/asterisk-users/attachments/20100601/c1181f8b/attachment.htm

The simple fix your missing is to simply put your NICs on different 
layer 3 segments.

In a configuration wherein multiple adapters are bound to the same layer 
3 network (subnet), most IP stacks will only send traffic out the first 
NIC to bind to that network. I've seen this many times in data centers 
where server guys configure the NICs in the same subnet for "load 
balancing".

With some IP stacks, even arp responses are only sent from the first NIC 
to bind to the subnet. This creates problems at layer 2, especially in 
large traffic volume situations. Since a technically correct arp 
response is sent - the mac address in the response is for the NIC to 
which the arp request was sent - from the first NIC to bind to the 
subnet, the switch(es) never actually learn where the MAC addresses for 
the other NICs are located. This creates a situation where all traffic 
for these NICs is broadcast to all ports in the VLAN.

Joe


On 6/1/2010 10:27 AM, Mike wrote:
> Hi,
>
> Reward offered: 50$ (paypal), and I am sure this is a ridiculous thing I
> have missing.
>
> My goal: On a 2 NIC Asterisk box, to send packets that came in Asterisk
> on NIC1 back to NIC 1, and NIC2 back to NIC 2. (basically, send them
> back the same way they came from).
>
> I have been doing what was recommended to me and mangling packets left
> and right. I have reached a point where I am stuck, and can`t imagine
> why this last little step isnt working.
>
> As you know, Asterisk sends all packets "from" the default IP (in
my
> case, NIC 1 IP). So connections to NIC 1 work fine, to NIC 2 they don`t.
> I therefore put in some routing rules to help me. Some example, a phone
> (remote PBX setup) coming in from 65.77.77.77 works fine because of
> these (slightly obfuscated by changing IPs shown) routing rules:
>
> ip rule show:
>
> 0: from all lookup 255
>
> 32759: from all fwmark 0x64 lookup ISP2 (<----- this is key to my issue)
>
> 32760: from all to 65.77.77.77 lookup ISP2
>
> 32766: from all lookup main
>
> 32767: from all lookup default
>
> ip route show table ISP2:
>
> default via 22.22.22.22 dev eth1 src 22.22.22.21
>
> BUT I can't reliably know where the phones come from (long story), or
> what IP they use (ISP1 or ISP2) to connect to me. So instead, I have
> done this with iptables:
>
> MARK all -- anywhere STRING match "22.22.22.21" ALGO name bm TO
> 65535MARK set 0x64
>
> Basically saying to mark all packets that have the string
"22.22.22.21"
> in it's SIP content (meaning they came in on NIC2 originally because
the
> phone registered to 22.22.22.21) with mark 0x64. And that works fine,
> because another rule that LOGs these marked packets is logging them
> correctly.
>
> Because of my above routing rules, packets going out marked with 0x64 or
> those going to 65.77.77.77 should go to the same ip route (route table
> ISP2). Mysteriously, I see that packets going to 65.77.77.77 (using
> wireshark) are correctly mangled as coming from 22.22.22.21, but not
> those marked with 0x64. Those still go out through the default routing
> table.
>
> What the heck am I missing? I believe I have done my homework, but there
> is no more door left to bang my head on.
>
> Mike
>

On Tue, 1 Jun 2010, Mike wrote:
> Thanks Joe,
> 
> They are on different segments.  Those two NICs share nothing but the
> server.
> 
> But more to the point, it doesn't explain why a simple routing rule
matching
> the destination by IP address works wonderfully, but not one where I match
a
> fwmark that has been set (apparently correctly according to my logging)
with
> iptables.
> 
> Mike
Is this the same thread about having multiple ISP's, and you have external 
phones hitting the asterisk server on one or the other, and you want the 
replies to come back on the same segment they came in on?

I think IP mangling is making it way too complicated.  I suggested you front 
each segment with a NAT router.  Unless you are expecting very heavy traffic 
volumes, even a cheapo $50 router from Officemax should suffice.

Create two internal subnets - one for each interface. Set each router in 
"DMZ" mode, so it will send all inbound traffic to the asterisk server
on
the appropriate interface.  The asterisk server will then think that the 
connection is coming from a locally attached phone, and it will respond 
out the correct NIC, using the correct IP.  The NAT router will send it 
back out the right Internet connection using the appropriate public IP.

j

Mike,

I don't see the outgoing rule in your description. How does your system
knows how to change the outgoing address to match the originating nic ?

i.e.

0: from all lookup local
32766: from all lookup main
32767: from all lookup default

ip rule add from 10.10.70.38/32 table admin
ip rule add to 10.10.70.38/32 table admin

0: from all lookup local
32764: from all to 10.10.70.38 lookup admin
32765: from 10.10.70.38 lookup admin  <======================32766: from all
lookup main
32767: from all lookup default

Jimmy
> -----Original Message-----
> From: list at virtutel.ca
> Sent: Tue, 01 Jun 2010 10:58:52 -0400
> To: asterisk-users at lists.digium.com
> Subject: Re: [asterisk-users] Slightly OT: trying to mangle packets from
> Asterisk for a multiple ISP setup (reward)
> 
> Thanks Joe,
> 
> They are on different segments.  Those two NICs share nothing but the
> server.
> 
> But more to the point, it doesn't explain why a simple routing rule
> matching
> the destination by IP address works wonderfully, but not one where I
> match a
> fwmark that has been set (apparently correctly according to my logging)
> with
> iptables.
> 
> Mike
> 
>> -----Original Message-----
>> From: asterisk-users-bounces at lists.digium.com
[mailto:asterisk-users-
>> bounces at lists.digium.com] On Behalf Of Joe Freeman
>> Sent: Tuesday, June 01, 2010 10:56
>> To: asterisk-users at lists.digium.com
>> Subject: Re: [asterisk-users] Slightly OT: trying to mangle packets
from
>> Asterisk for a multiple ISP setup (reward)
>> 
>> The simple fix your missing is to simply put your NICs on different
>> layer 3 segments.
>> 
>> In a configuration wherein multiple adapters are bound to the same
layer
>> 3 network (subnet), most IP stacks will only send traffic out the first
>> NIC to bind to that network. I've seen this many times in data
centers
>> where server guys configure the NICs in the same subnet for "load
>> balancing".
>> 
>> With some IP stacks, even arp responses are only sent from the first
NIC
>> to bind to the subnet. This creates problems at layer 2, especially in
>> large traffic volume situations. Since a technically correct arp
>> response is sent - the mac address in the response is for the NIC to
>> which the arp request was sent - from the first NIC to bind to the
>> subnet, the switch(es) never actually learn where the MAC addresses for
>> the other NICs are located. This creates a situation where all traffic
>> for these NICs is broadcast to all ports in the VLAN.
>> 
>> Joe
>> 
>> 
>> On 6/1/2010 10:27 AM, Mike wrote:
>>> Hi,
>>> 
>>> Reward offered: 50$ (paypal), and I am sure this is a ridiculous
thing
>>> I
>>> have missing.
>>> 
>>> My goal: On a 2 NIC Asterisk box, to send packets that came in
Asterisk
>>> on NIC1 back to NIC 1, and NIC2 back to NIC 2. (basically, send
them
>>> back the same way they came from).
>>> 
>>> I have been doing what was recommended to me and mangling packets
left
>>> and right. I have reached a point where I am stuck, and can`t
imagine
>>> why this last little step isnt working.
>>> 
>>> As you know, Asterisk sends all packets "from" the
default IP (in my
>>> case, NIC 1 IP). So connections to NIC 1 work fine, to NIC 2 they
>>> don`t.
>>> I therefore put in some routing rules to help me. Some example, a
phone
>>> (remote PBX setup) coming in from 65.77.77.77 works fine because of
>>> these (slightly obfuscated by changing IPs shown) routing rules:
>>> 
>>> ip rule show:
>>> 
>>> 0: from all lookup 255
>>> 
>>> 32759: from all fwmark 0x64 lookup ISP2 (<----- this is key to
my
>>> issue)
>>> 
>>> 32760: from all to 65.77.77.77 lookup ISP2
>>> 
>>> 32766: from all lookup main
>>> 
>>> 32767: from all lookup default
>>> 
>>> ip route show table ISP2:
>>> 
>>> default via 22.22.22.22 dev eth1 src 22.22.22.21
>>> 
>>> BUT I can't reliably know where the phones come from (long
story), or
>>> what IP they use (ISP1 or ISP2) to connect to me. So instead, I
have
>>> done this with iptables:
>>> 
>>> MARK all -- anywhere STRING match "22.22.22.21" ALGO name
bm TO
>>> 65535MARK set 0x64
>>> 
>>> Basically saying to mark all packets that have the string
"22.22.22.21"
>>> in it's SIP content (meaning they came in on NIC2 originally
because
>>> the
>>> phone registered to 22.22.22.21) with mark 0x64. And that works
fine,
>>> because another rule that LOGs these marked packets is logging them
>>> correctly.
>>> 
>>> Because of my above routing rules, packets going out marked with
0x64
>>> or
>>> those going to 65.77.77.77 should go to the same ip route (route
table
>>> ISP2). Mysteriously, I see that packets going to 65.77.77.77 (using
>>> wireshark) are correctly mangled as coming from 22.22.22.21, but
not
>>> those marked with 0x64. Those still go out through the default
routing
>>> table.
>>> 
>>> What the heck am I missing? I believe I have done my homework, but
>>> there
>>> is no more door left to bang my head on.
>>> 
>>> Mike
>>> 
>> 
>> 
>> --
>> _____________________________________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>> New to Asterisk? Join us for a live introductory webinar every Thurs:
>>                http://www.asterisk.org/hello
>> 
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>>    http://lists.digium.com/mailman/listinfo/asterisk-users
> 
> 
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> New to Asterisk? Join us for a live introductory webinar every Thurs:
>                http://www.asterisk.org/hello
> 
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
____________________________________________________________
FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop!
Check it out at http://www.inbox.com/earth

On Tue, 1 Jun 2010, Mike wrote:
> Hi,
>
> Reward offered: 50$ (paypal), and I am sure this is a ridiculous thing I
> have missing.
Not looking for reward, but did you see/try the stuff I posted about a few 
days ago - I've not seen you make menion of it, so I'm now wondering if
my
post got through to the list..

This was the main bit:

      http://lartc.org/howto/lartc.rpdb.multiple-links.html

Gordon