Hi, Can I beg some help handling the some corner cases with the goal to achieve very rigid routing via specific interfaces To recap the goal: - I have a router connected to several internet connections, lets call them eth0, wlan1, ppp0. Some of the internet connections may have substantial cost, eg satellite - The use case will be end user, mobile, and we will assume that it''s fair game to add/remove interfaces while in use, eg wifi networks could drop, 3g router could drop - Desire that individual local IPs will be routed out through specific internet connections. User will agree to use a specific interface in advance and should not use another internet interface until they have confirmed agreement (web interface) - Currently using several ipsets RT1,RT2, etc to mark which provider an IP should be using. This applies a packet mark in tcrules that matches the provider number allocated in providers file. - Local network is via either eth0 or wlan0, these are part of a bridge br0 (no requirement to filter between eth0/wlan0) What has become clear in the last few hours, is that I don''t appear to understand linux routing as well as I thought... In particular, how should I setup shorewall if I want to *require* that a connection marked in tcrules:prerouting with a specific packet mark (say 0x10000 ie equiv of high_routemarks/wide_tc) will either exit through a specific interface, say ppp0. Simply marking with the correct provider number routes through that provider most of the time, but appears vulnerable to external processes breaking the routing table? For example, my understanding, given this (abbreviated) routing output which results when ppp changes its IP and butchers my routes: Routing Rules 0: from all lookup local 10000: from all fwmark 0x10000/0xff0000 lookup peth0 10007: from all fwmark 0x80000/0xff0000 lookup pppp0 10011: from all fwmark 0xc0000/0xff0000 lookup pppp10 20000: from 192.168.105.71 lookup peth0 20000: from 10.64.64.74 lookup pppp10 20000: from 10.64.64.64 lookup pppp0 32766: from all lookup main 32767: from all lookup default Table pppp0: 192.168.111.0/24 dev br0 proto kernel scope link src 192.168.111.1 Then my mark will cause the table pppp0 to be tried, however, because it doesn''t match anything outbound, then the main/default tables will be checked next and packets may exit via some other interface (remember the spec *requires* that this doesn''t happen) So I *think* I need to add additional fallback routes for each mark which will null route anything which escapes from the prescribed routing table? How might I best do this through shorewall features? I guess the "routes" file is an option, but this appears to alter the existing pppp0 table, which raises the question of how to deal with: If something is in say RT4, but that associated provider interface isn''t currently up, then we won''t have a rule in the route database at all for the relevant mark... (eg consider packets for eth1, ie fwmark 0x20000 which isn''t mentioned at all in the output above) Therefore I would expect that in the situation above, we would fallback to some other route (probably main) and packets could go out of some unexpected interface. I think again the solution is to add a second much lower priority "null" route which is pointed to by the same route marks. This way if either the desired table is missing, or simply fails to capture our packet then the second table will capture the same mark and null route it So if that were indeed the best solution, what might be the best way to implement it within shorewall? I have 15 fixed providers which map to all possible providers we could ever have, although not all will be around or up at any given moment (but we know them all the interfaces in advance, just not the ip addresses of the interfaces) I think I will need to achieve something like: 0: from all lookup local 10000: from all fwmark 0x10000/0xff0000 lookup peth0 10007: from all fwmark 0x80000/0xff0000 lookup pppp0 10011: from all fwmark 0xc0000/0xff0000 lookup pppp10 32000: from all fwmark 0x10000/0xff0000 lookup peth0_kill_it 32007: from all fwmark 0x80000/0xff0000 lookup pppp0_kill_it 32011: from all fwmark 0xc0000/0xff0000 lookup pppp10_kill_it Thanks for any suggestions on how to proceed? Ed W ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
Tom Eastep
2012-Apr-22 16:22 UTC
Re: How to rigidly lock down routing to specific interfaces
On 04/21/2012 04:15 PM, Ed W wrote:> Hi, Can I beg some help handling the some corner cases with the goal to > achieve very rigid routing via specific interfaces > > To recap the goal: > > - I have a router connected to several internet connections, lets call > them eth0, wlan1, ppp0. Some of the internet connections may have > substantial cost, eg satellite > - The use case will be end user, mobile, and we will assume that it''s > fair game to add/remove interfaces while in use, eg wifi networks could > drop, 3g router could drop > - Desire that individual local IPs will be routed out through specific > internet connections. User will agree to use a specific interface in > advance and should not use another internet interface until they have > confirmed agreement (web interface) > - Currently using several ipsets RT1,RT2, etc to mark which provider an > IP should be using. This applies a packet mark in tcrules that matches > the provider number allocated in providers file. > - Local network is via either eth0 or wlan0, these are part of a bridge > br0 (no requirement to filter between eth0/wlan0) > > What has become clear in the last few hours, is that I don''t appear to > understand linux routing as well as I thought... > > In particular, how should I setup shorewall if I want to *require* that > a connection marked in tcrules:prerouting with a specific packet mark > (say 0x10000 ie equiv of high_routemarks/wide_tc) will either exit > through a specific interface, say ppp0. Simply marking with the correct > provider number routes through that provider most of the time, but > appears vulnerable to external processes breaking the routing table?Reject traffic going out of an interface if it doesn''t have the correct mark. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
On 22/04/2012 00:15, Ed W wrote:> I think I will need to achieve something like: > > 0: from all lookup local > 10000: from all fwmark 0x10000/0xff0000 lookup peth0 > 10007: from all fwmark 0x80000/0xff0000 lookup pppp0 > 10011: from all fwmark 0xc0000/0xff0000 lookup pppp10 > 32000: from all fwmark 0x10000/0xff0000 lookup peth0_kill_it > 32007: from all fwmark 0x80000/0xff0000 lookup pppp0_kill_it > 32011: from all fwmark 0xc0000/0xff0000 lookup pppp10_kill_it >I''m struggling with this - would be grateful for input I tried adding to init: ip rule add blackhole from all fwmark 0x10000/0xff0000 priority 32000 ip rule add blackhole from all fwmark 0x20000/0xff0000 priority 32000 ..etc... These match my provider marks, so I had thought that the routing match would try something like: 10000: from all fwmark 0x10000/0xff0000 lookup peth0 then 32000: from all fwmark 0x10000/0xff0000 blackhole However, either I''m testing incorrectly, or that isn''t how the routing policy table actually works? Additionally the shorewall enable/disable restore_routing script is taking down this route, so I would need to investigate better integration anyway. Any offers on how to make it so that it''s "route via provider X or drop the packet"? Thanks Ed W ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
On 22/04/2012 17:22, Tom Eastep wrote:> Reject traffic going out of an interface if it doesn''t have the correct > mark.Seems too obvious... I''m just trying now. I really want to write this: DROP:info any !net:eth0 - - - - - - 0x10000/0xF0000 But I can''t negate a destination right? I need to pass the packet if it has no routemark or a specific routemark. I think I can do it if I create an action with a bunch of continues and a drop at the end - just going to test that now. However, is there a similarly much more obvious or neater solution? Thanks Ed W ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
Tom Eastep
2012-Apr-22 17:17 UTC
Re: How to rigidly lock down routing to specific interfaces
Sent from my iPad On Apr 22, 2012, at 10:05 AM, Ed W <lists@wildgooses.com> wrote:> On 22/04/2012 17:22, Tom Eastep wrote: >> Reject traffic going out of an interface if it doesn''t have the correct >> mark. > > Seems too obvious... > > I''m just trying now. I really want to write this: > > DROP:info any !net:eth0 - - > - - - - 0x10000/0xF0000 > > But I can''t negate a destination right?Invert the test! Tom ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
On 22/04/2012 18:17, Tom Eastep wrote:> > Sent from my iPad > > On Apr 22, 2012, at 10:05 AM, Ed W<lists@wildgooses.com> wrote: > >> On 22/04/2012 17:22, Tom Eastep wrote: >>> Reject traffic going out of an interface if it doesn''t have the correct >>> mark. >> Seems too obvious... >> >> I''m just trying now. I really want to write this: >> >> DROP:info any !net:eth0 - - >> - - - - 0x10000/0xF0000 >> >> But I can''t negate a destination right? > Invert the test! >I don''t see how? Note, some connections will have no mark at all (so they route through the main table). If I literally invert the test above I will drop connections with mark 0x20000 and with mark 0 (but we want to keep these) I am using an action with two continues and a drop, something like: CONTINUE:info - - - - - - - - 0x0/0xFF0000 CONTINUE:info - - - - - - - - - DROP:info I think this implements the correct behaviour (obviously setting src/dest/mark in the action call) Am I missing something simpler? Many thanks Ed W ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
Tom Eastep
2012-Apr-22 23:27 UTC
Re: How to rigidly lock down routing to specific interfaces
On 4/22/12 11:39 AM, Ed W wrote:> On 22/04/2012 18:17, Tom Eastep wrote: >> >> Sent from my iPad >> >> On Apr 22, 2012, at 10:05 AM, Ed W<lists@wildgooses.com> wrote: >> >>> On 22/04/2012 17:22, Tom Eastep wrote: >>>> Reject traffic going out of an interface if it doesn''t have the correct >>>> mark. >>> Seems too obvious... >>> >>> I''m just trying now. I really want to write this: >>> >>> DROP:info any !net:eth0 - - >>> - - - - 0x10000/0xF0000 >>> >>> But I can''t negate a destination right? >> Invert the test! >> > > I don''t see how? Note, some connections will have no mark at all (so > they route through the main table). If I literally invert the test > above I will drop connections with mark 0x20000 and with mark 0 (but we > want to keep these) > > I am using an action with two continues and a drop, something like: > > CONTINUE:info - - - - > - - - - 0x0/0xFF0000 > CONTINUE:info - - - - > - - - - - > DROP:info > > I think this implements the correct behaviour (obviously setting > src/dest/mark in the action call)I don''t see how those particular rules can work but I agree that if you want to pass A and B on to the following rules and drop the rest then you need to use an Action with at least two rules. The first is a CONTINUE if A. The second is a DROP if not B. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2