LARTC - Oct 2007 - One machine, two net feeds, outbound route selection

Hello, world!

  Sorry to subscribe to the list only to immediately ask a question,
but this one''s got me scratching my head and I can''t find the
answer
in the archives, the HOWTO, or on the web.  Maybe I''m just asking the
wrong question.

  Anyway, our Internet gateway is a Linux box.  We''ve got two Internet
feeds, a fast one from a crummy provider, and a slower one from a good
provider.  The goal is to control which feed gets used on a
per-service basis.  For the most part, I''ve been pretty successful at
this, and it''s worked well.  Score one for the good guys.

  However, I''m having trouble trying to get Sendmail -- which is
running on the gateway box itself -- to use the good provider when it
tries to deliver mail.  (If we use the crummy provider, too many
others reject us as a likely spammer.)

  I found plenty of docs that tell me what to do if Sendmail is
running on a different box -- just select packets using iptables as
they come in, fwmark them, and then use a routing rule to put them
into the appropriate routing table.  This is conceptually easy,
because the packets would already have well-defined characteristics.

  The problem is that Sendmail is running on the gateway itself.  I
don''t know how to tell the system what to select.  The packets
won''t
have IP address or interface info yet, because they haven''t gone
through the kernel router yet.  Right?  And once they''ve gone through
the kernel router, it''s too late to try and pick the route
they''ll
use.  Right?  Catch-22?

  I can''t bind Sendmail''s outgoing SMTP client mailer to a
specific
interface, because it has to be able to forward mail on to inside
systems, too.

  I can think of all sorts of possible combinations of iptables
options I might try (table, chain, interface, TCP port, etc.), but
there are literally hundreds of permutations.  Trial-and-error doesn''t
seem like a good way to do this.  Especially since it''s a production
box.  I tried a few ideas and got nowhere useful.

  Does someone here already know the answer?

Environment:
- CentOS 5
- kernel 2.6.18-8.1.10.el5
- iptables 1.3.5-1.2.1
- iproute 2.6.18-4.el5
- Both feeds connect to the gateway with plain old IP-over-Ethernet
- Static IP addresses for both feeds
- LAN is NAT''ed and using a 10/8 private subnet
- Successfully using multiple routing tables, iptables, and/or
interface-bindings to select route/feed for most services

  advTHANKSance!

-- Ben

Ben Scott wrote:
> 
> <snip>
>   I can''t bind Sendmail''s outgoing SMTP client mailer to
a specific
> interface, because it has to be able to forward mail on to inside
> systems, too.
> 
Of course you can. Remember that the kernel knows about both networks - 
the internal and external ones. Once you bind to the external IP, a 
packet destined to the inside would still be routed correctly, without 
regard to the source IP.

On 10/25/07, Peter Rabbitson <rabbit+list@rabbit.us>
wrote:
>> I can''t bind Sendmail''s outgoing SMTP client mailer
to a specific
>> interface, because it has to be able to forward mail on to inside
>> systems, too.
>
> Of course you can. Remember that the kernel knows about both networks ...
  Hmmm.  You''re right, of course.  And even better, it worked when I
tried it.  ;)  Thanks!

  Now, for the sake of knowledge, let us say that a piece of needed
software didn''t have an option to bind to a specific interface.  Would
it be possible to control the outgoing route/interface anyway, by
using iptables or some other mechanism external to the software?  For
example, what if Sendmail didn''t have a ClientPortOptions directive
(heh, I know, use Postfix, but work with me here)?

  And remember: This is for something running on the gateway, not
forwarding packets received from another machine.  :)

  (No, I don''t presently have such a piece of software, but
I''d like to know.)

  Thanks again, either way!

-- Ben

Ben Scott wrote:
>   Now, for the sake of knowledge, let us say that a piece of needed
> software didn''t have an option to bind to a specific interface. 
Would
> it be possible to control the outgoing route/interface anyway, by
> using iptables or some other mechanism external to the software?  For
> example, what if Sendmail didn''t have a ClientPortOptions
directive
> (heh, I know, use Postfix, but work with me here)?
> 
Unfortunately not easy without doing local NAT (from the local interface 
to another local interface). The problem lies in how the kernel sends 
packets without a specified source. I wrote an explanation some time 
ago: http://mailman.ds9a.nl/pipermail/lartc/2007q2/020941.html

On 10/25/07, Peter Rabbitson <rabbit+list@rabbit.us>
wrote:
> Unfortunately not easy without doing local NAT (from the local interface
> to another local interface).
  I thought that might be the case.  I even started to write a rule
about how the NAT might work... but then I ran into brain pain trying
to figure out how, because I didn''t know when the packets get what
address/interface info assigned to them, and I didn''t know how SNAT
would interact with the routing tables.  Normally, I do SNAT in the
POSTROUTING chain, but by then the routing rules have already run,
right?  So the packet would still be bound for the wrong interface,
even if the source address is translated.  No?

  In other words, let''s say $DEF_ADDR is the IP address of the
interface that is going to be picked by the default routing table, but
I really want the packets to go out the $ALT_ADDR interface.  So I try
this:

iptables -t nat -A POSTROUTING -s $DEF_ADDR -p tcp --dport smtp -j
SNAT --to $ALT_ADDR

  But the whole point of changing the source address/interface is to
influence which routing rules match, and those have already been
applied by the time the packet transverses the POSTROUTING chain,
right?  In any event, that didn''t work.

  So then I thought, well, maybe I can do SNAT in the PREROUTING chain
for this?  But in that case, the kernel won''t have assigned it an
address yet, right?  So there''s nothing to SNAT.  And I can''t
do "-s
0/0" because that actually means "match all packets", right?

  So then I thought, well, maybe I can mark the packet in the OUTPUT
chain of the mangle table, and match that in the routing rules, and
*also* match that in the POSTROUTING chain:

iptables -t mangle -A OUTPUT -s $DEF_ADDR -p tcp --dport smtp -j MARK
--set-mark 42
ip rule add fwmark 42 table 42
iptables -t nat -A POSTROUTING -m mark --mark 42 -j SNAT --to-source $ALT_ADDR

  I think I tried that and it didn''t work either.  It was getting late
and my maintenance window was closing and my brain hurt.

  If this is just one of those "you can''t do that"
situations, I''m
willing to accept that answer.  But if there is a way, I''d like to
know what it is.  :)

-- Ben

On Thu, Oct 25, 2007 at 02:00:14PM -0400, Ben Scott
wrote:
> On 10/25/07, Peter Rabbitson <rabbit+list@rabbit.us> wrote:
> > Unfortunately not easy without doing local NAT (from the local
interface
> > to another local interface).
Can you use marking, mark the packet in the mangle table, us iptables to select 
the which packets and then use ip rules fw mark -> routing table (sorry about
the syntax)


> 
>   I thought that might be the case.  I even started to write a rule
> about how the NAT might work... but then I ran into brain pain trying
> to figure out how, because I didn''t know when the packets get what
> address/interface info assigned to them, and I didn''t know how
SNAT
> would interact with the routing tables.  Normally, I do SNAT in the
> POSTROUTING chain, but by then the routing rules have already run,
> right?  So the packet would still be bound for the wrong interface,
> even if the source address is translated.  No?
> 
>   In other words, let''s say $DEF_ADDR is the IP address of the
> interface that is going to be picked by the default routing table, but
> I really want the packets to go out the $ALT_ADDR interface.  So I try
> this:
> 
> iptables -t nat -A POSTROUTING -s $DEF_ADDR -p tcp --dport smtp -j
> SNAT --to $ALT_ADDR
> 
>   But the whole point of changing the source address/interface is to
> influence which routing rules match, and those have already been
> applied by the time the packet transverses the POSTROUTING chain,
> right?  In any event, that didn''t work.
> 
>   So then I thought, well, maybe I can do SNAT in the PREROUTING chain
> for this?  But in that case, the kernel won''t have assigned it an
> address yet, right?  So there''s nothing to SNAT.  And I
can''t do "-s
> 0/0" because that actually means "match all packets", right?
> 
>   So then I thought, well, maybe I can mark the packet in the OUTPUT
> chain of the mangle table, and match that in the routing rules, and
> *also* match that in the POSTROUTING chain:
> 
> iptables -t mangle -A OUTPUT -s $DEF_ADDR -p tcp --dport smtp -j MARK
> --set-mark 42
> ip rule add fwmark 42 table 42
> iptables -t nat -A POSTROUTING -m mark --mark 42 -j SNAT --to-source
$ALT_ADDR
> 
>   I think I tried that and it didn''t work either.  It was getting
late
> and my maintenance window was closing and my brain hurt.
> 
>   If this is just one of those "you can''t do that"
situations, I''m
> willing to accept that answer.  But if there is a way, I''d like to
> know what it is.  :)
> 
> -- Ben
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> 

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Ben Scott wrote:
> On 10/25/07, Peter Rabbitson <rabbit+list@rabbit.us> wrote:
>> Unfortunately not easy without doing local NAT (from the local
interface
>> to another local interface).
> 
>   I thought that might be the case.  I even started to write a rule
> about how the NAT might work... but then I ran into brain pain trying
> to figure out how, because I didn''t know when the packets get what
> address/interface info assigned to them, and I didn''t know how
SNAT
> would interact with the routing tables.  Normally, I do SNAT in the
> POSTROUTING chain, but by then the routing rules have already run,
> right?  So the packet would still be bound for the wrong interface,
> even if the source address is translated.  No?
> 
I was not thorough enough. The NAT is necessary in order to make the 
packet come back through the link/interface you want (because as I noted 
previously you do not have control over the choice of a source address). 
Once this is out of the way the only problem is how to make an already 
routed packet to leave via a different interface. One way to do this is 
the ROUTE target: 
http://netfilter.org/documentation/HOWTO//netfilter-extensions-HOWTO-4.html#ss4.5
There might also be other ways to do this, but I never investigated, as 
this is a mostly theoretical exercise.

Peter