freebsd stable - Feb 2014 - IPFW fwd not working after upgrade from 9.2 to 10.0

I have been using IPFW FWD to do per-interface routing on a VM instance. The
default gateway is on interface vtnet0, but there is a second interface, vtnet1,
on a different network with its own public IP address. The second network has
its own gateway, which I'd like to use for responses to connections coming
on on vtnet1. Under 9.2, the below worked fine:

fwd ${GW2} ip from ${PUBIP2} to not table(120) out via vtnet0

Table 120 contains all the local networks for which I don't want the rule to
apply.

I updated the VM to 10.0-RELEASE, with no changes to the IPFW rules or network
configuration. The forwarding to the secondary router no longer works. Traffic
comes in on ${PUBIP2} fine, and the counter for the IPFW rule increments, but no
packets are actually sent out vtnet1. Instead, it's trying to do a weird ARP
query:


# tcpdump -n -p -i vtnet1
...
16:46:33.146324 IP ${OUTSIDE_IP}.55063 > ${PUBIP2}.22: Flags [S], seq
2242981455, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1978614336
ecr 0,sackOK,eol], length 0
16:46:33.146372 ARP, Request who-has ${GW1} tell ${PUBIP2}, length 28

If I try to SSH from an outside IP to the public IP on vtnet1, a response never
goes out either interface (vtnet0 or vtnet1). Instead, an ARP query is going out
(on vtnet1) looking for the default gateway IP, which is only reachable on
vtnet0.

On the off chance this is not a bug, is there a better way I should be doing
per-interface routing under FreeBSD 10? If it is a bug, can anyone suggest what
might be going on here and how to track it down further?

Thanks,

JN

compile a kernel with more than the default 2 FIB tables (16 for example), and

setfib 0 route add default $GATEWAY_A
setfib 1 route add default $GATEWAY_B
setfib 2 route add default $GATEWAY_C

[ ... ]

ipfw table 1 add $NET_LAN               0
ipfw table 1 add $NET_VOIP              2
ipfw table 1 add $NET_VPN               0
ipfw table 1 add $NET_WIFI              0
ipfw table 1 add $NET_GUEST             1
ipfw table 1 add $NET_SECURITY          0
ipfw table 1 add $NET_COMMON            1
ipfw table 1 add $NET_FINANCE           1
ipfw table 1 add $NET_CORE              2
ipfw table 1 add $NET_EVENT             0

[ ... ]

ipfw add 00500 setfib tablearg ip from table\(1\) to any in lookup src-ip 1

On 06.02.2014 04:08, John Nielsen wrote:
> I have been using IPFW FWD to do per-interface routing on a VM
> instance. The default gateway is on interface vtnet0, but there is a
> second interface, vtnet1, on a different network with its own public
> IP address. The second network has its own gateway, which I'd like to
> use for responses to connections coming on on vtnet1. Under 9.2, the
> below worked fine:
Hi,

you can apply this patch:
http://svnweb.freebsd.org/base?view=revision&revision=260702

-- 
WBR, Andrey V. Elsukov

On Thu, 06 Feb 2014 01:08:24 +0100, John Nielsen <lists at jnielsen.net>  
wrote:
> I have been using IPFW FWD to do per-interface routing on a VM instance.  
> The default gateway is on interface vtnet0, but there is a second  
> interface, vtnet1, on a different network with its own public IP  
> address. The second network has its own gateway, which I'd like to use
> for responses to connections coming on on vtnet1. Under 9.2, the below  
> worked fine:
>
> fwd ${GW2} ip from ${PUBIP2} to not table(120) out via vtnet0
>
> Table 120 contains all the local networks for which I don't want the  
> rule to apply.
>
> I updated the VM to 10.0-RELEASE, with no changes to the IPFW rules or  
> network configuration. The forwarding to the secondary router no longer  
> works. Traffic comes in on ${PUBIP2} fine, and the counter for the IPFW  
> rule increments, but no packets are actually sent out vtnet1. Instead,  
> it's trying to do a weird ARP query:
>
>
> # tcpdump -n -p -i vtnet1
> ...
> 16:46:33.146324 IP ${OUTSIDE_IP}.55063 > ${PUBIP2}.22: Flags [S], seq  
> 2242981455, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val  
> 1978614336 ecr 0,sackOK,eol], length 0
> 16:46:33.146372 ARP, Request who-has ${GW1} tell ${PUBIP2}, length 28
>
> If I try to SSH from an outside IP to the public IP on vtnet1, a  
> response never goes out either interface (vtnet0 or vtnet1). Instead, an  
> ARP query is going out (on vtnet1) looking for the default gateway IP,  
> which is only reachable on vtnet0.
>
> On the off chance this is not a bug, is there a better way I should be  
> doing per-interface routing under FreeBSD 10? If it is a bug, can anyone  
> suggest what might be going on here and how to track it down further?
>
> Thanks,
>
> JN
The errata of FreeBSD 10.0 mentions ipfw fwd.
http://www.freebsd.org/releases/10.0R/errata.html

Ronald.