I'm using iproute2-ss010824 and a 2.4.20 kernel.

Quite a subtle issue here, so I can imagine it has not been spotted before.

The setup:

I have set up a machine as a gateway. The 'external' interface uses a
dummy IP address (eg. 10.0.0.2) and the internal side is a normal
address. (it's more complex in real life using Zebra and such, but this is
the basic setup that shows the problem)

To make sure I can start connections to the outside world from this
machine I must make sure that the source address of the packets is from
the internal interface with the 'real' address range, or otherwise packets
won't come back as they would originate from the 10.0.0.x range.

I could not use iptables to translate/NAT the source address (other issues
there), but the 'scope' option in iproute2 seemed the best answer and indeed
it seems to work fine.

I have now set up the internal interface with a 'global' scope and
the external with a 'link' scope:

1: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 02:02:05:62:00:23 brd ff:ff:ff:ff:ff:ff
    inet 17.70.0.1/28 brd 17.70.0.15 scope global eth0
2: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:30:48:27:25:15 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd brd 10.0.0.255 scope link eth1

So according to the docs now it should select the 10.0.0.2 address
for directy connected machines and use the 17.70.0.1 address for 'remote'
destinations.

This basically works fine. When I connect to a directly-connected 10.0.0.xxx
address it indeed chooses the 10.0.0.2 source address (link-address), but when
connecting to a remote host (via a GW learned by Zebra) it uses the 17.70.0.1
source address.

Great! Problem solved! Well.. Almost..

There seems to be one snag: Incorrect ARP source address.

If there is no ARP entry for the gateway yet (no traffic has gone out, routes
learned from another BGP peer) and I try to reach a remote address immediately
then the ARP request that goes out on the 10.0.0.0 network for the
correct gateway does *not* contain the 10.0.0.2 source address, but
instead 17.70.0.1.

Well.. That obviously does not work as this IP address does not occur on
this LAN and as a reasult most other routers will (correctly) ignore this.

If I try to connect to the correct gateway on a 10.0.0.x adress directly
then it does work as it will use the correct 10.0.0.2 source for it's
ARP request.

It seems that the ARP code also chooses the 'global' scope address for the
ARP request, while it should really always choose the 'link' address
of this interface as the source of the broadcast.

I have now temporarily fixed this by either adding some static ARP entries
or ARP-table filtering using iptables, but I feel that's only a temporary
measure.

Have I overlooked something in my setup or should I start poking in the
kernel ARP code?

Thanx!

							Bye, Arno.

Hello,

On Mon, 17 Mar 2003, Arno Griffioen wrote:

> There seems to be one snag: Incorrect ARP source address.
>
> If there is no ARP entry for the gateway yet (no traffic has gone out,
routes
> learned from another BGP peer) and I try to reach a remote address
immediately
> then the ARP request that goes out on the 10.0.0.0 network for the
> correct gateway does *not* contain the 10.0.0.2 source address, but
> instead 17.70.0.1.
>
> Well.. That obviously does not work as this IP address does not occur on
> this LAN and as a reasult most other routers will (correctly) ignore this.

	No, the router should answer this request because it knows
where 17.70.0.1 is: it is on the LAN or at least reachable via
gateway on this LAN. So, what is the good reason to ignore ARP
requests with src 17.70.0.1? Linux ARP follows the routing and will
reply in this case (when used in place of the router).

> If I try to connect to the correct gateway on a 10.0.0.x adress directly
> then it does work as it will use the correct 10.0.0.2 source for it's
> ARP request.

	Correct

> It seems that the ARP code also chooses the 'global' scope address for the
> ARP request, while it should really always choose the 'link' address
> of this interface as the source of the broadcast.

	No, the check is: is the source address in the IP packet local?
If yes, use it as src for ARP - classic case where ARP must be accepted
in router if the IP packets are accepted: if you have the right to send
IP packets with src=17.70.0.1 is there a reason to ignore ARP with same
src? In your case there is no reason, of course, there are other cases
where tuning the ARP protocol is needed.

> I have now temporarily fixed this by either adding some static ARP entries
> or ARP-table filtering using iptables, but I feel that's only a temporary
> measure.

	You are in the right direction, there are no many solutions
to handle such case. The problem comes only if "router decides not
to accept ARP from valid source IP from valid input device".

> Have I overlooked something in my setup or should I start poking in the
> kernel ARP code?

	Take a look at arp_solicit()

Regards

--
Julian Anastasov <ja@ssi.bg>