LARTC - May 2005 - Multiple network cards on same subnet problem (arp_filter=1)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

this might seem like a classical problem but I''ve trouble getting this
working correctly:

# ifconfig  ### output stripped down:
eth0      Link encap:Ethernet  HWaddr 00:50:FC:2B:EB:1B
          inet addr:192.168.5.220  Bcast:192.168.5.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr 00:02:44:60:EC:58
          inet addr:192.168.5.221  Bcast:192.168.5.255  Mask:255.255.255.0

eth2      Link encap:Ethernet  HWaddr 00:20:ED:33:2B:6B
          inet addr:192.168.5.222  Bcast:192.168.5.255  Mask:255.255.255.0

# ip route
192.168.5.0/24 dev eth0  proto kernel  scope link  src 192.168.5.220
192.168.5.0/24 dev eth1  proto kernel  scope link  src 192.168.5.221
192.168.5.0/24 dev eth2  proto kernel  scope link  src 192.168.5.222
default via 192.168.5.11 dev eth0

# cat /proc/sys/net/ipv4/conf/all/arp_filter
1
(and just to be sure I set /conf/<every interface>/arp_filter to 1 as
well)

So basically what I want is just a box with 3 network cards with 3
different ips, no bonding/trunking, just simple.. :).

What I get is simply that eth0 is answering (with it''s own HW addr) to
arp requests for all ips (so including .221 and .222). Also, the other
interfaces are not answering (so it''s not like they all answer either,
only eth0 does). The interfaces do receive the request however (when I
look with tcpdump).
The effect is obvious, when I list the arp table on the client machine:
  192.168.5.220         00-50-fc-2b-eb-1b     dynamic
  192.168.5.221         00-50-fc-2b-eb-1b     dynamic
  192.168.5.222         00-50-fc-2b-eb-1b     dynamic
                            ^ that''s all eth0''s hw addr :|

I''ve also tried all /proc/sys/net/ipv4/conf/all/arp_ignore settings
(+flushing arp tables @ client of course), but none of them helped...
I''m not even sure if those options have anything to do with what
I''m
trying to accomplish.

Actually, when adding the hw addr manually (arp -s 192.168.5.221
00-02-44-60-EC-58) pinging doesn''t work either... while all interfaces
receive the request (because naturally the switch didn''t see the arp
request/reply it gets broadcasted) none of them replies.

Some of these are irrelevant, but just to be sure:
- - no iptables rules are present
- - all default iptables actions are at ACCEPT
- - IP Forwarding is disabled
- - I''ve reset my switch
- - I''ve flushed all arp tables on both the server and test client
machine

This is Linux 2.4.30 (w/grsecurity).

Fast switching is off btw, checked that as well.

Did I miss a setting? A kernel compile option? Or is there a problem
with arp_filter in 2.4.*? I hope I''m missing something obvious here ;).

Thanks,

	Bram.

- --
Bram Matthys
Software developer/IT consultant        syzop@vulnscan.org
PGP key:                       www.vulnscan.org/pubkey.asc
PGP fp: 8DD4 437E 9BA8 09AA 0A8D  1811 E1C3 D65F E6ED 2AA2
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (MingW32)

iD8DBQFCdpa04cPWX+btKqIRArOUAJ4iLOSclzAsw7JbFeK6tGq/4jTwxwCfVMqA
VBYs/M0H0VVAQD9AutJ29mA=/L4a
-----END PGP SIGNATURE-----

Hi Bram, 
I have a similar setup, and what I find works for me is adding ip
''rules'' for certain traffic to have specific default routes.
for instance :

# ip rule add from 192.168.5.220 lookup 10
# ip route add table 10 default src 192.168.5.220 dev eth0

# ip rule add from 192.168.5.221 lookup 11
# ip route add table 11 default src 192.168.5.221 dev eth1

# ip rule add from 192.168.5.222 lookup 12
# ip route add table 12 default src 192.168.5.222 dev eth2

I hope that helps.
-Joe

On 5/2/05, Bram Matthys (Syzop) <syzop@vulnscan.org>
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi,
> 
> this might seem like a classical problem but I''ve trouble getting
this
> working correctly:
> 
> # ifconfig  ### output stripped down:
> eth0      Link encap:Ethernet  HWaddr 00:50:FC:2B:EB:1B
>          inet addr:192.168.5.220  Bcast:192.168.5.255  Mask:255.255.255.0
> 
> eth1      Link encap:Ethernet  HWaddr 00:02:44:60:EC:58
>          inet addr:192.168.5.221  Bcast:192.168.5.255  Mask:255.255.255.0
> 
> eth2      Link encap:Ethernet  HWaddr 00:20:ED:33:2B:6B
>          inet addr:192.168.5.222  Bcast:192.168.5.255  Mask:255.255.255.0
> 
> # ip route
> 192.168.5.0/24 dev eth0  proto kernel  scope link  src 192.168.5.220
> 192.168.5.0/24 dev eth1  proto kernel  scope link  src 192.168.5.221
> 192.168.5.0/24 dev eth2  proto kernel  scope link  src 192.168.5.222
> default via 192.168.5.11 dev eth0
> 
> # cat /proc/sys/net/ipv4/conf/all/arp_filter
> 1
> (and just to be sure I set /conf/<every interface>/arp_filter to 1 as
well)
> 
> So basically what I want is just a box with 3 network cards with 3
> different ips, no bonding/trunking, just simple.. :).
> 
> What I get is simply that eth0 is answering (with it''s own HW
addr) to
> arp requests for all ips (so including .221 and .222). Also, the other
> interfaces are not answering (so it''s not like they all answer
either,
> only eth0 does). The interfaces do receive the request however (when I
> look with tcpdump).
> The effect is obvious, when I list the arp table on the client machine:
>  192.168.5.220         00-50-fc-2b-eb-1b     dynamic
>  192.168.5.221         00-50-fc-2b-eb-1b     dynamic
>  192.168.5.222         00-50-fc-2b-eb-1b     dynamic
>                            ^ that''s all eth0''s hw addr :|
> 
> I''ve also tried all /proc/sys/net/ipv4/conf/all/arp_ignore
settings
> (+flushing arp tables @ client of course), but none of them helped...
> I''m not even sure if those options have anything to do with what
I''m
> trying to accomplish.
> 
> Actually, when adding the hw addr manually (arp -s 192.168.5.221
> 00-02-44-60-EC-58) pinging doesn''t work either... while all
interfaces
> receive the request (because naturally the switch didn''t see the
arp
> request/reply it gets broadcasted) none of them replies.
> 
> Some of these are irrelevant, but just to be sure:
> - - no iptables rules are present
> - - all default iptables actions are at ACCEPT
> - - IP Forwarding is disabled
> - - I''ve reset my switch
> - - I''ve flushed all arp tables on both the server and test client
machine
> 
> This is Linux 2.4.30 (w/grsecurity).
> 
> Fast switching is off btw, checked that as well.
> 
> Did I miss a setting? A kernel compile option? Or is there a problem
> with arp_filter in 2.4.*? I hope I''m missing something obvious
here ;).
> 
> Thanks,
> 
>        Bram.
> 
> - --
> Bram Matthys
> Software developer/IT consultant        syzop@vulnscan.org
> PGP key:                       www.vulnscan.org/pubkey.asc
> PGP fp: 8DD4 437E 9BA8 09AA 0A8D  1811 E1C3 D65F E6ED 2AA2
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.5 (MingW32)
> 
> iD8DBQFCdpa04cPWX+btKqIRArOUAJ4iLOSclzAsw7JbFeK6tGq/4jTwxwCfVMqA
> VBYs/M0H0VVAQD9AutJ29mA> =/L4a
> -----END PGP SIGNATURE-----
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Joe Nuts wrote:
> Hi Bram, 
> I have a similar setup, and what I find works for me is adding ip
> ''rules'' for certain traffic to have specific default
routes.
> for instance :
> 
> # ip rule add from 192.168.5.220 lookup 10
> # ip route add table 10 default src 192.168.5.220 dev eth0
> 
> # ip rule add from 192.168.5.221 lookup 11
> # ip route add table 11 default src 192.168.5.221 dev eth1
> 
> # ip rule add from 192.168.5.222 lookup 12
> # ip route add table 12 default src 192.168.5.222 dev eth2
Thanks! That fixed it :)

Regards,

	Bram.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (MingW32)

iD8DBQFCdp7g4cPWX+btKqIRAq5BAJ4m5kh6fxBww80EApX+cGpvhCmuggCgtieK
2Jh98u5bdOv5M3ugH+yCvo8=jpvN
-----END PGP SIGNATURE-----

> Did I miss a setting? A kernel compile option? Or is there a problem
> with arp_filter in 2.4.*? I hope I''m missing something obvious
here ;).
The reason (as I know it) that this is happening is the way that Linux (for
sure) and most unicies in general (as far as I know) will receive the traffic
that comes in to the NICs on what ever physical NIC that it may be connected to.
The problem is the reply.  Unix will by default choose the first route that it
has available to a subnet as it''s route out to the world.  If you
updated your routing table and put eth2, eth1, and then eth0 all the traffic
would go out eth2.  In short the solution that Joe gave you is probably one that
will work.  I would possibly end up looking at setting up specific routing
tables for each NIC which would have different routes, namely source IPs and
interfaces, to the network.  But the solution that Joe came up with will work
just as well.  I''m not familiar enough to know which one is better
though.



Grant. . . .

I am having the same problems as detailed in the mailing list on the 2 May 
2005 with the subject   

[LARTC] Multiple network cards on same subnet problem (arp_filter=1)

http://mailman.ds9a.nl/pipermail/lartc/2005q2/015788.html

I want to set up three physical interfaces on the same subnet but only
one interface responds to all. I have tried setting arp_filter, arp_ignore and
arp _announce. I am running kernel 2.6.12-rc1 

This is my setup
th0      Link encap:Ethernet  HWaddr 00:05:70:00:00:02
          inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.0.0.0

eth1      Link encap:Ethernet  HWaddr 00:05:70:80:00:01
          inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.0.0.0

eth2      Link encap:Ethernet  HWaddr 00:05:70:C0:00:01
          inet addr:10.0.0.3  Bcast:10.255.255.255  Mask:255.0.0.0


I have tried to use the fix detailed in the list  

# ip rule add from 10.0.0.1 lookup 10
# ip route add table 10 default src 10.0.0.1 dev eth0

etc.

but the command fails as shown

-bash-2.05b# ip rule add from 10.0.0.1 lookup 10
RTNETLINK answers: Invalid argument
-bash-2.05b#

Is there something else I need to do ?

Cheers Simon