Is anyone aware of some sort of facility in either FreeBSD 8.1-RELEASE
or the em(4) driver which would cause it to cache MAC addresses / ARP
entries for hosts on a per-protocol basis? We've been doing some
testing with new routers, and almost every time we switch them in or out
our FreeBSD machines get stuck sending UDP DNS requests to the MAC
address of the old default gateway in complete disregard for the current
contents of the ARP table. New TCP connections do not seem to be affected:
[spolyack@web01 ~]$ uname -a
FreeBSD web01 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #3: Mon Dec 6
08:58:21 EST 2010 root@web01:/usr/obj/usr/src/sys/WEB-AMD64 amd64
Current ARP table after the router replacement, the default gateway is
10.0.1.254, which it has the correct new MAC address for already:
[spolyack@web01 ~]$ arp -an
? (10.0.1.17) at 00:0c:29:47:74:3a on em2 permanent [ethernet]
? (10.0.1.130) at 00:0c:29:47:74:26 on em0 permanent [ethernet]
? (10.0.1.254) at 00:a0:c9:00:01:01 on em0 expires in 915 seconds [ethernet]
? (10.0.0.17) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.15) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.11) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.2) at 00:1f:a0:10:28:70 on em1 expires in 1162 seconds [ethernet]
? (10.0.0.1) at 00:1f:a0:10:28:70 on em1 expires in 943 seconds [ethernet]
tcpdump shows the DNS requests heading to the *old* router's MAC address
despite the contents of the ARP table:
[spolyack@web01 ~]$ sudo tcpdump -i em0 -s 256 -vvv -e -n -ttt 'port 53'
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 256
bytes
00:00:00.000000 00:0c:29:47:74:26 > 54:75:d0:a3:7c:8c, ethertype IPv4
(0x0800), length 86: (tos 0x0, ttl 64, id 55590, offset 0, flags [none],
proto UDP (17), length 72)
10.0.1.130.52419 > 10.0.2.80.53: [bad udp cksum fdc5!] 52051+ A?
db-testing-lab. (44)
[spolyack@web01 ~]$ arp -an | grep 54:75:d0
[spolyack@web01 ~]$
Using telnet to form a TCP connection to port 53 on the same DNS server
which the above UDP requests are headed to works just fine:
[spolyack@web01 ~]$ telnet 10.0.2.80 53
Trying 10.0.2.80...
Connected to 10.0.2.80.
Escape character is '^]'.
00:03:43.272134 00:0c:29:47:74:26 > 00:a0:c9:00:01:01, ethertype IPv4
(0x0800), length 74: (tos 0x10, ttl 64, id 24383, offset 0, flags [DF],
proto TCP (6), length 60)
10.0.2.130.20130 > 10.0.2.80.53: Flags [S], cksum 0x0353 (incorrect
-> 0xf74d), seq 2674341615, win 65535, options [mss 1460,nop,wscale
3,sackOK,TS val 60433610 ecr 0], length
...
Even after the above, standard UDP DNS requests are still headed to the
old default gateway MAC address. tcpdumping and looking at ARP
requests/responses doesn't show any traces of the old MAC address
either. The switch which connects the server and router doesn't have
any entries referencing the old router's MAC address anywhere either.
I'm assuming this isn't related to the new flowtable features included
in 8.x, since they appear to function on a 4-tuple of src addr, src
port, dst addr, and dst port, which isn't going to match for new DNS
requests.
If anyone has any ideas or suggestions on what additional data I can
grab, I'd be happy to investigate further, but I've run out of ideas
here.