# uname -a
FreeBSD gorodok.kes.net.ua 7.0-RELEASE-p3 FreeBSD 7.0-RELEASE-p3 #0: Sun Aug 3
13:18:21 EEST 2008 kes@gorodok.kes.net.ua:/usr/obj/usr/src/sys/KES_KERN_v7
i386
# netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.11.16.1 UGS 0 3758 rl0
10.0.0.0/16 10.11.16.2 UG 0 150 rl0
10.11.15.0/24 link#2 UC 0 0 rl1
10.11.16.0/24 link#1 UC 0 0 rl0
10.11.16.1 00:e0:4c:59:50:7e UHLW 2 421 rl0 953
10.11.16.2 00:03:79:01:9b:d0 UHLW 2 0 rl0 786
127.0.0.1 127.0.0.1 UH 0 122 lo0
Internet6:
Destination Gateway Flags Netif
Expire
::1 ::1 UHL lo0
fe80::%lo0/64 fe80::1%lo0 U lo0
fe80::1%lo0 link#4 UHL lo0
ff01:4::/32 fe80::1%lo0 UC lo0
ff02::%lo0/32 fe80::1%lo0 UC lo0
# ifconfig rl0
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:0e:2e:db:4f:d4
inet 10.11.16.14 netmask 0xffffff00 broadcast 10.11.16.255
inet 10.11.16.9 netmask 0xffffff00 broadcast 10.11.16.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
# ifconfig rl0 add 10.10.16.3/28
# ifconfig rl0
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 00:0e:2e:db:4f:d4
inet 10.11.16.14 netmask 0xffffff00 broadcast 10.11.16.255
inet 10.11.16.9 netmask 0xffffff00 broadcast 10.11.16.255
inet 10.10.16.3 netmask 0xfffffff0 broadcast 10.10.16.15
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
# netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.11.16.1 UGS 0 2751 rl0
10.0.0.0/16 10.11.16.2 UG 0 142 rl0
10.10.16.0/28 link#1 UC 0 0 rl0
10.10.16.3 00:0e:2e:db:4f:d4 UHLW 1 6 lo0
10.11.15.0/24 link#2 UC 0 0 rl1
10.11.16.0/24 link#1 UC 0 0 rl0
10.11.16.1 00:e0:4c:59:50:7e UHLW 2 119 rl0 1176
10.11.16.2 00:03:79:01:9b:d0 UHLW 2 0 rl0 1093
127.0.0.1 127.0.0.1 UH 0 122 lo0
Internet6:
Destination Gateway Flags Netif
Expire
::1 ::1 UHL lo0
fe80::%lo0/64 fe80::1%lo0 U lo0
fe80::1%lo0 link#4 UHL lo0
ff01:4::/32 fe80::1%lo0 UC lo0
ff02::%lo0/32 fe80::1%lo0 UC lo0
After 5 min:
# netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.11.16.1 UGS 0 2859 rl0
10.0.0.0/16 10.11.16.2 UG 0 142 rl0
10.10.16.0/28 link#1 UC 0 0 rl0
10.10.16.3 00:0e:2e:db:4f:d4 UHLW 1 6 lo0
10.11.15.0/24 link#2 UC 0 0 rl1
10.11.16.0/24 10.11.16.14 UGC 1 0 rl0
10.11.16.1 10.11.16.14 UGHW 1 298 rl0
127.0.0.1 127.0.0.1 UH 0 122 lo0
Internet6:
Destination Gateway Flags Netif
Expire
::1 ::1 UHL lo0
fe80::%lo0/64 fe80::1%lo0 U lo0
fe80::1%lo0 link#4 UHL lo0
ff01:4::/32 fe80::1%lo0 UC lo0
ff02::%lo0/32 fe80::1%lo0 UC lo0
Notice second link#1 disappeared
# cat /var/log/messages
Aug 9 12:54:58 gorodok kernel: arplookup 10.11.16.14 failed: host is not on
local network
Aug 9 12:54:58 gorodok kernel: arpresolve: can't allocate route for
10.11.16.14
Aug 9 12:54:59 gorodok kernel: arplookup 10.11.16.14 failed: host is not on
local network
Aug 9 12:54:59 gorodok kernel: arpresolve: can't allocate route for
10.11.16.14
Aug 9 12:55:00 gorodok kernel: arplookup 10.11.16.14 failed: host is not on
local network
Aug 9 12:55:00 gorodok kernel: arpresolve: can't allocate route for
10.11.16.14
Aug 9 12:59:59 gorodok kernel: arplookup 10.11.16.1 failed: host is not on
local network
Aug 9 12:59:59 gorodok kernel: arpresolve: can't allocate route for
10.11.16.1
Aug 9 12:59:59 gorodok kernel: arplookup 10.11.16.1 failed: host is not on
local network
Aug 9 12:59:59 gorodok kernel: arpresolve: can't allocate route for
10.11.16.1
And this repeats each 5min
#netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.11.16.1 UGS 0 2902 rl0
10.0.0.0/16 10.11.16.2 UG 0 145 rl0
10.10.16.0/28 link#1 UC 0 0 rl0
10.10.16.3 00:0e:2e:db:4f:d4 UHLW 1 6 lo0
10.11.15.0/24 link#2 UC 0 0 rl1
10.11.16.0/24 link#1 UC 0 0 rl0
10.11.16.1 00:e0:4c:59:50:7e UHLW 1 2 rl0 1199
127.0.0.1 127.0.0.1 UH 0 122 lo0
Internet6:
Destination Gateway Flags Netif
Expire
::1 ::1 UHL lo0
fe80::%lo0/64 fe80::1%lo0 U lo0
fe80::1%lo0 link#4 UHL lo0
ff01:4::/32 fe80::1%lo0 UC lo0
ff02::%lo0/32 fe80::1%lo0 UC lo0
Another 5 min left and this line appears again
10.11.16.0/24 link#1 UC 0 0 rl0
It seems this bug appears when multiple nets appear on same interface on
7.0-RELEASE-p3
on 7.0-STABLE this bug appears when you start routed daemon.
In FreeBSD 6.3 this bug does not appear
Usually if there is more than IP in a given subnet on an interface, you
give it a /32 netmask. Only the first IP in a subnet should have the
full netmask.
So your example should look like this:
inet 10.11.16.14 netmask 0xffffff00 broadcast 10.11.16.255
inet 10.11.16.9 netmask 0xffffffff broadcast 10.11.16.9
KES wrote:> # uname -a
> FreeBSD gorodok.kes.net.ua 7.0-RELEASE-p3 FreeBSD 7.0-RELEASE-p3 #0: Sun
Aug 3 13:18:21 EEST 2008
kes@gorodok.kes.net.ua:/usr/obj/usr/src/sys/KES_KERN_v7 i386
> # netstat -nr
> Routing tables
>
> Internet:
> Destination Gateway Flags Refs Use Netif Expire
> default 10.11.16.1 UGS 0 3758 rl0
> 10.0.0.0/16 10.11.16.2 UG 0 150 rl0
> 10.11.15.0/24 link#2 UC 0 0 rl1
> 10.11.16.0/24 link#1 UC 0 0 rl0
> 10.11.16.1 00:e0:4c:59:50:7e UHLW 2 421 rl0 953
> 10.11.16.2 00:03:79:01:9b:d0 UHLW 2 0 rl0 786
> 127.0.0.1 127.0.0.1 UH 0 122 lo0
>
> Internet6:
> Destination Gateway Flags
Netif Expire
> ::1 ::1 UHL
lo0
> fe80::%lo0/64 fe80::1%lo0 U
lo0
> fe80::1%lo0 link#4 UHL
lo0
> ff01:4::/32 fe80::1%lo0 UC
lo0
> ff02::%lo0/32 fe80::1%lo0 UC
lo0
> # ifconfig rl0
> rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
> options=8<VLAN_MTU>
> ether 00:0e:2e:db:4f:d4
> inet 10.11.16.14 netmask 0xffffff00 broadcast 10.11.16.255
> inet 10.11.16.9 netmask 0xffffff00 broadcast 10.11.16.255
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
09.08.08, 22:37, "Clifton Royston" <cliftonr@lava.net>:> On Sat, Aug 09, 2008 at 05:23:32PM +0400, KES wrote: > > 09.08.08, 16:22, "Matthew Seaman" <m.seaman@infracaninophile.co.uk>: > > > Andrew Snow wrote: > > > > Usually if there is more than IP in a given subnet on an interface, you > > > > give it a /32 netmask. Only the first IP in a subnet should have the > > > > full netmask. > > > > > > > > So your example should look like this: > > > > > > > > inet 10.11.16.14 netmask 0xffffff00 broadcast 10.11.16.255 > > > > inet 10.11.16.9 netmask 0xffffffff broadcast 10.11.16.9 > > > /32 netmasks for 2nd and subsequent IP alias addresses used to be > > > mandatory and are arguably more correct, but nowadays you can use > > > the actual netmask for the network instead. Was fixed a year or > > > two ago. It's a wetware compatibility thing -- other unixoid OSes > > > never had the /32 netmask requirement, and it kept tripping people up > > > when swapping between OSes. > > > Unfortunately I can't say exactly what the problem the OP is experiencing > > > is due to, but the way routes are appearing and disappearing on a 5 > > > minute timescale does suggest dynamic routing problems to me. As a > > > work-around, if the OP wanted to override the information routed gets > > > from the network, then he could use /etc/gateways to have the local > > > routed append some static routes to the routing table -- see routed(8) > > > for the gory details. Losing a route for a directly attached network > > > looks like a bug to me though. > ... > > > > > > inet 10.11.16.14 netmask 0xffffff00 broadcast 10.11.16.255 > > > > inet 10.11.16.9 netmask 0xffffffff broadcast 10.11.16.9 > > /24 mask on each IPs on same interfaces is working fine on FreeBSD 6.3 > > So I do not think that problem is with the network mask. Because of even ping 10.11.16.14 > > returns network is unreachable! > > Now when I upgraded to v7 I see trouble described earlier. > > So this is must be counted as BUG of v7 > I happened to see recently a report of a similar problem with 7.0 on > a private mailing list. Again, there were multiple IP addresses > configured within the main subnet of the interface (this time > configured as /32s on other physical interfaces) and again, after a > while the system lost connectivity to its main subnet and "forgot" how > to ARP for addresses on the interface. An important similarity - the > routing info like yours showed the attached network with the G flag, as > being reachable via the gateway address within the same subnet. > I can't troubleshoot this, no access to the system in question, but I > thought it might help to know that others have run into the same > problem. > > The thing which is very interesting is: > > Why period is 5 min? > Might be something to do with ARP? Not sure. > -- Clifton>I can't troubleshoot this, no access to the system in questionYou mean you can try to resolve trouble if you get access to machine? I also have tryed /32, but this do not help: gorodok# ifconfig rl0 rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 00:0e:2e:db:4f:d4 inet 10.11.16.14 netmask 0xffffff00 broadcast 10.11.16.255 inet 10.11.16.9 netmask 0xffffffff broadcast 10.11.16.9 media: Ethernet autoselect (100baseTX <full-duplex>) status: active gorodok# ifconfig rl0 add 10.10.16.3/28 gorodok# ping 10.10.16.3 PING 10.10.16.3 (10.10.16.3): 56 data bytes ping: sendto: Network is unreachable ping: sendto: Network is unreachable ^C --- 10.10.16.3 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss gorodok# netstat -nr Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.11.16.1 UGS 0 32727 rl0 10.0.0.0/16 10.11.16.2 UG 0 0 rl0 10.10.16.0/28 10.10.16.3 UGC 0 2 rl0 10.11.15.0/24 link#2 UC 0 0 rl1 10.11.16.0/24 link#1 UC 0 0 rl0 10.11.16.1 00:e0:4c:59:50:7e UHLW 2 0 rl0 1193 10.11.16.2 00:03:79:01:9b:d0 UHLW 2 0 rl0 1126 10.11.16.9 10.11.16.9 UH 0 0 rl0 => 10.11.16.9/32 link#1 UC 0 0 rl0 10.11.16.12 00:0c:6e:ff:0b:35 UHLW 1 2472 rl0 1127 10.11.16.14 00:0e:2e:db:4f:d4 UHLW 1 31 lo0 127.0.0.1 127.0.0.1 UH 0 314 lo0 Internet6: Destination Gateway Flags Netif Expire ::1 ::1 UHL lo0 fe80::%lo0/64 fe80::1%lo0 U lo0 fe80::1%lo0 link#4 UHL lo0 ff01:4::/32 fe80::1%lo0 UC lo0 ff02::%lo0/32 fe80::1%lo0 UC lo0 It seems v7 thinks that 10.10.16.3 is not local IP So it add route to 10.10.16.0/28 via gateway 10.10.16.3 I think something in wrong in algorithm of adding IP to system And somebody broke something in right code of FreeBSD 6.3 So I think if you diff file FreeBSD 6.3 and FreeBSD 7.0 that responsible for routing you can find mistake and it may be fixed easily =)