freebsd stable - Dec 2004 - Strange networking problems after update 5.2.1->5.3

Hi folks,

I recently updated my old Compaq Armada 1500c from 5.2.1 to 5-stable. 5.2.1
worked fine, the update went without any noticable problem according to the
docs. 5.3 behaves well apart from a strange networking problem.
The notebook lives in a /16 subnet with a /16 netmask and has a 16bit
NE2000 PCMCIA-card (Longshine).

Things that do work:
- ping to hosts in /24
- ssh to hosts in /24
- nis with a server in /24

Things that don't work:
- ping from any host
- ping to hosts outside /24
- nfs
- query dns in /16
- connecting ntp server in /16

My networking setup hasn't changed, netmask and everything are ok (and
worked before the update). I don't use any kind of filtering.
Even a portscan from a machine in /24 connected to the same switch doesn't
show a single thing. The packets are received by the notebook (I see the
LED of the card flashing), but ping -f doesn't create any noticable irq
load. netstat doesn't show any running daemons, though e.g. ssh and lpd are
definitely running.

I definitely don't have a clue what is going on here... anyone else?


cu
  Gerrit
--

On Thu, 30 Dec 2004, Gerrit K?hn wrote:
> I recently updated my old Compaq Armada 1500c from 5.2.1 to 5-stable. 5.2.1
> worked fine, the update went without any noticable problem according to the
> docs. 5.3 behaves well apart from a strange networking problem.
> The notebook lives in a /16 subnet with a /16 netmask and has a 16bit
> NE2000 PCMCIA-card (Longshine).
> 
> Things that do work:
> - ping to hosts in /24
> - ssh to hosts in /24
> - nis with a server in /24
> 
> Things that don't work:
> - ping from any host
> - ping to hosts outside /24
> - nfs
> - query dns in /16
> - connecting ntp server in /16
The summary appears to be "known local things work, less local things
don't", although for the NFS instance it's unclear if that's
local or not.
This suggests a routing or ARP problem.

I think I'd begin diagnosing the problem by checking the routing and arp
configurations to make sure that the configuration seems alright (or at
least, to see if any symptoms are visible).  This would mean doing things
like:

  route -n get default
  route -n get {host in /24}
  route -n get {host in /16}

Check "arp -a" and make sure that the default gateway is what you
expect,
and check to make sure it's hardware address is right.  You may want to
compare against what you see on another machine on the segment.  Make sure
you can ping the default gateway.

Next, I'd get out a packet sniffer and look for on-the-wire problems -- in
particular, to make sure that packets destined for non-local destinations
are getting stamped with the right destination hardware address (that of
the right default gateway).  I'd load up a sniffer on the remote system
and see if the problem is that your outgoing traffic doesn't get there, or
if it's the return traffic that's failing to be properly received. 
I'd
use the sniffer also to inspect the return traffic and make sure it's what
is expected.

Somewhere during all of this, you will probably find the broken bit --
packets missing at some step, the wrong address, or the like.  If you find
anything that isn't fixed via a configuration change (i.e., failed
checksums, no way to explain the address being put in the packet, etc),
let us know.

Robert N M Watson

>> I recently updated my old Compaq Armada 1500c from 5.2.1 to 5-stable.
5.2
>> worked fine, the update went without any noticable problem according to
the
>> docs. 5.3 behaves well apart from a strange networking problem.
>> The notebook lives in a /16 subnet with a /16 netmask and has a 16bit
>> NE2000 PCMCIA-card (Longshine).
>> Things that do work:
>> - ping to hosts in /24
>> - ssh to hosts in /24
>> - nis with a server in /24
>> Things that don't work:
>> - ping from any host
>> - ping to hosts outside /24
>> - nfs
>> - query dns in /16
>> - connecting ntp server in /16
> The summary appears to be "known local things work, less local things
> don't", although for the NFS instance it's unclear if
that's local or not.
Well, what surprised me most was the fact that nothing gets to this notebook
from the outside. I cannot ping it even from a host connected to the same
switch being in the same /24 subnet.
The nfs server was in /24, too.
>This suggests a routing or ARP problem.
I think if there are such problems, they're the result of something else
that is broken with the pcmcia card itself (wrong initialization or
whatever).

I took the notebook home via New Year's and tried to hunt the problem down.
My network at home is somewhat simpler (192.168.1.0/24 is local,
192.168.1.253 is another notebook that is acting as NAT and default router).
>  route -n get default
>  route -n get {host in /24}
>  route -n get {host in /16}
Looks ok for me. netstat doesn't show any problems either.
>Check "arp -a" and make sure that the default gateway is what you
expect,
>and check to make sure it's hardware address is right.  You may want to
>compare against what you see on another machine on the segment.  Make sure
>you can ping the default gateway.
arp things look ok, but I cannot ping the router, though I can ping any
other host (the same thing I already noticed here at work). But my router at
home complains about this:

arp: ether address is multicast for IP address 192.168.1.4!

.4 is the IP of my notebook. Obviously the ping packets reach the router,
but I don't know what should be wrong with the MAC. The output of ifconfig
looks ok.
I tried diagnosing via tcpdump and noticed a rather strange behaviour on the
notebook (the host with the problem, not the router): Quite often tcpdumps
sees nothing at all, not even the obviously outgoing ping packets. In this
state I cannot interrupt it with CTRL-C. Waiting some time (minutes) lets it
suddenly see the packets, but they still don't get through to ping as
desired.

At home I was able to use nfs with .5 as server. However, I noticed that the
packets are broad-(or multi-?)casted to each and every host in my subnet. I
guess this is closely related to the arp-message above from my router.

I have two further pcmcia-cards in the router (dlink de-660 and dlink
dfe-650) which I took out to try them in the Compaq. The dfe-650 was
recognized as 8 bit ne1000 card and didn't work at all. The de-660 was
recognized correctly and worked fine.

I built an OLDCARD kernel, which was complaining about an interrupt storm on
irq 11 (this is the one all the network cards attached to). It found the
pcmcia-bridge, but didn't find any cards, so there was no network device.
>Somewhere during all of this, you will probably find the broken bit --
>packets missing at some step, the wrong address, or the like.  If you find
>anything that isn't fixed via a configuration change (i.e., failed
>checksums, no way to explain the address being put in the packet, etc),
>let us know.
The best thing I can guess here is that 5.3 in contrast to 5.2.1 (all three
cards worked flawlessly with 5.2.1) has some problems dealing with my 
hardware. :-)

I'm anything but a network guru and will see if I have some time to dig
further into the packets. Perhaps I can find the broken bit then.


cu
  Gerrit
--