thr3ads.net - freebsd stable - em network issues [Oct 2006]

If this information is useful, please help other people find it:
Share via:

Jack Vogel

2006-Oct-18 10:46 UTC

em network issues

I think there may be a few different problems going on with the em driver
on 6.2 that are being lumped under the general description of network
hangs. In order to solve these I need a reproducible failure, either on a
system here at Intel, or someone who is willing to be a remote guinea
pig :)

I need detailed reports, meaning EXACT system data, if its an OEM
box, what model, what addons, a pciconf list, description of the
network, and anything special that is connected with the problem
occurence.  OH, and if you have a 'before and after' situation, then
please give driver deltas that worked, and which failed.

I know that there are systems out there that have management
hardware that can interfere on the network, it grabs certain packets
as being 'management' and doesnt pass them on to the OS.
Specifically packets for port 623 and 664 get 'eaten' by this
hardware. There is a fix for this, you tell the portmapper to
not use ports below 665, in particular:

          sysctl net.inet.ip.portrange.lowlast 665 (default is 600)

So, if you have IPMI or AMT hardware, you should try this
change and see if it fixes hangs.

There is also a hardware eeprom issue on systems with an 82573
type NIC on SOME systems. There is a utility to fix that, if you
have a problem, and have that NIC email me and I can send that
out to you.

Lastly, our Linux crew have long believed that there are lurking
issues on some AMD based systems, we have problems with
these because we dont have easy access to this hardware (as
you can imagine :). But we now have evidence that SOMETIMES
completion on transmit descriptors is not being written back, and
this causes hangs. They (the linux team) have a modified transmit
cleanup algorithm that does not use the DONE bit, instead it just
using the head and tail pointers. If I can get a case where someone
has this kind of hardware and has hangs AND is willing to test
then perhaps I can try coding something similar up.

Also, remember to let everyone know if something gets fixed :)

Cheers,

Jack

Kip Macy

2006-Oct-18 15:23 UTC

head link

em network issues

I have a Sun T2000 that I generally run with the em driver from as of
July in order to avoid watchdog timeouts. One trivial scenario that
reproduces the problem with 100% consistency is running the ghc
configure script (a 20kloc shell script) over NFS. As the T2000
doesn't exactly represent "typical" PC hardware it may not be the
most
desirable test platform. Nonetheless, let me know if you're
interested.
Thanks for looking into this issue.

    -Kip

On 10/18/06, Jack Vogel <jfvogel@gmail.com> wrote:> I think there may be a few different problems going on with the em driver
> on 6.2 that are being lumped under the general description of network
> hangs. In order to solve these I need a reproducible failure, either on a
> system here at Intel, or someone who is willing to be a remote guinea
> pig :)
>
> I need detailed reports, meaning EXACT system data, if its an OEM
> box, what model, what addons, a pciconf list, description of the
> network, and anything special that is connected with the problem
> occurence.  OH, and if you have a 'before and after' situation,
then
> please give driver deltas that worked, and which failed.
>
> I know that there are systems out there that have management
> hardware that can interfere on the network, it grabs certain packets
> as being 'management' and doesnt pass them on to the OS.
> Specifically packets for port 623 and 664 get 'eaten' by this
> hardware. There is a fix for this, you tell the portmapper to
> not use ports below 665, in particular:
>
>           sysctl net.inet.ip.portrange.lowlast 665 (default is 600)
>
> So, if you have IPMI or AMT hardware, you should try this
> change and see if it fixes hangs.
>
> There is also a hardware eeprom issue on systems with an 82573
> type NIC on SOME systems. There is a utility to fix that, if you
> have a problem, and have that NIC email me and I can send that
> out to you.
>
> Lastly, our Linux crew have long believed that there are lurking
> issues on some AMD based systems, we have problems with
> these because we dont have easy access to this hardware (as
> you can imagine :). But we now have evidence that SOMETIMES
> completion on transmit descriptors is not being written back, and
> this causes hangs. They (the linux team) have a modified transmit
> cleanup algorithm that does not use the DONE bit, instead it just
> using the head and tail pointers. If I can get a case where someone
> has this kind of hardware and has hangs AND is willing to test
> then perhaps I can try coding something similar up.
>
> Also, remember to let everyone know if something gets fixed :)
>
> Cheers,
>
> Jack
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
>

Albert Shih

2006-Oct-18 16:03 UTC

head link

em network issues

Le 18/10/2006  10:46:30-0700, Jack Vogel a écrit> I think there may be a few different problems going on with the em driver
> on 6.2 that are being lumped under the general description of network
> hangs. In order to solve these I need a reproducible failure, either on a
> system here at Intel, or someone who is willing to be a remote guinea
> pig :)
> 
> I need detailed reports, meaning EXACT system data, if its an OEM
> box, what model, what addons, a pciconf list, description of the
> network, and anything special that is connected with the problem
> occurence.  OH, and if you have a 'before and after' situation,
then
> please give driver deltas that worked, and which failed.
Well....

BOX : HP Proliant ML350 G4
All addons is HP 

Here the network config

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=8<VLAN_MTU>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
fxp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=8<VLAN_MTU>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

The pciconf -l

[root@nfs3 ~]# pciconf -l
hostb0@pci0:0:0:        class=0x060000 card=0x32000e11 chip=0x35908086 rev=0x0c
hdr=0x00
pcib1@pci0:2:0: class=0x060400 card=0x00000050 chip=0x35958086 rev=0x0c hdr=0x01
pcib4@pci0:4:0: class=0x060400 card=0x00000050 chip=0x35978086 rev=0x0c hdr=0x01
pcib5@pci0:6:0: class=0x060400 card=0x00000050 chip=0x35998086 rev=0x0c hdr=0x01
pcib6@pci0:28:0:        class=0x060400 card=0x00000050 chip=0x25ae8086 rev=0x02
hdr=0x01
none0@pci0:29:0:        class=0x0c0300 card=0x32010e11 chip=0x25a98086 rev=0x02
hdr=0x00
none1@pci0:29:1:        class=0x0c0300 card=0x32010e11 chip=0x25aa8086 rev=0x02
hdr=0x00
none2@pci0:29:4:        class=0x088000 card=0x32010e11 chip=0x25ab8086 rev=0x02
hdr=0x00
none3@pci0:29:5:        class=0x080020 card=0x32010e11 chip=0x25ac8086 rev=0x02
hdr=0x00
none4@pci0:29:7:        class=0x0c0320 card=0x32010e11 chip=0x25ad8086 rev=0x02
hdr=0x00
pcib8@pci0:30:0:        class=0x060400 card=0x00000000 chip=0x244e8086 rev=0x0a
hdr=0x01
isab0@pci0:31:0:        class=0x060100 card=0x00000000 chip=0x25a18086 rev=0x02
hdr=0x00
atapci0@pci0:31:1:      class=0x01018a card=0x32010e11 chip=0x25a28086 rev=0x02
hdr=0x00
pcib2@pci5:0:0: class=0x060400 card=0x00000044 chip=0x03298086 rev=0x09 hdr=0x01
pcib3@pci5:0:2: class=0x060400 card=0x00000044 chip=0x032a8086 rev=0x09 hdr=0x01
isp0@pci6:1:0:  class=0x0c0400 card=0x01000e11 chip=0x23121077 rev=0x02 hdr=0x00
em0@pci9:1:0:   class=0x020000 card=0x00db0e11 chip=0x10108086 rev=0x01 hdr=0x00
em1@pci9:1:1:   class=0x020000 card=0x00db0e11 chip=0x10108086 rev=0x01 hdr=0x00
ciss0@pci9:2:0: class=0x010400 card=0x409a0e11 chip=0x00460e11 rev=0x01 hdr=0x00
pcib7@pci2:2:0: class=0x060400 card=0x000000dc chip=0xb1548086 rev=0x00 hdr=0x01
mpt0@pci2:3:0:  class=0x010000 card=0x00da0e11 chip=0x00301000 rev=0x08 hdr=0x00
mpt1@pci2:3:1:  class=0x010000 card=0x00da0e11 chip=0x00301000 rev=0x08 hdr=0x00
fxp0@pci3:4:0:  class=0x020000 card=0xb1630e11 chip=0x12298086 rev=0x08 hdr=0x00
fxp1@pci3:5:0:  class=0x020000 card=0xb1630e11 chip=0x12298086 rev=0x08 hdr=0x00
bge0@pci1:2:0:  class=0x020000 card=0x00e30e11 chip=0x165414e4 rev=0x03 hdr=0x00
none5@pci1:3:0: class=0x030000 card=0x001e0e11 chip=0x47521002 rev=0x27 hdr=0x00
none6@pci1:4:0: class=0x088000 card=0x00d70e11 chip=0x00d70e11 rev=0x01 hdr=0x00
[root@nfs3 ~]# 

This server has only one purpose : nfsd.

There are a MSA1000 (disk array) connected in FC with Qlogic FC card.

History of my problem :

The server is buy on january 2006 (in replacement of old HP) only the
server is news, the MSA1000 is the old one.

I've install FreeBSD 6.x on this server.

Because I've 0 problem with the old server, I directly put my server in
production (I known bad idea...)

After some weeks the problem begin with lost em interface (watchdog),
sometime it's fxp (but very rarely).

When this append nothing can fix (without reboot).

I make many cvs, to swapp on 6-Stable, but nothing change. After some weeks
the server just hang-on or the network em is on watchdog status.

In ~march 2006 someone on this mailing list tel me I can put the interface
on polling mode with no SMP.

This thing work very fine until september 2006. Without any change on my
server (no cvs, no buildkernel, no reboot), no change on the nfs clients
(linux/FreebSD) the problem come again. 

The first crash is the server hang-on.

I've make cvsup/buildworld/buildkernel.

After some day the server hang-on again.

When we are in polling mode I don't have the message «em* watchdog», but
the server just hang-on (event on the console).

When I make no polling I've got the em* watchdog message.

Now I run (from yeasterday) in this mode :

	no-SMP
	no-polling
	the patch
http://lists.freebsd.org/pipermail/freebsd-stable/2006-October/029224.html
	and I build a kernel without USB (because I've got many IRQ on usb).

Of course it's to short to tell if the problem is solve.
> hardware. There is a fix for this, you tell the portmapper to
> not use ports below 665, in particular:
> 
>          sysctl net.inet.ip.portrange.lowlast 665 (default is 600)
> 
> So, if you have IPMI or AMT hardware, you should try this
> change and see if it fixes hangs.
I don't known if I've AMT but I put this on my sysctl.conf.
> 
> There is also a hardware eeprom issue on systems with an 82573
> type NIC on SOME systems. There is a utility to fix that, if you
and on HP ?

I'm sorry for :

	1/ My bad english
	2/ The server is on production ... I can make many change or test

but if I can help....

Thanks for all_FB_dev 

Regards.



--
Albert SHIH
Universite de Paris 7 (Denis DIDEROT)
U.F.R. de Mathematiques.
7 ième étage, plateau D, bureau 10
Heure local/Local time:
Thu Oct 19 00:43:39 CEST 2006

Jack Vogel

2006-Oct-18 16:54 UTC

head link

em network issues

On 10/18/06, Albert Shih <shih@math.jussieu.fr>
wrote:> >
> > There is also a hardware eeprom issue on systems with an 82573
> > type NIC on SOME systems. There is a utility to fix that, if you
>
> and on HP ?
your system does not have 573 NICs, (what you show are 546) do you
have others that are?

Jack

Mikhail Teterin

2006-Oct-21 10:00 UTC

head link

em network issues

= I'd appreciate if people who are observing the problem will report
= whether adding DEVICE_POLLING option to kernel config helps them
= or not. This will help to tell whether the problem is in the above
= quote or in the import of new versions from vendor.

I tried this yesterday -- before writing to net@. I saw the "system"
component
of the total load being rather high and then enabled polling.

Again, I did not wait long enough to check, whether the system will cease 
communicating completely before enabling polling on em, but the
"system" load
was shooting way up upon starting my backup procedure even when I switched to 
DEVICE_POLLING-using kernel.

	-mi

freebsd stable - Oct 2006 - em network issues

em network issues

em network issues

em network issues

em network issues

em network issues