Lev Serebryakov
2011-Feb-23 09:35 UTC
em0 with latest driver hangs again and again (without "Watchdog timeout" message!)
Hello, Freebsd-net.
It's me again, as problem is not solved and no "clear" answer
was
received.
em0 NIC on my storage server hangs every several (2-3) days.
Symptoms are simple: no packets can be send, mbufs are overfilled, "No
buf space to send" error for any program.
Configuration now is VERY BASIC: no polling, no sysctls or
loader.conf tunables AT ALL. No jumbo frames.
nic doesn't show any "Watchdog timeout" / "resetting"
messages.
Driver from "em driver, 82574L chip, and possibly ASPM" thread
doesn't help, really: it seems, that it decrease frequincy of hangs,
but doesn't eliminate them, but I can not say for sure, may be
frequency change is only illusion, as it random process.
I've added diagnostic patch from Michael Tuxen.
System is cvsupped and built Mon Feb 21, it is FreeBSD 8-STABLE
(RELENG_8).
Hardware is:
em0@pci0:0:25:0: class=0x020000 card=0x82681043 chip=0x10bd8086 rev=0x02
hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82566DM Gigabit Ethernet Adapter (82566DM)'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfeb40000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xfeb7a000, size 4096, enabled
bar [18] = type I/O Port, range 32, base 0xe880, size 32, enabled
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit
cap 09[e0] = vendor (length 6) Intel cap 2 version 0
Output of ifconfig, vmstat -m, netstat -m, top -Snd 1, sysctl
dev.em.0 is attached.
Interesting part of diagnostic sysctls:
dev.em.0.queue0.rxd_head: 896
dev.em.0.queue0.rxd_tail: 895
dev.em.0.queue0.rx_irq: 0
dev.em.0.queue0.rx_nxt_refresh: 896
dev.em.0.queue0.rx_nxt_check: 896
--
// Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: em0.log
Type: application/octet-stream
Size: 13014 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110223/473236cc/em0.obj
Mike Tancsa
2011-Feb-23 11:16 UTC
em0 with latest driver hangs again and again (without "Watchdog timeout" message!)
On 2/23/2011 4:16 AM, Lev Serebryakov wrote:> Driver from "em driver, 82574L chip, and possibly ASPM" thread > doesn't help, really: it seems, that it decrease frequincy of hangs,Looking at your sysctl output, you are not using the test drivers posted in that thread.>>> sysctl dev.em.0dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.1.9 dev.em.0.%driver: em It should show dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.1.9-test If you want to try 7.1.9-test, you can download it at http://www.tancsa.com/if_em-8.c for releng_8. However, there is a newer one Jack has, 7.2.2 which seems to work for me as well so far and has additional fixes that the 7.1.9-test cvsup to RELENG_8, then copy if_em-8.c to /usr/src/sys/dev/e1000/if_em.c ---Mike
Jack Vogel
2011-Feb-23 17:57 UTC
em0 with latest driver hangs again and again (without "Watchdog timeout" message!)
Anyone in net and stable that wants it, limits blocked it, so send me personal email and I'll send to you. Jack On Wed, Feb 23, 2011 at 9:47 AM, Jack Vogel <jfvogel@gmail.com> wrote:> Here is the 7.2.2 tarball. IMPORTANT: if you use this DO NOT try and put it > > into your kernel source tree, it will break that. What you must do is > config the > em driver OUT of your kernel, then untar this, build it standalone, and > then > load it. > > This is just a temporary thing, once I have data to decide on this change > vs > the earlier one it will get integrated. > > Jack > > > 2011/2/23 ?zkan KIRIK <ozkan.kirik@gmail.com> > > Hi, >> >> How can we get 7.2.2. version of if_em driver ? >> I wanna test it. >> >> I can help you for testing changes to em drivers. >> >> >> Regards, >> Ozkan KIRIK >> >> On Wed, Feb 23, 2011 at 1:36 PM, Lev Serebryakov <lev@serebryakov.spb.ru> >> wrote: >> > Hello, Mike. >> > You wrote 23 ??????? 2011 ?., 14:16:28: >> > >> >>> Driver from "em driver, 82574L chip, and possibly ASPM" thread >> >>> doesn't help, really: it seems, that it decrease frequincy of hangs, >> >> Looking at your sysctl output, you are not using the test drivers >> posted >> >> in that thread. >> > Yes, as it doesn't help, I've reverted to "stock" one. >> > >> >> If you want to try 7.1.9-test, you can download it at >> >> http://www.tancsa.com/if_em-8.c for releng_8. >> > I've tried it. It has worked without hangs for 7-8 days, and after >> > that hangs 2 times in 3 days with "7.1.9-test" :( >> > >> > -- >> > // Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru> >> > >> > _______________________________________________ >> > freebsd-stable@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> > To unsubscribe, send any mail to " >> freebsd-stable-unsubscribe@freebsd.org" >> > >> > >
Özkan KIRIK
2011-Feb-24 08:03 UTC
em0 with latest driver hangs again and again (without "Watchdog timeout" message!)
Thank you. I'll test and share my experiences with you. On Wed, Feb 23, 2011 at 7:47 PM, Jack Vogel <jfvogel@gmail.com> wrote:> Here is the 7.2.2 tarball. IMPORTANT: if you use this DO NOT try and put it > into your kernel source tree, it will break that. What you must do is config > the > em driver OUT of your kernel, then untar this, build it standalone, and then > load it. > > This is just a temporary thing, once I have data to decide on this change vs > the earlier one it will get integrated. > > Jack > > > 2011/2/23 ?zkan KIRIK <ozkan.kirik@gmail.com> >> >> Hi, >> >> How can we get 7.2.2. version of if_em driver ? >> I wanna test it. >> >> I can help you for testing changes to em drivers. >> >> >> Regards, >> Ozkan KIRIK >> >> On Wed, Feb 23, 2011 at 1:36 PM, Lev Serebryakov <lev@serebryakov.spb.ru> >> wrote: >> > Hello, Mike. >> > You wrote 23 ??????? 2011 ?., 14:16:28: >> > >> >>> ? Driver from "em driver, 82574L chip, and possibly ASPM" thread >> >>> ?doesn't help, really: it seems, that it decrease frequincy of hangs, >> >> Looking at your sysctl output, you are not using the test drivers >> >> posted >> >> in that thread. >> > ?Yes, as it doesn't help, I've reverted to "stock" one. >> > >> >> If you want to try 7.1.9-test, you can download it at >> >> http://www.tancsa.com/if_em-8.c for releng_8. >> > ?I've tried it. It has worked without hangs for 7-8 days, and after >> > that hangs 2 times in 3 days with "7.1.9-test" ?:( >> > >> > -- >> > // Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru> >> > >> > _______________________________________________ >> > freebsd-stable@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> > To unsubscribe, send any mail to >> > "freebsd-stable-unsubscribe@freebsd.org" >> > > >
Mike Tancsa
2011-Mar-01 14:21 UTC
em0 with latest driver hangs again and again (without "Watchdog timeout" message!)
I have been running with 7.2.2 and so far so good. However, its hard to say in my case as the box I would only periodically see the issue. Jan, have you had a chance to try 7.2.2 ? You seemed to hit the issue the most frequently. There are also some alternate patches in http://www.freebsd.org/cgi/query-pr.cgi?pr=150516 But I think Jack said 7.2.2 takes a similar strategy ? ---Mike On 2/24/2011 3:03 AM, ?zkan KIRIK wrote:> Thank you. I'll test and share my experiences with you. > > On Wed, Feb 23, 2011 at 7:47 PM, Jack Vogel <jfvogel@gmail.com> wrote: >> Here is the 7.2.2 tarball. IMPORTANT: if you use this DO NOT try and put it >> into your kernel source tree, it will break that. What you must do is config >> the >> em driver OUT of your kernel, then untar this, build it standalone, and then >> load it. >> >> This is just a temporary thing, once I have data to decide on this change vs >> the earlier one it will get integrated. >> >> Jack >> >> >> 2011/2/23 ?zkan KIRIK <ozkan.kirik@gmail.com> >>> >>> Hi, >>> >>> How can we get 7.2.2. version of if_em driver ? >>> I wanna test it. >>> >>> I can help you for testing changes to em drivers. >>> >>> >>> Regards, >>> Ozkan KIRIK >>> >>> On Wed, Feb 23, 2011 at 1:36 PM, Lev Serebryakov <lev@serebryakov.spb.ru> >>> wrote: >>>> Hello, Mike. >>>> You wrote 23 ??????? 2011 ?., 14:16:28: >>>> >>>>>> Driver from "em driver, 82574L chip, and possibly ASPM" thread >>>>>> doesn't help, really: it seems, that it decrease frequincy of hangs, >>>>> Looking at your sysctl output, you are not using the test drivers >>>>> posted >>>>> in that thread. >>>> Yes, as it doesn't help, I've reverted to "stock" one. >>>> >>>>> If you want to try 7.1.9-test, you can download it at >>>>> http://www.tancsa.com/if_em-8.c for releng_8. >>>> I've tried it. It has worked without hangs for 7-8 days, and after >>>> that hangs 2 times in 3 days with "7.1.9-test" :( >>>> >>>> -- >>>> // Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru> >>>> >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to >>>> "freebsd-stable-unsubscribe@freebsd.org" >>>> >> >> > >-- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
Arnaud Lacombe
2011-Mar-02 07:21 UTC
em0 with latest driver hangs again and again (without "Watchdog timeout" message!)
Hi, On Tue, Mar 1, 2011 at 2:52 PM, Lev Serebryakov <lev@serebryakov.spb.ru> wrote:> Hello, Mike. > You wrote 1 ????? 2011 ?., 17:20:49: > >> I have been running with 7.2.2 and so far so good. ?However, its hard to >> say in my case as the box I would only periodically see the issue. > ?As I wrote to Jack, my NIC hangs today with 7.2.2 >Do you have any detailed error ? What the output of sysctl "dev.em.X" where X is the index of the hung interface ? Thanks, - Arnaud> -- > // Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Lev Serebryakov
2011-Mar-03 08:11 UTC
em0 with latest driver hangs again and again (without "Watchdog timeout" message!)
Hello, Arnaud. You wrote 2 ????? 2011 ?., 9:55:50:>>> I have been running with 7.2.2 and so far so good. ?However, its hard to >>> say in my case as the box I would only periodically see the issue. >> ?As I wrote to Jack, my NIC hangs today with 7.2.2 > Do you have any detailed error ? What the output of sysctl "dev.em.X" > where X is the index of the hung interface ?One more hang. Two logs are attached. -- // Black Lion AKA Lev Serebryakov <lev@serebryakov.spb.ru> -------------- next part -------------- A non-text attachment was scrubbed... Name: em0.7.2.2.hang1.log Type: application/octet-stream Size: 12972 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110303/a334daec/em0.7.2.2.hang1-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: em0.7.2.2.hang2.log Type: application/octet-stream Size: 12985 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110303/a334daec/em0.7.2.2.hang2-0001.obj