-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
hi,
on important production server I've started to experience weird regular
(once a day-two) sporadic reboots after last upgrade (2015-07-17), and
would appreciate if someone could help me to analyze further and fix the
issue.
Collected crash information below
full logs are available upon request.
Thank you!
backtrace:
- ---[ quotation start ]-------------------------------------------
fo02.ibs dumped core - see /var/crash/vmcore.2
Wed Jul 22 08:01:15 EEST 2015
FreeBSD fo02.ibs 10.1-RELEASE-p14 FreeBSD 10.1-RELEASE-p14 #4 r285499M: Fri Jul
17 22:45:54 EEST 2015 root at
fo02.ibs:/usr/obj/usr/src/sys/FO02.20150307233500 amd64
panic: page fault
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address = 0xffff80403c82b740
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80730e9c
stack pointer = 0x28:0xfffffe000023d790
frame pointer = 0x28:0xfffffe000023d7a0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (irq283: igb3:que 3)
trap number = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff804fe970 at kdb_backtrace+0x60
#1 0xffffffff804c52a5 at panic+0x155
#2 0xffffffff8073f1cf at trap_fatal+0x38f
#3 0xffffffff8073f4e8 at trap_pfault+0x308
#4 0xffffffff8073eb4a at trap+0x47a
#5 0xffffffff80724cb2 at calltrap+0x8
#6 0xffffffff807910cb at bounce_bus_dmamap_load_buffer+0x1bb
#7 0xffffffff804f9c82 at bus_dmamap_load_mbuf_sg+0x72
#8 0xffffffff810c37fe at igb_refresh_mbufs+0x19e
#9 0xffffffff810c3603 at igb_rxeof+0x683
#10 0xffffffff810c3ab4 at igb_msix_que+0xe4
#11 0xffffffff80498e1b at intr_event_execute_handlers+0xab
#12 0xffffffff80499266 at ithread_loop+0x96
#13 0xffffffff80496a3a at fork_exit+0x9a
#14 0xffffffff807251ee at fork_trampoline+0xe
Uptime: 1d10h23m19s
Dumping 1191 out of 8136 MB:..2%..11%..21%..31%..41%..52%..61%..72%..81%..92%
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/pf.ko.symbols...done.
Loaded symbols for /boot/kernel/pf.ko.symbols
Reading symbols from /boot/kernel/if_igb.ko.symbols...done.
Loaded symbols for /boot/kernel/if_igb.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/aesni.ko.symbols...done.
Loaded symbols for /boot/kernel/aesni.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel/cryptodev.ko.symbols
Reading symbols from /boot/kernel/pflog.ko.symbols...done.
Loaded symbols for /boot/kernel/pflog.ko.symbols
Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel/ichsmb.ko.symbols
Reading symbols from /boot/kernel/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel/smbus.ko.symbols
Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
Loaded symbols for /boot/kernel/if_bridge.ko.symbols
Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
Loaded symbols for /boot/kernel/bridgestp.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
#0 doadump (textdump=<value optimized out>) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219
#1 0xffffffff804c4f22 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:452
#2 0xffffffff804c52e4 in panic (fmt=<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff8073f1cf in trap_fatal (frame=<value optimized out>,
eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865
#4 0xffffffff8073f4e8 in trap_pfault (frame=0xfffffe000023d6e0,
usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676
#5 0xffffffff8073eb4a in trap (frame=0xfffffe000023d6e0)
at /usr/src/sys/amd64/amd64/trap.c:440
#6 0xffffffff80724cb2 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:232
#7 0xffffffff80730e9c in pmap_kextract (va=729286318152418877)
at /usr/src/sys/amd64/amd64/pmap.c:639
#8 0xffffffff807910cb in bounce_bus_dmamap_load_buffer (
dmat=0xfffff800034c1800, map=0xffffffff80be01e8,
buf=<value optimized out>, buflen=<value optimized out>,
pmap=0xffffffff80c52ce0, flags=<value optimized out>,
segs=<value optimized out>) at
/usr/src/sys/x86/x86/busdma_bounce.c:690
#9 0xffffffff804f9c82 in bus_dmamap_load_mbuf_sg (dmat=0xfffff800034c1800,
map=0x0, m0=<value optimized out>, segs=0xfffffe000023d8b0,
nsegs=0xfffffe000023d8ac, flags=<value optimized out>)
at /usr/src/sys/kern/subr_bus_dma.c:110
#10 0xffffffff810c37fe in igb_refresh_mbufs (rxr=0xfffff8000349f330,
limit=2101) at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:4144
#11 0xffffffff810c3603 in igb_rxeof (count=<value optimized out>)
at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:4998
#12 0xffffffff810c3ab4 in igb_msix_que (arg=0xfffff800034a8b38)
at /usr/src/sys/modules/igb/../../dev/e1000/if_igb.c:1621
#13 0xffffffff80498e1b in intr_event_execute_handlers (
p=<value optimized out>, ie=0xfffff80003473c00)
at /usr/src/sys/kern/kern_intr.c:1263
#14 0xffffffff80499266 in ithread_loop (arg=0xfffff800034a53e0)
at /usr/src/sys/kern/kern_intr.c:1276
#15 0xffffffff80496a3a in fork_exit (
callout=0xffffffff804991d0 <ithread_loop>, arg=0xfffff800034a53e0,
frame=0xfffffe000023dac0) at /usr/src/sys/kern/kern_fork.c:996
#16 0xffffffff807251ee in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:606
#17 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb)
- ---[ quotation end ]-------------------------------------------
#> uname -a
FreeBSD 10.1-RELEASE-p14 #4 r285499M amd64
#> dmesg
...
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz (3100.06-MHz K8-class CPU)
...
real memory = 8589934592 (8192 MB)
avail memory = 8245338112 (7863 MB)
...
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci4: <ACPI PCI bus> on pcib4
igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd000-0xd01f mem 0xf7400000-0xf747ffff,0xf7480000-0xf7483fff irq 18 at device
0.0 on pci4
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:11:22:33:44:51
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci5: <ACPI PCI bus> on pcib5
igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xc000-0xc01f mem 0xf7300000-0xf737ffff,0xf7380000-0xf7383fff irq 19 at device
0.0 on pci5
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 00:11:22:33:44:52
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci6: <ACPI PCI bus> on pcib6
igb2: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xb000-0xb01f mem 0xf7200000-0xf727ffff,0xf7280000-0xf7283fff irq 18 at device
0.0 on pci6
igb2: Using MSIX interrupts with 5 vectors
igb2: Ethernet address: 00:11:22:33:44:53
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
pcib7: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci7: <ACPI PCI bus> on pcib7
igb3: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xa000-0xa01f mem 0xf7100000-0xf717ffff,0xf7180000-0xf7183fff irq 19 at device
0.0 on pci7
igb3: Using MSIX interrupts with 5 vectors
igb3: Ethernet address: 00:11:22:33:44:54
igb3: Bound queue 0 to cpu 0
igb3: Bound queue 1 to cpu 1
igb3: Bound queue 2 to cpu 2
igb3: Bound queue 3 to cpu 3
...
#> dmidecode
...
System Information
Manufacturer: Supermicro
Product Name: X10SLM+-LN4F
...
#> pciconf -lv
...
igb0 at pci0:4:0:0: class=0x020000 card=0x153315d9 chip=0x15338086
rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
subclass = ethernet
igb1 at pci0:5:0:0: class=0x020000 card=0x153315d9 chip=0x15338086
rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
subclass = ethernet
igb2 at pci0:6:0:0: class=0x020000 card=0x153315d9 chip=0x15338086
rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
subclass = ethernet
igb3 at pci0:7:0:0: class=0x020000 card=0x153315d9 chip=0x15338086
rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
device = 'I210 Gigabit Network Connection'
class = network
subclass = ethernet
...
- --
Zeus V. Panchenko jid:zeus at im.ibs.dn.ua
IT Dpt., I.B.S. LLC GMT+2 (EET)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iEYEARECAAYFAlWvdkEACgkQr3jpPg/3oyraCwCg8OZFNErBWJXmjSHLm+Qm8ruL
IdcAoLG5kxdQpRjv1lPGN3UO27vFY7+H
=ahV7
-----END PGP SIGNATURE-----