[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x1002b
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0909ee7
stack pointer = 0x28:0xeb8cba9c
frame pointer = 0x28:0xeb8cbaf4
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1257 (httpd)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 1d22h29m29s
Physical memory: 2025 MB
Dumping 308 MB: 293 277 261 245 229 213 197 181 165 149 133 117 101 85
69 53 37 21 5
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc06eef57 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc06ef219 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc095730c in trap_fatal (frame=0xeb8cba5c, eva=65579) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc0957590 in trap_pfault (frame=0xeb8cba5c, usermode=0, eva=65579)
at /usr/src/sys/i386/i386/trap.c:812
#5 0xc0957f3c in trap (frame=0xeb8cba5c) at
/usr/src/sys/i386/i386/trap.c:490
#6 0xc093debb in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc0909ee7 in vm_page_splay (pindex=Unhandled dwarf expression
opcode 0x93
) at /usr/src/sys/vm/vm_page.c:576
#8 0xc090a49d in vm_page_remove (m=0xc14faca0) at
/usr/src/sys/vm/vm_page.c:718
#9 0xc090a6e1 in vm_page_free_toq (m=0xc14faca0) at
/usr/src/sys/vm/vm_page.c:1291
#10 0xc090a8b6 in vm_page_free (m=0xc14faca0) at
/usr/src/sys/vm/vm_page.c:498
#11 0xc0908ea5 in vm_object_terminate (object=0xcd8656c8) at
/usr/src/sys/vm/vm_object.c:647
#12 0xc0909713 in vm_object_deallocate (object=0xcd8656c8) at
/usr/src/sys/vm/vm_object.c:580
#13 0xc0901a38 in vm_map_delete (map=Variable "map" is not available.
) at /usr/src/sys/vm/vm_map.c:2315
#14 0xc0901ac1 in vm_map_remove (map=0xca0f02b8, start=0,
end=3217031168) at /usr/src/sys/vm/vm_map.c:2423
#15 0xc09041bf in vmspace_exit (td=0xca672c60) at
/usr/src/sys/vm/vm_map.c:324
#16 0xc06cda1a in exit1 (td=0xca672c60, rv=0) at
/usr/src/sys/kern/kern_exit.c:294
#17 0xc06ced6d in sys_exit (td=Could not find the frame base for
"sys_exit".
) at /usr/src/sys/kern/kern_exit.c:98
#18 0xc09578e5 in syscall (frame=0xeb8cbd38) at
/usr/src/sys/i386/i386/trap.c:1035
#19 0xc093df20 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:196
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
--
second crash 36 hours later.
--
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0xfff77a88
fault code = supervisor write, page not present
instruction pointer = 0x20:0xc090bee2
stack pointer = 0x28:0xe8df4bf4
frame pointer = 0x28:0xe8df4c24
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 43 (pagedaemon)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 1d3h12m13s
Physical memory: 2025 MB
Dumping 321 MB: 306 290 274 258 242 226 210 194 178 162 146 130 114 98
82 66 50 34 18 2
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc06eef57 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc06ef219 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc095730c in trap_fatal (frame=0xe8df4bb4, eva=4294408840) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc0957590 in trap_pfault (frame=0xe8df4bb4, usermode=0,
eva=4294408840) at /usr/src/sys/i386/i386/trap.c:812
#5 0xc0957f3c in trap (frame=0xe8df4bb4) at
/usr/src/sys/i386/i386/trap.c:490
#6 0xc093debb in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc090bee2 in vm_page_cache (m=0xc1a6c860) at
/usr/src/sys/vm/vm_page.c:1556
#8 0xc090df9a in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:862
#9 0xc06cef79 in fork_exit (callout=0xc090d450 <vm_pageout>, arg=0x0,
frame=0xe8df4d38) at /usr/src/sys/kern/kern_fork.c:781
#10 0xc093df30 in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
--
crash a few hours later
--
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0xac2248
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc075c0a6
stack pointer = 0x28:0xeb2468ac
frame pointer = 0x28:0xeb2468d4
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 27668 (httpd)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 9h16m52s
Physical memory: 2025 MB
Dumping 297 MB: 282 266 250 234 218 202 186 170 154 138 122 106 90 74 58
42 26 10
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc06eef57 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc06ef219 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc095730c in trap_fatal (frame=0xeb24686c, eva=11280968) at
/usr/src/sys/i386/i386/trap.c:899
#4 0xc0957590 in trap_pfault (frame=0xeb24686c, usermode=0,
eva=11280968) at /usr/src/sys/i386/i386/trap.c:812
#5 0xc0957f3c in trap (frame=0xeb24686c) at
/usr/src/sys/i386/i386/trap.c:490
#6 0xc093debb in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc075c0a6 in vfs_hash_get (mp=0xc8abf29c, hash=17431063,
flags=Variable "flags" is not available.
) at /usr/src/sys/kern/vfs_hash.c:72
#8 0xc08dae99 in ffs_vget (mp=0xc8abf29c, ino=17431063, flags=2,
vpp=0xeb2469bc) at /usr/src/sys/ufs/ffs/ffs_vfsops.c:1320
#9 0xc08e74ad in ufs_lookup (ap=0xeb246a00) at
/usr/src/sys/ufs/ufs/ufs_lookup.c:573
#10 0xc096b8c2 in VOP_CACHEDLOOKUP_APV (vop=0xc0a74720, a=0xeb246a00) at
vnode_if.c:153
#11 0xc0756bb0 in vfs_cache_lookup (ap=0xeb246a84) at vnode_if.h:83
#12 0xc096d506 in VOP_LOOKUP_APV (vop=0xc0a74c40, a=0xeb246a84) at
vnode_if.c:99
#13 0xc075d3f1 in lookup (ndp=0xeb246ba8) at vnode_if.h:57
#14 0xc075e0ff in namei (ndp=0xeb246ba8) at
/usr/src/sys/kern/vfs_lookup.c:219
#15 0xc076bbed in kern_stat (td=0xc9830a50, path=0x29b066d8 <Address
0x29b066d8 out of bounds>, pathseg=UIO_USERSPACE, sbp=0xeb246c18) at
/usr/src/sys/kern/vfs_syscalls.c:2109
#16 0xc076bd9f in stat (td=0xc9830a50, uap=0xeb246cfc) at
/usr/src/sys/kern/vfs_syscalls.c:2093
#17 0xc09578e5 in syscall (frame=0xeb246d38) at
/usr/src/sys/i386/i386/trap.c:1035
#18 0xc093df20 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:196
#19 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit
--
this is in about 3-4 days, it is a production server.
/etc/sysctl.conf contains
kern.ipc.nmbclusters=65536
kern.maxfiles=35000
net.inet.icmp.icmplim=500
/boot/loader.conf conTAINED
vm.pmap.shpgperproc=1000
as i get these errors:
Approaching the limit on PV entries, consider increasing either the
vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
--
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-RELEASE-p6 #0: Mon Dec 15 21:39:02 PST 2008
root@dual2.box.com:/usr/obj/usr/src/sys/DUAL
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) Dual CPU E2180 @ 2.00GHz (1995.01-MHz
686-class CPU)
Origin = "GenuineIntel" Id = 0x6fd Stepping = 13
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe39d<SSE3,RSVD2,MON,DS_CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
real memory = 2137522176 (2038 MB)
avail memory = 2082156544 (1985 MB)
ACPI APIC Table: <INTELB R_WATERP>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr a280a2806000a28
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr a280a2806000a28
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x30b0-0x30b7 mem
0xd0000000-0xd00fffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
agp0: <Intel 946GZ SVGA controller> on vgapci0
agp0: detected 7676k stolen memory
agp0: aperture size is 256M
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci6: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x4000-0x401f mem 0xd0100000-0xd011ffff irq 16 at device 0.0 on pci6
em0: Using MSI interrupt
em0: Ethernet address: 00:30:48:91:d0:48
em0: [FILTER]
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci7: <ACPI PCI bus> on pcib3
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x5000-0x501f mem 0xd0200000-0xd021ffff irq 17 at device 0.0 on pci7
em1: Using MSI interrupt
em1: Ethernet address: 00:30:48:91:d0:49
em1: [FILTER]
uhci0: <UHCI (generic) USB controller> port 0x3000-0x301f irq 16 at
device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x3020-0x303f irq 17 at
device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x3040-0x305f irq 18 at
device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x3060-0x307f irq 19 at
device 29.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem
0xd0500000-0xd05003ff irq 16 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci8: <ACPI PCI bus> on pcib4
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 SATA300 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.2 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcbfff pnpid
ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, rule-based forwarding
disabled, default to accept, logging limited to 10000 packets/entry by
default
ad2: 238475MB <WDC WD2500KS-00MJB0 02.01C03> at ata1-master SATA150
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad2s1a
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
-
this system was migrated from another company, apache is configured for
4000 clients.. server worked well, now it only does 5-20mbit. i have
another server doing 130mbit with same config, except for
vm.pmap.shpgperproc=1000 on the bigger server is
vm.pmap.shpgperproc=2000 and it's a QUAD processor, same amount of ram.
i have since removed the loader.conf file and will test to see if that
has anything to do with it.
during the last crash, i had set kern.ipc.shm_use_phys=1 and it crashed
a few hours later. now i'm not sure if any of this is linked.. for all i
know, it could be a client doing something.
any suggestions would be appreciated.
thank you.