Did cvsup on a machine that does just mail processing (well, a lot of spam
scanning) and it crashed not too much later. This kernel does not include
MFC src/sys/kern/sys_process.c revisions 1.111 and 1.112:
Use kmem_alloc_nofault() rather than kmem_alloc_pageable() in
procfs_rwmem().
Use vm_page_hold() in place of vm_page_wire() since the page can be freed.
Don't hold extra reference to the containing vm object while page is
mapped.
which went in after I cvsup'd. Is there a change the crash is due to the
above ? Hardware is pretty generic, we have the same configs on a number
of other boxes. I know there were quite a few VM changes as well as twe
changes since the late May kernel it was running prior to today.
ns4# gdb -k /kernel.debug /var/crash/vmcore.1
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated
bfd_read
called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
line 2627 in elfstab_build_psymtabs
Deprecated bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
line 933 in fill_symbuf
IdlePTD at phsyical address 0x003ad000
initial pcb at physical address 0x0030cb40
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xc408b0c8
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc025fa19
stack pointer = 0x10:0xe8ba6e24
frame pointer = 0x10:0xe8ba6e28
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 31044 (perl)
interrupt mask = net tty bio cam
trap number = 12
panic: page fault
syncing disks... 11 2 2 2 2 2 2 2 8
done
Uptime: 4h39m31s
twe0: failed to delete unit 0
dumping to dev #twed/1, offset 2176
dump 1022 1021 1020 1019 1018 1017 1016 1015 1014 1013 1012 1011 1010 1009
1008 1007 1006 1005 1004 6 5 4 3 2 1 0
---
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) bt
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0161d49 in panic (fmt=0xc02db8cc "%s") at
/usr/src/sys/kern/kern_shutdown.c:595
#3 0xc028cb94 in trap_fatal (frame=0xe8ba6de4, eva=3288903880) at
/usr/src/sys/i386/i386/trap.c:974
#4 0xc028c829 in trap_pfault (frame=0xe8ba6de4, usermode=0,
eva=3288903880) at /usr/src/sys/i386/i386/trap.c:867
#5 0xc028c38f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi =
7314456, tf_esi = -386904664, tf_ebp = -390435288,
tf_isp = -390435312, tf_ebx = -1053617864, tf_edx = -1006063416,
tf_ecx = -386904664, tf_eax = -1006063424, tf_trapno = 12,
tf_err = 0, tf_eip = -1071252967, tf_cs = 8, tf_eflags = 66182,
tf_esp = -1053617864, tf_ss = -390435260})
at /usr/src/sys/i386/i386/trap.c:466
#6 0xc025fa19 in vm_page_remove (m=0xc1331138) at
/usr/src/sys/vm/vm_page.c:462
#7 0xc02600ac in vm_page_free_toq (m=0xc1331138) at
/usr/src/sys/vm/vm_page.c:1104
#8 0xc025de22 in vm_object_terminate (object=0xe8f04da8) at
/usr/src/sys/vm/vm_page.h:514
#9 0xc025dce0 in vm_object_deallocate (object=0xe8f04da8) at
/usr/src/sys/vm/vm_object.c:399
#10 0xc025af21 in vm_map_entry_delete (map=0xe8b3e040, entry=0xe8906c60) at
/usr/src/sys/vm/vm_map.c:2054
#11 0xc025b0d2 in vm_map_delete (map=0xe8b3e040, start=0, end=3217031168)
at /usr/src/sys/vm/vm_map.c:2174
#12 0xc025b161 in vm_map_remove (map=0xe8b3e040, start=0, end=3217031168)
at /usr/src/sys/vm/vm_map.c:2199
#13 0xc0159c3d in exit1 (p=0xe8a89ea0, rv=1054720) at
/usr/src/sys/kern/kern_exit.c:226
#14 0xc0159a09 in sys_exit (p=0xe8a89ea0, uap=0xe8ba6f80) at
/usr/src/sys/kern/kern_exit.c:104
#15 0xc028ce05 in syscall2 (frame={tf_fs = -1071185873, tf_es = 47, tf_ds =
47, tf_edi = 0, tf_esi = -1, tf_ebp = -1077936848,
tf_isp = -390434860, tf_ebx = 672950156, tf_edx = 673025784, tf_ecx
= 5, tf_eax = 1, tf_trapno = 12, tf_err = 2,
tf_eip = 672622500, tf_cs = 31, tf_eflags = 647, tf_esp =
-1077936892, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
#16 0xc027fb05 in Xint0x80_syscall ()
Cannot access memory at address 0xbfbffd30.
(kgdb)
(kgdb) list
482 dumpsys(void)
483 {
484 int error;
485
486 savectx(&dumppcb);
487 if (dumping++) {
488 printf("Dump already in progress,
bailing...\n");
489 return;
490 }
491 if (!dodump)
(kgdb) up
#1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
316 dumpsys();
(kgdb) list
311 * been completed.
312 */
313 EVENTHANDLER_INVOKE(shutdown_post_sync, howto);
314 splhigh();
315 if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP && !cold)
316 dumpsys();
317
318 /* Now that we're going to really halt the system... */
319 EVENTHANDLER_INVOKE(shutdown_final, howto);
320
(kgdb) up
#2 0xc0161d49 in panic (fmt=0xc02db8cc "%s") at
/usr/src/sys/kern/kern_shutdown.c:595
595 boot(bootopt);
(kgdb) list
590
591 #if defined(DDB)
592 if (debugger_on_panic)
593 Debugger ("panic");
594 #endif
595 boot(bootopt);
596 }
597
598 /*
599 * Support for poweroff delay.
(kgdb) up
#3 0xc028cb94 in trap_fatal (frame=0xe8ba6de4, eva=3288903880) at
/usr/src/sys/i386/i386/trap.c:974
974 panic("%s", trap_msg[type]);
(kgdb) list
969 if ((debugger_on_panic || db_active) && kdb_trap(type,
0,
frame))
970 return;
971 #endif
972 printf("trap number = %d\n", type);
973 if (type <= MAX_TRAP_MSG)
974 panic("%s", trap_msg[type]);
975 else
976 panic("unknown/reserved trap");
977 }
978
(kgdb) up
#4 0xc028c829 in trap_pfault (frame=0xe8ba6de4, usermode=0,
eva=3288903880) at /usr/src/sys/i386/i386/trap.c:867
867 trap_fatal(frame, eva);
(kgdb) list
862 if (!usermode) {
863 if (intr_nesting_level == 0 && curpcb &&
curpcb->pcb_onfault) {
864 frame->tf_eip = (int)curpcb->pcb_onfault;
865 return (0);
866 }
867 trap_fatal(frame, eva);
868 return (-1);
869 }
870
871 /* kludge to pass faulting virtual address to sendsig */
(kgdb) up
#5 0xc028c38f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi =
7314456, tf_esi = -386904664, tf_ebp = -390435288,
tf_isp = -390435312, tf_ebx = -1053617864, tf_edx = -1006063416,
tf_ecx = -386904664, tf_eax = -1006063424, tf_trapno = 12,
tf_err = 0, tf_eip = -1071252967, tf_cs = 8, tf_eflags = 66182,
tf_esp = -1053617864, tf_ss = -390435260})
at /usr/src/sys/i386/i386/trap.c:466
466 (void) trap_pfault(&frame, FALSE, eva);
(kgdb) list
461 kernel_trap:
462 /* kernel trap */
463
464 switch (type) {
465 case T_PAGEFLT: /* page fault */
466 (void) trap_pfault(&frame, FALSE, eva);
467 return;
468
469 case T_DNA:
470 #if NNPX > 0
(kgdb) up
#6 0xc025fa19 in vm_page_remove (m=0xc1331138) at
/usr/src/sys/vm/vm_page.c:462
462 bucket = &(*bucket)->hnext;
(kgdb) list
457
458 bucket = &vm_page_buckets[vm_page_hash(m->object,
m->pindex)];
459 while (*bucket != m) {
460 if (*bucket == NULL)
461 panic("vm_page_remove(): page not
found in hash");
462 bucket = &(*bucket)->hnext;
463 }
464 *bucket = m->hnext;
465 m->hnext = NULL;
466 vm_page_bucket_generation++;
(kgdb) up
#7 0xc02600ac in vm_page_free_toq (m=0xc1331138) at
/usr/src/sys/vm/vm_page.c:1104
1104 vm_page_remove(m);
(kgdb) list
1099 * callback routine until after we've put the page on the
1100 * appropriate free queue.
1101 */
1102
1103 vm_page_unqueue_nowakeup(m);
1104 vm_page_remove(m);
1105
1106 /*
1107 * If fictitious remove object association and
1108 * return, otherwise delay object association removal.
(kgdb)
#8 0xc025de22 in vm_object_terminate (object=0xe8f04da8) at
/usr/src/sys/vm/vm_page.h:514
514 vm_page_free_toq(m);
(kgdb) list
509 static __inline void
510 vm_page_free(m)
511 vm_page_t m;
512 {
513 vm_page_flag_clear(m, PG_ZERO);
514 vm_page_free_toq(m);
515 }
516
517 /*
518 * vm_page_free_zero:
(kgdb) up
#9 0xc025dce0 in vm_object_deallocate (object=0xe8f04da8) at
/usr/src/sys/vm/vm_object.c:399
399 vm_object_terminate(object);
(kgdb) list
394 * Don't double-terminate, we could be in a
termination
395 * recursion due to the terminate having to sync data
396 * to disk.
397 */
398 if ((object->flags & OBJ_DEAD) == 0)
399 vm_object_terminate(object);
400 object = temp;
401 }
402 }
403
(kgdb) up
#10 0xc025af21 in vm_map_entry_delete (map=0xe8b3e040, entry=0xe8906c60) at
/usr/src/sys/vm/vm_map.c:2054
2054 vm_object_deallocate(entry->object.vm_object);
(kgdb) list
2049 {
2050 vm_map_entry_unlink(map, entry);
2051 map->size -= entry->end - entry->start;
2052
2053 if ((entry->eflags & MAP_ENTRY_IS_SUB_MAP) == 0) {
2054 vm_object_deallocate(entry->object.vm_object);
2055 }
2056
2057 vm_map_entry_dispose(map, entry);
2058 }
(kgdb) up
#11 0xc025b0d2 in vm_map_delete (map=0xe8b3e040, start=0, end=3217031168)
at /usr/src/sys/vm/vm_map.c:2174
2174 vm_map_entry_delete(map, entry);
(kgdb) list
2169 * Delete the entry (which may delete the object)
only after
2170 * removing all pmap entries pointing to its pages.
2171 * (Otherwise, its page frames may be reallocated,
and any
2172 * modify bits will be set in the wrong object!)
2173 */
2174 vm_map_entry_delete(map, entry);
2175 entry = next;
2176 }
2177 return (KERN_SUCCESS);
2178 }
(kgdb) up
#12 0xc025b161 in vm_map_remove (map=0xe8b3e040, start=0, end=3217031168)
at /usr/src/sys/vm/vm_map.c:2199
2199 result = vm_map_delete(map, start, end);
(kgdb) list
2194 if (map == kmem_map || map == mb_map)
2195 s = splvm();
2196
2197 vm_map_lock(map);
2198 VM_MAP_RANGE_CHECK(map, start, end);
2199 result = vm_map_delete(map, start, end);
2200 vm_map_unlock(map);
2201
2202 if (map == kmem_map || map == mb_map)
2203 splx(s);
(kgdb) up
#13 0xc0159c3d in exit1 (p=0xe8a89ea0, rv=1054720) at
/usr/src/sys/kern/kern_exit.c:226
226 (void) vm_map_remove(&vm->vm_map, VM_MIN_ADDRESS,
(kgdb) list
221 if (--vm->vm_refcnt == 0) {
222 if (vm->vm_shm)
223 shmexit(p);
224 pmap_remove_pages(vmspace_pmap(vm), VM_MIN_ADDRESS,
225 VM_MAXUSER_ADDRESS);
226 (void) vm_map_remove(&vm->vm_map, VM_MIN_ADDRESS,
227 VM_MAXUSER_ADDRESS);
228 }
229
230 if (SESS_LEADER(p)) {
(kgdb) up
#14 0xc0159a09 in sys_exit (p=0xe8a89ea0, uap=0xe8ba6f80) at
/usr/src/sys/kern/kern_exit.c:104
104 exit1(p, W_EXITCODE(uap->rval, 0));
(kgdb) list
99 struct sys_exit_args /* {
100 int rval;
101 } */ *uap;
102 {
103
104 exit1(p, W_EXITCODE(uap->rval, 0));
105 /* NOTREACHED */
106 }
107
108 /*
(kgdb)
ns4# cat /var/run/dmesg.boot
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.8-STABLE #0: Tue Aug 12 11:15:45 EDT 2003
mdtancsa@ns4.recycle.net:/usr/obj/usr/src/sys/ns4
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 2398856892 Hz
CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2398.86-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf27 Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
real memory = 1072627712 (1047488K bytes)
config> q
avail memory = 1040453632 (1016068K bytes)
Preloaded elf kernel "kernel" at 0xc038e000.
Preloaded userconfig_script "/boot/kernel.conf" at 0xc038e09c.
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 7 entries at 0xc00fded0
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
agp0: <Intel 82845G (845G GMCH) SVGA controller> mem
0xd8d00000-0xd8d7ffff,0xd0000000-0xd7ffffff irq 12 at device 2.0 on pci0
agp0: detected 892k stolen memory
agp0: aperture size is 128M
pcib1: <Intel 82801BA/BAM (ICH2) Hub to PCI bridge> at device 30.0 on pci0
pci1: <PCI bus> on pcib1
fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
0xc000-0xc01f
mem 0xd8800000-0xd88fffff,0xd8c00000-0xd8c00fff irq 12 at device 0.0 on pci1
fxp0: Ethernet address 00:a0:c9:e7:a6:e6
inphy0: <i82555 10/100 media interface> on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
0xc400-0xc43f
mem 0xd8a00000-0xd8afffff,0xd8b00000-0xd8b00fff irq 10 at device 1.0 on pci1
fxp1: Ethernet address 00:02:b3:07:fd:8d
inphy1: <i82555 10/100 media interface> on miibus1
inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
twe0: <3ware Storage Controller> port 0xc800-0xc80f irq 15 at device 5.0
on
pci1
twe0: 2 ports, Firmware FE6X 1.02.00.029, BIOS BEXX 1.07.00.009
fxp2: <Intel 82801DB (ICH4) Pro/100 Ethernet> port 0xcc00-0xcc3f mem
0xd8b01000-0xd8b01fff irq 11 at device 8.0 on pci1
fxp2: Ethernet address 00:01:80:38:46:36
inphy2: <i82562ET 10/100 media interface> on miibus2
inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <PCI to ISA bridge (vendor=8086 device=24c0)> at device 31.0 on
pci0
isa0: <ISA bus> on isab0
pci0: <unknown card> (vendor=0x8086, dev=0x24c3) at 31.3 irq 10
orm0: <Option ROMs> at iomem
0xc0000-0xcafff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcf7ff on isa0
fdc0: ready for input in output
fdc0: cmd 3 failed at out byte 1 of 3
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ipfw2 initialized, divert enabled, rule-based forwarding enabled, default
to accept, logging limited to 33100 packets/entry by default
IPsec: Initialized Security Association Processing.
twed0: <Unit 0, TwinStor, Normal> on twe0
twed0: 76318MB (156299440 sectors)
Mounting root from ufs:/dev/twed0a
WARNING: / was not properly dismounted
ns4#
machine i386
cpu I386_CPU
cpu I486_CPU
cpu I586_CPU
cpu I686_CPU
ident ns4
maxusers 0
makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols
options INET #InterNETworking
options FFS #Berkeley Fast Filesystem
options FFS_ROOT #FFS usable as root device [keep this!]
options SOFTUPDATES #Enable FFS soft updates support
options UFS_DIRHASH #Improve performance on big directories
options MFS #Memory Filesystem
options MD_ROOT #MD is a potential root device
options NFS #Network Filesystem
options NFS_ROOT #NFS usable as root device, NFS
required
options CD9660 #ISO 9660 Filesystem
options CD9660_ROOT #CD-ROM usable as root, CD9660 required
options PROCFS #Process filesystem
options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!]
options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI
options UCONSOLE #Allow users to grab the console
options USERCONFIG #boot -c editor
options VISUAL_USERCONFIG #visual boot -c editor
options KTRACE #ktrace(1) support
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options P1003_1B #Posix P1003_1B real-time extensions
options _KPOSIX_PRIORITY_SCHEDULING
options ICMP_BANDLIM #Rate limit bad replies
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
# output. Adds ~128k to driver.
# output. Adds ~215k to driver.
device isa
device pci
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
device ata
device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
options ATA_STATIC_ID #Static device numbering
# Allow ncr to attach legacy NCR devices when
# both sym and ncr are configured
device twe # 3ware Escalade
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1 flags 0x1
device psm0 at atkbdc? irq 12
device vga0 at isa?
pseudo-device splash
device sc0 at isa? flags 0x100
device agp # support several AGP chipsets
device npx0 at nexus? port IO_NPX irq 13
device card
device pcic0 at isa? irq 0 port 0x3e0 iomem 0xd0000
device pcic1 at isa? irq 0 port 0x3e2 iomem 0xd4000 disable
device sio0 at isa? port IO_COM1 flags 0x10 irq 4
device sio1 at isa? port IO_COM2 irq 3
device miibus # MII bus support
device dc # DEC/Intel 21143 and various workalikes
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device rl # RealTek 8129/8139
device sis # Silicon Integrated Systems SiS 900/SiS 7016
pseudo-device loop # Network loopback
pseudo-device ether # Ethernet support
pseudo-device disc 1 # Kernel SLIP
pseudo-device ppp 1 # Kernel PPP
pseudo-device tun # Packet tunnel.
pseudo-device pty # Pseudo-ttys (telnet etc)
pseudo-device md # Memory "disks"
pseudo-device gif # IPv6 and IPv4 tunneling
pseudo-device faith 1 # IPv6-to-IPv4 relaying (translation)
pseudo-device bpf #Berkeley packet filter
options NETGRAPH #netgraph(4) system
options NETGRAPH_ASYNC
options NETGRAPH_BPF
options NETGRAPH_CISCO
options NETGRAPH_ECHO
options NETGRAPH_ETHER
options NETGRAPH_FRAME_RELAY
options NETGRAPH_HOLE
options NETGRAPH_IFACE
options NETGRAPH_KSOCKET
options NETGRAPH_L2TP
options NETGRAPH_LMI
options NETGRAPH_MPPC_ENCRYPTION
options NETGRAPH_ONE2MANY
options NETGRAPH_PPP
options NETGRAPH_PPPOE
options NETGRAPH_PPTPGRE
options NETGRAPH_RFC1490
options NETGRAPH_SOCKET
options NETGRAPH_TEE
options NETGRAPH_TTY
options NETGRAPH_UI
options NETGRAPH_VJC
options IPSEC #IP security
options IPSEC_ESP #IP security (crypto; define w/ IPSEC)
options IPSEC_DEBUG #debug for IP security
options IPFIREWALL #firewall
options IPFW2 #firewall
options IPFIREWALL_VERBOSE #enable logging to syslogd(8)
options IPFIREWALL_FORWARD #enable transparent proxy support
options IPFIREWALL_VERBOSE_LIMIT=33100 #limit verbosity
options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default
options IPV6FIREWALL #firewall for IPv6
options IPV6FIREWALL_VERBOSE
options IPV6FIREWALL_VERBOSE_LIMIT=100
options IPDIVERT #divert sockets
options DDB
options DDB_UNATTENDED
options BREAK_TO_DEBUGGER
options ALT_BREAK_TO_DEBUGGER
--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike
Mike Tancsa
2003-Aug-21 09:13 UTC
panic with today's stable (sort of resolved) PAE related ?
OK, I cvsup'd back to August 7th (default date=2003.08.07.00.00.00),
buildworld / kernel etc and the box has been stable for almost 24hrs
now. Prior to that, it would be rebooting a few times a day.
The panics would be like the one below. I also tried a stable up to the
19th, but still panics.
Below are some more panics and some output tegge requested
-------------------------------------------------
(kgdb) bt
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0161d49 in panic (fmt=0xc02db8cc "%s") at
/usr/src/sys/kern/kern_shutdown.c:595
#3 0xc028cb94 in trap_fatal (frame=0xe9cade3c, eva=16) at
/usr/src/sys/i386/i386/trap.c:974
#4 0xc028c829 in trap_pfault (frame=0xe9cade3c, usermode=0, eva=16) at
/usr/src/sys/i386/i386/trap.c:867
#5 0xc028c38f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = -372637680,
tf_edi = 8, tf_esi = 1143, tf_ebp = -372580732,
tf_isp = -372580760, tf_ebx = -366795420, tf_edx = -4, tf_ecx =
93583622, tf_eax = 524281, tf_trapno = 12, tf_err = 0,
tf_eip = -1071252840, tf_cs = 8, tf_eflags = 66182, tf_esp =
-372353324, tf_ss = 1143}) at /usr/src/sys/i386/i386/trap.c:466
#6 0xc025fa98 in vm_page_lookup (object=0xea232564, pindex=1143) at
/usr/src/sys/vm/vm_page.c:515
#7 0xc0257846 in vm_fault (map=0xea15b140, vaddr=148422656, fault_type=1
'\001', fault_flags=0) at /usr/src/sys/vm/vm_fault.c:426
#8 0xc028c7bb in trap_pfault (frame=0xe9cadfa8, usermode=1, eva=148424256)
at /usr/src/sys/i386/i386/trap.c:847
#9 0xc028c23b in trap (frame={tf_fs = -1078001617, tf_es = 142606383,
tf_ds = -1078001617, tf_edi = 135422592,
tf_esi = 141122551, tf_ebp = -1077937232, tf_isp = -372580396,
tf_ebx = 672134608, tf_edx = 148424256, tf_ecx = 758396440,
tf_eax = 148306460, tf_trapno = 12, tf_err = 4, tf_eip = 671848534,
tf_cs = 31, tf_eflags = 66050, tf_esp = -1077937256,
tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:377
#10 0x280b9856 in ?? ()
#11 0x280e61a6 in ?? ()
#12 0x280e6674 in ?? ()
#13 0x280e6568 in ?? ()
#14 0x280b780c in ?? ()
#15 0x2807ef5d in ?? ()
#16 0x280ea9ab in ?? ()
#17 0x8048e8b in ?? ()
#18 0x8048d5a in ?? ()
(kgdb) up 6
#6 0xc025fa98 in vm_page_lookup (object=0xea232564, pindex=1143) at
/usr/src/sys/vm/vm_page.c:515
515 for (m = *bucket; m != NULL; m = m->hnext) {
(kgdb) print ((struct vm_page *) 0xc1331138)->object
$1 = 0xe991f284
(kgdb) print ((struct vm_page *) 0xc1331138)->object[0]
$2 = {object_list = {tqe_next = 0xe8eb7564, tqe_prev = 0xe9586e04},
shadow_head = {lh_first = 0x0}, shadow_list = {
le_next = 0xe7fccbdc, le_prev = 0xe94bb234}, memq = {tqh_first =
0xc0de94a0, tqh_last = 0xc1179770}, generation = 192,
size = 74, ref_count = 1, shadow_count = 0, hash_rand = -441882534, type
= 1 '\001', flags = 8576, pg_color = 22,
paging_in_progress = 0, resident_page_count = 51, backing_object =
0xe7fa233c, backing_object_offset = 0, pager_object_list = {
tqe_next = 0xe825f000, tqe_prev = 0xe98818e8}, handle = 0x0, un_pager
= {vnp = {vnp_size = 3}, devp = {devp_pglist = {
tqh_first = 0x3, tqh_last = 0x0}}, swp = {swp_bcount = 3}}}
(kgdb) print ((struct vm_page *) 0xc1331138)->pindex
$3 = 61
(kgdb) print vm_page_hash_mask
$4 = 524287
(kgdb) print vm_page_buckets
$5 = (struct vm_page **) 0xc041d000
(kgdb) print vm_page_bucket_count
$6 = 524288
(kgdb) print vm_page_buckets + vm_page_bucket_count
$7 = (struct vm_page **) 0xc061d000
(kgdb) print vm_page_array
$8 = 0xc061d000
(kgdb) print vm_page_array_size
$9 = 261111
(kgdb) print vm_page_array + vm_page_array_size
$10 = (struct vm_page *) 0xc150dde4
(kgdb) print m
$11 = 0x0
(kgdb)
and
(kgdb) bt
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0161d49 in panic (fmt=0xc02db8cc "%s") at
/usr/src/sys/kern/kern_shutdown.c:595
#3 0xc028cb94 in trap_fatal (frame=0xe85c0e3c, eva=12582934) at
/usr/src/sys/i386/i386/trap.c:974
#4 0xc028c829 in trap_pfault (frame=0xe85c0e3c, usermode=0, eva=12582934)
at /usr/src/sys/i386/i386/trap.c:867
#5 0xc028c38f in trap (frame={tf_fs = 16, tf_es = -390856688, tf_ds =
-390856688, tf_edi = 134873088, tf_esi = 84,
tf_ebp = -396620156, tf_isp = -396620184, tf_ebx = -400233508,
tf_edx = 12582914, tf_ecx = 20568904, tf_eax = 524283,
tf_trapno = 12, tf_err = 0, tf_eip = -1071252840, tf_cs = 8,
tf_eflags = 66050, tf_esp = 0, tf_ss = -391880320})
at /usr/src/sys/i386/i386/trap.c:466
#6 0xc025fa98 in vm_page_lookup (object=0xe824ebdc, pindex=84) at
/usr/src/sys/vm/vm_page.c:515
#7 0xc02575ea in vm_fault (map=0xe8a46180, vaddr=134873088, fault_type=1
'\001', fault_flags=0) at /usr/src/sys/vm/vm_fault.c:292
#8 0xc028c7bb in trap_pfault (frame=0xe85c0fa8, usermode=1, eva=134876672)
at /usr/src/sys/i386/i386/trap.c:847
#9 0xc028c23b in trap (frame={tf_fs = 47, tf_es = 135004207, tf_ds =
-1078001617, tf_edi = -1077936692, tf_esi = 672112688,
tf_ebp = -1077936992, tf_isp = -396619820, tf_ebx = 672134608,
tf_edx = 134621060, tf_ecx = 672140532, tf_eax = 134876672,
tf_trapno = 12, tf_err = 4, tf_eip = 671837895, tf_cs = 31,
tf_eflags = 66050, tf_esp = -1077937080, tf_ss = 47})
at /usr/src/sys/i386/i386/trap.c:377
#10 0x280b6ec7 in ?? ()
#11 0x2807ef5d in ?? ()
#12 0x280ea9ab in ?? ()
#13 0x8048e8b in ?? ()
#14 0x8048d5a in ?? ()
(kgdb) up 6
#6 0xc025fa98 in vm_page_lookup (object=0xe824ebdc, pindex=84) at
/usr/src/sys/vm/vm_page.c:515
515 for (m = *bucket; m != NULL; m = m->hnext) {
(kgdb) print ((struct vm_page *) 0xc1331138)->object
$1 = 0xc0318d6c
(kgdb) print ((struct vm_page *) 0xc1331138)->object[0]
$2 = {object_list = {tqe_next = 0xc0318e24, tqe_prev = 0xc0317f0c},
shadow_head = {lh_first = 0x0}, shadow_list = {le_next = 0x0,
le_prev = 0xc0318c60}, memq = {tqh_first = 0xc0934a54, tqh_last =
0xc08f37a4}, generation = 68154, size = 68091,
ref_count = 1, shadow_count = 0, hash_rand = -170022, type = 1
'\001',
flags = 8576, pg_color = 2, paging_in_progress = 0,
resident_page_count = 68026, backing_object = 0x0, backing_object_offset
= 0, pager_object_list = {tqe_next = 0xe7dfd8fc,
tqe_prev = 0xc0318be8}, handle = 0x0, un_pager = {vnp = {vnp_size =
5}, devp = {devp_pglist = {tqh_first = 0x5,
tqh_last = 0x0}}, swp = {swp_bcount = 5}}}
(kgdb) print ((struct vm_page *) 0xc1331138)->pindex
$3 = 36689
(kgdb) print vm_page_hash_mask
$4 = 524287
(kgdb) print vm_page_buckets
$5 = (struct vm_page **) 0xc041d000
(kgdb) print vm_page_bucket_count
$6 = 524288
(kgdb) print vm_page_buckets + vm_page_bucket_count
$7 = (struct vm_page **) 0xc061d000
(kgdb) print vm_page_array
$8 = 0xc061d000
(kgdb) print vm_page_array_size
$9 = 261111
(kgdb) print vm_page_array + vm_page_array_size
$10 = (struct vm_page *) 0xc150dde4
(kgdb) print m
$11 = 0x0
(kgdb)
---Mike
ns4# cat /boot/loader.conf
# -- sysinstall generated deltas -- #
userconfig_script_load="YES"
hw.ata.wc="1"
hw.physmem="1020m"
ns4#
ns4# gdb -k /kernel.debug vmcore.3
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated
bfd_read
called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
line 2627 in elfstab_build_psymtabs
Deprecated bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
line 933 in fill_symbuf
IdlePTD at phsyical address 0x003ad000
initial pcb at physical address 0x0030cb40
panicstr: vm_page_remove(): page not found in hash
panic messages:
---
panic: vm_page_remove(): page not found in hash
syncing disks... 14
stray irq 7
2 2 2 2 2 2 2 7
done
Uptime: 14h21m23s
twe0: failed to delete unit 0
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) bt
#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2 0xc0161d49 in panic (fmt=0xc02d2f60 "vm_page_remove(): page not found
in hash") at /usr/src/sys/kern/kern_shutdown.c:595
#3 0xc025fa13 in vm_page_remove (m=0xc06b0378) at
/usr/src/sys/vm/vm_page.c:461
#4 0xc02600ac in vm_page_free_toq (m=0xc06b0378) at
/usr/src/sys/vm/vm_page.c:1104
#5 0xc025de22 in vm_object_terminate (object=0xe8712d4c) at
/usr/src/sys/vm/vm_page.h:514
#6 0xc025dce0 in vm_object_deallocate (object=0xe8712d4c) at
/usr/src/sys/vm/vm_object.c:399
#7 0xc025af21 in vm_map_entry_delete (map=0xe85ba100, entry=0xe8605780) at
/usr/src/sys/vm/vm_map.c:2054
#8 0xc025b0d2 in vm_map_delete (map=0xe85ba100, start=0, end=3217031168)
at /usr/src/sys/vm/vm_map.c:2174
#9 0xc025b161 in vm_map_remove (map=0xe85ba100, start=0, end=3217031168)
at /usr/src/sys/vm/vm_map.c:2199
#10 0xc0159c3d in exit1 (p=0xe84fa8a0, rv=11) at
/usr/src/sys/kern/kern_exit.c:226
#11 0xc01638bb in sigexit (p=0xe84fa8a0, sig=11) at
/usr/src/sys/kern/kern_sig.c:1504
#12 0xc016362c in postsig (sig=11) at /usr/src/sys/kern/kern_sig.c:1407
#13 0xc028c5db in trap (frame={tf_fs = -1078001617, tf_es = -1078001617,
tf_ds = -1078001617, tf_edi = 135456085,
tf_esi = 135455758, tf_ebp = -1077948736, tf_isp = -397344812,
tf_ebx = 135455744, tf_edx = 135455744, tf_ecx = 0,
tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = 134919195, tf_cs =
31, tf_eflags = 66195, tf_esp = -1077948744, tf_ss = 47})
at /usr/src/sys/i386/i386/trap.c:174
#14 0x80ab41b in ?? ()
Cannot access memory at address 0xbfbfcec0.
(kgdb) up 5
#5 0xc025de22 in vm_object_terminate (object=0xe8712d4c) at
/usr/src/sys/vm/vm_page.h:514
514 vm_page_free_toq(m);
(kgdb) print ((struct vm_page *) 0xc1331138)->object
$1 = 0x0
(kgdb) print ((struct vm_page *) 0xc1331138)->object[0]
Cannot access memory at address 0x0.
(kgdb) print ((struct vm_page *) 0xc1331138)->pindex
$2 = 74
(kgdb) print vm_page_hash_mask
$3 = 524287
(kgdb) print vm_page_buckets
$4 = (struct vm_page **) 0xc041d000
(kgdb) print vm_page_bucket_count
$5 = 524288
(kgdb) print vm_page_buckets + vm_page_bucket_count
$6 = (struct vm_page **) 0xc061d000
(kgdb) print vm_page_array
$7 = 0xc061d000
(kgdb) print vm_page_array_size
$8 = 261111
(kgdb) print vm_page_array + vm_page_array_size
$9 = (struct vm_page *) 0xc150dde4
(kgdb) print m
$10 = 0x0
(kgdb)
dmesg now looks like
ns4# dmesg
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.8-STABLE #0: Tue Aug 12 11:15:45 EDT 2003
mdtancsa@ns4.recycle.net:/usr/obj/usr/src/sys/ns4
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 2398857928 Hz
CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2398.86-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf27 Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
real memory = 1069547520 (1044480K bytes)
config> q
avail memory = 1037451264 (1013136K bytes)
Preloaded elf kernel "kernel" at 0xc038e000.
Preloaded userconfig_script "/boot/kernel.conf" at 0xc038e09c.
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 7 entries at 0xc00fded0
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
agp0: <Intel 82845G (845G GMCH) SVGA controller> mem
0xd8d00000-0xd8d7ffff,0xd0000000-0xd7ffffff irq 12 at device 2.0 on pci0
agp0: detected 892k stolen memory
agp0: aperture size is 128M
pcib1: <Intel 82801BA/BAM (ICH2) Hub to PCI bridge> at device 30.0 on pci0
pci1: <PCI bus> on pcib1
fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
0xc000-0xc01f
mem 0xd8800000-0xd88fffff,0xd8c00000-0xd8c00fff irq 12 at device 0.0 on pci1
fxp0: Ethernet address 00:a0:c9:e7:a6:e6
inphy0: <i82555 10/100 media interface> on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
0xc400-0xc43f
mem 0xd8a00000-0xd8afffff,0xd8b00000-0xd8b00fff irq 10 at device 1.0 on pci1
fxp1: Ethernet address 00:02:b3:07:fd:8d
inphy1: <i82555 10/100 media interface> on miibus1
inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
twe0: <3ware Storage Controller> port 0xc800-0xc80f irq 15 at device 5.0
on
pci1
twe0: 2 ports, Firmware FE6X 1.02.00.029, BIOS BEXX 1.07.00.009
fxp2: <Intel 82801DB (ICH4) Pro/100 Ethernet> port 0xcc00-0xcc3f mem
0xd8b01000-0xd8b01fff irq 11 at device 8.0 on pci1
fxp2: Ethernet address 00:01:80:38:46:36
inphy2: <i82562ET 10/100 media interface> on miibus2
inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <PCI to ISA bridge (vendor=8086 device=24c0)> at device 31.0 on
pci0
isa0: <ISA bus> on isab0
pci0: <unknown card> (vendor=0x8086, dev=0x24c3) at 31.3 irq 10
orm0: <Option ROMs> at iomem
0xc0000-0xcafff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcf7ff on isa0
fdc0: ready for input in output
fdc0: cmd 3 failed at out byte 1 of 3
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ipfw2 initialized, divert enabled, rule-based forwarding enabled, default
to accept, logging limited to 33100 packets/entry by default
IPsec: Initialized Security Association Processing.
twed0: <Unit 0, TwinStor, Normal> on twe0
twed0: 76318MB (156299440 sectors)
Mounting root from ufs:/dev/twed0a
WARNING: / was not properly dismounted
ns4#
At 04:31 PM 12/08/2003 -0400, Mike Tancsa wrote:
>Did cvsup on a machine that does just mail processing (well, a lot of spam
>scanning) and it crashed not too much later. This kernel does not include
>
>MFC src/sys/kern/sys_process.c revisions 1.111 and 1.112:
> Use kmem_alloc_nofault() rather than kmem_alloc_pageable() in
> procfs_rwmem().
> Use vm_page_hold() in place of vm_page_wire() since the page can be
freed.
> Don't hold extra reference to the containing vm object while page is
> mapped.
>
>which went in after I cvsup'd. Is there a change the crash is due to the
>above ? Hardware is pretty generic, we have the same configs on a number
>of other boxes. I know there were quite a few VM changes as well as twe
>changes since the late May kernel it was running prior to today.
>
>ns4# gdb -k /kernel.debug /var/crash/vmcore.1
>GNU gdb 4.18 (FreeBSD)
>Copyright 1998 Free Software Foundation, Inc.
>GDB is free software, covered by the GNU General Public License, and you are
>welcome to change it and/or distribute copies of it under certain
conditions.
>Type "show copying" to see the conditions.
>There is absolutely no warranty for GDB. Type "show warranty" for
details.
>This GDB was configured as "i386-unknown-freebsd"...Deprecated
bfd_read
>called at
>/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
>line 2627 in elfstab_build_psymtabs
>Deprecated bfd_read called at
>/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c
>line 933 in fill_symbuf
>
>IdlePTD at phsyical address 0x003ad000
>initial pcb at physical address 0x0030cb40
>panicstr: page fault
>panic messages:
>---
>Fatal trap 12: page fault while in kernel mode
>fault virtual address = 0xc408b0c8
>fault code = supervisor read, page not present
>instruction pointer = 0x8:0xc025fa19
>stack pointer = 0x10:0xe8ba6e24
>frame pointer = 0x10:0xe8ba6e28
>code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, def32 1, gran 1
>processor eflags = interrupt enabled, resume, IOPL = 0
>current process = 31044 (perl)
>interrupt mask = net tty bio cam
>trap number = 12
>panic: page fault
>
>syncing disks... 11 2 2 2 2 2 2 2 8
>done
>Uptime: 4h39m31s
>twe0: failed to delete unit 0
>
>dumping to dev #twed/1, offset 2176
>dump 1022 1021 1020 1019 1018 1017 1016 1015 1014 1013 1012 1011 1010 1009
>1008 1007 1006 1005 1004 6 5 4 3 2 1 0
>---
>#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
>487 if (dumping++) {
>(kgdb) bt
>#0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
>#1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
>#2 0xc0161d49 in panic (fmt=0xc02db8cc "%s") at
>/usr/src/sys/kern/kern_shutdown.c:595
>#3 0xc028cb94 in trap_fatal (frame=0xe8ba6de4, eva=3288903880) at
>/usr/src/sys/i386/i386/trap.c:974
>#4 0xc028c829 in trap_pfault (frame=0xe8ba6de4, usermode=0,
>eva=3288903880) at /usr/src/sys/i386/i386/trap.c:867
>#5 0xc028c38f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi
>= 7314456, tf_esi = -386904664, tf_ebp = -390435288,
> tf_isp = -390435312, tf_ebx = -1053617864, tf_edx = -1006063416,
> tf_ecx = -386904664, tf_eax = -1006063424, tf_trapno = 12,
> tf_err = 0, tf_eip = -1071252967, tf_cs = 8, tf_eflags = 66182,
> tf_esp = -1053617864, tf_ss = -390435260})
> at /usr/src/sys/i386/i386/trap.c:466
>#6 0xc025fa19 in vm_page_remove (m=0xc1331138) at
>/usr/src/sys/vm/vm_page.c:462
>#7 0xc02600ac in vm_page_free_toq (m=0xc1331138) at
>/usr/src/sys/vm/vm_page.c:1104
>#8 0xc025de22 in vm_object_terminate (object=0xe8f04da8) at
>/usr/src/sys/vm/vm_page.h:514
>#9 0xc025dce0 in vm_object_deallocate (object=0xe8f04da8) at
>/usr/src/sys/vm/vm_object.c:399
>#10 0xc025af21 in vm_map_entry_delete (map=0xe8b3e040, entry=0xe8906c60)
>at /usr/src/sys/vm/vm_map.c:2054
>#11 0xc025b0d2 in vm_map_delete (map=0xe8b3e040, start=0, end=3217031168)
>at /usr/src/sys/vm/vm_map.c:2174
>#12 0xc025b161 in vm_map_remove (map=0xe8b3e040, start=0, end=3217031168)
>at /usr/src/sys/vm/vm_map.c:2199
>#13 0xc0159c3d in exit1 (p=0xe8a89ea0, rv=1054720) at
>/usr/src/sys/kern/kern_exit.c:226
>#14 0xc0159a09 in sys_exit (p=0xe8a89ea0, uap=0xe8ba6f80) at
>/usr/src/sys/kern/kern_exit.c:104
>#15 0xc028ce05 in syscall2 (frame={tf_fs = -1071185873, tf_es = 47, tf_ds
>= 47, tf_edi = 0, tf_esi = -1, tf_ebp = -1077936848,
> tf_isp = -390434860, tf_ebx = 672950156, tf_edx = 673025784, tf_ecx
> = 5, tf_eax = 1, tf_trapno = 12, tf_err = 2,
> tf_eip = 672622500, tf_cs = 31, tf_eflags = 647, tf_esp =
> -1077936892, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
>#16 0xc027fb05 in Xint0x80_syscall ()
>Cannot access memory at address 0xbfbffd30.
>(kgdb)
>
>(kgdb) list
>482 dumpsys(void)
>483 {
>484 int error;
>485
>486 savectx(&dumppcb);
>487 if (dumping++) {
>488 printf("Dump already in progress,
bailing...\n");
>489 return;
>490 }
>491 if (!dodump)
>(kgdb) up
>#1 0xc01618fc in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
>316 dumpsys();
>(kgdb) list
>311 * been completed.
>312 */
>313 EVENTHANDLER_INVOKE(shutdown_post_sync, howto);
>314 splhigh();
>315 if ((howto & (RB_HALT|RB_DUMP)) == RB_DUMP &&
!cold)
>316 dumpsys();
>317
>318 /* Now that we're going to really halt the system... */
>319 EVENTHANDLER_INVOKE(shutdown_final, howto);
>320
>(kgdb) up
>#2 0xc0161d49 in panic (fmt=0xc02db8cc "%s") at
>/usr/src/sys/kern/kern_shutdown.c:595
>595 boot(bootopt);
>(kgdb) list
>590
>591 #if defined(DDB)
>592 if (debugger_on_panic)
>593 Debugger ("panic");
>594 #endif
>595 boot(bootopt);
>596 }
>597
>598 /*
>599 * Support for poweroff delay.
>(kgdb) up
>#3 0xc028cb94 in trap_fatal (frame=0xe8ba6de4, eva=3288903880) at
>/usr/src/sys/i386/i386/trap.c:974
>974 panic("%s", trap_msg[type]);
>(kgdb) list
>969 if ((debugger_on_panic || db_active) &&
kdb_trap(type, 0,
>frame))
>970 return;
>971 #endif
>972 printf("trap number = %d\n", type);
>973 if (type <= MAX_TRAP_MSG)
>974 panic("%s", trap_msg[type]);
>975 else
>976 panic("unknown/reserved trap");
>977 }
>978
>(kgdb) up
>#4 0xc028c829 in trap_pfault (frame=0xe8ba6de4, usermode=0,
>eva=3288903880) at /usr/src/sys/i386/i386/trap.c:867
>867 trap_fatal(frame, eva);
>(kgdb) list
>862 if (!usermode) {
>863 if (intr_nesting_level == 0 && curpcb
&&
>curpcb->pcb_onfault) {
>864 frame->tf_eip =
(int)curpcb->pcb_onfault;
>865 return (0);
>866 }
>867 trap_fatal(frame, eva);
>868 return (-1);
>869 }
>870
>871 /* kludge to pass faulting virtual address to sendsig */
>(kgdb) up
>#5 0xc028c38f in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi
>= 7314456, tf_esi = -386904664, tf_ebp = -390435288,
> tf_isp = -390435312, tf_ebx = -1053617864, tf_edx = -1006063416,
> tf_ecx = -386904664, tf_eax = -1006063424, tf_trapno = 12,
> tf_err = 0, tf_eip = -1071252967, tf_cs = 8, tf_eflags = 66182,
> tf_esp = -1053617864, tf_ss = -390435260})
> at /usr/src/sys/i386/i386/trap.c:466
>466 (void) trap_pfault(&frame, FALSE, eva);
>(kgdb) list
>461 kernel_trap:
>462 /* kernel trap */
>463
>464 switch (type) {
>465 case T_PAGEFLT: /* page fault */
>466 (void) trap_pfault(&frame, FALSE, eva);
>467 return;
>468
>469 case T_DNA:
>470 #if NNPX > 0
>(kgdb) up
>#6 0xc025fa19 in vm_page_remove (m=0xc1331138) at
>/usr/src/sys/vm/vm_page.c:462
>462 bucket = &(*bucket)->hnext;
>(kgdb) list
>457
>458 bucket =
&vm_page_buckets[vm_page_hash(m->object,
>m->pindex)];
>459 while (*bucket != m) {
>460 if (*bucket == NULL)
>461 panic("vm_page_remove(): page
not
>found in hash");
>462 bucket = &(*bucket)->hnext;
>463 }
>464 *bucket = m->hnext;
>465 m->hnext = NULL;
>466 vm_page_bucket_generation++;
>(kgdb) up
>#7 0xc02600ac in vm_page_free_toq (m=0xc1331138) at
>/usr/src/sys/vm/vm_page.c:1104
>1104 vm_page_remove(m);
>(kgdb) list
>1099 * callback routine until after we've put the page on
the
>1100 * appropriate free queue.
>1101 */
>1102
>1103 vm_page_unqueue_nowakeup(m);
>1104 vm_page_remove(m);
>1105
>1106 /*
>1107 * If fictitious remove object association and
>1108 * return, otherwise delay object association removal.
>(kgdb)
>#8 0xc025de22 in vm_object_terminate (object=0xe8f04da8) at
>/usr/src/sys/vm/vm_page.h:514
>514 vm_page_free_toq(m);
>(kgdb) list
>509 static __inline void
>510 vm_page_free(m)
>511 vm_page_t m;
>512 {
>513 vm_page_flag_clear(m, PG_ZERO);
>514 vm_page_free_toq(m);
>515 }
>516
>517 /*
>518 * vm_page_free_zero:
>(kgdb) up
>#9 0xc025dce0 in vm_object_deallocate (object=0xe8f04da8) at
>/usr/src/sys/vm/vm_object.c:399
>399 vm_object_terminate(object);
>(kgdb) list
>394 * Don't double-terminate, we could be in a
>termination
>395 * recursion due to the terminate having to sync
data
>396 * to disk.
>397 */
>398 if ((object->flags & OBJ_DEAD) == 0)
>399 vm_object_terminate(object);
>400 object = temp;
>401 }
>402 }
>403
>(kgdb) up
>#10 0xc025af21 in vm_map_entry_delete (map=0xe8b3e040, entry=0xe8906c60)
>at /usr/src/sys/vm/vm_map.c:2054
>2054 vm_object_deallocate(entry->object.vm_object);
>(kgdb) list
>2049 {
>2050 vm_map_entry_unlink(map, entry);
>2051 map->size -= entry->end - entry->start;
>2052
>2053 if ((entry->eflags & MAP_ENTRY_IS_SUB_MAP) == 0) {
>2054 vm_object_deallocate(entry->object.vm_object);
>2055 }
>2056
>2057 vm_map_entry_dispose(map, entry);
>2058 }
>(kgdb) up
>#11 0xc025b0d2 in vm_map_delete (map=0xe8b3e040, start=0, end=3217031168)
>at /usr/src/sys/vm/vm_map.c:2174
>2174 vm_map_entry_delete(map, entry);
>(kgdb) list
>2169 * Delete the entry (which may delete the object)
>only after
>2170 * removing all pmap entries pointing to its pages.
>2171 * (Otherwise, its page frames may be reallocated,
>and any
>2172 * modify bits will be set in the wrong object!)
>2173 */
>2174 vm_map_entry_delete(map, entry);
>2175 entry = next;
>2176 }
>2177 return (KERN_SUCCESS);
>2178 }
>(kgdb) up
>#12 0xc025b161 in vm_map_remove (map=0xe8b3e040, start=0, end=3217031168)
>at /usr/src/sys/vm/vm_map.c:2199
>2199 result = vm_map_delete(map, start, end);
>(kgdb) list
>2194 if (map == kmem_map || map == mb_map)
>2195 s = splvm();
>2196
>2197 vm_map_lock(map);
>2198 VM_MAP_RANGE_CHECK(map, start, end);
>2199 result = vm_map_delete(map, start, end);
>2200 vm_map_unlock(map);
>2201
>2202 if (map == kmem_map || map == mb_map)
>2203 splx(s);
>(kgdb) up
>#13 0xc0159c3d in exit1 (p=0xe8a89ea0, rv=1054720) at
>/usr/src/sys/kern/kern_exit.c:226
>226 (void) vm_map_remove(&vm->vm_map,
VM_MIN_ADDRESS,
>(kgdb) list
>221 if (--vm->vm_refcnt == 0) {
>222 if (vm->vm_shm)
>223 shmexit(p);
>224 pmap_remove_pages(vmspace_pmap(vm), VM_MIN_ADDRESS,
>225 VM_MAXUSER_ADDRESS);
>226 (void) vm_map_remove(&vm->vm_map,
VM_MIN_ADDRESS,
>227 VM_MAXUSER_ADDRESS);
>228 }
>229
>230 if (SESS_LEADER(p)) {
>(kgdb) up
>#14 0xc0159a09 in sys_exit (p=0xe8a89ea0, uap=0xe8ba6f80) at
>/usr/src/sys/kern/kern_exit.c:104
>104 exit1(p, W_EXITCODE(uap->rval, 0));
>(kgdb) list
>99 struct sys_exit_args /* {
>100 int rval;
>101 } */ *uap;
>102 {
>103
>104 exit1(p, W_EXITCODE(uap->rval, 0));
>105 /* NOTREACHED */
>106 }
>107
>108 /*
>(kgdb)
>
>ns4# cat /var/run/dmesg.boot
>Copyright (c) 1992-2003 The FreeBSD Project.
>Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
>FreeBSD 4.8-STABLE #0: Tue Aug 12 11:15:45 EDT 2003
> mdtancsa@ns4.recycle.net:/usr/obj/usr/src/sys/ns4
>Timecounter "i8254" frequency 1193182 Hz
>Timecounter "TSC" frequency 2398856892 Hz
>CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2398.86-MHz 686-class CPU)
> Origin = "GenuineIntel" Id = 0xf27 Stepping = 7
>
>Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>real memory = 1072627712 (1047488K bytes)
>config> q
>avail memory = 1040453632 (1016068K bytes)
>Preloaded elf kernel "kernel" at 0xc038e000.
>Preloaded userconfig_script "/boot/kernel.conf" at 0xc038e09c.
>Pentium Pro MTRR support enabled
>md0: Malloc disk
>Using $PIR table, 7 entries at 0xc00fded0
>npx0: <math processor> on motherboard
>npx0: INT 16 interface
>pcib0: <Host to PCI bridge> on motherboard
>pci0: <PCI bus> on pcib0
>agp0: <Intel 82845G (845G GMCH) SVGA controller> mem
>0xd8d00000-0xd8d7ffff,0xd0000000-0xd7ffffff irq 12 at device 2.0 on pci0
>agp0: detected 892k stolen memory
>agp0: aperture size is 128M
>pcib1: <Intel 82801BA/BAM (ICH2) Hub to PCI bridge> at device 30.0 on
pci0
>pci1: <PCI bus> on pcib1
>fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
>0xc000-0xc01f mem 0xd8800000-0xd88fffff,0xd8c00000-0xd8c00fff irq 12 at
>device 0.0 on pci1
>fxp0: Ethernet address 00:a0:c9:e7:a6:e6
>inphy0: <i82555 10/100 media interface> on miibus0
>inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
>fxp1: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
>0xc400-0xc43f mem 0xd8a00000-0xd8afffff,0xd8b00000-0xd8b00fff irq 10 at
>device 1.0 on pci1
>fxp1: Ethernet address 00:02:b3:07:fd:8d
>inphy1: <i82555 10/100 media interface> on miibus1
>inphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
>twe0: <3ware Storage Controller> port 0xc800-0xc80f irq 15 at device
5.0
>on pci1
>twe0: 2 ports, Firmware FE6X 1.02.00.029, BIOS BEXX 1.07.00.009
>fxp2: <Intel 82801DB (ICH4) Pro/100 Ethernet> port 0xcc00-0xcc3f mem
>0xd8b01000-0xd8b01fff irq 11 at device 8.0 on pci1
>fxp2: Ethernet address 00:01:80:38:46:36
>inphy2: <i82562ET 10/100 media interface> on miibus2
>inphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
>isab0: <PCI to ISA bridge (vendor=8086 device=24c0)> at device 31.0 on
pci0
>isa0: <ISA bus> on isab0
>pci0: <unknown card> (vendor=0x8086, dev=0x24c3) at 31.3 irq 10
>orm0: <Option ROMs> at iomem
>0xc0000-0xcafff,0xcc000-0xccfff,0xcd000-0xcdfff,0xce000-0xcf7ff on isa0
>fdc0: ready for input in output
>fdc0: cmd 3 failed at out byte 1 of 3
>atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
>vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
>sc0: <System console> at flags 0x100 on isa0
>sc0: VGA <16 virtual consoles, flags=0x100>
>sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
>sio0: type 16550A, console
>sio1 at port 0x2f8-0x2ff irq 3 on isa0
>sio1: type 16550A
>ipfw2 initialized, divert enabled, rule-based forwarding enabled, default
>to accept, logging limited to 33100 packets/entry by default
>IPsec: Initialized Security Association Processing.
>twed0: <Unit 0, TwinStor, Normal> on twe0
>twed0: 76318MB (156299440 sectors)
>Mounting root from ufs:/dev/twed0a
>WARNING: / was not properly dismounted
>ns4#
>
>
>machine i386
>cpu I386_CPU
>cpu I486_CPU
>cpu I586_CPU
>cpu I686_CPU
>ident ns4
>maxusers 0
>makeoptions DEBUG=-g #Build kernel with gdb(1) debug
>symbols
>options INET #InterNETworking
>options FFS #Berkeley Fast Filesystem
>options FFS_ROOT #FFS usable as root device [keep
>this!]
>options SOFTUPDATES #Enable FFS soft updates support
>options UFS_DIRHASH #Improve performance on big
>directories
>options MFS #Memory Filesystem
>options MD_ROOT #MD is a potential root device
>options NFS #Network Filesystem
>options NFS_ROOT #NFS usable as root device, NFS
>required
>options CD9660 #ISO 9660 Filesystem
>options CD9660_ROOT #CD-ROM usable as root, CD9660
>required
>options PROCFS #Process filesystem
>options COMPAT_43 #Compatible with BSD 4.3 [KEEP
THIS!]
>options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI
>options UCONSOLE #Allow users to grab the console
>options USERCONFIG #boot -c editor
>options VISUAL_USERCONFIG #visual boot -c editor
>options KTRACE #ktrace(1) support
>options SYSVSHM #SYSV-style shared memory
>options SYSVMSG #SYSV-style message queues
>options SYSVSEM #SYSV-style semaphores
>options P1003_1B #Posix P1003_1B real-time extensions
>options _KPOSIX_PRIORITY_SCHEDULING
>options ICMP_BANDLIM #Rate limit bad replies
>options KBD_INSTALL_CDEV # install a CDEV entry in /dev
> # output. Adds ~128k to driver.
> # output. Adds ~215k to driver.
>device isa
>device pci
>device fdc0 at isa? port IO_FD1 irq 6 drq 2
>device fd0 at fdc0 drive 0
>device ata
>device atadisk # ATA disk drives
>device atapicd # ATAPI CDROM drives
>options ATA_STATIC_ID #Static device numbering
> # Allow ncr to attach legacy NCR devices
when
> # both sym and ncr are configured
>device twe # 3ware Escalade
>device atkbdc0 at isa? port IO_KBD
>device atkbd0 at atkbdc? irq 1 flags 0x1
>device psm0 at atkbdc? irq 12
>device vga0 at isa?
>pseudo-device splash
>device sc0 at isa? flags 0x100
>device agp # support several AGP chipsets
>device npx0 at nexus? port IO_NPX irq 13
>device card
>device pcic0 at isa? irq 0 port 0x3e0 iomem 0xd0000
>device pcic1 at isa? irq 0 port 0x3e2 iomem 0xd4000 disable
>device sio0 at isa? port IO_COM1 flags 0x10 irq 4
>device sio1 at isa? port IO_COM2 irq 3
>device miibus # MII bus support
>device dc # DEC/Intel 21143 and various workalikes
>device fxp # Intel EtherExpress PRO/100B (82557, 82558)
>device rl # RealTek 8129/8139
>device sis # Silicon Integrated Systems SiS 900/SiS
7016
>pseudo-device loop # Network loopback
>pseudo-device ether # Ethernet support
>pseudo-device disc 1 # Kernel SLIP
>pseudo-device ppp 1 # Kernel PPP
>pseudo-device tun # Packet tunnel.
>pseudo-device pty # Pseudo-ttys (telnet etc)
>pseudo-device md # Memory "disks"
>pseudo-device gif # IPv6 and IPv4 tunneling
>pseudo-device faith 1 # IPv6-to-IPv4 relaying (translation)
>pseudo-device bpf #Berkeley packet filter
>options NETGRAPH #netgraph(4) system
>options NETGRAPH_ASYNC
>options NETGRAPH_BPF
>options NETGRAPH_CISCO
>options NETGRAPH_ECHO
>options NETGRAPH_ETHER
>options NETGRAPH_FRAME_RELAY
>options NETGRAPH_HOLE
>options NETGRAPH_IFACE
>options NETGRAPH_KSOCKET
>options NETGRAPH_L2TP
>options NETGRAPH_LMI
>options NETGRAPH_MPPC_ENCRYPTION
>options NETGRAPH_ONE2MANY
>options NETGRAPH_PPP
>options NETGRAPH_PPPOE
>options NETGRAPH_PPTPGRE
>options NETGRAPH_RFC1490
>options NETGRAPH_SOCKET
>options NETGRAPH_TEE
>options NETGRAPH_TTY
>options NETGRAPH_UI
>options NETGRAPH_VJC
>options IPSEC #IP security
>options IPSEC_ESP #IP security (crypto; define w/
IPSEC)
>options IPSEC_DEBUG #debug for IP security
>options IPFIREWALL #firewall
>options IPFW2 #firewall
>options IPFIREWALL_VERBOSE #enable logging to syslogd(8)
>options IPFIREWALL_FORWARD #enable transparent proxy support
>options IPFIREWALL_VERBOSE_LIMIT=33100 #limit verbosity
>options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by default
>options IPV6FIREWALL #firewall for IPv6
>options IPV6FIREWALL_VERBOSE
>options IPV6FIREWALL_VERBOSE_LIMIT=100
>options IPDIVERT #divert sockets
>options DDB
>options DDB_UNATTENDED
>options BREAK_TO_DEBUGGER
>options ALT_BREAK_TO_DEBUGGER
>--------------------------------------------------------------------
>Mike Tancsa, tel +1 519 651 3400
>Sentex Communications, mike@sentex.net
>Providing Internet since 1994 www.sentex.net
>Cambridge, Ontario Canada www.sentex.net/mike
>
>_______________________________________________
>freebsd-stable@freebsd.org mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
Mike Tancsa
2003-Aug-21 11:47 UTC
panic with today's stable (sort of resolved) PAE related ?
One other note about this issue. I have a number of other boxes running
post PAE commits without problem, but this machine's function is a little
different in that
a) it does spam scanning via a milter app
b) load averages can spike up dramatically due to bursts of spam.
I occurred to me the other big commits in this period were the libc_r
MFC: Add wrapper for kqueue() to keep track of the allocated fd and allow
it to be closed. This fixes a file descriptor leak when closing a
kqueue() fd.
I have saved all the crash files (7 of them) as well as corresponding debug
kernels in case other information is required.
---Mike
At 12:13 PM 21/08/2003 -0400, Mike Tancsa wrote:
>OK, I cvsup'd back to August 7th (default date=2003.08.07.00.00.00),
>buildworld / kernel etc and the box has been stable for almost 24hrs
>now. Prior to that, it would be rebooting a few times a day.
>
>The panics would be like the one below. I also tried a stable up to the
>19th, but still panics.
>
>Below are some more panics and some output tegge requested