freebsd stable - Sep 2003 - panics on 24 hour boundaries

Hi stable, nice you see you again.  I was one of those guys who was seeing 
constand panics on 24 hour boundaries but couldn't provide a backtrace due 
to the ar device not taking a dump.  I installed a dedicated drive just to 
take the dump, and then didn't have a panic for a couple weeks.  Now, I am 
back with, and I have traces to share.

The first two, from 2003-09-27 and 2003-09-28 are from an older kernel... 
the same one that was working great for a couple weeks.  Due to some severe 
storms around here, I shut the maching down voluntarily, and once rebooted, 
it started the 24 hour panic crap again.

So, since I didn't have anything much to lose, I cvsuped my src and rebuilt 
and installed world and kernel on the evening of 2003-09-28.  Lo and 
behold, I had another 24 hour panic.  The trace for this one is 
phoenix_2003-09-29.txt.  The full uname from this latest panic is:

FreeBSD phoenix.gargantuan.com 4.9-PRERELEASE FreeBSD 4.9-PRERELEASE #1: Sun 
Sep 28 18:16:30 EDT 2003     
root@phoenix.gargantuan.com:/usr/obj/usr/src/sys/PHOENIX  i386

If this information isn't sufficient, please let me know what else I can do 
to help troubleshoot this problem.  Thanks in advance for your help, it is 
much appreciated!

-- 
Mike
perl -e 'print
unpack("u","88V]N=&%C=\"!I;F9O(&EN(&AE861E<G,*");'


-------------- next part --------------
$ gdb -k kernel.debug vmcore.0
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated
bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line
2627 in elfstab_build_psymtabs
Deprecated bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933
in fill_symbuf

IdlePTD at phsyical address 0x0040f000
initial pcb at physical address 0x003618e0
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x121a2673
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc0283ad3
stack pointer           = 0x10:0xdf65dc3c
frame pointer           = 0x10:0xdf65dc50
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 115 (ntpd)
interrupt mask          = none
trap number             = 12
panic: page fault

syncing disks... 1 
done
Uptime: 1d0h0m17s

dumping to dev #ad/0x20001, offset 3237369
dump ata0: resetting devices .. done
639 638 637 636 635 634 633 632 631 630 629 628 627 626 625 624 623 622 621 620
619 618 617 616 615 614 613 612 611 610 609 608 607 606 605 604 603 602 601 600
599 598 597 596 595 594 593 592 591 590 589 588 587 586 585 584 583 582 581 580
579 578 577 576 575 574 573 572 571 570 569 568 567 566 565 564 563 562 561 560
559 558 557 556 555 554 553 552 551 550 549 548 547 546 545 544 543 542 541 540
539 538 537 536 535 534 533 532 531 530 529 528 527 526 525 524 523 522 521 520
519 518 517 516 515 514 513 512 511 510 509 508 507 506 505 504 503 502 501 500
499 498 497 496 495 494 493 492 491 490 489 488 487 486 485 484 483 482 481 480
479 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464 463 462 461 460
459 458 457 456 455 454 453 452 451 450 449 448 447 446 445 444 443 442 441 440
439 438 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421 420
419 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404 403 402 401 400
399 398 397 396 395 394 393 392 391 390 389 388 387 386 385 384 383 382 381 380
379 378 377 376 375 374 373 372 371 370 369 368 367 366 365 364 363 362 361 360
359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 342 341 340
339 338 337 336 335 334 333 332 331 330 329 328 327 326 325 324 323 322 321 320
319 318 317 316 315 314 313 312 311 310 309 308 307 306 305 304 303 302 301 300
299 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284 283 282 281 280
279 278 277 276 275 274 273 272 271 270 269 268 267 266 265 264 263 262 261 260
259 258 257 256 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240
239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220
219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200
199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180
179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160
159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140
139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120
119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100
99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73
72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46
45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19
18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487             if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc01a1c4f in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc01a2074 in poweroff_wait (junk=0xc031a58c, howto=-1070489457) at
/usr/src/sys/kern/kern_shutdown.c:595
#3  0xc02c1006 in trap_fatal (frame=0xdf65dbfc, eva=303703667) at
/usr/src/sys/i386/i386/trap.c:974
#4  0xc02c0cd9 in trap_pfault (frame=0xdf65dbfc, usermode=0, eva=303703667) at
/usr/src/sys/i386/i386/trap.c:867
#5  0xc02c0897 in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = 512,
tf_esi = -1030041600, tf_ebp = -546972592,
      tf_isp = -546972632, tf_ebx = 11, tf_edx = -1029989860, tf_ecx =
303703647, tf_eax = 1, tf_trapno = 12, tf_err = 0,
      tf_eip = -1071105325, tf_cs = 8, tf_eflags = 66070, tf_esp = 0, tf_ss =
-1030041600}) at /usr/src/sys/i386/i386/trap.c:466
#6  0xc0283ad3 in ufsdirhash_findfree (ip=0xc29ad000, slotneeded=24,
slotsize=0xdf65dcc4) at /usr/src/sys/ufs/ufs/ufs_dirhash.c:467
#7  0xc027df16 in ufs_lookup (ap=0xdf65dd18) at
/usr/src/sys/ufs/ufs/ufs_lookup.c:200
#8  0xc0283105 in ufs_vnoperate (ap=0xdf65dd18) at
/usr/src/sys/ufs/ufs/ufs_vnops.c:2376
#9  0xc01cc14e in vfs_cache_lookup (ap=0xdf65dd70) at vnode_if.h:77
#10 0xc0283105 in ufs_vnoperate (ap=0xdf65dd70) at
/usr/src/sys/ufs/ufs/ufs_vnops.c:2376
#11 0xc01cf08d in lookup (ndp=0xdf65dec8) at vnode_if.h:52
#12 0xc01ceb80 in namei (ndp=0xdf65dec8) at /usr/src/sys/kern/vfs_lookup.c:153
#13 0xc01d73be in vn_open (ndp=0xdf65dec8, fmode=1538, cmode=420) at
/usr/src/sys/kern/vfs_vnops.c:99
#14 0xc01d3688 in open (p=0xd865dc20, uap=0xdf65df80) at
/usr/src/sys/kern/vfs_syscalls.c:1029
#15 0xc02c12b5 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi =
8, tf_esi = 672455168, tf_ebp = -1077937200,
      tf_isp = -546971692, tf_ebx = 672379980, tf_edx = 672455168, tf_ecx = 16,
tf_eax = 5, tf_trapno = 22, tf_err = 2,
      tf_eip = 672287832, tf_cs = 31, tf_eflags = 663, tf_esp = -1077937244,
tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
#16 0xc02b2705 in Xint0x80_syscall ()
#17 0x805ead8 in ?? ()
#18 0x805e83c in ?? ()
#19 0x805f7ab in ?? ()
#20 0x805f370 in ?? ()
#21 0x8049b0e in ?? ()
(kgdb)
-------------- next part --------------
root phoenix Mon Sep 29 21:03:26 /usr/var/crash
$ gdb -k kernel.debug vmcore.0 
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated
bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line
2627 in elfstab_build_psymtabs
Deprecated bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933
in fill_symbuf

IdlePTD at phsyical address 0x00408000
initial pcb at physical address 0x0035c060
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0xa4cf520b
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc01f360c
stack pointer           = 0x10:0xc031ef00
frame pointer           = 0x10:0xc031ef08
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = 
trap number             = 12
panic: page fault

syncing disks... 

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x30
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc026ff98
stack pointer           = 0x10:0xc031ed24
frame pointer           = 0x10:0xc031ed2c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = bio 
trap number             = 12
panic: page fault
Uptime: 1d0h0m17s

dumping to dev #ad/0x20001, offset 3237369
dump ata0: resetting devices .. done
639 638 637 636 635 634 633 632 631 630 629 628 627 626 625 624 623 622 621 620
619 618 617 616 615 614 613 612 611 610 609 608 607 606 605 604 603 602 601 600
599 598 597 596 595 594 593 592 591 590 589 588 587 586 585 584 583 582 581 580
579 578 577 576 575 574 573 572 571 570 569 568 567 566 565 564 563 562 561 560
559 558 557 556 555 554 553 552 551 550 549 548 547 546 545 544 543 542 541 540
539 538 537 536 535 534 533 532 531 530 529 528 527 526 525 524 523 522 521 520
519 518 517 516 515 514 513 512 511 510 509 508 507 506 505 504 503 502 501 500
499 498 497 496 495 494 493 492 491 490 489 488 487 486 485 484 483 482 481 480
479 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464 463 462 461 460
459 458 457 456 455 454 453 452 451 450 449 448 447 446 445 444 443 442 441 440
439 438 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421 420
419 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404 403 402 401 400
399 398 397 396 395 394 393 392 391 390 389 388 387 386 385 384 383 382 381 380
379 378 377 376 375 374 373 372 371 370 369 368 367 366 365 364 363 362 361 360
359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 342 341 340
339 338 337 336 335 334 333 332 331 330 329 328 327 326 325 324 323 322 321 320
319 318 317 316 315 314 313 312 311 310 309 308 307 306 305 304 303 302 301 300
299 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284 283 282 281 280
279 278 277 276 275 274 273 272 271 270 269 268 267 266 265 264 263 262 261 260
259 258 257 256 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240
239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220
219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200
199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180
179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160
159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140
139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120
119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100
99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73
72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46
45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19
18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487             if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc01a19bf in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc01a1de4 in poweroff_wait (junk=0xc03153ec, howto=-1070510353) at
/usr/src/sys/kern/kern_shutdown.c:595
#3  0xc02bcb8a in trap_fatal (frame=0xc031ece4, eva=48) at
/usr/src/sys/i386/i386/trap.c:974
#4  0xc02bc85d in trap_pfault (frame=0xc031ece4, usermode=0, eva=48) at
/usr/src/sys/i386/i386/trap.c:867
#5  0xc02bc41b in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi =
-1070101248, tf_esi = 0, tf_ebp = -1070469844,
      tf_isp = -1070469872, tf_ebx = -1070360356, tf_edx = 6869056, tf_ecx = 28,
tf_eax = 0, tf_trapno = 12, tf_err = 0,
      tf_eip = -1071186024, tf_cs = 8, tf_eflags = 66050, tf_esp = 0, tf_ss =
0}) at /usr/src/sys/i386/i386/trap.c:466
#6  0xc026ff98 in acquire_lock (lk=0xc03398dc) at
/usr/src/sys/ufs/ffs/ffs_softdep.c:266
#7  0xc02745d6 in softdep_fsync_mountdev (vp=0xda9e7a80) at
/usr/src/sys/ufs/ffs/ffs_softdep.c:4024
#8  0xc0278816 in ffs_fsync (ap=0xc031eda0) at
/usr/src/sys/ufs/ffs/ffs_vnops.c:134
#9  0xc027749f in ffs_sync (mp=0xc29b4200, waitfor=2, cred=0xc1818680,
p=0xc0378d00) at vnode_if.h:558
#10 0xc01d2d17 in sync (p=0xc0378d00, uap=0x0) at
/usr/src/sys/kern/vfs_syscalls.c:577
#11 0xc01a1782 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
#12 0xc01a1de4 in poweroff_wait (junk=0xc03153ec, howto=-1070510353) at
/usr/src/sys/kern/kern_shutdown.c:595
#13 0xc02bcb8a in trap_fatal (frame=0xc031eec0, eva=2765050379) at
/usr/src/sys/i386/i386/trap.c:974
#14 0xc02bc85d in trap_pfault (frame=0xc031eec0, usermode=0, eva=2765050379) at
/usr/src/sys/i386/i386/trap.c:867
#15 0xc02bc41b in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi =
4194304, tf_esi = 4194304, tf_ebp = -1070469368,
      tf_isp = -1070469396, tf_ebx = -1529916925, tf_edx = -1529916925, tf_ecx =
-821203904, tf_eax = 1064884209, tf_trapno = 12,
      tf_err = 0, tf_eip = -1071696372, tf_cs = 8, tf_eflags = 66182, tf_esp =
0, tf_ss = -1071696420})
    at /usr/src/sys/i386/i386/trap.c:466
#16 0xc01f360c in arptimer (ignored_arg=0x0) at
/usr/src/sys/netinet/if_ether.c:150
#17 0xc01a7a4d in softclock () at /usr/src/sys/kern/kern_timeout.c:131
#18 0xc02af6b3 in doreti_swi ()
(kgdb) 
-------------- next part --------------
root phoenix Sun Sep 28 13:11:43 /usr/var/crash
$ gdb -k kernel.debug vmcore.1
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-unknown-freebsd"...Deprecated
bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line
2627 in elfstab_build_psymtabs
Deprecated bfd_read called at
/usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933
in fill_symbuf

IdlePTD at phsyical address 0x0040f000
initial pcb at physical address 0x003618e0
panicstr: clist reservation botch
panic messages:
---
panic: clist reservation botch

syncing disks... 3 
done
Uptime: 1d5h7m46s

dumping to dev #ad/0x20001, offset 3237369
dump ata0: resetting devices .. done
639 638 637 636 635 634 633 632 631 630 629 628 627 626 625 624 623 622 621 620
619 618 617 616 615 614 613 612 611 610 609 608 607 606 605 604 603 602 601 600
599 598 597 596 595 594 593 592 591 590 589 588 587 586 585 584 583 582 581 580
579 578 577 576 575 574 573 572 571 570 569 568 567 566 565 564 563 562 561 560
559 558 557 556 555 554 553 552 551 550 549 548 547 546 545 544 543 542 541 540
539 538 537 536 535 534 533 532 531 530 529 528 527 526 525 524 523 522 521 520
519 518 517 516 515 514 513 512 511 510 509 508 507 506 505 504 503 502 501 500
499 498 497 496 495 494 493 492 491 490 489 488 487 486 485 484 483 482 481 480
479 478 477 476 475 474 473 472 471 470 469 468 467 466 465 464 463 462 461 460
459 458 457 456 455 454 453 452 451 450 449 448 447 446 445 444 443 442 441 440
439 438 437 436 435 434 433 432 431 430 429 428 427 426 425 424 423 422 421 420
419 418 417 416 415 414 413 412 411 410 409 408 407 406 405 404 403 402 401 400
399 398 397 396 395 394 393 392 391 390 389 388 387 386 385 384 383 382 381 380
379 378 377 376 375 374 373 372 371 370 369 368 367 366 365 364 363 362 361 360
359 358 357 356 355 354 353 352 351 350 349 348 347 346 345 344 343 342 341 340
339 338 337 336 335 334 333 332 331 330 329 328 327 326 325 324 323 322 321 320
319 318 317 316 315 314 313 312 311 310 309 308 307 306 305 304 303 302 301 300
299 298 297 296 295 294 293 292 291 290 289 288 287 286 285 284 283 282 281 280
279 278 277 276 275 274 273 272 271 270 269 268 267 266 265 264 263 262 261 260
259 258 257 256 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240
239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220
219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200
199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180
179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160
159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140
139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120
119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100
99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73
72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46
45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19
18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487             if (dumping++) {
(kgdb) where
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc01a1c4f in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc01a2074 in poweroff_wait (junk=0xc02fc304, howto=-1070385024) at
/usr/src/sys/kern/kern_shutdown.c:595
#3  0xc01bc8c5 in cblock_free_cblocks (number=31215) at
/usr/src/sys/kern/tty_subr.c:103
#4  0xc01bc88e in clist_alloc_cblocks (clistp=0xc29ad100, ccmax=1024,
ccreserved=1024) at /usr/src/sys/kern/tty_subr.c:178
#5  0xc01ba448 in ttsetwater (tp=0xc29ad100) at /usr/src/sys/kern/tty.c:2264
#6  0xc01b7a3f in ttyopen (device=0xc035fe24, tp=0xc29ad100) at
/usr/src/sys/kern/tty.c:221
#7  0xc02d2503 in sioopen (dev=0xc035fe24, flag=10, mode=8192, p=0xd865df60) at
/usr/src/sys/isa/sio.c:1633
#8  0xc01bb5a4 in cnopen (dev=0xc035feac, flag=10, mode=8192, p=0xd865df60) at
/usr/src/sys/kern/tty_cons.c:279
#9  0xc01db81d in spec_open (ap=0xdf654dfc) at
/usr/src/sys/miscfs/specfs/spec_vnops.c:193
#10 0xc01db71d in spec_vnoperate (ap=0xdf654dfc) at
/usr/src/sys/miscfs/specfs/spec_vnops.c:119
#11 0xc0283135 in ufs_vnoperatespec (ap=0xdf654dfc) at
/usr/src/sys/ufs/ufs/ufs_vnops.c:2394
#12 0xc01d773b in vn_open (ndp=0xdf654ec8, fmode=10, cmode=0) at vnode_if.h:189
#13 0xc01d3688 in open (p=0xd865df60, uap=0xdf654f80) at
/usr/src/sys/kern/vfs_syscalls.c:1029
#14 0xc02c12b5 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi =
-1077940772, tf_esi = -1077940774,
      tf_ebp = -1077942188, tf_isp = -547008556, tf_ebx = -1077943290, tf_edx =
2, tf_ecx = 134561838, tf_eax = 5, tf_trapno = 7,
      tf_err = 2, tf_eip = 672029784, tf_cs = 31, tf_eflags = 659, tf_esp =
-1077943416, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1175
#15 0xc02b2705 in Xint0x80_syscall ()
#16 0x804be71 in ?? ()
#17 0xbfbfffac in ?? ()
#18 0x804972a in ?? ()
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: signature
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20030929/316323bc/attachment.bin

Folks,

I received grief a while back for not posting traces with panic reports, so 
would someone please confirm that what I posted last evening is what is 
needed?  If further information is required, I will do my best to help 
gather and post it.

Thanks much.

-- 
Mike
perl -e 'print
unpack("u","88V]N=&%C=\"!I;F9O(&EN(&AE861E<G,*");'


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: signature
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20030930/178f2344/attachment.bin

On Mon, 29 Sep 2003, Michael W. Oliver wrote:
> Hi stable, nice you see you again.  I was one of those guys who was
> seeing constand panics on 24 hour boundaries but couldn't provide a
> backtrace due to the ar device not taking a dump.  I installed a
> dedicated drive just to take the dump, and then didn't have a panic for
> a couple weeks.  Now, I am back with, and I have traces to share. 
> 
> The first two, from 2003-09-27 and 2003-09-28 are from an older
> kernel...  the same one that was working great for a couple weeks.  Due
> to some severe storms around here, I shut the maching down voluntarily,
> and once rebooted, it started the 24 hour panic crap again.
> 
> So, since I didn't have anything much to lose, I cvsuped my src and
> rebuilt and installed world and kernel on the evening of 2003-09-28.  Lo
> and behold, I had another 24 hour panic.  The trace for this one is
> phoenix_2003-09-29.txt.  The full uname from this latest panic is: 
> 
> FreeBSD phoenix.gargantuan.com 4.9-PRERELEASE FreeBSD 4.9-PRERELEASE #1: 
> Sun Sep 28 18:16:30 EDT 2003
> root@phoenix.gargantuan.com:/usr/obj/usr/src/sys/PHOENIX i386
> 
> If this information isn't sufficient, please let me know what else I
can
> do to help troubleshoot this problem.  Thanks in advance for your help,
> it is much appreciated! 
Initial reactions: panics on 24 hour boundaries are, in my experience,
often associated with the daily event.  Once a day, the daily scripts run
find several times on your file systems, causing every file and directory
to be inspected for changes in setuid scripts, etc.  This can trigger
certain classes of race conditions and resource limits that you might
otherwise not hit in normal operation -- and conviently, they run 24 hours
apart :-).  To try and confirm this suspicion, it would be interesting to
know what time of day exactly the panics take place, and whether you can
reproduce the panic by manually running the daily or security script.

Also, out of curiosity: since you're experiencing crashes, I assume fsck
has run on all your file systems.  If not, you might want to boot to
single user and run fsck on each file system manually to make sure there's
no on-disk corruption of UFS meta-data.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert@fledge.watson.org      Network Associates Laboratories



  [NON-Text Body part not included]

On Wed, Oct 01, 2003 at 11:05 , Men gasped, women
fainted, and small children were reduced to tears as
freebsd-stable-request@freebsd.org confessed to all:
> Message: 18
> Date: Tue, 30 Sep 2003 23:19:34 -0400
> From: "Michael W. Oliver" <michael@gargantuan.com>
> Subject: Re: panics on 24 hour boundaries
[severly edited - wjv]
> +--- On Tuesday, September 30, 2003 22:35 ---
> | Robert Watson proclaimed:
> | Initial reactions: panics on 24 hour boundaries are, in my
> | experience, often associated with the daily event. Once a
> | day, the daily scripts run find several times on your file
> | systems, causing every file and directory to be inspected
> | for changes in setuid scripts, etc. This can trigger certain
> | classes of race conditions and resource limits that you might
> | otherwise not hit in normal operation -- and conviently, they
> | run 24 hours apart :-). To try and confirm this suspicion,
> | it would be interesting to know what time of day exactly the
> | panics take place, and whether you can reproduce the panic by
> | manually running the daily or security script.
> All of the panics happened in the evening hours, between 1800
> and 2200 EDT. I am also able to successfully run the daily
> periodic scripts at any time of the day without issue.
One part in this thread said the panics happened 24 hours after
reboot and that would imply something in scripts that depend upon
a length of time being powered up.

However if all the pnaics occur in the 1800-2200 time frame
this could be caused by an external event.

It could be any large device on the same electric circuits you are
on.  By the 'same circuit' I mean anyone and/or anything connected
to the same power transformer.  In a residential area this could be
several houses.

Anything that could put a spike on the line could cuase this.
And even if the computer if filter and on a UPS if any device
connected to the computer is not also on the same filter those
could be the culprits.  I've seen [in the far past when I
maintained many machines with serial terminal] terminals and also
printers cause this.

One place had contruction going on next door and that was alway
in early afternoon when one piece of equipment was fired up.

And one of the legendary stories is about the systems that paniced
every day between noon and about 10 after.  That was traced to a
microwave in the lunchroom.

It's been my experience that often time related crashes are
external to the machines involved unless each and everything
connected to the machine is coming from the same filterer/protected
source, including all phone lines for DSL and or cable.

Bill
> End of freebsd-stable Digest, Vol 28, Issue 4

-- 
Bill Vermillion - bv @ wjv . com

At 03:59 PM 01/10/2003, Daniel Eischen wrote:
>You can also try setting your clocks ahead (or back) and see
>if the crashes move with the time change.
Tried that and it didnt make a difference.  Also, at the time, the box 
seemed to panic when periodic was running.  BUT if I changed the daily 
scripts to run a few hours after boot time, it would not die.  I could run 
daily to my hearts content manually, but no crash.  (I also made sure than 
the disk cache would be flushed by running md5 on a 1 gig file a couple of 
time in case it was some combo disk / cache issue).  In my case, removing 
INET6 from the kernel totally solved the problem.

         ---Mike