Yesterday I begin a couple of update to the latest 4.8-STABLE. After that the two boxes continues to go in panics as soon as Apache (1.3 from the ports, also freshly recompiled, 2.0.x seems NOT to hang) starts. I don't know if it is related to the other thread : "Kernel core dump in recent 4.8-STABLE" but it is easily reproducible by cvsupping to a today -STABLE and then running apache 1.3 ... Thanks very much for any info you con provide. Best Regards, Gianmarco Giovannelli , "Unix expert since yesterday" http://www.gufi.org/~gmarco
On Thu, 26 Jun 2003, Gianmarco Giovannelli wrote:> Yesterday I begin a couple of update to the latest 4.8-STABLE. After > that the two boxes continues to go in panics as soon as Apache (1.3 from > the ports, also freshly recompiled, 2.0.x seems NOT to hang) starts. > > I don't know if it is related to the other thread : "Kernel core dump in > recent 4.8-STABLE" but it is easily reproducible by cvsupping to a today > -STABLE and then running apache 1.3 ... > > Thanks very much for any info you con provide.Any chance you have a serial console set up on one or more of the machines and can copy/paste the panic output into an e-mail? A stack trace from DDB would also be good; there's quite a decent chapter in the Developer's Handbook on getting stack traces, etc, from dumps: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html Thanks, Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories
At 27/06/2003, Robert Watson wrote:>On Thu, 26 Jun 2003, Gianmarco Giovannelli wrote: > > > Yesterday I begin a couple of update to the latest 4.8-STABLE. After > > that the two boxes continues to go in panics as soon as Apache (1.3 from > > the ports, also freshly recompiled, 2.0.x seems NOT to hang) starts. > > > > I don't know if it is related to the other thread : "Kernel core dump in > > recent 4.8-STABLE" but it is easily reproducible by cvsupping to a today > > -STABLE and then running apache 1.3 ... > > > > Thanks very much for any info you con provide. > >Any chance you have a serial console set up on one or more of the machines >and can copy/paste the panic output into an e-mail? A stack trace from >DDB would also be good; there's quite a decent chapter in the Developer's >Handbook on getting stack traces, etc, from dumps: > >http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.htmlI am organizing for the kernel debugging... In the meantime I have isolated the problem. It starts happening with cvsup of 2003.06.07.00.00.00, infact I have made some world/kernel using the following cvsup date (the ok mean that apache is running fine, the ko it panics the freebsd boxes I have) The tests was made on a P4 -1.7, but the panics happened also on an epia 533, epia 933, 2 boxes with p4 - 2.4 (one with 1.024 ddr ram the other one with 1.024 pc133 ram). 01.06 --> ok 15.06 --> ko 10.06 --> ko 05.06 --> ok ( date=2003.06.05.00.00.00 ) Edit src/sys/modules/hifn/Makefile Add delta 1.2.2.2 2003.06.05.04.13.47 sam Edit src/sys/modules/ubsec/Makefile Add delta 1.2.2.2 2003.06.05.04.13.48 sam 06.06 --> ok ( date=2003.06.06.00.00.00 ) Edit src/lib/libfetch/common.c Add delta 1.7.2.13 2003.06.06.06.45.25 des Edit src/lib/libfetch/common.h Add delta 1.7.2.10 2003.06.06.06.45.25 des Edit src/lib/libfetch/fetch.3 Add delta 1.11.2.28 2003.06.06.06.45.25 des Edit src/lib/libfetch/fetch.c Add delta 1.10.2.14 2003.06.06.06.45.25 des Edit src/lib/libfetch/ftp.c Add delta 1.16.2.31 2003.06.06.06.45.25 des Edit src/lib/libfetch/http.c Add delta 1.13.2.23 2003.06.06.06.45.25 des Edit src/sbin/vinum/commands.c Add delta 1.31.2.6 2003.06.06.05.13.29 grog Edit src/sys/dev/ata/ata-pci.c Add delta 1.32.2.15 2003.06.06.13.27.05 fjoe Edit src/sys/kern/init_main.c Add delta 1.134.2.8 2003.06.06.20.21.32 tegge Edit src/sys/kern/kern_descrip.c Add delta 1.81.2.17 2003.06.06.20.21.32 tegge Edit src/sys/kern/kern_fork.c Add delta 1.72.2.13 2003.06.06.20.21.32 tegge Edit src/sys/sys/filedesc.h Add delta 1.19.2.5 2003.06.06.20.21.32 tegge Edit src/sys/sys/proc.h Add delta 1.99.2.9 2003.06.06.20.21.32 tegge Edit src/usr.bin/fetch/fetch.1 Add delta 1.33.2.12 2003.06.06.06.48.42 des Edit src/usr.bin/fetch/fetch.c Add delta 1.10.2.21 2003.06.06.06.48.42 des 06.07 --> KO ( date=2003.06.07.00.00.00 ) So I presume the panics are related to these changes, even if I can't say why they happens ... Any idea ? Best Regards, Gianmarco Giovannelli , "Unix expert since yesterday" http://www.gufi.org/~gmarco
At 30/06/2003, you wrote:>I am organizing for the kernel debugging... >In the meantime I have isolated the problem. It starts happening with >cvsup of 2003.06.07.00.00.00, infact I have made some world/kernel using >the following cvsup date (the ok mean that apache is running fine, the ko >it panics the freebsd boxes I have) >The tests was made on a P4 -1.7, but the panics happened also on an epia >533, epia 933, 2 boxes with p4 - 2.4 (one with 1.024 ddr ram the other one >with 1.024 pc133 ram). > >01.06 --> ok > >15.06 --> ko >10.06 --> ko > >05.06 --> ok ( date=2003.06.05.00.00.00 ) > > Edit src/sys/modules/hifn/Makefile > Add delta 1.2.2.2 2003.06.05.04.13.47 sam > Edit src/sys/modules/ubsec/Makefile > Add delta 1.2.2.2 2003.06.05.04.13.48 sam > >06.06 --> ok ( date=2003.06.06.00.00.00 ) > > Edit src/lib/libfetch/common.c > Add delta 1.7.2.13 2003.06.06.06.45.25 des > Edit src/lib/libfetch/common.h > Add delta 1.7.2.10 2003.06.06.06.45.25 des > Edit src/lib/libfetch/fetch.3 > Add delta 1.11.2.28 2003.06.06.06.45.25 des > Edit src/lib/libfetch/fetch.c > Add delta 1.10.2.14 2003.06.06.06.45.25 des > Edit src/lib/libfetch/ftp.c > Add delta 1.16.2.31 2003.06.06.06.45.25 des > Edit src/lib/libfetch/http.c > Add delta 1.13.2.23 2003.06.06.06.45.25 des > Edit src/sbin/vinum/commands.c > Add delta 1.31.2.6 2003.06.06.05.13.29 grog > Edit src/sys/dev/ata/ata-pci.c > Add delta 1.32.2.15 2003.06.06.13.27.05 fjoe > Edit src/sys/kern/init_main.c > Add delta 1.134.2.8 2003.06.06.20.21.32 tegge > Edit src/sys/kern/kern_descrip.c > Add delta 1.81.2.17 2003.06.06.20.21.32 tegge > Edit src/sys/kern/kern_fork.c > Add delta 1.72.2.13 2003.06.06.20.21.32 tegge > Edit src/sys/sys/filedesc.h > Add delta 1.19.2.5 2003.06.06.20.21.32 tegge > Edit src/sys/sys/proc.h > Add delta 1.99.2.9 2003.06.06.20.21.32 tegge > Edit src/usr.bin/fetch/fetch.1 > Add delta 1.33.2.12 2003.06.06.06.48.42 des > Edit src/usr.bin/fetch/fetch.c > Add delta 1.10.2.21 2003.06.06.06.48.42 des > >06.07 --> KO ( date=2003.06.07.00.00.00 ) > >So I presume the panics are related to these changes, even if I can't say >why they happens ... >Any idea ?Ok digging any further. It is not apache that panics but "apachectl" and in particular it is the line (at least this line panics, dunno if there are others :-) : eval `limits -e -U www` >/dev/null 2>&1 Infact making a little script in this way: ---> begin <--- #!/bin/sh eval `limits -e -U www` >/dev/null 2>&1 ---> end <--- panics the box too ... Best Regards, Gianmarco Giovannelli , "Unix expert since yesterday" http://www.gufi.org/~gmarco
At 30/06/2003, Gianmarco Giovannelli wrote:>Infact making a little script in this way: >---> begin <--- >#!/bin/sh > >eval `limits -e -U www` >/dev/null 2>&1 >---> end <--- > >panics the box too ...If you use csh instead of sh it works ... This crashes... ---> begin <--- #!/bin/sh limits -e -U www --> end <--- This doesn't crash ---> begin <--- #!/bin/csh limits -e -U www ---> end <--- A little of kernel debugging: freebsd:/usr/obj/usr/src/sys/FREEBSD# gdb -k GNU gdb 4.18 (FreeBSD) Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-freebsd". (kgdb) symbol-file kernel.debug Reading symbols from kernel.debug...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 in elfstab_build_psymtabs Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933 in fill_symbuf done. [...] orm0: <Option ROM> at iomem 0xcc800-0xd47ff on isa0 fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> on fdc0 drive 0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0 kbd0 at atkbd0 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/16 bytes threshold lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port DUMMYNET initialized (011031) ipfw2 initialized, divert enabled, rule-based forwarding enabled, default to deny, logging unlimited ad0: 19092MB <ST320413A> [38792/16/63] at ata0-master PIO4 acd0: CDROM <CD-ROM Drive/F5E> at ata1-master PIO4 ata1-slave: <IOMEGA ZIP 100 ATAPI/13.A> - NO DRIVER! Mounting root from ufs:/dev/ad0s1a Fatal trap 12: page fault while in kernel mode fault virtual address = 0x14 fault code = supervisor read, page not present instruction pointer = 0x8:0xc124ea3f stack pointer = 0x10:0xd16d9ce8 frame pointer = 0x10:0xd16d9e18 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 121 (limits) interrupt mask = none trap number = 12 panic: page fault syncing disks... 12 12 12 2 1 1 1 done Uptime: 46s dumping to dev #ad/0x20001, offset 524416 dump ata0: resetting devices .. done 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 --- #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) where #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 #1 0xc0169bd3 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316 #2 0xc0169ff8 in poweroff_wait (junk=0xc026090c, howto=-1071250417) at /usr/src/sys/kern/kern_shutdown.c:595 #3 0xc0220c6e in trap_fatal (frame=0xd16d9ca8, eva=20) at /usr/src/sys/i386/i386/trap.c:974 #4 0xc0220941 in trap_pfault (frame=0xd16d9ca8, usermode=0, eva=20) at /usr/src/sys/i386/i386/trap.c:867 #5 0xc02204ff in trap (frame={tf_fs = -1072103408, tf_es = -781582320, tf_ds = 16, tf_edi = -1054504672, tf_esi = -872841856, tf_ebp = -781345256, tf_isp = -781345580, tf_ebx = -872841856, tf_edx = 3, tf_ecx = -872841856, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1054545345, tf_cs = 8, tf_eflags = 66178, tf_esp = -781345068, tf_ss = -872841856}) at /usr/src/sys/i386/i386/trap.c:466 #6 0xc124ea3f in ?? () #7 0xc124f2a8 in ?? () #8 0xc019edec in vn_read (fp=0xc1315b00, uio=0xd16d9ed4, cred=0xc125d400, flags=0, p=0xcbf97be0) at vnode_if.h:334 #9 0xc0178924 in dofileread (p=0xcbf97be0, fp=0xc1315b00, fd=3, buf=0x8050000, nbyte=4096, offset=-1, flags=0) at /usr/src/sys/sys/file.h:147 #10 0xc01787ea in read (p=0xcbf97be0, uap=0xd16d9f80) at /usr/src/sys/kern/sys_generic.c:117 #11 0xc0220f1d in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 672172128, tf_esi = 672172128, tf_ebp = -1077939296, tf_isp = -781344812, tf_ebx = 672096940, tf_edx = 672040824, tf_ecx = 134545408, tf_eax = 3, tf_trapno = 12, tf_err = 2, tf_eip = 672050184, tf_cs = 31, tf_eflags = 663, tf_esp = -1077939340, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175 #12 0xc02144b5 in Xint0x80_syscall () #13 0x280d8b92 in ?? () #14 0x280d0ae8 in ?? () #15 0x8049dab in ?? () #16 0x804951e in ?? () #17 0x8048caa in ?? () I am really not able to provide any further information because my kernel knowledge is soo small. If someone need something else, please ask (providing a step by step istruction :-) Thanks... Best Regards, Gianmarco Giovannelli , "Unix expert since yesterday" http://www.gufi.org/~gmarco
gmarco@scotty.masternet.it said:>06.06 --> ok ( date=2003.06.06.00.00.00 ) >06.07 --> KO ( date=2003.06.07.00.00.00 )> eval `limits -e -U www` >/dev/null 2>&1 > panics the box too ...The only commit in that window that affects the kernel is Tor's filedesc_to_leader stuff. A quick eyeball doesn't show any problems with the ulimit code in these commits, but I'm hardly an expert on this. Perhaps Tor can comment? [btw: the PR referenced in these commits (kern/50923) is still open.]