Adrian Chadd
2016-Aug-11 23:37 UTC
Panic in stable/11 (amd64) @r303903: page fault while in kernel mode
Eep. Is this anotehr case where there's a race and ifp is NULL or the ll pointer for ifp is NULL or use-after-free'd? I remember bumping into these here and there because we don't seem to have a well defined lifecycle for lladdr access. ;( -adrian On 10 August 2016 at 12:10, Bryan Drewery <bdrewery at freebsd.org> wrote:> On 8/10/16 9:54 AM, David Wolfskill wrote: >> Happened after a few iterations of {"pkill dhclient" followed by >> "dhclient wlan0"}. >> >> Gory details (both "normal" and gzipped, and including the crash >> dump and crashinfo) are in >> <http://www.catwhisker.org/~david/FreeBSD/stable_11/2016.08.10/>. >> >> Summary: >> Wed Aug 10 15:56:26 UTC 2016 >> >> FreeBSD 11.0-BETA4 FreeBSD 11.0-BETA4 #69 r303902M/303903:1100120: Wed Aug 10 04:00:09 PDT 2016 root at g1-252.catwhisker.org:/common/S3/obj/usr/src/sys/CANARY amd64 >> >> panic: page fault >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> >> >> Fatal trap 12: page fault while in kernel mode >> cpuid = 7; apic id = 07 >> fault virtual address = 0x0 >> fault code = supervisor read data, page not present >> instruction pointer = 0x20:0xffffffff80bdaaa1 >> stack pointer = 0x28:0xfffffe060bc956e0 >> frame pointer = 0x28:0xfffffe060bc957b0 >> code segment = base 0x0, limit 0xfffff, type 0x1b >> = DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags = interrupt enabled, resume, IOPL = 0 >> current process = 20685 (wpa_supplicant) >> trap number = 12 >> panic: page fault >> cpuid = 7 >> KDB: stack backtrace: >> #0 0xffffffff80add787 at kdb_backtrace+0x67 >> #1 0xffffffff80a950e2 at vpanic+0x182 >> #2 0xffffffff80a94f53 at panic+0x43 >> #3 0xffffffff80eead51 at trap_fatal+0x351 >> #4 0xffffffff80eeaf43 at trap_pfault+0x1e3 >> #5 0xffffffff80eea4ec at trap+0x26c >> #6 0xffffffff80ece0d1 at calltrap+0x8 >> #7 0xffffffff80b9811c at ifioctl+0x133c >> #8 0xffffffff80afc914 at kern_ioctl+0x2d4 >> #9 0xffffffff80afc5d1 at sys_ioctl+0x171 >> #10 0xffffffff80eeb6c9 at amd64_syscall+0x4e9 >> #11 0xffffffff80ece3bb at Xfast_syscall+0xfb >> Uptime: 3h0m4s >> ... >> Reading symbols from /boot/kernel/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux64.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/linux64.ko >> #0 doadump (textdump=<value optimized out>) at pcpu.h:221 >> 221 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:221 >> #1 0xffffffff80a94b69 in kern_reboot (howto=260) >> at /usr/src/sys/kern/kern_shutdown.c:366 >> #2 0xffffffff80a9511b in vpanic (fmt=<value optimized out>, >> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 >> #3 0xffffffff80a94f53 in panic (fmt=0x0) >> at /usr/src/sys/kern/kern_shutdown.c:690 >> #4 0xffffffff80eead51 in trap_fatal (frame=0xfffffe060bc95630, eva=0) >> at /usr/src/sys/amd64/amd64/trap.c:841 >> #5 0xffffffff80eeaf43 in trap_pfault (frame=0xfffffe060bc95630, usermode=0) >> at /usr/src/sys/amd64/amd64/trap.c:691 >> #6 0xffffffff80eea4ec in trap (frame=0xfffffe060bc95630) >> at /usr/src/sys/amd64/amd64/trap.c:442 >> #7 0xffffffff80ece0d1 in calltrap () >> at /usr/src/sys/amd64/amd64/exception.S:236 >> #8 0xffffffff80bdaaa1 in ieee80211_ioctl (ifp=0xfffff80007991800, >> cmd=<value optimized out>, data=<value optimized out>) >> at /usr/src/sys/net80211/ieee80211_ioctl.c:3398 > > The code crashing is quite recent: > >> commit c6321695321bae43c0cd024db564c5207a7e8e31 >> Author: avos <avos at ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f> >> Date: Mon May 2 20:46:05 2016 +0000 >> >> net80211: fix MAC address change via SIOCSIFLLADDR ioctl. >> >> Recheck MAC address on SIOCSIFFLAGS; as a result, >> 'ifconfig wlan0 ether <addr>' can be used after interface startup. >> >> PR: 208933 >> >> >> git-svn-id: svn+ssh://svn.freebsd.org/base/head at 298941 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f >> >> diff --git sys/net80211/ieee80211_ioctl.c sys/net80211/ieee80211_ioctl.c >> index c3b02e8..823906b 100644 >> --- sys/net80211/ieee80211_ioctl.c >> +++ sys/net80211/ieee80211_ioctl.c >> @@ -3382,8 +3382,18 @@ ieee80211_ioctl(struct ifnet *ifp, u_long cmd, caddr_t data) >> } >> IEEE80211_UNLOCK(ic); >> /* Wait for parent ioctl handler if it was queued */ >> - if (wait) >> + if (wait) { >> ieee80211_waitfor_parent(ic); >> + >> + /* >> + * Check if the MAC address was changed >> + * via SIOCSIFLLADDR ioctl. >> + */ >> + if ((ifp->if_flags & IFF_UP) == 0 && >> + !IEEE80211_ADDR_EQ(vap->iv_myaddr, IF_LLADDR(ifp))) >> + IEEE80211_ADDR_COPY(vap->iv_myaddr, >> + IF_LLADDR(ifp)); >> + } >> break; >> case SIOCADDMULTI: >> case SIOCDELMULTI: > > >> #9 0xffffffff80b9811c in ifioctl (so=<value optimized out>, >> cmd=<value optimized out>, data=<value optimized out>, >> td=<value optimized out>) at /usr/src/sys/net/if.c:2447 >> #10 0xffffffff80afc914 in kern_ioctl (td=<value optimized out>, >> fd=<value optimized out>, com=2149607696, data=0xfffffe060bc958e0 "wlan0") >> at file.h:327 >> #11 0xffffffff80afc5d1 in sys_ioctl (td=<value optimized out>, >> uap=0xfffffe060bc95a40) at /usr/src/sys/kern/sys_generic.c:743 >> #12 0xffffffff80eeb6c9 in amd64_syscall (td=<value optimized out>, >> traced=<value optimized out>) at subr_syscall.c:135 >> #13 0xffffffff80ece3bb in Xfast_syscall () >> at /usr/src/sys/amd64/amd64/exception.S:396 >> #14 0x00000008015c448a in ?? () >> Previous frame inner to this frame (corrupt stack?) >> Current language: auto; currently minimal >> (kgdb) >> >> This was on my laptop, which I'm actively using at work as I type >> -- though it's now connected via wired NIC (em0). I had experienced >> no trouble with wlan0 at home (before coming in to work) or on the >> bus (en route to work). (I didn't attempt it while cycling to the >> bus stop. :-}) >> >> Also, I had no issues running stable/11 (amd64) @303870 -- either >> at home or at work -- yesterday. On the other hand, this is (so >> far) a one-off, so alleging a "pattern" at this point is not something >> I'm willing to do. >> >> Peace, >> david >> > > > -- > Regards, > Bryan Drewery >
Adrian Chadd
2016-Aug-11 23:37 UTC
Panic in stable/11 (amd64) @r303903: page fault while in kernel mode
.. and maybe we should revert or comment out the code until we figure out what to do about LLADDR checks. (I see this in the detach path too; same kind of race. Sigh.) -adrian On 11 August 2016 at 16:37, Adrian Chadd <adrian.chadd at gmail.com> wrote:> Eep. Is this anotehr case where there's a race and ifp is NULL or the > ll pointer for ifp is NULL or use-after-free'd? > > I remember bumping into these here and there because we don't seem to > have a well defined lifecycle for lladdr access. ;( > > > -adrian > > > On 10 August 2016 at 12:10, Bryan Drewery <bdrewery at freebsd.org> wrote: >> On 8/10/16 9:54 AM, David Wolfskill wrote: >>> Happened after a few iterations of {"pkill dhclient" followed by >>> "dhclient wlan0"}. >>> >>> Gory details (both "normal" and gzipped, and including the crash >>> dump and crashinfo) are in >>> <http://www.catwhisker.org/~david/FreeBSD/stable_11/2016.08.10/>. >>> >>> Summary: >>> Wed Aug 10 15:56:26 UTC 2016 >>> >>> FreeBSD 11.0-BETA4 FreeBSD 11.0-BETA4 #69 r303902M/303903:1100120: Wed Aug 10 04:00:09 PDT 2016 root at g1-252.catwhisker.org:/common/S3/obj/usr/src/sys/CANARY amd64 >>> >>> panic: page fault >>> >>> GNU gdb 6.1.1 [FreeBSD] >>> Copyright 2004 Free Software Foundation, Inc. >>> GDB is free software, covered by the GNU General Public License, and you are >>> welcome to change it and/or distribute copies of it under certain conditions. >>> Type "show copying" to see the conditions. >>> There is absolutely no warranty for GDB. Type "show warranty" for details. >>> This GDB was configured as "amd64-marcel-freebsd"... >>> >>> Unread portion of the kernel message buffer: >>> >>> >>> Fatal trap 12: page fault while in kernel mode >>> cpuid = 7; apic id = 07 >>> fault virtual address = 0x0 >>> fault code = supervisor read data, page not present >>> instruction pointer = 0x20:0xffffffff80bdaaa1 >>> stack pointer = 0x28:0xfffffe060bc956e0 >>> frame pointer = 0x28:0xfffffe060bc957b0 >>> code segment = base 0x0, limit 0xfffff, type 0x1b >>> = DPL 0, pres 1, long 1, def32 0, gran 1 >>> processor eflags = interrupt enabled, resume, IOPL = 0 >>> current process = 20685 (wpa_supplicant) >>> trap number = 12 >>> panic: page fault >>> cpuid = 7 >>> KDB: stack backtrace: >>> #0 0xffffffff80add787 at kdb_backtrace+0x67 >>> #1 0xffffffff80a950e2 at vpanic+0x182 >>> #2 0xffffffff80a94f53 at panic+0x43 >>> #3 0xffffffff80eead51 at trap_fatal+0x351 >>> #4 0xffffffff80eeaf43 at trap_pfault+0x1e3 >>> #5 0xffffffff80eea4ec at trap+0x26c >>> #6 0xffffffff80ece0d1 at calltrap+0x8 >>> #7 0xffffffff80b9811c at ifioctl+0x133c >>> #8 0xffffffff80afc914 at kern_ioctl+0x2d4 >>> #9 0xffffffff80afc5d1 at sys_ioctl+0x171 >>> #10 0xffffffff80eeb6c9 at amd64_syscall+0x4e9 >>> #11 0xffffffff80ece3bb at Xfast_syscall+0xfb >>> Uptime: 3h0m4s >>> ... >>> Reading symbols from /boot/kernel/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux64.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/linux64.ko >>> #0 doadump (textdump=<value optimized out>) at pcpu.h:221 >>> 221 pcpu.h: No such file or directory. >>> in pcpu.h >>> (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:221 >>> #1 0xffffffff80a94b69 in kern_reboot (howto=260) >>> at /usr/src/sys/kern/kern_shutdown.c:366 >>> #2 0xffffffff80a9511b in vpanic (fmt=<value optimized out>, >>> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 >>> #3 0xffffffff80a94f53 in panic (fmt=0x0) >>> at /usr/src/sys/kern/kern_shutdown.c:690 >>> #4 0xffffffff80eead51 in trap_fatal (frame=0xfffffe060bc95630, eva=0) >>> at /usr/src/sys/amd64/amd64/trap.c:841 >>> #5 0xffffffff80eeaf43 in trap_pfault (frame=0xfffffe060bc95630, usermode=0) >>> at /usr/src/sys/amd64/amd64/trap.c:691 >>> #6 0xffffffff80eea4ec in trap (frame=0xfffffe060bc95630) >>> at /usr/src/sys/amd64/amd64/trap.c:442 >>> #7 0xffffffff80ece0d1 in calltrap () >>> at /usr/src/sys/amd64/amd64/exception.S:236 >>> #8 0xffffffff80bdaaa1 in ieee80211_ioctl (ifp=0xfffff80007991800, >>> cmd=<value optimized out>, data=<value optimized out>) >>> at /usr/src/sys/net80211/ieee80211_ioctl.c:3398 >> >> The code crashing is quite recent: >> >>> commit c6321695321bae43c0cd024db564c5207a7e8e31 >>> Author: avos <avos at ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f> >>> Date: Mon May 2 20:46:05 2016 +0000 >>> >>> net80211: fix MAC address change via SIOCSIFLLADDR ioctl. >>> >>> Recheck MAC address on SIOCSIFFLAGS; as a result, >>> 'ifconfig wlan0 ether <addr>' can be used after interface startup. >>> >>> PR: 208933 >>> >>> >>> git-svn-id: svn+ssh://svn.freebsd.org/base/head at 298941 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f >>> >>> diff --git sys/net80211/ieee80211_ioctl.c sys/net80211/ieee80211_ioctl.c >>> index c3b02e8..823906b 100644 >>> --- sys/net80211/ieee80211_ioctl.c >>> +++ sys/net80211/ieee80211_ioctl.c >>> @@ -3382,8 +3382,18 @@ ieee80211_ioctl(struct ifnet *ifp, u_long cmd, caddr_t data) >>> } >>> IEEE80211_UNLOCK(ic); >>> /* Wait for parent ioctl handler if it was queued */ >>> - if (wait) >>> + if (wait) { >>> ieee80211_waitfor_parent(ic); >>> + >>> + /* >>> + * Check if the MAC address was changed >>> + * via SIOCSIFLLADDR ioctl. >>> + */ >>> + if ((ifp->if_flags & IFF_UP) == 0 && >>> + !IEEE80211_ADDR_EQ(vap->iv_myaddr, IF_LLADDR(ifp))) >>> + IEEE80211_ADDR_COPY(vap->iv_myaddr, >>> + IF_LLADDR(ifp)); >>> + } >>> break; >>> case SIOCADDMULTI: >>> case SIOCDELMULTI: >> >> >>> #9 0xffffffff80b9811c in ifioctl (so=<value optimized out>, >>> cmd=<value optimized out>, data=<value optimized out>, >>> td=<value optimized out>) at /usr/src/sys/net/if.c:2447 >>> #10 0xffffffff80afc914 in kern_ioctl (td=<value optimized out>, >>> fd=<value optimized out>, com=2149607696, data=0xfffffe060bc958e0 "wlan0") >>> at file.h:327 >>> #11 0xffffffff80afc5d1 in sys_ioctl (td=<value optimized out>, >>> uap=0xfffffe060bc95a40) at /usr/src/sys/kern/sys_generic.c:743 >>> #12 0xffffffff80eeb6c9 in amd64_syscall (td=<value optimized out>, >>> traced=<value optimized out>) at subr_syscall.c:135 >>> #13 0xffffffff80ece3bb in Xfast_syscall () >>> at /usr/src/sys/amd64/amd64/exception.S:396 >>> #14 0x00000008015c448a in ?? () >>> Previous frame inner to this frame (corrupt stack?) >>> Current language: auto; currently minimal >>> (kgdb) >>> >>> This was on my laptop, which I'm actively using at work as I type >>> -- though it's now connected via wired NIC (em0). I had experienced >>> no trouble with wlan0 at home (before coming in to work) or on the >>> bus (en route to work). (I didn't attempt it while cycling to the >>> bus stop. :-}) >>> >>> Also, I had no issues running stable/11 (amd64) @303870 -- either >>> at home or at work -- yesterday. On the other hand, this is (so >>> far) a one-off, so alleging a "pattern" at this point is not something >>> I'm willing to do. >>> >>> Peace, >>> david >>> >> >> >> -- >> Regards, >> Bryan Drewery >>
David Wolfskill
2016-Aug-11 23:42 UTC
Panic in stable/11 (amd64) @r303903: page fault while in kernel mode
On Thu, Aug 11, 2016 at 04:37:26PM -0700, Adrian Chadd wrote:> Eep. Is this anotehr case where there's a race and ifp is NULL or the > ll pointer for ifp is NULL or use-after-free'd?I don't know, but it's the only panic of a similar nature I've seen -- though since on any given day, I'm usually running FreeBSD freshly-built that morning, "trends" might be a little more difficult to determine.> I remember bumping into these here and there because we don't seem to > have a well defined lifecycle for lladdr access. ;(Well, you're quite welcome to grab the crash data & poke around, or tell me where to look, and I'll do so.....> ... > >> Gory details (both "normal" and gzipped, and including the crash > >> dump and crashinfo) are in > >> <http://www.catwhisker.org/~david/FreeBSD/stable_11/2016.08.10/>. > ...Peace, david -- David H. Wolfskill david at catwhisker.org Those who would murder in the name of God or prophet are blasphemous cowards. See http://www.catwhisker.org/~david/publickey.gpg for my public key. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 603 bytes Desc: not available URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160811/87f79a4e/attachment.sig>