Pawe³ Ma³achowski
2004-Mar-02 13:39 UTC
Using read-only NULLFS leads to panic. gdb output included, easy to reproduce.
>Submitter-Id: current-users >Originator: Pawel Malachowski >Organization: ZiN >Confidential: no >Synopsis: Using read-only NULLFS leads to panic. gdb output included, easy to reproduce. >Severity: serious >Priority: medium >Category: kern >Class: sw-bug >Release: FreeBSD 4.7-RELEASE-p25 i386 >Environment:RELENG_4>Description:I know NULLFS is documented as broken and incoming PRs are usually put in suspended state, awaiting a patch. However, there are people claiming that using NULLFS in read-only mode is safe. It seems, they are wrong. I'm not too familiar with debugging, however I decided to use my free time and try to provide more than backtrace, in hope someone will take a look at this for a while (maybe it is trivial to fix?). Environmnet: (A) FreeBSD 4.9-RELEASE, null.ko. (B) FreeBSD 4.9-STABLE, NULLFS, almost GENERIC (+IPFIREWALL, IPFILTER...) (C) FreeBSD 4.8-RELEASE, GENERIC, nullfs.ko (+ipfw.ko) Original problem touched me on machine A: % mount | grep -c 'null, local, read-only' 23 It usually comes at night, when cron is doing its job, especially periodic tasks. However, I took machine B (completly different, pure routing) and C (GENERIC+debug), and successfully reproduced this crash with identical backtrace this way: mount_null -o ro /usr/ports /mnt/1 mount_null -o ro /usr/ports /mnt/2 mount_null -o ro /usr/ports /mnt/3 find /usr/ports -type f -perm -u+s & find /usr/ports -type f -perm -u+s & ... find /mnt/1 -type f -perm -u+s & find /mnt/1 -type f -perm -u+s & ... find /mnt/2 -type f -perm -u+s & find /mnt/2 -type f -perm -u+s & ... (Machine C crashed after few minutes). (C) Fatal trap 12: page fault while in kernel mode fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x8:0xc0255ab7 stack pointer = 0x10:0xcbb38e90 frame pointer = 0x10:0xcbb38ea4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 58363 (find) interrupt mask = none trap number = 12 panic: page fault syncing disks... 65 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 giving up on 1 buffers Uptime: 24d9h54m57s (kgdb) add-symbol-file /sys/modules/nullfs/null.ko 0xC1424388 add symbol table from file "/sys/modules/nullfs/null.ko" at text_addr = 0xc1424388? (y or n) y Reading symbols from /sys/modules/nullfs/null.ko...done. (kgdb) bt #0 dumpsys () at ../../kern/kern_shutdown.c:487 #1 0xc0227653 in boot (howto=256) at ../../kern/kern_shutdown.c:316 #2 0xc0227a78 in poweroff_wait (junk=0xc0421bec, howto=-1069410545) at ../../kern/kern_shutdown.c:595 #3 0xc03a522e in trap_fatal (frame=0xcbb38e50, eva=4) at ../../i386/i386/trap.c:974 #4 0xc03a4f01 in trap_pfault (frame=0xcbb38e50, usermode=0, eva=4) at ../../i386/i386/trap.c:867 #5 0xc03a4abf in trap (frame={tf_fs = 65552, tf_es = 16842768, tf_ds = -877461488, tf_edi = -877520608, tf_esi = -875975552, tf_ebp = -877424988, tf_isp = -877425028, tf_ebx = 0, tf_edx = 6, tf_ecx = -877520608, tf_eax = -877520608, tf_trapno = 12, tf_err = 0, tf_eip = -1071293769, tf_cs = 8, tf_eflags = 66178, tf_esp = -1054023424, tf_ss = 58363}) at ../../i386/i386/trap.c:466 #6 0xc0255ab7 in vput (vp=0x0) at ../../kern/vfs_subr.c:1608 #7 0xc14252e2 in null_inactive (ap=0xcbb38ee4) at /usr/src/sys/modules/nullfs/../../miscfs/nullfs/null_vnops.c:728 #8 0xc0255a57 in vrele (vp=0xcbc9ac80) at vnode_if.h:815 #9 0xc0257e47 in fchdir (p=0xcbb21920, uap=0xcbb38f80) at ../../kern/vfs_syscalls.c:842 #10 0xc03a54dd in syscall2 (frame={tf_fs = 134545455, tf_es = 47, tf_ds = -1078001617, tf_edi = 134626560, tf_esi = 5, tf_ebp = -1077938908, tf_isp = -877424684, tf_ebx = 672079852, tf_edx = 134561920, tf_ecx = 672154432, tf_eax = 13, tf_trapno = 7, tf_err = 2, tf_eip = 671764044, tf_cs = 31, tf_eflags = 663, tf_esp = -1077939048, tf_ss = 47}) at ../../i386/i386/trap.c:1175 #11 0xc03962f5 in Xint0x80_syscall () #12 0x280a074d in ?? () (kgdb) frame 0 #0 dumpsys () at ../../kern/kern_shutdown.c:487 487 if (dumping++) { (kgdb) up 6 #6 0xc0255ab7 in vput (vp=0x0) at ../../kern/vfs_subr.c:1608 1608 struct proc *p = curproc; /* XXX */ (kgdb) l 1603 1604 void 1605 vput(vp) 1606 struct vnode *vp; 1607 { 1608 struct proc *p = curproc; /* XXX */ 1609 1610 KASSERT(vp != NULL, ("vput: null vp")); 1611 1612 simple_lock(&vp->v_interlock); (kgdb) p vp $1 = (struct vnode *) 0x0 (kgdb) up #7 0xc14252e2 in null_inactive (ap=0xcbb38ee4) at /usr/src/sys/modules/nullfs/../../miscfs/nullfs/null_vnops.c:728 728 vput(lowervp); (kgdb) l 723 if (vp->v_vnlock != NULL) { 724 vp->v_vnlock = &xp->null_lock; /* we no longer share the lock */ 725 } else 726 VOP_UNLOCK(vp, LK_THISLAYER, p); 727 728 vput(lowervp); 729 /* 730 * Now it is safe to drop references to the lower vnode. 731 * VOP_INACTIVE() will be called by vrele() if necessary. 732 */ (kgdb) p lowervp $2 = (struct vnode *) 0x0 (kgdb) l - 713 struct vnode *vp = ap->a_vp; 714 struct proc *p = ap->a_p; 715 struct null_node *xp = VTONULL(vp); 716 struct vnode *lowervp = xp->null_lowervp; 717 718 lockmgr(&null_hashlock, LK_EXCLUSIVE, NULL, p); 719 LIST_REMOVE(xp, null_hash); 720 lockmgr(&null_hashlock, LK_RELEASE, NULL, p); 721 722 xp->null_lowervp = NULLVP; (kgdb) p *xp $4 = {null_lock = {lk_interlock = {lock_data = -1054640128}, lk_flags = 64, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_prio = 8, lk_wmesg = 0xc142548d "nullnode", lk_timo = 0, lk_lockholder = -1}, null_vnlock = 0x0, null_hash = {le_next = 0x0, le_prev = 0xc12c4de4}, null_lowervp = 0x0, null_vnode = 0xcbc9ac80} (kgdb) p xp->null_lowervp $5 = (struct vnode *) 0x0 (kgdb) p vp $7 = (struct vnode *) 0xcbc9ac80 (kgdb) p vp->v_data $8 = (void *) 0xc12ce100 (kgdb) p (struct null_node) vp->v_data $10 = {null_lock = {lk_interlock = {lock_data = -1054023424}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = -875975424, lk_exclusivecount = -21376, lk_prio = -13367, lk_wmesg = 0x0, lk_timo = 0, lk_lockholder = 0}, null_vnlock = 0x0, null_hash = {le_next = 0x0, le_prev = 0x0}, null_lowervp = 0x0, null_vnode = 0x0} (kgdb) p ((struct null_node)vp->v_data)->null_lowervp $11 = (struct vnode *) 0x0 (kgdb) up #9 0xc0257e47 in fchdir (p=0xcbb21920, uap=0xcbb38f80) at ../../kern/vfs_syscalls.c:842 842 vrele(fdp->fd_cdir); (kgdb) l 837 if (error) { 838 vput(vp); 839 return (error); 840 } 841 VOP_UNLOCK(vp, 0, p); 842 vrele(fdp->fd_cdir); 843 fdp->fd_cdir = vp; 844 return (0); 845 } 846 (kgdb) p (struct null_node) fdp->fd_cdir->v_data $16 = {null_lock = {lk_interlock = {lock_data = -1054023424}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = -875975424, lk_exclusivecount = -21376, lk_prio = -13367, lk_wmesg = 0x0, lk_timo = 0, lk_lockholder = 0}, null_vnlock = 0x0, null_hash = {le_next = 0x0, le_prev = 0x0}, null_lowervp = 0x0, null_vnode = 0x0} (kgdb) l fchdir 806 fchdir(p, uap) 807 struct proc *p; 808 struct fchdir_args /* { 809 syscallarg(int) fd; 810 } */ *uap; 811 { 812 register struct filedesc *fdp = p->p_fd; 813 struct vnode *vp, *tdp; 814 struct mount *mp; 815 struct file *fp; (kgdb) p (struct null_node) p->p_fd->fd_cdir->v_data $20 = {null_lock = {lk_interlock = {lock_data = -1054023424}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = -875975424, lk_exclusivecount = -21376, lk_prio = -13367, lk_wmesg = 0x0, lk_timo = 0, lk_lockholder = 0}, null_vnlock = 0x0, null_hash = {le_next = 0x0, le_prev = 0x0}, null_lowervp = 0x0, null_vnode = 0x0} (kgdb) p *p $22 = {p_procq = {tqe_next = 0xcbb20f60, tqe_prev = 0xc04a97d0}, p_list = { le_next = 0xcbb20f60, le_prev = 0xc04a9778}, p_cred = 0xc0f731e0, p_fd = 0xc10ee500, p_stats = 0xcbb36cd0, p_limit = 0xc11e9e00, p_upages_obj = 0xc049b5c0, p_procsig = 0xc1387880, p_flag = 16390, p_stat = 2 '\002', p_pad1 = "\000\000", p_pid = 58363, p_hash = {le_next = 0x0, le_prev = 0xc0a815ec}, p_pglist = {le_next = 0x0, le_prev = 0xc13ecc28}, p_pptr = 0xcbb1fd80, p_sibling = {le_next = 0xcbb20f60, le_prev = 0xcbb1fdd0}, p_children = {lh_first = 0x0}, p_ithandle = {callout = 0xc2befd50}, p_oppid = 0, p_dupfd = 0, p_vmspace = 0xcbb52880, p_estcpu = 295, p_cpticks = 75, p_pctcpu = 1182, p_wchan = 0x0, p_wmesg = 0xc04113ea "inode", p_swtime = 54, p_slptime = 0, p_realtimer = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}, p_runtime = 5487340, p_uu = 0, p_su = 136, p_iu = 0, p_uticks = 99, p_sticks = 2561, p_iticks = 7, p_traceflag = 0, p_tracep = 0x0, p_siglist = {__bits = {0, 0, 0, 0}}, p_textvp = 0xcb96f300, p_lock = 0 '\000', p_oncpu = 0 '\000', p_lastcpu = 0 '\000', p_rqindex = 2 '\002', p_locks = -175, p_simple_locks = 0, p_stops = 0, p_stype = 0, p_step = 0 '\000', p_pfsflags = 0 '\000', p_pad3 = "\000", p_retval = {0, 134561920}, p_sigiolst = {slh_first = 0x0}, p_sigparent = 20, p_oldsigmask = {__bits = {0, 0, 0, 0}}, p_sig = 0, p_code = 0, p_klist = {slh_first = 0x0}, p_sigmask = {__bits = {0, 0, 0, 0}}, p_sigstk = { ss_sp = 0x0, ss_size = 0, ss_flags = 4}, p_priority = 8 '\b', p_usrpri = 86 'V', p_nice = 0 '\000', p_comm = "find\000n\000\000\000\000\000\000\000\000\000\000", p_pgrp = 0xc13ecc20, p_sysent = 0xc044b420, p_rtprio = {type = 1, prio = 0}, p_prison = 0x0, p_args = 0xc12dc300, p_addr = 0xcbb36000, p_md = { md_regs = 0xcbb38fa8}, p_xstat = 0, p_acflag = 2, p_ru = 0x0, p_nthreads = 0, p_aioinfo = 0x0, p_wakeup = 0, p_peers = 0x0, p_leader = 0xcbb21920, p_asleep = { as_priority = 0, as_timo = 0}, p_emuldata = 0x0} (kgdb) Why is null_lowervp NULL? It may be significant that problem appears when I search non-null /usr/ports and null /mnt/x at the same time. It may be also interesting, on machine B there were about 30 find(1) processess around once a time, and all of them stuck into inode state, becoming zombie. Also new process were not able to go into /usr/ports (`cd /usr/ports' -> frozen shell). After performing reboot(8) machine failed to reboot because of these inode-state processess. Power-off/on cycle was necessery... Other panic messages: (A, this _one_ is less common) Fatal trap 12: page fault while in kernel mode fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x8:0xc02766eb stack pointer = 0x10:0xe9589dd0 frame pointer = 0x10:0xe9589de4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 80250 (cron) interrupt mask = none trap number = 12 panic: page fault syncing disks... 28 3 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 giving up on 1 buffers Uptime: 2d20h42m48s (kgdb) add-symbol-file /sys/modules/nullfs/null.ko 0xC3811390 add symbol table from file "/sys/modules/nullfs/null.ko" at text_addr = 0xc3811390? (y or n) y Reading symbols from /sys/modules/nullfs/null.ko...done. (kgdb) bt #0 dumpsys () at ../../kern/kern_shutdown.c:487 #1 0xc0247b4b in boot (howto=256) at ../../kern/kern_shutdown.c:316 #2 0xc0247f70 in poweroff_wait (junk=0xc044a62c, howto=-1069244113) at ../../kern/kern_shutdown.c:595 #3 0xc03c2dba in trap_fatal (frame=0xe9589d90, eva=4) at ../../i386/i386/trap.c:974 #4 0xc03c2a8d in trap_pfault (frame=0xe9589d90, usermode=0, eva=4) at ../../i386/i386/trap.c:867 #5 0xc03c264b in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = -388392512, tf_esi = -374358784, tf_ebp = -380068380, tf_isp = -380068420, tf_ebx = 0, tf_edx = 6, tf_ecx = -388392512, tf_eax = -388392512, tf_trapno = 12, tf_err = 0, tf_eip = -1071159573, tf_cs = 8, tf_eflags = 66182, tf_esp = -1007055424, tf_ss = 80250}) at ../../i386/i386/trap.c:466 #6 0xc02766eb in vput (vp=0x0) at ../../kern/vfs_subr.c:1629 #7 0xc38122ea in null_inactive (ap=0xe9589e24) at /src/sys/modules/nullfs/../../miscfs/nullfs/null_vnops.c:728 #8 0xc027668b in vrele (vp=0xe9afbd00) at vnode_if.h:815 #9 0xc027cf23 in vn_close (vp=0xe9afbd00, flags=1, cred=0xc54d3100, p=0xe8d999c0) at ../../kern/vfs_vnops.c:235 #10 0xc027d843 in vn_closefile (fp=0xc4f78ac0, p=0xe8d999c0) at ../../kern/vfs_vnops.c:693 #11 0xc023d6c3 in fdrop (fp=0xc4f78ac0, p=0xe8d999c0) at ../../sys/file.h:218 #12 0xc023d60c in closef (fp=0xc4f78ac0, p=0xe8d999c0) at ../../kern/kern_descrip.c:1441 #13 0xc023c743 in close (p=0xe8d999c0, uap=0xe9589f80) at ../../kern/kern_descrip.c:623 #14 0xc03c3069 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134574392, tf_esi = 1, tf_ebp = -1077941168, tf_isp = -380067884, tf_ebx = 672113388, tf_edx = 134574080, tf_ecx = 134574080, tf_eax = 6, tf_trapno = 12, tf_err = 2, tf_eip = 672066564, tf_cs = 31, tf_eflags = 643, tf_esp = -1077941212, tf_ss = 47}) at ../../i386/i386/trap.c:1175 #15 0xc03b40e5 in Xint0x80_syscall () #16 0x280df523 in ?? () (kgdb) up 6 #6 0xc02766eb in vput (vp=0x0) at ../../kern/vfs_subr.c:1629 1629 struct proc *p = curproc; /* XXX */ (kgdb) l 1624 1625 void 1626 vput(vp) 1627 struct vnode *vp; 1628 { 1629 struct proc *p = curproc; /* XXX */ 1630 1631 KASSERT(vp != NULL, ("vput: null vp")); 1632 1633 simple_lock(&vp->v_interlock); (kgdb) p vp $1 = (struct vnode *) 0x0 (kgdb) p (struct null_node) vp->v_data $2 = {null_lock = {lk_interlock = {lock_data = -1007055424}, lk_flags = 0, lk_sharecount = 0, lk_waitcount = -374358656, lk_exclusivecount = -17152, lk_prio = -5713, lk_wmesg = 0x0, lk_timo = 0, lk_lockholder = 0}, null_vnlock = 0x0, null_hash = {le_next = 0x0, le_prev = 0x0}, null_lowervp = 0x0, null_vnode = 0x0} (kgdb) up #9 0xc027cf23 in vn_close (vp=0xe9afbd00, flags=1, cred=0xc54d3100, p=0xe8d999c0) at ../../kern/vfs_vnops.c:235 235 vrele(vp); (kgdb) l 230 int error; 231 232 if (flags & FWRITE) 233 vp->v_writecount--; 234 error = VOP_CLOSE(vp, flags, cred, p); 235 vrele(vp); 236 return (error); 237 } 238 239 static __inline (kgdb) up #10 0xc027d843 in vn_closefile (fp=0xc4f78ac0, p=0xe8d999c0) at ../../kern/vfs_vnops.c:693 693 return (vn_close(((struct vnode *)fp->f_data), fp->f_flag, (kgdb) l 688 struct file *fp; 689 struct proc *p; 690 { 691 692 fp->f_ops = &badfileops; 693 return (vn_close(((struct vnode *)fp->f_data), fp->f_flag, 694 fp->f_cred, p)); 695 } 696 697 static int (kgdb) p (struct vnode) fp->f_data $11 = {v_flag = 3920608512, v_usecount = 0, v_writecount = 0, v_holdcnt = 858863156, v_id = 0, v_mount = 0x0, v_op = 0xc34adbc8, v_freelist = { tqe_next = 0xc4fe7c00, tqe_prev = 0xc3fa04c8}, v_nmntvnodes = {tqe_next = 0x0, tqe_prev = 0xe9032180}, v_cleanblkhd = {tqh_first = 0xe905a680, tqh_last = 0xe9032100}, v_dirtyblkhd = {tqh_first = 0x33730a00, tqh_last = 0x6d373639}, v_synclist = {le_next = 0x67706a2e, le_prev = 0x0}, v_numoutput = -385670912, v_type = VNON, v_un = {vu_mountedhere = 0x0, vu_socket = 0x0, vu_spec = {vu_specinfo = 0x0, vu_specnext = { sle_next = 0x67616d00}}, vu_fifoinfo = 0x0}, v_lease = 0x0, v_lastw = -1018520864, v_cstart = 0, v_lasta = -994427576, v_clen = -986729152, v_object = 0xc3e98450, v_interlock = {lock_data = -374519936}, v_vnlock = 0x0, v_tag = 1747847424, v_data = 0x63636174, v_cache_src = {lh_first = 0x737365}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0x0}, v_dd = 0x0, v_ddid = 1747873904, v_pollinfo = {vpi_lock = {lock_data = 1093599266}, vpi_selinfo = {si_pid = 0, si_note = {slh_first = 0xc3a6fd00}, si_flags = 4352}, vpi_events = -28088, vpi_revents = -15367}, v_vxproc = 0x0} (kgdb) p (struct null_node)((struct vnode) fp->f_data)->v_data $13 = {null_lock = {lk_interlock = {lock_data = 1667457396}, lk_flags = 7566181, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_prio = 0, lk_wmesg = 0x682e7070 <Address 0x682e7070 out of bounds>, lk_timo = 1093599266, lk_lockholder = 0}, null_vnlock = 0xc3a6fd00, null_hash = {le_next = 0xc4b71100, le_prev = 0xc3f99248}, null_lowervp = 0x0, null_vnode = 0xe9baaf80} (kgdb) up #12 0xc023d60c in closef (fp=0xc4f78ac0, p=0xe8d999c0) at ../../kern/kern_descrip.c:1441 1441 return (fdrop(fp, p)); (kgdb) l 1436 wakeup(fdtol); 1437 } 1438 } 1439 } 1440 } 1441 return (fdrop(fp, p)); 1442 } 1443 1444 int 1445 fdrop(fp, p) (kgdb) p (struct null_node)((struct vnode) fp->f_data)->v_data $15 = {null_lock = {lk_interlock = {lock_data = 1667457396}, lk_flags = 7566181, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_prio = 0, lk_wmesg = 0x682e7070 <Address 0x682e7070 out of bounds>, lk_timo = 1093599266, lk_lockholder = 0}, null_vnlock = 0xc3a6fd00, null_hash = {le_next = 0xc4b71100, le_prev = 0xc3f99248}, null_lowervp = 0x0, null_vnode = 0xe9baaf80} (kgdb) up #13 0xc023c743 in close (p=0xe8d999c0, uap=0xe9589f80) at ../../kern/kern_descrip.c:623 623 error = closef(fp, p); (kgdb) l 618 fdp->fd_lastfile--; 619 if (fd < fdp->fd_freefile) 620 fdp->fd_freefile = fd; 621 if (fd < fdp->fd_knlistsize) 622 knote_fdclose(p, fd); 623 error = closef(fp, p); 624 if (holdleaders) { 625 fdp->fd_holdleaderscount--; 626 if (fdp->fd_holdleaderscount == 0 && 627 fdp->fd_holdleaderswakeup != 0) { (kgdb) p (struct null_node)((struct vnode) fp->f_data)->v_data $18 = {null_lock = {lk_interlock = {lock_data = 1667457396}, lk_flags = 7566181, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_prio = 0, lk_wmesg = 0x682e7070 <Address 0x682e7070 out of bounds>, lk_timo = 1093599266, lk_lockholder = 0}, null_vnlock = 0xc3a6fd00, null_hash = {le_next = 0xc4b71100, le_prev = 0xc3f99248}, null_lowervp = 0x0, null_vnode = 0xe9baaf80} (kgdb) up #14 0xc03c3069 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134574392, tf_esi = 1, tf_ebp = -1077941168, tf_isp = -380067884, tf_ebx = 672113388, tf_edx = 134574080, tf_ecx = 134574080, tf_eax = 6, tf_trapno = 12, tf_err = 2, tf_eip = 672066564, tf_cs = 31, tf_eflags = 643, tf_esp = -1077941212, tf_ss = 47}) at ../../i386/i386/trap.c:1175 1175 error = (*callp->sy_call)(p, args); (A) Fatal trap 12: page fault while in kernel mode fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x8:0xc02766eb stack pointer = 0x10:0xe8dcfe90 frame pointer = 0x10:0xe8dcfea4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 91056 (find) interrupt mask = none trap number = 12 panic: page fault syncing disks... 73 27 1 1 1 1 1 1 1 5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 giving up on 1 buffers Uptime: 5d8h9m51s (kgdb) bt #0 dumpsys () at ../../kern/kern_shutdown.c:487 #1 0xc0247b4b in boot (howto=256) at ../../kern/kern_shutdown.c:316 #2 0xc0247f70 in poweroff_wait (junk=0xc044a62c, howto=-1069244113) at ../../kern/kern_shutdown.c:595 #3 0xc03c2dba in trap_fatal (frame=0xe8dcfe50, eva=4) at ../../i386/i386/trap.c:974 #4 0xc03c2a8d in trap_pfault (frame=0xe8dcfe50, usermode=0, eva=4) at ../../i386/i386/trap.c:867 #5 0xc03c264b in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = -388593440, tf_esi = -373419328, tf_ebp = -388170076, tf_isp = -388170116, tf_ebx = 0, tf_edx = 6, tf_ecx = -388593440, tf_eax = -388593440, tf_trapno = 12, tf_err = 0, tf_eip = -1071159573, tf_cs = 8, tf_eflags = 66178, tf_esp = -1013564992, tf_ss = 91056}) at ../../i386/i386/trap.c:466 #6 0xc02766eb in vput (vp=0x0) at ../../kern/vfs_subr.c:1629 #7 0xc38262ea in ?? () #8 0xc027668b in vrele (vp=0xe9be12c0) at vnode_if.h:815 #9 0xc0278a83 in fchdir (p=0xe8d688e0, uap=0xe8dcff80) at ../../kern/vfs_syscalls.c:843 #10 0xc03c3069 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134623232, tf_esi = 5, tf_ebp = -1077937660, tf_isp = -388169772, tf_ebx = 672080620, tf_edx = 134557696, tf_ecx = 672155200, tf_eax = 13, tf_trapno = 7, tf_err = 2, tf_eip = 671764800, tf_cs = 31, tf_eflags = 659, tf_esp = -1077937800, tf_ss = 47}) at ../../i386/i386/trap.c:1175 #11 0xc03b40e5 in Xint0x80_syscall () #12 0x280a0a41 in ?? () (B) Fatal trap 12: page fault while in kernel mode fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x8:0xc02766eb stack pointer = 0x10:0xe8dcfe90 frame pointer = 0x10:0xe8dcfea4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 91056 (find) interrupt mask = none trap number = 12 panic: page fault syncing disks... 73 27 1 1 1 1 1 1 1 5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 giving up on 1 buffers Uptime: 5d8h9m51s (B) instruction pointer = 0x8:0xc0269bc7 stack pointer = 0x10:0xd5d45e90 frame pointer = 0x10:0xd5d45ea4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 558 (find) interrupt mask = none trap number = 12 panic: page fault syncing disks... 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 giving up on 1 buffers Uptime: 21m42s>How-To-Repeat:mount_null -o ro /usr/ports /mnt/1 mount_null -o ro /usr/ports /mnt/2 mount_null -o ro /usr/ports /mnt/3 find /usr/ports -type f -perm -u+s & find /usr/ports -type f -perm -u+s & ... find /mnt/1 -type f -perm -u+s & find /mnt/1 -type f -perm -u+s & ... find /mnt/2 -type f -perm -u+s & find /mnt/2 -type f -perm -u+s & ...>Fix:Unknown.
James Read
2004-Mar-02 15:04 UTC
Using read-only NULLFS leads to panic. gdb output included, easy toreproduce.
> find /usr/ports -type f -perm -u+s & > find /usr/ports -type f -perm -u+s & > ... > find /mnt/1 -type f -perm -u+s & > find /mnt/1 -type f -perm -u+s & > ... > find /mnt/2 -type f -perm -u+s & > find /mnt/2 -type f -perm -u+s & > ... > > (Machine C crashed after few minutes).All I can say is that I've had this happen to me before. 'Me too' The ports were mounted in exact the same way, but with rw instead of ro. Also a few jails were running at the time, infact 3 were. All had /usr/ports mount_null'ed inside there jail. After running find from the usual periodic scripts it brought the machine down every time it ran that script. What I did to stop the box from panicing all the time at that one particular place, was just to disable the script that does the 'find' / locate. Off the top of my head I think it was /etc/periodic/weekly/310.locate. Once this was disabled (from inside and outside the jails), I didnt get any more panics from 'find'. Granted this isnt a fix, but it did save me from panic hell every week. If there is a better way / another way to 'mount_null' /usr/ports (or any other mount point for that matter) to other places in/on the filesystem, by using NFS or other such things, then speak up! I don't like getting panics more then anyone else does ;> Regards, James. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. Mailscanner thanks transtec Computers for their support.
Tim Robbins
2004-Mar-02 16:23 UTC
Using read-only NULLFS leads to panic. gdb output included, easy to reproduce.
On Tue, Mar 02, 2004 at 10:39:36PM +0100, Pawe Maachowski wrote:> > I know NULLFS is documented as broken and incoming PRs are usually put > in suspended state, awaiting a patch. > However, there are people claiming that using NULLFS in read-only mode > is safe. It seems, they are wrong. [...] > Environmnet: > (A) FreeBSD 4.9-RELEASE, null.ko. > (B) FreeBSD 4.9-STABLE, NULLFS, almost GENERIC (+IPFIREWALL, IPFILTER...) > (C) FreeBSD 4.8-RELEASE, GENERIC, nullfs.ko (+ipfw.ko) [...]There are known bugs in nullfs in all 4.x releases to date, and in 5.0. If I have time, I may MFC the fixes some time before 4.10 is released. Can you reproduce these problems on 5.1 or 5.2? Tim
Pawel Jakub Dawidek
2004-Mar-07 15:22 UTC
kern/63662: Using read-only NULLFS leads to panic. gdb output included, easy to reproduce.
On Tue, Mar 02, 2004 at 10:39:36PM +0100, Pawe? Ma?achowski wrote: +> >Synopsis: Using read-only NULLFS leads to panic. gdb output included, easy to reproduce. I'm not able to reproduce it on -CURRENT. tjr@ was working on nullfs problems (mostly deadlocks), but all his work was done in -CURRENT AFAIR. I'm afraid that noone will dare to touch nullfs in 4.x. -- Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20040308/0653f1cf/attachment.bin