I plan to MFC all of this lovely stuff for 6.1: chesapeake.net/~jroberson/vfsmfc.diff I'm looking for people who are willing to patch their stable boxes and test this. This has the following changes in it: 1) Improved debugging with DEBUG_LOCKS via the new stack(9) api. 2) Fixed an INACTIVE leak. 3) Fixed several unmount races. 4) Fixed several nullfs unmount issues. 5) Some more Giant related VFS fixes and asserts. 6) Fixed the quota deadlock. These problems should be rare enough that most of you have not seen them. So just let me know if this introduces any new problems etc. I will be MFCing within a week. Thanks, Jeff
On Fri, Mar 03, 2006 at 03:41:55PM -0800, Jeff Roberson wrote:> I plan to MFC all of this lovely stuff for 6.1: > > chesapeake.net/~jroberson/vfsmfc.diff > > I'm looking for people who are willing to patch their stable boxes and > test this. This has the following changes in it: > > 1) Improved debugging with DEBUG_LOCKS via the new stack(9) api. > 2) Fixed an INACTIVE leak. > 3) Fixed several unmount races. > 4) Fixed several nullfs unmount issues. > 5) Some more Giant related VFS fixes and asserts. > 6) Fixed the quota deadlock. > > These problems should be rare enough that most of you have not seen them. > So just let me know if this introduces any new problems etc. I will be > MFCing within a week. > > Thanks, > JeffI applied the patch to the today 6-STABLE and now testing it on the (relatively slow, k6/266Mhz) machine, by cvs-ing the sources and building the world. Kernel config is custom (see below issue #2), I added DEBUG_* and WITNESS* options from your patch. Config does not include QUOTAS. 1. The patch breaks the kernel ABI, even for the case when DEBUG_VFS_LOCK is not defined. This is due to changes inside struct mount, adding mnt_ref and rearranging several existing fields. This issue shall be at least mentioned in release notes. 2. I built custom kernel with "options MAC". After some fs activity, I got the LOR: lock order reversal: 1st 0xc1a018f0 vnode interlock (vnode interlock) @ /usr/home/kostik/work/bsd/sys/kern/vfs_subr.c:2449 2nd 0xc0c43144 system map (system map) @ /usr/home/kostik/work/bsd/sys/vm/vm_kern.c:295 KDB: stack backtrace: kdb_backtrace(0,ffffffff,c06676b0,c0667700,c0636024) at 0xc049d3c9 = kdb_backtrace+0x29 witness_checkorder(c0c43144,9,c061fe28,127) at 0xc04a80c2 = witness_checkorder+0x582 _mtx_lock_flags(c0c43144,0,c061fe28,127) at 0xc047b998 = _mtx_lock_flags+0x58 _vm_map_lock(c0c430c0,c061fe28,127) at 0xc059eb46 = _vm_map_lock+0x26 kmem_malloc(c0c430c0,1000,101,c819fbe0,c059679f) at 0xc059e0d2 = kmem_malloc+0x32 page_alloc(c0c4d300,1000,c819fbd3,101,c06a3bf8) at 0xc0596bda = page_alloc+0x1a slab_zalloc(c0c4d300,101,c0c4d300,c0647a64,c0c4e460) at 0xc059679f = slab_zalloc+0x9f uma_zone_slab(c0c4d300,1,c0c4e468,0,c061f05a,8a2) at 0xc0597dec = uma_zone_slab+0xec uma_zalloc_internal(c0c4d300,0,1,0,c0c4dc48) at 0xc0598129 = uma_zalloc_internal+0x29 bucket_alloc(80,1,c0c380a0,0,c19ab6a4) at 0xc0595eac = bucket_alloc+0x2c uma_zfree_arg(c0c4dc00,c19ab6a4,0) at 0xc0598483 = uma_zfree_arg+0x283 mac_labelzone_free(c19ab6a4,c1a01828,e8,c819fc9c,c0565ad2) at 0xc055dab3 = mac_labelzone_free+0x13 mac_vnode_label_free(c19ab6a4,c1a01828,c819fcac,c04d8766,c1a01828) at 0xc0565aaa = mac_vnode_label_free+0x6a mac_destroy_vnode(c1a01828) at 0xc0565ad2 = mac_destroy_vnode+0x12 vdestroy(c1a01828,c1a01828,c819fcec,c04d8142,c1a01828) at 0xc04d8766 = vdestroy+0x1c6 vdropl(c1a01828,7,a8,c0653ee0,c1a01828) at 0xc04dad1e = vdropl+0x3e vlrureclaim(c15e8000,c1529000,c156f000,c04d8360,c156f000) at 0xc04d8142 = vlrureclaim+0x282 vnlru_proc(0,c819fd38,0,c04d8360,0) at 0xc04d84e3 = vnlru_proc+0x183 fork_exit(c04d8360,0,c819fd38) at 0xc046de7d = fork_exit+0x9d fork_trampoline() at 0xc05d33bc = fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xc819fd6c, ebp = 0 --- A patch to fix the LOR (seems to be relevant for CURRENT too): --- sys/kern/vfs_subr.c.orig Sat Mar 4 10:44:47 2006 +++ sys/kern/vfs_subr.c Sat Mar 4 10:45:21 2006 @@ -787,9 +787,6 @@ VNASSERT(bo->bo_dirty.bv_root == NULL, vp, ("dirtyblkroot not NULL")); VNASSERT(TAILQ_EMPTY(&vp->v_cache_dst), vp, ("vp has namecache dst")); VNASSERT(LIST_EMPTY(&vp->v_cache_src), vp, ("vp has namecache src")); -#ifdef MAC - mac_destroy_vnode(vp); -#endif if (vp->v_pollinfo != NULL) { knlist_destroy(&vp->v_pollinfo->vpi_selinfo.si_note); mtx_destroy(&vp->v_pollinfo->vpi_lock); @@ -801,6 +798,9 @@ #endif lockdestroy(vp->v_vnlock); mtx_destroy(&vp->v_interlock); +#ifdef MAC + mac_destroy_vnode(vp); +#endif uma_zfree(vnode_zone, vp); } Up to the moment (uptime 11:09AM up 1:24, 1 user, load averages: 1.32, 1.56, 1.56) everything else seems to be okey. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : lists.freebsd.org/pipermail/freebsd-stable/attachments/20060304/2537dc9d/attachment.bin
Kris,>> If relevant, I may be able to test your patch, but the problem is >> occurring only rarely. Do you have any suggestions for isolating and >> reproducing this bug? > > Run your script in a loop?And I assume I can meaningfully test w/o the rsync call? I.e., just mounting and unmounting the drive over and over again should trigger the error, no? chad
Jeff, Kris Kennaway directed me to this thread from FreeBSD-questions. I am seeing a "panic: unmount: dangling vnode" with 6.0-RELEASE. Here are the relevant threads: 2 probs w/ backup.sh: "Device busy" and "dangling vnode" lists.freebsd.org/pipermail/freebsd-questions/2006-March/114825.html "Panic: unmount: dangling vnode" lists.freebsd.org/pipermail/freebsd-questions/2006-March/115060.html If relevant, I may be able to test your patch, but the problem is occurring only rarely. Do you have any suggestions for isolating and reproducing this bug? Also, we are seeing this problem on a production box. I notice that the patch fixes 6 issues, and apparently "breaks the kernel ABI," which sounds nasty from out here in userland. Any chance of getting a patch that isolates this specific issue? I'll be more likely able to apply such a patch. Our alternative is to simply keep our backup drive always mounted until 6.1 comes out and test your patch then. :^) Thoughts? Thanks for your work on this. Chad Whitacre zetadev.com Jeff Roberson wrote:> I plan to MFC all of this lovely stuff for 6.1: > > chesapeake.net/~jroberson/vfsmfc.diff > > I'm looking for people who are willing to patch their stable boxes and > test this. This has the following changes in it: > > 1) Improved debugging with DEBUG_LOCKS via the new stack(9) api. > 2) Fixed an INACTIVE leak. > 3) Fixed several unmount races. > 4) Fixed several nullfs unmount issues. > 5) Some more Giant related VFS fixes and asserts. > 6) Fixed the quota deadlock. > > These problems should be rare enough that most of you have not seen > them. So just let me know if this introduces any new problems etc. I > will be MFCing within a week. > > Thanks, > Jeff > _______________________________________________ > freebsd-stable@freebsd.org mailing list > lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
On Tue, Mar 07, 2006 at 09:58:35AM -0500, Chad Whitacre wrote:> Kris, > > >>If relevant, I may be able to test your patch, but the problem is > >>occurring only rarely. Do you have any suggestions for isolating and > >>reproducing this bug? > > > >Run your script in a loop? > > And I assume I can meaningfully test w/o the rsync call? I.e., just > mounting and unmounting the drive over and over again should trigger the > error, no?No, the rsync (i.e. activity on the filesystem) is important. Kris -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : lists.freebsd.org/pipermail/freebsd-stable/attachments/20060307/805517b3/attachment.bin
Kris,> No, the rsync (i.e. activity on the filesystem) is important.Yeah, just ran a test w/o it actually. We are planning to run a test w/ some disk activity later this afternoon. I suppose the more activity, the more likely to see the bug, eh? chad
On Tue, Mar 07, 2006 at 12:26:47PM -0500, Chad Whitacre wrote:> Kris, > > >No, the rsync (i.e. activity on the filesystem) is important. > > Yeah, just ran a test w/o it actually. We are planning to run a test w/ > some disk activity later this afternoon. I suppose the more activity, > the more likely to see the bug, eh?Yes. FYI, I am fairly confident this is fixed, because I make extensive use of mount/umount+filesystem activity, and I am no longer seeing problems like this. Kris -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : lists.freebsd.org/pipermail/freebsd-stable/attachments/20060307/0074aa77/attachment.bin
On 3/3/06, Jeff Roberson <jroberson@chesapeake.net> wrote:> I plan to MFC all of this lovely stuff for 6.1: > > chesapeake.net/~jroberson/vfsmfc.diff > > I'm looking for people who are willing to patch their stable boxes and > test this. This has the following changes in it: > > 1) Improved debugging with DEBUG_LOCKS via the new stack(9) api. > 2) Fixed an INACTIVE leak. > 3) Fixed several unmount races. > 4) Fixed several nullfs unmount issues. > 5) Some more Giant related VFS fixes and asserts. > 6) Fixed the quota deadlock. > > These problems should be rare enough that most of you have not seen them. > So just let me know if this introduces any new problems etc. I will be > MFCing within a week.Do you have a list of the PRs that this affects and/or resolves? I'm curious, specifically, about kern/84589. I believe it may be related to snapshots in some way, but I don't know. I'm not going to be able to test the patch on these servers, unfortunately, as they're fully in production now. The servers are stable now with background_fsck="NO", but "YES" is still the default (last I checked). FWIW: the deadlock in 84589 doesn't involve quotas, as there were no quotas enabled in the kernel. The URLs referenced in the PR are no longer valid (they were never clicked) but I can produce them if necessary. Also: freebsd.org/releases/6.1R/todo.html The todo page mentions deadlocks but doesn't link to any specific PRs/threads that discuss them. Are these fixed with this MFC?