Hi, I believe I have found a race condition in the ZFS ARC code. The problem manifests itself only when debugging is turned on and arc_mru->arcs_size is very close to arc_mru->arcs_lsize. It causes these assertions in arc.c to fail: 1) In remove_reference(): ASSERT3U(state->arcs_size, >=, state->arcs_lsize); 2) In arc_change_state(): ASSERT3U(new_state->arcs_size + to_delta, >=, new_state->arcs_lsize); Steps to reproduce: Well, in zfs-fuse it''s just a matter of compiling in debug mode and running ''bonnie++ -f -d /pool''. It fails a couple of minutes into the rewrite test with the arc_change_state() assertion. Currently, in zfs-fuse, there is a lot of context switching, which may trigger this bug more frequently than in Solaris. I have found that, with the bonnie++ workload, as much as 6 threads enter arc_change_state() at the same time, just before it fails. Also, I have a relatively low (64 MB) arc_c_max value, which might be relevant. It is much easier to reproduce if you apply this patch (relative to the current mercurial tip): diff -r 773fb303fd36 usr/src/uts/common/fs/zfs/arc.c --- a/usr/src/uts/common/fs/zfs/arc.c Mon Feb 19 05:28:47 2007 -0800 +++ b/usr/src/uts/common/fs/zfs/arc.c Tue Feb 20 05:23:03 2007 +0000 @@ -834,6 +834,8 @@ arc_change_state(arc_state_t *new_state, if (use_mutex) mutex_exit(&new_state->arcs_mtx); + if (new_state == arc_mru) + delay(hz); // sleep 1 second } } With this patch, zfs-fuse will always crash in the remove_reference() assertion simply by mounting a filesystem (when compiled in debug mode, of course). Unfortunately, even with that patch, it''s a little hard to trigger the bug with ztest because arc_mru->arcs_size gets much bigger than arc_mru->arcs_lsize after a while. I had moderate success with the following steps: 1) Applying the patch 2) Changing the ztest_dmu_write_parallel test frequency from zopt_always to zopt_rarely (I''m unsure how much this actually helps) 3) Compiling in debug mode 4) Running ztest with parameters "-T600 -P3" 5) Keep retrying. Usually if it doesn''t fail in the first 10 minutes, I think it''s better to start ztest from the beginning.. I have fixed this bug with the attached patch, which I don''t really like very much, but it fixes the race. -------------- next part -------------- A non-text attachment was scrubbed... Name: arc.patch Type: text/x-diff Size: 823 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-code/attachments/20070220/7be98875/attachment.bin>