I observe a strange reading of the ZFS memory stats: Mem: 298M Active, 207M Inact, 446M Wired, 10M Cache, 91M Buf, 29M Free ARC: 339M Total, 8758K MFU, 43M MRU, 52K Anon, 35M Header, 40M Other Swap: 2441M Total, 402M Used, 2040M Free, 16% Inuse Usually I perceived the "Total" value being approx. the sum of the other values. Now this is still the case after system start, but after a day the significant difference appears like shown above. (40+35+43+9 = 127 << 339) Also it seems the ARC is reluctant to grow when free mem is avail nor does it shrink much while paging out. The build is r309023M. Definitely the behaviour is different than what I tried before (r306589:306943M), but that one was probably unstable, and I see a bunch of ZFS related commits interim. Also I now see some counts on "l2_cksum_bad" which weren't there before. BTW: is there some specific mailing-list where ZFS changes are pronounced? Machine is i386 with 1GB mem. Probably hardware is somehow crappy, but at least the mem readings are difficult to explain by hardware weakness. Config (in case it matters): vm.kmem_size="576M" vm.kmem_size_max="576M" vfs.zfs.arc_max="320M" vfs.zfs.arc_min="120M" vfs.zfs.vdev.cache.size="5M" vfs.zfs.prefetch_disable="0" vfs.zfs.l2arc_norw="0" vfs.zfs.l2arc_noprefetch="0" kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 1016019 kstat.zfs.misc.arcstats.sync_wait_for_async: 1157 kstat.zfs.misc.arcstats.arc_meta_min: 62914560 kstat.zfs.misc.arcstats.arc_meta_max: 242711832 kstat.zfs.misc.arcstats.arc_meta_limit: 83886080 kstat.zfs.misc.arcstats.arc_meta_used: 133996612 kstat.zfs.misc.arcstats.memory_throttle_count: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 272242 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 489828 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 3372460809216 kstat.zfs.misc.arcstats.l2_write_pios: 14313 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 122496 kstat.zfs.misc.arcstats.l2_write_full: 177 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 4673385 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 925 kstat.zfs.misc.arcstats.l2_write_in_l2: 93122523 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 196362282 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 57198 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 20575 kstat.zfs.misc.arcstats.l2_padding_needed: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 33567112 kstat.zfs.misc.arcstats.l2_asize: 4040757248 kstat.zfs.misc.arcstats.l2_size: 4472570880 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 61 kstat.zfs.misc.arcstats.l2_abort_lowmem: 15 kstat.zfs.misc.arcstats.l2_free_on_write: 26703 kstat.zfs.misc.arcstats.l2_evict_l1cached: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_writes_lock_retry: 173 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_done: 14313 kstat.zfs.misc.arcstats.l2_writes_sent: 14313 kstat.zfs.misc.arcstats.l2_write_bytes: 6030606336 kstat.zfs.misc.arcstats.l2_read_bytes: 11140009984 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_feeds: 122496 kstat.zfs.misc.arcstats.l2_misses: 4370503 kstat.zfs.misc.arcstats.l2_hits: 2932017 kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 46062080 kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 1047040 kstat.zfs.misc.arcstats.mfu_ghost_size: 47109120 kstat.zfs.misc.arcstats.mfu_evictable_metadata: 0 kstat.zfs.misc.arcstats.mfu_evictable_data: 114688 kstat.zfs.misc.arcstats.mfu_size: 9073664 kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 178836480 kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 86231040 kstat.zfs.misc.arcstats.mru_ghost_size: 265067520 kstat.zfs.misc.arcstats.mru_evictable_metadata: 5632 kstat.zfs.misc.arcstats.mru_evictable_data: 1155072 kstat.zfs.misc.arcstats.mru_size: 49945088 kstat.zfs.misc.arcstats.anon_evictable_metadata: 0 kstat.zfs.misc.arcstats.anon_evictable_data: 0 kstat.zfs.misc.arcstats.anon_size: 53248 kstat.zfs.misc.arcstats.other_size: 44759120 kstat.zfs.misc.arcstats.metadata_size: 50840064 kstat.zfs.misc.arcstats.data_size: 231464448 kstat.zfs.misc.arcstats.hdr_size: 4830316 kstat.zfs.misc.arcstats.overhead_size: 41351168 kstat.zfs.misc.arcstats.uncompressed_size: 52131328 kstat.zfs.misc.arcstats.compressed_size: 17729024 kstat.zfs.misc.arcstats.size: 365461060 kstat.zfs.misc.arcstats.c_max: 335544320 kstat.zfs.misc.arcstats.c_min: 125829120 kstat.zfs.misc.arcstats.c: 315017029 kstat.zfs.misc.arcstats.p: 145334923 kstat.zfs.misc.arcstats.hash_chain_max: 17 kstat.zfs.misc.arcstats.hash_chains: 119135 kstat.zfs.misc.arcstats.hash_collisions: 6453863 kstat.zfs.misc.arcstats.hash_elements_max: 538227 kstat.zfs.misc.arcstats.hash_elements: 525460 kstat.zfs.misc.arcstats.evict_l2_skip: 4277 kstat.zfs.misc.arcstats.evict_l2_ineligible: 7410790400 kstat.zfs.misc.arcstats.evict_l2_eligible: 14946466816 kstat.zfs.misc.arcstats.evict_l2_cached: 26608123904 kstat.zfs.misc.arcstats.evict_not_enough: 7622153 kstat.zfs.misc.arcstats.evict_skip: 534700987 kstat.zfs.misc.arcstats.mutex_miss: 3099525 kstat.zfs.misc.arcstats.deleted: 3411987 kstat.zfs.misc.arcstats.allocated: 10956315 kstat.zfs.misc.arcstats.mfu_ghost_hits: 1498760 kstat.zfs.misc.arcstats.mfu_hits: 2421694 kstat.zfs.misc.arcstats.mru_ghost_hits: 719390 kstat.zfs.misc.arcstats.mru_hits: 1031308 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 556165 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 2573587 kstat.zfs.misc.arcstats.prefetch_data_misses: 2813624 kstat.zfs.misc.arcstats.prefetch_data_hits: 632368 kstat.zfs.misc.arcstats.demand_metadata_misses: 3353524 kstat.zfs.misc.arcstats.demand_metadata_hits: 565240 kstat.zfs.misc.arcstats.demand_data_misses: 579231 kstat.zfs.misc.arcstats.demand_data_hits: 1050510 kstat.zfs.misc.arcstats.misses: 7302544 kstat.zfs.misc.arcstats.hits: 4821705