Are there any tweaks that can be done to speed up or improve zfs metadata performance ? I have a backup server with a lot of snapshots (40,000)? and just doing a listing can take a great deal of time.? Best case scenario is about 24 seconds, worst case, I have seen it up to 15 minutes.? (FreeBSD 12.1-STABLE r363078) ARC Efficiency:???????????????????????????????? 79.33b ??????? Cache Hit Ratio:??????????????? 92.81%? 73.62b ??????? Cache Miss Ratio:?????????????? 7.19%?? 5.71b ??????? Actual Hit Ratio:?????????????? 92.78%? 73.60b ??????? Data Demand Efficiency:???????? 96.47%? 461.91m ??????? Data Prefetch Efficiency:?????? 1.00%?? 262.73m ??????? CACHE HITS BY CACHE LIST: ????????? Anonymously Used:???????????? 0.01%?? 3.86m ????????? Most Recently Used:?????????? 3.91%?? 2.88b ????????? Most Frequently Used:???????? 96.06%? 70.72b ????????? Most Recently Used Ghost:???? 0.01%?? 5.31m ????????? Most Frequently Used Ghost:?? 0.01%?? 10.47m ??????? CACHE HITS BY DATA TYPE: ????????? Demand Data:????????????????? 0.61%?? 445.60m ????????? Prefetch Data:??????????????? 0.00%?? 2.63m ????????? Demand Metadata:????????????? 99.36%? 73.15b ????????? Prefetch Metadata:??????????? 0.03%?? 21.00m ??????? CACHE MISSES BY DATA TYPE: ????????? Demand Data:????????????????? 0.29%?? 16.31m ????????? Prefetch Data:??????????????? 4.56%?? 260.10m ????????? Demand Metadata:????????????? 95.02%? 5.42b ????????? Prefetch Metadata:??????????? 0.14%?? 7.75m Other than increase the metadata max, I havent really changed any tuneables ZFS Tunables (sysctl): ??????? kern.maxusers?????????????????????????? 4416 ??????? vm.kmem_size??????????????????????????? 66691842048 ??????? vm.kmem_size_scale????????????????????? 1 ??????? vm.kmem_size_min??????????????????????? 0 ??????? vm.kmem_size_max??????????????????????? 1319413950874 ??????? vfs.zfs.trim.max_interval?????????????? 1 ??????? vfs.zfs.trim.timeout??????????????????? 30 ??????? vfs.zfs.trim.txg_delay????????????????? 32 ??????? vfs.zfs.trim.enabled??????????????????? 1 ??????? vfs.zfs.vol.immediate_write_sz????????? 32768 ??????? vfs.zfs.vol.unmap_sync_enabled????????? 0 ??????? vfs.zfs.vol.unmap_enabled?????????????? 1 ??????? vfs.zfs.vol.recursive?????????????????? 0 ??????? vfs.zfs.vol.mode??????????????????????? 1 ??????? vfs.zfs.version.zpl???????????????????? 5 ??????? vfs.zfs.version.spa???????????????????? 5000 ??????? vfs.zfs.version.acl???????????????????? 1 ??????? vfs.zfs.version.ioctl?????????????????? 7 ??????? vfs.zfs.debug?????????????????????????? 0 ??????? vfs.zfs.super_owner???????????????????? 0 ??????? vfs.zfs.immediate_write_sz????????????? 32768 ??????? vfs.zfs.sync_pass_rewrite?????????????? 2 ??????? vfs.zfs.sync_pass_dont_compress???????? 5 ??????? vfs.zfs.sync_pass_deferred_free???????? 2 ??????? vfs.zfs.zio.dva_throttle_enabled??????? 1 ??????? vfs.zfs.zio.exclude_metadata??????????? 0 ??????? vfs.zfs.zio.use_uma???????????????????? 1 ??????? vfs.zfs.zio.taskq_batch_pct???????????? 75 ??????? vfs.zfs.zil_maxblocksize??????????????? 131072 ??????? vfs.zfs.zil_slog_bulk?????????????????? 786432 ??????? vfs.zfs.zil_nocacheflush??????????????? 0 ??????? vfs.zfs.zil_replay_disable????????????? 0 ??????? vfs.zfs.cache_flush_disable???????????? 0 ??????? vfs.zfs.standard_sm_blksz?????????????? 131072 ??????? vfs.zfs.dtl_sm_blksz??????????????????? 4096 ??????? vfs.zfs.min_auto_ashift???????????????? 9 ??????? vfs.zfs.max_auto_ashift???????????????? 13 ??????? vfs.zfs.vdev.trim_max_pending?????????? 10000 ??????? vfs.zfs.vdev.bio_delete_disable???????? 0 ??????? vfs.zfs.vdev.bio_flush_disable????????? 0 ??????? vfs.zfs.vdev.def_queue_depth??????????? 32 ??????? vfs.zfs.vdev.queue_depth_pct??????????? 1000 ??????? vfs.zfs.vdev.write_gap_limit??????????? 4096 ??????? vfs.zfs.vdev.read_gap_limit???????????? 32768 ??????? vfs.zfs.vdev.aggregation_limit_non_rotating131072 ??????? vfs.zfs.vdev.aggregation_limit????????? 1048576 ??????? vfs.zfs.vdev.initializing_max_active??? 1 ??????? vfs.zfs.vdev.initializing_min_active??? 1 ??????? vfs.zfs.vdev.removal_max_active???????? 2 ??????? vfs.zfs.vdev.removal_min_active???????? 1 ??????? vfs.zfs.vdev.trim_max_active??????????? 64 ??????? vfs.zfs.vdev.trim_min_active??????????? 1 ??????? vfs.zfs.vdev.scrub_max_active?????????? 2 ??????? vfs.zfs.vdev.scrub_min_active?????????? 1 ??????? vfs.zfs.vdev.async_write_max_active???? 10 ??????? vfs.zfs.vdev.async_write_min_active???? 1 ??????? vfs.zfs.vdev.async_read_max_active????? 3 ??????? vfs.zfs.vdev.async_read_min_active????? 1 ??????? vfs.zfs.vdev.sync_write_max_active????? 10 ??????? vfs.zfs.vdev.sync_write_min_active????? 10 ??????? vfs.zfs.vdev.sync_read_max_active?????? 10 ??????? vfs.zfs.vdev.sync_read_min_active?????? 10 ??????? vfs.zfs.vdev.max_active???????????????? 1000 ??????? vfs.zfs.vdev.async_write_active_max_dirty_percent60 ??????? vfs.zfs.vdev.async_write_active_min_dirty_percent30 ??????? vfs.zfs.vdev.mirror.non_rotating_seek_inc1 ??????? vfs.zfs.vdev.mirror.non_rotating_inc??? 0 ??????? vfs.zfs.vdev.mirror.rotating_seek_offset1048576 ??????? vfs.zfs.vdev.mirror.rotating_seek_inc?? 5 ??????? vfs.zfs.vdev.mirror.rotating_inc??????? 0 ??????? vfs.zfs.vdev.trim_on_init?????????????? 1 ??????? vfs.zfs.vdev.cache.bshift?????????????? 16 ??????? vfs.zfs.vdev.cache.size???????????????? 0 ??????? vfs.zfs.vdev.cache.max????????????????? 16384 ??????? vfs.zfs.vdev.validate_skip????????????? 0 ??????? vfs.zfs.vdev.max_ms_shift?????????????? 34 ??????? vfs.zfs.vdev.default_ms_shift?????????? 29 ??????? vfs.zfs.vdev.max_ms_count_limit???????? 131072 ??????? vfs.zfs.vdev.min_ms_count?????????????? 16 ??????? vfs.zfs.vdev.default_ms_count?????????? 200 ??????? vfs.zfs.txg.timeout???????????????????? 5 ??????? vfs.zfs.space_map_ibs?????????????????? 14 ??????? vfs.zfs.special_class_metadata_reserve_pct25 ??????? vfs.zfs.user_indirect_is_special??????? 1 ??????? vfs.zfs.ddt_data_is_special???????????? 1 ??????? vfs.zfs.spa_allocators????????????????? 4 ??????? vfs.zfs.spa_min_slop??????????????????? 134217728 ??????? vfs.zfs.spa_slop_shift????????????????? 5 ??????? vfs.zfs.spa_asize_inflation???????????? 24 ??????? vfs.zfs.deadman_enabled???????????????? 1 ??????? vfs.zfs.deadman_checktime_ms??????????? 5000 ??????? vfs.zfs.deadman_synctime_ms???????????? 1000000 ??????? vfs.zfs.debugflags????????????????????? 0 ??????? vfs.zfs.recover???????????????????????? 0 ??????? vfs.zfs.spa_load_verify_data??????????? 1 ??????? vfs.zfs.spa_load_verify_metadata??????? 1 ??????? vfs.zfs.spa_load_verify_maxinflight???? 10000 ??????? vfs.zfs.max_missing_tvds_scan?????????? 0 ??????? vfs.zfs.max_missing_tvds_cachefile????? 2 ??????? vfs.zfs.max_missing_tvds??????????????? 0 ??????? vfs.zfs.spa_load_print_vdev_tree??????? 0 ??????? vfs.zfs.ccw_retry_interval????????????? 300 ??????? vfs.zfs.check_hostid??????????????????? 1 ??????? vfs.zfs.multihost_fail_intervals??????? 10 ??????? vfs.zfs.multihost_import_intervals????? 20 ??????? vfs.zfs.multihost_interval????????????? 1000 ??????? vfs.zfs.mg_fragmentation_threshold????? 85 ??????? vfs.zfs.mg_noalloc_threshold??????????? 0 ??????? vfs.zfs.condense_pct??????????????????? 200 ??????? vfs.zfs.metaslab_sm_blksz?????????????? 4096 ??????? vfs.zfs.metaslab.bias_enabled?????????? 1 ??????? vfs.zfs.metaslab.lba_weighting_enabled? 1 ??????? vfs.zfs.metaslab.fragmentation_factor_enabled1 ??????? vfs.zfs.metaslab.preload_enabled??????? 1 ??????? vfs.zfs.metaslab.preload_limit????????? 3 ??????? vfs.zfs.metaslab.unload_delay?????????? 8 ??????? vfs.zfs.metaslab.load_pct?????????????? 50 ??????? vfs.zfs.metaslab.min_alloc_size???????? 33554432 ??????? vfs.zfs.metaslab.df_free_pct??????????? 4 ??????? vfs.zfs.metaslab.df_alloc_threshold???? 131072 ??????? vfs.zfs.metaslab.debug_unload?????????? 0 ??????? vfs.zfs.metaslab.debug_load???????????? 0 ??????? vfs.zfs.metaslab.fragmentation_threshold70 ??????? vfs.zfs.metaslab.force_ganging????????? 16777217 ??????? vfs.zfs.free_bpobj_enabled????????????? 1 ??????? vfs.zfs.free_max_blocks???????????????? -1 ??????? vfs.zfs.zfs_scan_checkpoint_interval??? 7200 ??????? vfs.zfs.zfs_scan_legacy???????????????? 0 ??????? vfs.zfs.no_scrub_prefetch?????????????? 0 ??????? vfs.zfs.no_scrub_io???????????????????? 0 ??????? vfs.zfs.resilver_min_time_ms??????????? 3000 ??????? vfs.zfs.free_min_time_ms??????????????? 1000 ??????? vfs.zfs.scan_min_time_ms??????????????? 1000 ??????? vfs.zfs.scan_idle?????????????????????? 50 ??????? vfs.zfs.scrub_delay???????????????????? 4 ??????? vfs.zfs.resilver_delay????????????????? 2 ??????? vfs.zfs.zfetch.array_rd_sz????????????? 1048576 ??????? vfs.zfs.zfetch.max_idistance??????????? 67108864 ??????? vfs.zfs.zfetch.max_distance???????????? 8388608 ??????? vfs.zfs.zfetch.min_sec_reap???????????? 2 ??????? vfs.zfs.zfetch.max_streams????????????? 8 ??????? vfs.zfs.prefetch_disable??????????????? 0 ??????? vfs.zfs.delay_scale???????????????????? 500000 ??????? vfs.zfs.delay_min_dirty_percent???????? 60 ??????? vfs.zfs.dirty_data_sync_pct???????????? 20 ??????? vfs.zfs.dirty_data_max_percent????????? 10 ??????? vfs.zfs.dirty_data_max_max????????????? 4294967296 ??????? vfs.zfs.dirty_data_max????????????????? 4294967296 ??????? vfs.zfs.max_recordsize????????????????? 1048576 ??????? vfs.zfs.default_ibs???????????????????? 17 ??????? vfs.zfs.default_bs????????????????????? 9 ??????? vfs.zfs.send_holes_without_birth_time?? 1 ??????? vfs.zfs.mdcomp_disable????????????????? 0 ??????? vfs.zfs.per_txg_dirty_frees_percent???? 5 ??????? vfs.zfs.nopwrite_enabled??????????????? 1 ??????? vfs.zfs.dedup.prefetch????????????????? 1 ??????? vfs.zfs.dbuf_cache_lowater_pct????????? 10 ??????? vfs.zfs.dbuf_cache_hiwater_pct????????? 10 ??????? vfs.zfs.dbuf_metadata_cache_overflow??? 0 ??????? vfs.zfs.dbuf_metadata_cache_shift?????? 6 ??????? vfs.zfs.dbuf_cache_shift??????????????? 5 ??????? vfs.zfs.dbuf_metadata_cache_max_bytes?? 1025282816 ??????? vfs.zfs.dbuf_cache_max_bytes??????????? 2050565632 ??????? vfs.zfs.arc_min_prescient_prefetch_ms?? 6 ??????? vfs.zfs.arc_min_prefetch_ms???????????? 1 ??????? vfs.zfs.l2c_only_size?????????????????? 0 ??????? vfs.zfs.mfu_ghost_data_esize??????????? 7778263552 ??????? vfs.zfs.mfu_ghost_metadata_esize??????? 16851792896 ??????? vfs.zfs.mfu_ghost_size????????????????? 24630056448 ??????? vfs.zfs.mfu_data_esize????????????????? 3059418112 ??????? vfs.zfs.mfu_metadata_esize????????????? 28641792 ??????? vfs.zfs.mfu_size??????????????????????? 6399023104 ??????? vfs.zfs.mru_ghost_data_esize??????????? 2199812096 ??????? vfs.zfs.mru_ghost_metadata_esize??????? 6289682432 ??????? vfs.zfs.mru_ghost_size????????????????? 8489494528 ??????? vfs.zfs.mru_data_esize????????????????? 22781456384 ??????? vfs.zfs.mru_metadata_esize????????????? 309155840 ??????? vfs.zfs.mru_size??????????????????????? 23847875584 ??????? vfs.zfs.anon_data_esize???????????????? 0 ??????? vfs.zfs.anon_metadata_esize???????????? 0 ??????? vfs.zfs.anon_size?????????????????????? 8556544 ??????? vfs.zfs.l2arc_norw????????????????????? 1 ??????? vfs.zfs.l2arc_feed_again??????????????? 1 ??????? vfs.zfs.l2arc_noprefetch??????????????? 1 ??????? vfs.zfs.l2arc_feed_min_ms?????????????? 200 ??????? vfs.zfs.l2arc_feed_secs???????????????? 1 ??????? vfs.zfs.l2arc_headroom????????????????? 2 ??????? vfs.zfs.l2arc_write_boost?????????????? 8388608 ??????? vfs.zfs.l2arc_write_max???????????????? 8388608 ??????? vfs.zfs.arc_meta_strategy?????????????? 1 ??????? vfs.zfs.arc_meta_limit????????????????? 15833624576 ??????? vfs.zfs.arc_free_target???????????????? 346902 ??????? vfs.zfs.arc_kmem_cache_reap_retry_ms??? 1000 ??????? vfs.zfs.compressed_arc_enabled????????? 1 ??????? vfs.zfs.arc_grow_retry????????????????? 60 ??????? vfs.zfs.arc_shrink_shift??????????????? 7 ??????? vfs.zfs.arc_average_blocksize?????????? 8192 ??????? vfs.zfs.arc_no_grow_shift?????????????? 5 ??????? vfs.zfs.arc_min???????????????????????? 8202262528 ??????? vfs.zfs.arc_max???????????????????????? 39334498304 ??????? vfs.zfs.abd_chunk_size????????????????? 4096 ??????? vfs.zfs.abd_scatter_enabled???????????? 1
Hi, My first suggestion would be to remove a lot of snapshots. But that my not match your business case. Maybe you can provide more information about your setup: Amount of RAM, CPU? output of "zpool status" output of "zfs list" if possible to share Type of disks/ssds? What is the load of the system? I/O per second, etc. Do you use dedup, GELI? Something else special about the setup. output of "top -b" That kind of information. Regards, Ronald. Van: mike tancsa <mike at sentex.net> Datum: zondag, 19 juli 2020 16:17 Aan: FreeBSD-STABLE Mailing List <freebsd-stable at freebsd.org> Onderwerp: zfs meta data slowness> > Are there any tweaks that can be done to speed up or improve zfs > metadata performance ? I have a backup server with a lot of snapshots > (40,000) and just doing a listing can take a great deal of time. Best > case scenario is about 24 seconds, worst case, I have seen it up to 15 > minutes. (FreeBSD 12.1-STABLE r363078) > > > ARC Efficiency: 79.33b > Cache Hit Ratio: 92.81% 73.62b > Cache Miss Ratio: 7.19% 5.71b > Actual Hit Ratio: 92.78% 73.60b > > Data Demand Efficiency: 96.47% 461.91m > Data Prefetch Efficiency: 1.00% 262.73m > > CACHE HITS BY CACHE LIST: > Anonymously Used: 0.01% 3.86m > Most Recently Used: 3.91% 2.88b > Most Frequently Used: 96.06% 70.72b > Most Recently Used Ghost: 0.01% 5.31m > Most Frequently Used Ghost: 0.01% 10.47m > > CACHE HITS BY DATA TYPE: > Demand Data: 0.61% 445.60m > Prefetch Data: 0.00% 2.63m > Demand Metadata: 99.36% 73.15b > Prefetch Metadata: 0.03% 21.00m > > CACHE MISSES BY DATA TYPE: > Demand Data: 0.29% 16.31m > Prefetch Data: 4.56% 260.10m > Demand Metadata: 95.02% 5.42b > Prefetch Metadata: 0.14% 7.75m > > > Other than increase the metadata max, I havent really changed any tuneables > > > ZFS Tunables (sysctl): > kern.maxusers 4416 > vm.kmem_size 66691842048 > vm.kmem_size_scale 1 > vm.kmem_size_min 0 > vm.kmem_size_max 1319413950874 > vfs.zfs.trim.max_interval 1 > vfs.zfs.trim.timeout 30 > vfs.zfs.trim.txg_delay 32 > vfs.zfs.trim.enabled 1 > vfs.zfs.vol.immediate_write_sz 32768 > vfs.zfs.vol.unmap_sync_enabled 0 > vfs.zfs.vol.unmap_enabled 1 > vfs.zfs.vol.recursive 0 > vfs.zfs.vol.mode 1 > vfs.zfs.version.zpl 5 > vfs.zfs.version.spa 5000 > vfs.zfs.version.acl 1 > vfs.zfs.version.ioctl 7 > vfs.zfs.debug 0 > vfs.zfs.super_owner 0 > vfs.zfs.immediate_write_sz 32768 > vfs.zfs.sync_pass_rewrite 2 > vfs.zfs.sync_pass_dont_compress 5 > vfs.zfs.sync_pass_deferred_free 2 > vfs.zfs.zio.dva_throttle_enabled 1 > vfs.zfs.zio.exclude_metadata 0 > vfs.zfs.zio.use_uma 1 > vfs.zfs.zio.taskq_batch_pct 75 > vfs.zfs.zil_maxblocksize 131072 > vfs.zfs.zil_slog_bulk 786432 > vfs.zfs.zil_nocacheflush 0 > vfs.zfs.zil_replay_disable 0 > vfs.zfs.cache_flush_disable 0 > vfs.zfs.standard_sm_blksz 131072 > vfs.zfs.dtl_sm_blksz 4096 > vfs.zfs.min_auto_ashift 9 > vfs.zfs.max_auto_ashift 13 > vfs.zfs.vdev.trim_max_pending 10000 > vfs.zfs.vdev.bio_delete_disable 0 > vfs.zfs.vdev.bio_flush_disable 0 > vfs.zfs.vdev.def_queue_depth 32 > vfs.zfs.vdev.queue_depth_pct 1000 > vfs.zfs.vdev.write_gap_limit 4096 > vfs.zfs.vdev.read_gap_limit 32768 > vfs.zfs.vdev.aggregation_limit_non_rotating131072 > vfs.zfs.vdev.aggregation_limit 1048576 > vfs.zfs.vdev.initializing_max_active 1 > vfs.zfs.vdev.initializing_min_active 1 > vfs.zfs.vdev.removal_max_active 2 > vfs.zfs.vdev.removal_min_active 1 > vfs.zfs.vdev.trim_max_active 64 > vfs.zfs.vdev.trim_min_active 1 > vfs.zfs.vdev.scrub_max_active 2 > vfs.zfs.vdev.scrub_min_active 1 > vfs.zfs.vdev.async_write_max_active 10 > vfs.zfs.vdev.async_write_min_active 1 > vfs.zfs.vdev.async_read_max_active 3 > vfs.zfs.vdev.async_read_min_active 1 > vfs.zfs.vdev.sync_write_max_active 10 > vfs.zfs.vdev.sync_write_min_active 10 > vfs.zfs.vdev.sync_read_max_active 10 > vfs.zfs.vdev.sync_read_min_active 10 > vfs.zfs.vdev.max_active 1000 > vfs.zfs.vdev.async_write_active_max_dirty_percent60 > vfs.zfs.vdev.async_write_active_min_dirty_percent30 > vfs.zfs.vdev.mirror.non_rotating_seek_inc1 > vfs.zfs.vdev.mirror.non_rotating_inc 0 > vfs.zfs.vdev.mirror.rotating_seek_offset1048576 > vfs.zfs.vdev.mirror.rotating_seek_inc 5 > vfs.zfs.vdev.mirror.rotating_inc 0 > vfs.zfs.vdev.trim_on_init 1 > vfs.zfs.vdev.cache.bshift 16 > vfs.zfs.vdev.cache.size 0 > vfs.zfs.vdev.cache.max 16384 > vfs.zfs.vdev.validate_skip 0 > vfs.zfs.vdev.max_ms_shift 34 > vfs.zfs.vdev.default_ms_shift 29 > vfs.zfs.vdev.max_ms_count_limit 131072 > vfs.zfs.vdev.min_ms_count 16 > vfs.zfs.vdev.default_ms_count 200 > vfs.zfs.txg.timeout 5 > vfs.zfs.space_map_ibs 14 > vfs.zfs.special_class_metadata_reserve_pct25 > vfs.zfs.user_indirect_is_special 1 > vfs.zfs.ddt_data_is_special 1 > vfs.zfs.spa_allocators 4 > vfs.zfs.spa_min_slop 134217728 > vfs.zfs.spa_slop_shift 5 > vfs.zfs.spa_asize_inflation 24 > vfs.zfs.deadman_enabled 1 > vfs.zfs.deadman_checktime_ms 5000 > vfs.zfs.deadman_synctime_ms 1000000 > vfs.zfs.debugflags 0 > vfs.zfs.recover 0 > vfs.zfs.spa_load_verify_data 1 > vfs.zfs.spa_load_verify_metadata 1 > vfs.zfs.spa_load_verify_maxinflight 10000 > vfs.zfs.max_missing_tvds_scan 0 > vfs.zfs.max_missing_tvds_cachefile 2 > vfs.zfs.max_missing_tvds 0 > vfs.zfs.spa_load_print_vdev_tree 0 > vfs.zfs.ccw_retry_interval 300 > vfs.zfs.check_hostid 1 > vfs.zfs.multihost_fail_intervals 10 > vfs.zfs.multihost_import_intervals 20 > vfs.zfs.multihost_interval 1000 > vfs.zfs.mg_fragmentation_threshold 85 > vfs.zfs.mg_noalloc_threshold 0 > vfs.zfs.condense_pct 200 > vfs.zfs.metaslab_sm_blksz 4096 > vfs.zfs.metaslab.bias_enabled 1 > vfs.zfs.metaslab.lba_weighting_enabled 1 > vfs.zfs.metaslab.fragmentation_factor_enabled1 > vfs.zfs.metaslab.preload_enabled 1 > vfs.zfs.metaslab.preload_limit 3 > vfs.zfs.metaslab.unload_delay 8 > vfs.zfs.metaslab.load_pct 50 > vfs.zfs.metaslab.min_alloc_size 33554432 > vfs.zfs.metaslab.df_free_pct 4 > vfs.zfs.metaslab.df_alloc_threshold 131072 > vfs.zfs.metaslab.debug_unload 0 > vfs.zfs.metaslab.debug_load 0 > vfs.zfs.metaslab.fragmentation_threshold70 > vfs.zfs.metaslab.force_ganging 16777217 > vfs.zfs.free_bpobj_enabled 1 > vfs.zfs.free_max_blocks -1 > vfs.zfs.zfs_scan_checkpoint_interval 7200 > vfs.zfs.zfs_scan_legacy 0 > vfs.zfs.no_scrub_prefetch 0 > vfs.zfs.no_scrub_io 0 > vfs.zfs.resilver_min_time_ms 3000 > vfs.zfs.free_min_time_ms 1000 > vfs.zfs.scan_min_time_ms 1000 > vfs.zfs.scan_idle 50 > vfs.zfs.scrub_delay 4 > vfs.zfs.resilver_delay 2 > vfs.zfs.zfetch.array_rd_sz 1048576 > vfs.zfs.zfetch.max_idistance 67108864 > vfs.zfs.zfetch.max_distance 8388608 > vfs.zfs.zfetch.min_sec_reap 2 > vfs.zfs.zfetch.max_streams 8 > vfs.zfs.prefetch_disable 0 > vfs.zfs.delay_scale 500000 > vfs.zfs.delay_min_dirty_percent 60 > vfs.zfs.dirty_data_sync_pct 20 > vfs.zfs.dirty_data_max_percent 10 > vfs.zfs.dirty_data_max_max 4294967296 > vfs.zfs.dirty_data_max 4294967296 > vfs.zfs.max_recordsize 1048576 > vfs.zfs.default_ibs 17 > vfs.zfs.default_bs 9 > vfs.zfs.send_holes_without_birth_time 1 > vfs.zfs.mdcomp_disable 0 > vfs.zfs.per_txg_dirty_frees_percent 5 > vfs.zfs.nopwrite_enabled 1 > vfs.zfs.dedup.prefetch 1 > vfs.zfs.dbuf_cache_lowater_pct 10 > vfs.zfs.dbuf_cache_hiwater_pct 10 > vfs.zfs.dbuf_metadata_cache_overflow 0 > vfs.zfs.dbuf_metadata_cache_shift 6 > vfs.zfs.dbuf_cache_shift 5 > vfs.zfs.dbuf_metadata_cache_max_bytes 1025282816 > vfs.zfs.dbuf_cache_max_bytes 2050565632 > vfs.zfs.arc_min_prescient_prefetch_ms 6 > vfs.zfs.arc_min_prefetch_ms 1 > vfs.zfs.l2c_only_size 0 > vfs.zfs.mfu_ghost_data_esize 7778263552 > vfs.zfs.mfu_ghost_metadata_esize 16851792896 > vfs.zfs.mfu_ghost_size 24630056448 > vfs.zfs.mfu_data_esize 3059418112 > vfs.zfs.mfu_metadata_esize 28641792 > vfs.zfs.mfu_size 6399023104 > vfs.zfs.mru_ghost_data_esize 2199812096 > vfs.zfs.mru_ghost_metadata_esize 6289682432 > vfs.zfs.mru_ghost_size 8489494528 > vfs.zfs.mru_data_esize 22781456384 > vfs.zfs.mru_metadata_esize 309155840 > vfs.zfs.mru_size 23847875584 > vfs.zfs.anon_data_esize 0 > vfs.zfs.anon_metadata_esize 0 > vfs.zfs.anon_size 8556544 > vfs.zfs.l2arc_norw 1 > vfs.zfs.l2arc_feed_again 1 > vfs.zfs.l2arc_noprefetch 1 > vfs.zfs.l2arc_feed_min_ms 200 > vfs.zfs.l2arc_feed_secs 1 > vfs.zfs.l2arc_headroom 2 > vfs.zfs.l2arc_write_boost 8388608 > vfs.zfs.l2arc_write_max 8388608 > vfs.zfs.arc_meta_strategy 1 > vfs.zfs.arc_meta_limit 15833624576 > vfs.zfs.arc_free_target 346902 > vfs.zfs.arc_kmem_cache_reap_retry_ms 1000 > vfs.zfs.compressed_arc_enabled 1 > vfs.zfs.arc_grow_retry 60 > vfs.zfs.arc_shrink_shift 7 > vfs.zfs.arc_average_blocksize 8192 > vfs.zfs.arc_no_grow_shift 5 > vfs.zfs.arc_min 8202262528 > vfs.zfs.arc_max 39334498304 > vfs.zfs.abd_chunk_size 4096 > vfs.zfs.abd_scatter_enabled 1 > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" > > >
19.07.2020 21:17, mike tancsa wrote:> Are there any tweaks that can be done to speed up or improve zfs > metadata performance ? I have a backup server with a lot of snapshots > (40,000) and just doing a listing can take a great deal of time. Best > case scenario is about 24 seconds, worst case, I have seen it up to 15 > minutes. (FreeBSD 12.1-STABLE r363078)I believe you have to perform kernel profiling for your specific use case.