Hi,
??? Thanks for the response. Reply in line
On 7/20/2020 9:04 AM, Ronald Klop wrote:> Hi,
>
> My first suggestion would be to remove a lot of snapshots. But that my
> not match your business case.
As its a backup server, its sort of the point to have all those snapshots.
> Maybe you can provide more information about your setup:
> Amount of RAM, CPU?
64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz> output of "zpool status"
# zpool status -x
all pools are healthy
> output of "zfs list" if possible to share
its a big list
# zfs list | wc
???? 824??? 4120? 107511
> Type of disks/ssds?
old school Device Model:???? WDC WD80EFAX-68KNBN0> What is the load of the system? I/O per second, etc.
its not cpu bound, disks are sometimes running at 100% based on gstat,
but not always> Do you use dedup, GELI?
no and no
> Something else special about the setup.
> output of "top -b"
>
ports are right now being built in a VM, but the problem (zrepl hanging)
and zfs list -t snapshots taking forever happens regardless
? PID USERNAME??? THR PRI NICE?? SIZE??? RES STATE??? C?? TIME??? WCPU
COMMAND
?4439 root???????? 12? 40?? 20? 6167M? 5762M kqread?? 3 535:13 200.00% bhyve
98783 root????????? 2? 21??? 0??? 16M? 5136K hdr->b?? 4?? 0:01?? 1.95% zfs
76489 root???????? 21? 23??? 0?? 738M??? 54M uwait??? 1?? 2:18?? 0.88% zrepl
98784 root????????? 1? 21??? 0??? 13M? 3832K piperd?? 3?? 0:01?? 0.59% zfs
99563 root????????? 1? 20??? 0??? 13M? 4136K zio->i?? 4?? 0:00?? 0.39% zfs
16136 root???????? 18? 25??? 0?? 705M??? 56M uwait??? 3? 29:58?? 0.00%
zrepl-freebsd-amd64
?1845 root????????? 1? 20??? 0??? 12M? 3772K nanslp?? 7?? 5:54?? 0.00%
ossec-syscheckd
?1567 root????????? 1? 20??? 0??? 11M? 2744K select?? 0?? 2:22?? 0.00%
syslogd
?1737 root???????? 32? 20??? 0??? 11M? 2844K rpcsvc?? 6?? 1:40?? 0.00% nfsd
?1660 root????????? 1 -52?? r0??? 11M??? 11M nanslp?? 5?? 1:18?? 0.00%
watchdogd
?1434 root????????? 1? 20??? 0? 9988K?? 988K select?? 3?? 0:27?? 0.00% devd
?2435 mdtancsa????? 1? 20??? 0??? 20M? 8008K select?? 0?? 0:21?? 0.00% sshd
?1754 root????????? 3? 20??? 0??? 18M? 3556K select?? 1?? 0:11?? 0.00%
apcupsd
?5917 root????????? 1? 20??? 0??? 11M? 2672K select?? 2?? 0:06?? 0.00%
script
?1449 _pflogd?????? 1? 20??? 0??? 12M? 3572K bpf????? 3?? 0:05?? 0.00%
pflogd
??? ---Mike
> That kind of information.
>
> Regards,
> Ronald.
>
>
> Van: mike tancsa <mike at sentex.net>
> Datum: zondag, 19 juli 2020 16:17
> Aan: FreeBSD-STABLE Mailing List <freebsd-stable at freebsd.org>
> Onderwerp: zfs meta data slowness
>>
>> Are there any tweaks that can be done to speed up or improve zfs
>> metadata performance ? I have a backup server with a lot of snapshots
>> (40,000)? and just doing a listing can take a great deal of time.? Best
>> case scenario is about 24 seconds, worst case, I have seen it up to 15
>> minutes.? (FreeBSD 12.1-STABLE r363078)
>>
>>
>> ARC Efficiency:???????????????????????????????? 79.33b
>> ??????? Cache Hit Ratio:??????????????? 92.81%? 73.62b
>> ??????? Cache Miss Ratio:?????????????? 7.19%?? 5.71b
>> ??????? Actual Hit Ratio:?????????????? 92.78%? 73.60b
>>
>> ??????? Data Demand Efficiency:???????? 96.47%? 461.91m
>> ??????? Data Prefetch Efficiency:?????? 1.00%?? 262.73m
>>
>> ??????? CACHE HITS BY CACHE LIST:
>> ????????? Anonymously Used:???????????? 0.01%?? 3.86m
>> ????????? Most Recently Used:?????????? 3.91%?? 2.88b
>> ????????? Most Frequently Used:???????? 96.06%? 70.72b
>> ????????? Most Recently Used Ghost:???? 0.01%?? 5.31m
>> ????????? Most Frequently Used Ghost:?? 0.01%?? 10.47m
>>
>> ??????? CACHE HITS BY DATA TYPE:
>> ????????? Demand Data:????????????????? 0.61%?? 445.60m
>> ????????? Prefetch Data:??????????????? 0.00%?? 2.63m
>> ????????? Demand Metadata:????????????? 99.36%? 73.15b
>> ????????? Prefetch Metadata:??????????? 0.03%?? 21.00m
>>
>> ??????? CACHE MISSES BY DATA TYPE:
>> ????????? Demand Data:????????????????? 0.29%?? 16.31m
>> ????????? Prefetch Data:??????????????? 4.56%?? 260.10m
>> ????????? Demand Metadata:????????????? 95.02%? 5.42b
>> ????????? Prefetch Metadata:??????????? 0.14%?? 7.75m
>>
>>
>> Other than increase the metadata max, I havent really changed any
>> tuneables
>>
>>
>> ZFS Tunables (sysctl):
>> ??????? kern.maxusers?????????????????????????? 4416
>> ??????? vm.kmem_size??????????????????????????? 66691842048
>> ??????? vm.kmem_size_scale????????????????????? 1
>> ??????? vm.kmem_size_min??????????????????????? 0
>> ??????? vm.kmem_size_max??????????????????????? 1319413950874
>> ??????? vfs.zfs.trim.max_interval?????????????? 1
>> ??????? vfs.zfs.trim.timeout??????????????????? 30
>> ??????? vfs.zfs.trim.txg_delay????????????????? 32
>> ??????? vfs.zfs.trim.enabled??????????????????? 1
>> ??????? vfs.zfs.vol.immediate_write_sz????????? 32768
>> ??????? vfs.zfs.vol.unmap_sync_enabled????????? 0
>> ??????? vfs.zfs.vol.unmap_enabled?????????????? 1
>> ??????? vfs.zfs.vol.recursive?????????????????? 0
>> ??????? vfs.zfs.vol.mode??????????????????????? 1
>> ??????? vfs.zfs.version.zpl???????????????????? 5
>> ??????? vfs.zfs.version.spa???????????????????? 5000
>> ??????? vfs.zfs.version.acl???????????????????? 1
>> ??????? vfs.zfs.version.ioctl?????????????????? 7
>> ??????? vfs.zfs.debug?????????????????????????? 0
>> ??????? vfs.zfs.super_owner???????????????????? 0
>> ??????? vfs.zfs.immediate_write_sz????????????? 32768
>> ??????? vfs.zfs.sync_pass_rewrite?????????????? 2
>> ??????? vfs.zfs.sync_pass_dont_compress???????? 5
>> ??????? vfs.zfs.sync_pass_deferred_free???????? 2
>> ??????? vfs.zfs.zio.dva_throttle_enabled??????? 1
>> ??????? vfs.zfs.zio.exclude_metadata??????????? 0
>> ??????? vfs.zfs.zio.use_uma???????????????????? 1
>> ??????? vfs.zfs.zio.taskq_batch_pct???????????? 75
>> ??????? vfs.zfs.zil_maxblocksize??????????????? 131072
>> ??????? vfs.zfs.zil_slog_bulk?????????????????? 786432
>> ??????? vfs.zfs.zil_nocacheflush??????????????? 0
>> ??????? vfs.zfs.zil_replay_disable????????????? 0
>> ??????? vfs.zfs.cache_flush_disable???????????? 0
>> ??????? vfs.zfs.standard_sm_blksz?????????????? 131072
>> ??????? vfs.zfs.dtl_sm_blksz??????????????????? 4096
>> ??????? vfs.zfs.min_auto_ashift???????????????? 9
>> ??????? vfs.zfs.max_auto_ashift???????????????? 13
>> ??????? vfs.zfs.vdev.trim_max_pending?????????? 10000
>> ??????? vfs.zfs.vdev.bio_delete_disable???????? 0
>> ??????? vfs.zfs.vdev.bio_flush_disable????????? 0
>> ??????? vfs.zfs.vdev.def_queue_depth??????????? 32
>> ??????? vfs.zfs.vdev.queue_depth_pct??????????? 1000
>> ??????? vfs.zfs.vdev.write_gap_limit??????????? 4096
>> ??????? vfs.zfs.vdev.read_gap_limit???????????? 32768
>> ??????? vfs.zfs.vdev.aggregation_limit_non_rotating131072
>> ??????? vfs.zfs.vdev.aggregation_limit????????? 1048576
>> ??????? vfs.zfs.vdev.initializing_max_active??? 1
>> ??????? vfs.zfs.vdev.initializing_min_active??? 1
>> ??????? vfs.zfs.vdev.removal_max_active???????? 2
>> ??????? vfs.zfs.vdev.removal_min_active???????? 1
>> ??????? vfs.zfs.vdev.trim_max_active??????????? 64
>> ??????? vfs.zfs.vdev.trim_min_active??????????? 1
>> ??????? vfs.zfs.vdev.scrub_max_active?????????? 2
>> ??????? vfs.zfs.vdev.scrub_min_active?????????? 1
>> ??????? vfs.zfs.vdev.async_write_max_active???? 10
>> ??????? vfs.zfs.vdev.async_write_min_active???? 1
>> ??????? vfs.zfs.vdev.async_read_max_active????? 3
>> ??????? vfs.zfs.vdev.async_read_min_active????? 1
>> ??????? vfs.zfs.vdev.sync_write_max_active????? 10
>> ??????? vfs.zfs.vdev.sync_write_min_active????? 10
>> ??????? vfs.zfs.vdev.sync_read_max_active?????? 10
>> ??????? vfs.zfs.vdev.sync_read_min_active?????? 10
>> ??????? vfs.zfs.vdev.max_active???????????????? 1000
>> ??????? vfs.zfs.vdev.async_write_active_max_dirty_percent60
>> ??????? vfs.zfs.vdev.async_write_active_min_dirty_percent30
>> ??????? vfs.zfs.vdev.mirror.non_rotating_seek_inc1
>> ??????? vfs.zfs.vdev.mirror.non_rotating_inc??? 0
>> ??????? vfs.zfs.vdev.mirror.rotating_seek_offset1048576
>> ??????? vfs.zfs.vdev.mirror.rotating_seek_inc?? 5
>> ??????? vfs.zfs.vdev.mirror.rotating_inc??????? 0
>> ??????? vfs.zfs.vdev.trim_on_init?????????????? 1
>> ??????? vfs.zfs.vdev.cache.bshift?????????????? 16
>> ??????? vfs.zfs.vdev.cache.size???????????????? 0
>> ??????? vfs.zfs.vdev.cache.max????????????????? 16384
>> ??????? vfs.zfs.vdev.validate_skip????????????? 0
>> ??????? vfs.zfs.vdev.max_ms_shift?????????????? 34
>> ??????? vfs.zfs.vdev.default_ms_shift?????????? 29
>> ??????? vfs.zfs.vdev.max_ms_count_limit???????? 131072
>> ??????? vfs.zfs.vdev.min_ms_count?????????????? 16
>> ??????? vfs.zfs.vdev.default_ms_count?????????? 200
>> ??????? vfs.zfs.txg.timeout???????????????????? 5
>> ??????? vfs.zfs.space_map_ibs?????????????????? 14
>> ??????? vfs.zfs.special_class_metadata_reserve_pct25
>> ??????? vfs.zfs.user_indirect_is_special??????? 1
>> ??????? vfs.zfs.ddt_data_is_special???????????? 1
>> ??????? vfs.zfs.spa_allocators????????????????? 4
>> ??????? vfs.zfs.spa_min_slop??????????????????? 134217728
>> ??????? vfs.zfs.spa_slop_shift????????????????? 5
>> ??????? vfs.zfs.spa_asize_inflation???????????? 24
>> ??????? vfs.zfs.deadman_enabled???????????????? 1
>> ??????? vfs.zfs.deadman_checktime_ms??????????? 5000
>> ??????? vfs.zfs.deadman_synctime_ms???????????? 1000000
>> ??????? vfs.zfs.debugflags????????????????????? 0
>> ??????? vfs.zfs.recover???????????????????????? 0
>> ??????? vfs.zfs.spa_load_verify_data??????????? 1
>> ??????? vfs.zfs.spa_load_verify_metadata??????? 1
>> ??????? vfs.zfs.spa_load_verify_maxinflight???? 10000
>> ??????? vfs.zfs.max_missing_tvds_scan?????????? 0
>> ??????? vfs.zfs.max_missing_tvds_cachefile????? 2
>> ??????? vfs.zfs.max_missing_tvds??????????????? 0
>> ??????? vfs.zfs.spa_load_print_vdev_tree??????? 0
>> ??????? vfs.zfs.ccw_retry_interval????????????? 300
>> ??????? vfs.zfs.check_hostid??????????????????? 1
>> ??????? vfs.zfs.multihost_fail_intervals??????? 10
>> ??????? vfs.zfs.multihost_import_intervals????? 20
>> ??????? vfs.zfs.multihost_interval????????????? 1000
>> ??????? vfs.zfs.mg_fragmentation_threshold????? 85
>> ??????? vfs.zfs.mg_noalloc_threshold??????????? 0
>> ??????? vfs.zfs.condense_pct??????????????????? 200
>> ??????? vfs.zfs.metaslab_sm_blksz?????????????? 4096
>> ??????? vfs.zfs.metaslab.bias_enabled?????????? 1
>> ??????? vfs.zfs.metaslab.lba_weighting_enabled? 1
>> ??????? vfs.zfs.metaslab.fragmentation_factor_enabled1
>> ??????? vfs.zfs.metaslab.preload_enabled??????? 1
>> ??????? vfs.zfs.metaslab.preload_limit????????? 3
>> ??????? vfs.zfs.metaslab.unload_delay?????????? 8
>> ??????? vfs.zfs.metaslab.load_pct?????????????? 50
>> ??????? vfs.zfs.metaslab.min_alloc_size???????? 33554432
>> ??????? vfs.zfs.metaslab.df_free_pct??????????? 4
>> ??????? vfs.zfs.metaslab.df_alloc_threshold???? 131072
>> ??????? vfs.zfs.metaslab.debug_unload?????????? 0
>> ??????? vfs.zfs.metaslab.debug_load???????????? 0
>> ??????? vfs.zfs.metaslab.fragmentation_threshold70
>> ??????? vfs.zfs.metaslab.force_ganging????????? 16777217
>> ??????? vfs.zfs.free_bpobj_enabled????????????? 1
>> ??????? vfs.zfs.free_max_blocks???????????????? -1
>> ??????? vfs.zfs.zfs_scan_checkpoint_interval??? 7200
>> ??????? vfs.zfs.zfs_scan_legacy???????????????? 0
>> ??????? vfs.zfs.no_scrub_prefetch?????????????? 0
>> ??????? vfs.zfs.no_scrub_io???????????????????? 0
>> ??????? vfs.zfs.resilver_min_time_ms??????????? 3000
>> ??????? vfs.zfs.free_min_time_ms??????????????? 1000
>> ??????? vfs.zfs.scan_min_time_ms??????????????? 1000
>> ??????? vfs.zfs.scan_idle?????????????????????? 50
>> ??????? vfs.zfs.scrub_delay???????????????????? 4
>> ??????? vfs.zfs.resilver_delay????????????????? 2
>> ??????? vfs.zfs.zfetch.array_rd_sz????????????? 1048576
>> ??????? vfs.zfs.zfetch.max_idistance??????????? 67108864
>> ??????? vfs.zfs.zfetch.max_distance???????????? 8388608
>> ??????? vfs.zfs.zfetch.min_sec_reap???????????? 2
>> ??????? vfs.zfs.zfetch.max_streams????????????? 8
>> ??????? vfs.zfs.prefetch_disable??????????????? 0
>> ??????? vfs.zfs.delay_scale???????????????????? 500000
>> ??????? vfs.zfs.delay_min_dirty_percent???????? 60
>> ??????? vfs.zfs.dirty_data_sync_pct???????????? 20
>> ??????? vfs.zfs.dirty_data_max_percent????????? 10
>> ??????? vfs.zfs.dirty_data_max_max????????????? 4294967296
>> ??????? vfs.zfs.dirty_data_max????????????????? 4294967296
>> ??????? vfs.zfs.max_recordsize????????????????? 1048576
>> ??????? vfs.zfs.default_ibs???????????????????? 17
>> ??????? vfs.zfs.default_bs????????????????????? 9
>> ??????? vfs.zfs.send_holes_without_birth_time?? 1
>> ??????? vfs.zfs.mdcomp_disable????????????????? 0
>> ??????? vfs.zfs.per_txg_dirty_frees_percent???? 5
>> ??????? vfs.zfs.nopwrite_enabled??????????????? 1
>> ??????? vfs.zfs.dedup.prefetch????????????????? 1
>> ??????? vfs.zfs.dbuf_cache_lowater_pct????????? 10
>> ??????? vfs.zfs.dbuf_cache_hiwater_pct????????? 10
>> ??????? vfs.zfs.dbuf_metadata_cache_overflow??? 0
>> ??????? vfs.zfs.dbuf_metadata_cache_shift?????? 6
>> ??????? vfs.zfs.dbuf_cache_shift??????????????? 5
>> ??????? vfs.zfs.dbuf_metadata_cache_max_bytes?? 1025282816
>> ??????? vfs.zfs.dbuf_cache_max_bytes??????????? 2050565632
>> ??????? vfs.zfs.arc_min_prescient_prefetch_ms?? 6
>> ??????? vfs.zfs.arc_min_prefetch_ms???????????? 1
>> ??????? vfs.zfs.l2c_only_size?????????????????? 0
>> ??????? vfs.zfs.mfu_ghost_data_esize??????????? 7778263552
>> ??????? vfs.zfs.mfu_ghost_metadata_esize??????? 16851792896
>> ??????? vfs.zfs.mfu_ghost_size????????????????? 24630056448
>> ??????? vfs.zfs.mfu_data_esize????????????????? 3059418112
>> ??????? vfs.zfs.mfu_metadata_esize????????????? 28641792
>> ??????? vfs.zfs.mfu_size??????????????????????? 6399023104
>> ??????? vfs.zfs.mru_ghost_data_esize??????????? 2199812096
>> ??????? vfs.zfs.mru_ghost_metadata_esize??????? 6289682432
>> ??????? vfs.zfs.mru_ghost_size????????????????? 8489494528
>> ??????? vfs.zfs.mru_data_esize????????????????? 22781456384
>> ??????? vfs.zfs.mru_metadata_esize????????????? 309155840
>> ??????? vfs.zfs.mru_size??????????????????????? 23847875584
>> ??????? vfs.zfs.anon_data_esize???????????????? 0
>> ??????? vfs.zfs.anon_metadata_esize???????????? 0
>> ??????? vfs.zfs.anon_size?????????????????????? 8556544
>> ??????? vfs.zfs.l2arc_norw????????????????????? 1
>> ??????? vfs.zfs.l2arc_feed_again??????????????? 1
>> ??????? vfs.zfs.l2arc_noprefetch??????????????? 1
>> ??????? vfs.zfs.l2arc_feed_min_ms?????????????? 200
>> ??????? vfs.zfs.l2arc_feed_secs???????????????? 1
>> ??????? vfs.zfs.l2arc_headroom????????????????? 2
>> ??????? vfs.zfs.l2arc_write_boost?????????????? 8388608
>> ??????? vfs.zfs.l2arc_write_max???????????????? 8388608
>> ??????? vfs.zfs.arc_meta_strategy?????????????? 1
>> ??????? vfs.zfs.arc_meta_limit????????????????? 15833624576
>> ??????? vfs.zfs.arc_free_target???????????????? 346902
>> ??????? vfs.zfs.arc_kmem_cache_reap_retry_ms??? 1000
>> ??????? vfs.zfs.compressed_arc_enabled????????? 1
>> ??????? vfs.zfs.arc_grow_retry????????????????? 60
>> ??????? vfs.zfs.arc_shrink_shift??????????????? 7
>> ??????? vfs.zfs.arc_average_blocksize?????????? 8192
>> ??????? vfs.zfs.arc_no_grow_shift?????????????? 5
>> ??????? vfs.zfs.arc_min???????????????????????? 8202262528
>> ??????? vfs.zfs.arc_max???????????????????????? 39334498304
>> ??????? vfs.zfs.abd_chunk_size????????????????? 4096
>> ??????? vfs.zfs.abd_scatter_enabled???????????? 1
>>
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to
>> "freebsd-stable-unsubscribe at freebsd.org"
>>
>>
>>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
>