Hi.
snv_44, v440
filebench/varmail results for ZFS RAID10 with 6 disks and 32 disks.
What is suprising is that the results for both cases are almost the same!
6 disks:
IO Summary: 566997 ops 9373.6 ops/s, (1442/1442 r/w) 45.7mb/s, 299us
cpu/op, 5.1ms latency
IO Summary: 542398 ops 8971.4 ops/s, (1380/1380 r/w) 43.9mb/s, 300us
cpu/op, 5.4ms latency
32 disks:
IO Summary: 572429 ops 9469.7 ops/s, (1457/1457 r/w) 46.2mb/s, 301us
cpu/op, 5.1ms latency
IO Summary: 560491 ops 9270.6 ops/s, (1426/1427 r/w) 45.4mb/s, 300us
cpu/op, 5.2ms latency
Using iostat I can see that with 6 disks in a pool I get about 100-200 IO/s per
disk in a pool, and with 32 disk pool I get only 30-70 IO/s per disk in a pool.
Each CPU is used at about 25% in SYS (there''re 4 CPUs).
Something is wrong here.
# zpool status
pool: zfs_raid10_32disks
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zfs_raid10_32disks ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t16d0 ONLINE 0 0 0
c3t17d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t18d0 ONLINE 0 0 0
c3t19d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t20d0 ONLINE 0 0 0
c3t21d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t22d0 ONLINE 0 0 0
c3t23d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t24d0 ONLINE 0 0 0
c3t25d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t26d0 ONLINE 0 0 0
c3t27d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t32d0 ONLINE 0 0 0
c3t33d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t34d0 ONLINE 0 0 0
c3t35d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t36d0 ONLINE 0 0 0
c3t37d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t38d0 ONLINE 0 0 0
c3t39d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t40d0 ONLINE 0 0 0
c3t41d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t42d0 ONLINE 0 0 0
c3t43d0 ONLINE 0 0 0
errors: No known data errors
bash-3.00# zpool destroy zfs_raid10_32disks
bash-3.00# zpool create zfs_raid10_6disks mirror c3t42d0 c3t43d0 mirror c3t40d0
c3t41d0 mirror c3t38d0 c3t39d0
bash-3.00# zfs set atime=off zfs_raid10_6disks
bash-3.00# zfs create zfs_raid10_6disks/t1
bash-3.00#
This message posted from opensolaris.org
Looks like somewhere between the CPU and your disks you have a limitation of <9500 ops/sec. How did you connect 32 disks to your v440? Doug> Hi. > > snv_44, v440 > lebench/varmail results for ZFS RAID10 with 6 disks > and 32 disks. > What is suprising is that the results for both cases > are almost the same! > > > > 6 disks: > > IO Summary: 566997 ops 9373.6 ops/s, > (1442/1442 r/w) 45.7mb/s, 299us cpu/op, 5.1ms > latency > IO Summary: 542398 ops 8971.4 ops/s, > (1380/1380 r/w) 43.9mb/s, 300us cpu/op, 5.4ms > latency > > > 32 disks: > IO Summary: 572429 ops 9469.7 ops/s, > (1457/1457 r/w) 46.2mb/s, 301us cpu/op, 5.1ms > latency > IO Summary: 560491 ops 9270.6 ops/s, > (1426/1427 r/w) 45.4mb/s, 300us cpu/op, 5.2ms > latencyThis message posted from opensolaris.org
On Tue, Aug 08, 2006 at 09:54:16AM -0700, Robert Milkowski wrote:> Hi. > > snv_44, v440 > > filebench/varmail results for ZFS RAID10 with 6 disks and 32 disks. > What is suprising is that the results for both cases are almost the same! > > > > 6 disks: > > IO Summary: 566997 ops 9373.6 ops/s, (1442/1442 r/w) 45.7mb/s, 299us cpu/op, 5.1ms latency > IO Summary: 542398 ops 8971.4 ops/s, (1380/1380 r/w) 43.9mb/s, 300us cpu/op, 5.4ms latency > > > 32 disks: > IO Summary: 572429 ops 9469.7 ops/s, (1457/1457 r/w) 46.2mb/s, 301us cpu/op, 5.1ms latency > IO Summary: 560491 ops 9270.6 ops/s, (1426/1427 r/w) 45.4mb/s, 300us cpu/op, 5.2ms latency > > > > Using iostat I can see that with 6 disks in a pool I get about 100-200 IO/s per disk in a pool, and with 32 disk pool I get only 30-70 IO/s per disk in a pool. Each CPU is used at about 25% in SYS (there''re 4 CPUs). > > Something is wrong here.It''s possible that you are CPU limited. I''m guessing that your test uses only one thread, so that may be the limiting factor. We can get a quick idea of where that CPU is being spent if you can run ''lockstat -kgIW sleep 60'' while your test is running, and send us the first 100 lines of output. It would be nice to see the output of ''iostat -xnpc 3'' while the test is running, too. --matt
Hello Doug,
Tuesday, August 8, 2006, 7:28:07 PM, you wrote:
DS> Looks like somewhere between the CPU and your disks you have a limitation
of <9500 ops/sec.
DS> How did you connect 32 disks to your v440?
Some 3510 JBODs connected directly over FC.
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
filebench in varmail by default creates 16 threads - I configrm it with prstat,
16 threrads are created and running.
bash-3.00# lockstat -kgIW sleep 60|less
Profiling interrupt: 23308 events in 60.059 seconds (388 events/sec)
Count genr cuml rcnt nsec Hottest CPU+PIL Caller
-------------------------------------------------------------------------------
17615 76% ---- 0.00 2016 cpu[2] thread_start
17255 74% ---- 0.00 2001 cpu[2] idle
14439 62% ---- 0.00 2015 cpu[2] disp_getwork
4726 20% ---- 0.00 2673 cpu[2] syscall_trap
1010 4% ---- 0.00 2625 cpu[2] fdsync
998 4% ---- 0.00 2630 cpu[2] fop_fsync
988 4% ---- 0.00 2632 cpu[2] zfs_fsync
958 4% ---- 0.00 2639 cpu[2] zil_commit
839 4% ---- 0.00 2814 cpu[2] fop_read
765 3% ---- 0.00 2624 cpu[0] write
755 3% ---- 0.00 2625 cpu[0] fop_write
746 3% ---- 0.00 2626 cpu[0] zfs_write
739 3% ---- 0.00 2751 cpu[1] copen
705 3% ---- 0.00 2712 cpu[1] vn_openat
601 3% ---- 0.00 2841 cpu[2] lookuppnat
599 3% ---- 0.00 2284 cpu0 (usermode)
585 3% ---- 0.00 2837 cpu[2] lookuppnvp
546 2% ---- 0.00 2653 cpu[2] zil_lwb_write_start
541 2% ---- 0.00 2726 cpu[0] pread
493 2% ---- 0.00 2762 cpu[1] read
481 2% ---- 0.00 2811 cpu[0] mutex_enter
451 2% ---- 0.00 2684 cpu0 zio_checksum_generate
451 2% ---- 0.00 2684 cpu0 fletcher_2_native
439 2% ---- 0.00 2740 cpu[2] uiomove
413 2% ---- 0.00 2523 cpu[1] zio_checksum
401 2% ---- 0.00 2969 cpu[0] lookupnameat
384 2% ---- 0.00 2755 cpu[1] zfs_read
372 2% ---- 0.00 2529 cpu[1] vn_createat
371 2% ---- 0.00 2653 cpu[0] lwp_mutex_timedlock
352 2% ---- 0.00 2914 cpu[2] pr_read_lwpusage
321 1% ---- 0.00 2777 cpu[0] bzero
317 1% ---- 0.00 2702 cpu[2] unlink
314 1% ---- 0.00 2695 cpu[2] vn_removeat
313 1% ---- 0.00 2760 cpu[1]+11 disp_getbest
311 1% ---- 0.00 2431 cpu[1] zil_lwb_commit
296 1% ---- 0.00 2774 cpu[2] bcopy_more
289 1% ---- 0.00 2796 cpu[1] copyout_more
280 1% ---- 0.00 2757 cpu0 zfs_grow_blocksize
277 1% ---- 0.00 2754 cpu0 dmu_object_set_blocksize
277 1% ---- 0.00 2592 cpu[1] dmu_write_uio
276 1% ---- 0.00 2912 cpu[2] traverse
274 1% ---- 0.00 2759 cpu0 dnode_set_blksz
269 1% ---- 0.00 2751 cpu0 fop_lookup
263 1% ---- 0.00 2675 cpu[0] lwp_upimutex_lock
262 1% ---- 0.00 2753 cpu0 dbuf_new_size
261 1% ---- 0.00 2942 cpu[1] dnode_hold_impl
246 1% ---- 0.00 2478 cpu[2] fop_create
244 1% ---- 0.00 2480 cpu[2] zfs_create
244 1% ---- 0.00 3080 cpu0 vdev_mirror_io_start
212 1% ---- 0.00 2755 cpu[2] mutex_vector_enter
201 1% ---- 0.00 2709 cpu0 zfs_lookup
197 1% ---- 0.00 2723 cpu[2] fop_remove
194 1% ---- 0.00 3007 cpu[2] zfs_zget
194 1% ---- 0.00 2720 cpu[2] zfs_remove
182 1% ---- 0.00 3040 cpu[2] fsop_root
176 1% ---- 0.00 3073 cpu[2] zfs_root
174 1% ---- 0.00 2841 cpu[1]+11 cv_wait
171 1% ---- 0.00 2593 cpu[0]+6 intr_thread
165 1% ---- 0.00 3246 cpu[1] dbuf_hold_impl
163 1% ---- 0.00 2465 cpu[2] zfs_get_data
162 1% ---- 0.00 2534 cpu[0] dbuf_read
160 1% ---- 0.00 2351 cpu[0] taskq_thread
151 1% ---- 0.00 3264 cpu[1] dbuf_hold
147 1% ---- 0.00 3162 cpu[2] dmu_bonus_hold
143 1% ---- 0.00 3770 cpu0 txg_sync_thread
143 1% ---- 0.00 3770 cpu0 spa_sync
143 1% ---- 0.00 3770 cpu0 dsl_pool_sync
143 1% ---- 0.00 3770 cpu0 dmu_objset_sync
143 1% ---- 0.00 3770 cpu0 dmu_objset_sync_dnodes
141 1% ---- 0.00 3798 cpu0 dsl_dataset_sync
141 1% ---- 0.00 2551 cpu[0] dbuf_read_impl
139 1% ---- 0.00 2683 cpu0 lock_set_spl_spin
130 1% ---- 0.00 2584 cpu[2] zfs_mknode
129 1% ---- 0.00 2693 cpu[2] swtch
127 1% ---- 0.00 2472 cpu[0] dbuf_will_dirty
125 1% ---- 0.00 2839 cpu[2] cstatat
120 1% ---- 0.00 3171 cpu[2] ssd_start_cmds
120 1% ---- 0.00 3177 cpu[2] ssd_core_iostart
117 1% ---- 0.00 2881 cpu[2] cstatat_getvp
117 1% ---- 0.00 2741 cpu[2] disp
115 0% ---- 0.00 3100 cpu[0]+6 pci_intr_wrapper
115 0% ---- 0.00 2786 cpu[2] lwp_mutex_unlock
114 0% ---- 0.00 2669 cpu[2] zfs_dirlook
114 0% ---- 0.00 3863 cpu0 dnode_sync
113 0% ---- 0.00 3081 cpu[0]+6 ql_isr_aif
113 0% ---- 0.00 3081 cpu[0]+6 ql_isr
112 0% ---- 0.00 2955 cpu0 kmem_zalloc
108 0% ---- 0.00 3078 cpu[0]+6 ql_response_pkt
108 0% ---- 0.00 3150 cpu[0] kmem_cache_alloc
106 0% ---- 0.00 2695 cpu[0]+11 turnstile_block
104 0% ---- 0.00 2858 cpu[2] lwp_upimutex_unlock
104 0% ---- 0.00 2695 cpu0 utl0
102 0% ---- 0.00 3120 cpu[0]+6 ql_status_entry
101 0% ---- 0.00 3396 cpu[2] dbuf_find
99 0% ---- 0.00 2685 cpu[1] dmu_buf_hold_array
99 0% ---- 0.00 2794 cpu0 cpu_update_pct
97 0% ---- 0.00 3179 cpu[0]+6 ql_fast_fcp_post
97 0% ---- 0.00 3884 cpu0 dbuf_sync
96 0% ---- 0.00 3209 cpu[2] ssd_xbuf_iostart
96 0% ---- 0.00 2565 cpu[0] zio_done
95 0% ---- 0.00 2748 cpu[1] zfs_dirent_lock
95 0% ---- 0.00 3926 cpu0 arc_write
92 0% ---- 0.00 3952 cpu0 zio_write_compress
90 0% ---- 0.00 2760 cpu[2] copyin_more
87 0% ---- 0.00 2724 cpu[0] dmu_tx_check_ioerr
86 0% ---- 0.00 2673 cpu0 zfs_zaccess
85 0% ---- 0.00 2783 cpu0 zfs_zaccess_common
84 0% ---- 0.00 2729 cpu0 new_mstate
82 0% ---- 0.00 3344 cpu[0]+6 fcp_cmd_callback
78 0% ---- 0.00 3302 cpu[0] fcp_scsi_start
75 0% ---- 0.00 2666 cpu[0] atomic_add_64
75 0% ---- 0.00 2594 cpu[0]+11 _resume_from_idle
74 0% ---- 0.00 2850 cpu[2] lwpchan_get_mapping
73 0% ---- 0.00 2588 cpu[1] dmu_tx_hold_write
72 0% ---- 0.00 2798 cpu[1] prcvtusage
70 0% ---- 0.00 2499 cpu[2] dmu_tx_hold_object_impl
69 0% ---- 0.00 2605 cpu[2] dmu_tx_count_write
69 0% ---- 0.00 3244 cpu[2] pr_p_lock
69 0% ---- 0.00 2782 cpu[2] dnlc_lookup
69 0% ---- 0.00 1829 cpu[1] generic_idle_cpu
67 0% ---- 0.00 2537 cpu[2] dmu_object_alloc
67 0% ---- 0.00 4225 cpu[0] ioctl
66 0% ---- 0.00 2693 cpu0 kmem_cache_free
61 0% ---- 0.00 1500 cpu0 fsflush
60 0% ---- 0.00 2569 cpu0 close
59 0% ---- 0.00 3000 cpu[1] kmem_alloc
59 0% ---- 0.00 2561 cpu0 atomic_add_32
59 0% ---- 0.00 2451 cpu[0]+11 lock_set_spl
58 0% ---- 0.00 2378 cpu0 syscall_mstate
58 0% ---- 0.00 2348 cpu[0] releasef
57 0% ---- 0.00 2599 cpu0 closeandsetf
56 0% ---- 0.00 2190 cpu0 zfs_link_create
56 0% ---- 0.00 2930 cpu[2]+11 scalehrtime
55 0% ---- 0.00 2677 cpu[1] dmu_tx_assign
54 0% ---- 0.00 2453 cpu[2] vdev_disk_io_done
54 0% ---- 0.00 4751 cpu[0] syscall_trap32
53 0% ---- 0.00 2528 cpu[1] dmu_tx_try_assign
53 0% ---- 0.00 2675 cpu[2] copyout
53 0% ---- 0.00 2986 cpu[2] vn_vfsrlock
53 0% ---- 0.00 2916 cpu[0]+11 upimutex_unlock
52 0% ---- 0.00 2473 cpu[2] vdev_queue_io_done
52 0% ---- 0.00 2762 cpu[0] dmu_tx_hold_zap
52 0% ---- 0.00 4686 cpu[0] fop_ioctl
52 0% ---- 0.00 1530 cpu0+0xa cyclic_softint
51 0% ---- 0.00 1510 cpu0+0xa clock
51 0% ---- 0.00 1510 cpu0+0xa cbe_level10
49 0% ---- 0.00 2907 cpu0 vn_vfslocks_getlock
48 0% ---- 0.00 2911 cpu[1] zio_create
48 0% ---- 0.00 2496 cpu0 dbuf_rele
48 0% ---- 0.00 2845 cpu[0]+11 taskq_dispatch
48 0% ---- 0.00 2605 cpu0 closef
48 0% ---- 0.00 1918 cpu0+0xb post_syscall
48 0% ---- 0.00 2642 cpu[0]+11 setfrontdq
47 0% ---- 0.00 2707 cpu[1] hrt2ts
47 0% ---- 0.00 2918 cpu0 cpu_grow
46 0% ---- 0.00 3200 cpu[0]+6 ssdintr
46 0% ---- 0.00 2802 cpu[2] lwpchan_cache_mapping
45 0% ---- 0.00 3605 cpu[2] fcp_prepare_pkt
45 0% ---- 0.00 2678 cpu[1] zfs_link_destroy
45 0% ---- 0.00 2786 cpu0 vdev_disk_io_start
45 0% ---- 0.00 2921 cpu0 prgetusage
44 0% ---- 0.00 5012 cpu[0] di_ioctl
44 0% ---- 0.00 2278 cpu[2] zap_add
[...]
bash-3.00# iostat -xnpcz 3
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 11.7 0.0 256.3 0.0 0.1 0.0 5.8 0 4 md3
0.0 11.7 0.0 256.3 0.0 0.1 0.0 6.4 0 4 md4
0.0 11.7 0.0 256.3 0.0 0.2 0.0 13.2 0 5 d5
0.0 11.7 0.0 256.3 0.0 0.1 0.0 5.8 0 4 c1t0d0
0.0 11.7 0.0 256.3 0.0 0.1 0.0 5.8 0 4 c1t0d0s3
0.0 11.7 0.0 256.3 0.0 0.1 0.0 6.4 0 4 c1t1d0
0.0 11.7 0.0 256.3 0.0 0.1 0.0 6.4 0 4 c1t1d0s3
0.0 26.7 0.0 2091.8 0.0 0.0 0.0 1.5 0 3 c3t19d0
0.0 26.7 0.0 2091.8 0.0 0.0 0.0 1.5 0 3 c3t19d0s0
0.0 26.4 0.0 2145.2 0.0 0.0 0.0 1.2 0 3 c3t42d0
0.0 26.4 0.0 2145.2 0.0 0.0 0.0 1.2 0 3 c3t42d0s0
0.0 28.4 0.0 2119.8 0.0 0.0 0.0 1.3 0 3 c3t20d0
0.0 28.4 0.0 2119.8 0.0 0.0 0.0 1.3 0 3 c3t20d0s0
0.0 26.0 0.0 2111.8 0.0 0.0 0.0 1.3 0 3 c3t16d0
0.0 26.0 0.0 2111.8 0.0 0.0 0.0 1.3 0 3 c3t16d0s0
0.0 26.4 0.0 2145.2 0.0 0.0 0.0 1.6 0 4 c3t43d0
0.0 26.4 0.0 2145.2 0.0 0.0 0.0 1.6 0 4 c3t43d0s0
0.0 30.7 0.0 2428.2 0.0 0.0 0.0 1.5 0 4 c3t25d0
0.0 30.7 0.0 2428.2 0.0 0.0 0.0 1.5 0 4 c3t25d0s0
0.0 26.0 0.0 2111.8 0.0 0.0 0.0 1.5 0 3 c3t17d0
0.0 26.0 0.0 2111.8 0.0 0.0 0.0 1.5 0 3 c3t17d0s0
0.0 26.4 0.0 2131.9 0.0 0.0 0.0 1.7 0 4 c3t39d0
0.0 26.4 0.0 2131.9 0.0 0.0 0.0 1.7 0 4 c3t39d0s0
0.0 26.4 0.0 2131.9 0.0 0.0 0.0 1.3 0 3 c3t38d0
0.0 26.4 0.0 2131.9 0.0 0.0 0.0 1.3 0 3 c3t38d0s0
0.0 30.7 0.0 2428.2 0.0 0.0 0.0 1.3 0 4 c3t24d0
0.0 30.7 0.0 2428.2 0.0 0.0 0.0 1.3 0 4 c3t24d0s0
0.0 28.4 0.0 2119.9 0.0 0.0 0.0 1.5 0 4 c3t21d0
0.0 28.4 0.0 2119.9 0.0 0.0 0.0 1.5 0 4 c3t21d0s0
0.0 26.7 0.0 2091.8 0.0 0.0 0.0 1.2 0 3 c3t18d0
0.0 26.7 0.0 2091.8 0.0 0.0 0.0 1.2 0 3 c3t18d0s0
0.0 26.7 0.0 2115.9 0.0 0.0 0.0 1.1 0 3 c3t40d0
0.0 26.7 0.0 2115.9 0.0 0.0 0.0 1.1 0 3 c3t40d0s0
0.0 29.4 0.0 2329.5 0.0 0.0 0.0 1.3 0 4 c3t36d0
0.0 29.4 0.0 2329.5 0.0 0.0 0.0 1.3 0 4 c3t36d0s0
0.0 35.0 0.0 2790.1 0.0 0.0 0.0 0.9 0 3 c3t26d0
0.0 35.0 0.0 2790.1 0.0 0.0 0.0 0.9 0 3 c3t26d0s0
0.0 29.0 0.0 2329.5 0.0 0.1 0.0 1.7 0 4 c3t37d0
0.0 29.0 0.0 2329.5 0.0 0.1 0.0 1.7 0 4 c3t37d0s0
0.0 40.4 0.0 2866.2 0.0 0.1 0.0 1.3 0 4 c3t35d0
0.0 40.4 0.0 2866.2 0.0 0.1 0.0 1.3 0 4 c3t35d0s0
0.0 35.0 0.0 2790.1 0.0 0.0 0.0 1.3 0 4 c3t27d0
0.0 35.0 0.0 2790.1 0.0 0.0 0.0 1.3 0 4 c3t27d0s0
0.0 26.7 0.0 2115.9 0.0 0.0 0.0 1.5 0 4 c3t41d0
0.0 26.7 0.0 2115.9 0.0 0.0 0.0 1.5 0 4 c3t41d0s0
0.0 25.4 0.0 2112.0 0.0 0.0 0.0 1.2 0 3 c3t22d0
0.0 25.4 0.0 2112.0 0.0 0.0 0.0 1.2 0 3 c3t22d0s0
0.0 37.4 0.0 2824.9 0.0 0.0 0.0 1.3 0 4 c3t33d0
0.0 37.4 0.0 2824.9 0.0 0.0 0.0 1.3 0 4 c3t33d0s0
0.0 25.4 0.0 2112.0 0.0 0.0 0.0 1.6 0 3 c3t23d0
0.0 25.4 0.0 2112.0 0.0 0.0 0.0 1.6 0 3 c3t23d0s0
0.0 37.4 0.0 2824.9 0.0 0.0 0.0 1.0 0 4 c3t32d0
0.0 37.4 0.0 2824.9 0.0 0.0 0.0 1.0 0 4 c3t32d0s0
0.0 40.4 0.0 2866.4 0.0 0.0 0.0 1.1 0 4 c3t34d0
0.0 40.4 0.0 2866.4 0.0 0.0 0.0 1.1 0 4 c3t34d0s0
cpu
us sy wt id
3 24 0 73
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 3.7 0.0 90.6 0.0 0.0 0.0 5.7 0 2 md3
0.0 3.7 0.0 90.6 0.0 0.0 0.0 4.8 0 2 md4
0.0 3.7 0.0 90.6 0.0 0.0 0.0 10.1 0 2 d5
0.0 3.7 0.0 90.6 0.0 0.0 0.0 5.7 0 2 c1t0d0
0.0 3.7 0.0 90.6 0.0 0.0 0.0 5.7 0 2 c1t0d0s3
0.0 3.7 0.0 90.6 0.0 0.0 0.0 4.8 0 2 c1t1d0
0.0 3.7 0.0 90.6 0.0 0.0 0.0 4.8 0 2 c1t1d0s3
0.0 27.6 0.0 2097.8 0.0 0.0 0.0 1.4 0 3 c3t19d0
0.0 27.6 0.0 2097.8 0.0 0.0 0.0 1.4 0 3 c3t19d0s0
0.0 59.9 0.0 2851.9 0.0 0.2 0.0 2.8 0 6 c3t42d0
0.0 59.9 0.0 2851.9 0.0 0.2 0.0 2.8 0 6 c3t42d0s0
0.0 26.6 0.0 2099.1 0.0 0.0 0.0 1.2 0 3 c3t20d0
0.0 26.6 0.0 2099.1 0.0 0.0 0.0 1.2 0 3 c3t20d0s0
0.0 40.0 0.0 2610.7 0.0 0.0 0.0 1.1 0 4 c3t16d0
0.0 40.0 0.0 2610.7 0.0 0.0 0.0 1.1 0 4 c3t16d0s0
0.0 58.3 0.0 2851.9 0.0 0.1 0.0 2.4 0 7 c3t43d0
0.0 58.3 0.0 2851.9 0.0 0.1 0.0 2.4 0 7 c3t43d0s0
0.0 27.0 0.0 2119.9 0.0 0.1 0.0 4.5 0 5 c3t25d0
0.0 27.0 0.0 2119.9 0.0 0.1 0.0 4.5 0 5 c3t25d0s0
0.0 40.0 0.0 2610.7 0.0 0.1 0.0 1.5 0 5 c3t17d0
0.0 40.0 0.0 2610.7 0.0 0.1 0.0 1.5 0 5 c3t17d0s0
0.0 40.0 0.0 2755.9 0.0 0.5 0.0 12.3 0 9 c3t39d0
0.0 40.0 0.0 2755.9 0.0 0.5 0.0 12.3 0 9 c3t39d0s0
0.0 40.0 0.0 2755.9 0.0 0.5 0.0 12.2 0 9 c3t38d0
0.0 40.0 0.0 2755.9 0.0 0.5 0.0 12.2 0 9 c3t38d0s0
0.0 26.3 0.0 2119.9 0.0 0.1 0.0 2.8 0 4 c3t24d0
0.0 26.3 0.0 2119.9 0.0 0.1 0.0 2.8 0 4 c3t24d0s0
0.0 26.6 0.0 2099.1 0.0 0.0 0.0 1.4 0 4 c3t21d0
0.0 26.6 0.0 2099.1 0.0 0.0 0.0 1.4 0 4 c3t21d0s0
0.0 27.6 0.0 2097.7 0.0 0.0 0.0 1.1 0 3 c3t18d0
0.0 27.6 0.0 2097.7 0.0 0.0 0.0 1.1 0 3 c3t18d0s0
0.0 55.9 0.0 2726.4 0.0 0.2 0.0 3.7 0 9 c3t40d0
0.0 55.9 0.0 2726.4 0.0 0.2 0.0 3.7 0 9 c3t40d0s0
0.0 37.3 0.0 2567.0 0.0 0.5 0.0 14.5 0 8 c3t36d0
0.0 37.3 0.0 2567.0 0.0 0.5 0.0 14.5 0 8 c3t36d0s0
0.0 35.0 0.0 2086.5 0.0 0.8 0.0 21.8 0 7 c3t26d0
0.0 35.0 0.0 2086.5 0.0 0.8 0.0 21.8 0 7 c3t26d0s0
0.0 37.0 0.0 2567.0 0.0 0.5 0.0 13.3 0 9 c3t37d0
0.0 37.0 0.0 2567.0 0.0 0.5 0.0 13.3 0 9 c3t37d0s0
0.0 37.3 0.0 2095.2 0.0 0.9 0.0 23.0 0 8 c3t35d0
0.0 37.3 0.0 2095.2 0.0 0.9 0.0 23.0 0 8 c3t35d0s0
0.0 34.0 0.0 2086.5 0.0 0.7 0.0 20.3 0 7 c3t27d0
0.0 34.0 0.0 2086.5 0.0 0.7 0.0 20.3 0 7 c3t27d0s0
0.0 58.3 0.0 2726.3 0.0 0.2 0.0 3.6 0 8 c3t41d0
0.0 58.3 0.0 2726.3 0.0 0.2 0.0 3.6 0 8 c3t41d0s0
0.0 28.3 0.0 2084.4 0.0 0.0 0.0 1.1 0 3 c3t22d0
0.0 28.3 0.0 2084.4 0.0 0.0 0.0 1.1 0 3 c3t22d0s0
0.0 32.6 0.0 2093.8 0.0 0.7 0.0 20.9 0 8 c3t33d0
0.0 32.6 0.0 2093.8 0.0 0.7 0.0 20.9 0 8 c3t33d0s0
0.0 28.3 0.0 2084.4 0.0 0.0 0.0 1.5 0 4 c3t23d0
0.0 28.3 0.0 2084.4 0.0 0.0 0.0 1.5 0 4 c3t23d0s0
0.0 32.6 0.0 2093.8 0.0 0.6 0.0 18.4 0 7 c3t32d0
0.0 32.6 0.0 2093.8 0.0 0.6 0.0 18.4 0 7 c3t32d0s0
0.0 36.6 0.0 2095.2 0.0 0.8 0.0 21.6 0 8 c3t34d0
0.0 36.6 0.0 2095.2 0.0 0.8 0.0 21.6 0 8 c3t34d0s0
cpu
us sy wt id
5 23 0 71
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 3.7 0.0 64.0 0.0 0.0 0.0 8.2 0 2 md3
0.0 3.7 0.0 64.0 0.0 0.0 0.0 6.8 0 2 md4
0.0 3.7 0.0 64.0 0.0 0.0 1.5 13.4 1 2 d5
0.0 4.0 0.0 64.2 0.0 0.0 0.0 8.8 0 3 c1t0d0
0.0 3.7 0.0 64.0 0.0 0.0 0.0 8.2 0 2 c1t0d0s3
0.0 0.3 0.0 0.2 0.0 0.0 0.0 16.2 0 1 c1t0d0s6
0.0 4.0 0.0 64.2 0.0 0.0 0.0 7.5 0 3 c1t1d0
0.0 3.7 0.0 64.0 0.0 0.0 0.0 6.7 0 2 c1t1d0s3
0.0 0.3 0.0 0.2 0.0 0.0 0.0 15.8 0 1 c1t1d0s6
0.0 33.3 0.0 2772.5 0.0 0.1 0.0 1.6 0 4 c3t19d0
0.0 33.3 0.0 2772.5 0.0 0.1 0.0 1.6 0 4 c3t19d0s0
0.0 29.0 0.0 2125.7 0.0 0.0 0.0 1.2 0 3 c3t42d0
0.0 29.0 0.0 2125.7 0.0 0.0 0.0 1.2 0 3 c3t42d0s0
0.0 37.3 0.0 2825.9 0.0 0.0 0.0 1.1 0 4 c3t20d0
0.0 37.3 0.0 2825.9 0.0 0.0 0.0 1.1 0 4 c3t20d0s0
0.0 29.3 0.0 2319.1 0.0 0.0 0.0 1.1 0 3 c3t16d0
0.0 29.3 0.0 2319.1 0.0 0.0 0.0 1.1 0 3 c3t16d0s0
0.0 29.0 0.0 2125.7 0.0 0.0 0.0 1.5 0 4 c3t43d0
0.0 29.0 0.0 2125.7 0.0 0.0 0.0 1.5 0 4 c3t43d0s0
0.0 29.7 0.0 2345.8 0.0 0.0 0.0 1.5 0 4 c3t25d0
0.0 29.7 0.0 2345.8 0.0 0.0 0.0 1.5 0 4 c3t25d0s0
0.0 29.3 0.0 2319.1 0.0 0.0 0.0 1.4 0 4 c3t17d0
0.0 29.3 0.0 2319.1 0.0 0.0 0.0 1.4 0 4 c3t17d0s0
0.0 28.0 0.0 2068.4 0.0 0.0 0.0 1.5 0 4 c3t39d0
0.0 28.0 0.0 2068.4 0.0 0.0 0.0 1.5 0 4 c3t39d0s0
0.0 28.0 0.0 2068.4 0.0 0.0 0.0 1.1 0 3 c3t38d0
0.0 28.0 0.0 2068.4 0.0 0.0 0.0 1.1 0 3 c3t38d0s0
0.0 30.0 0.0 2384.5 0.0 0.0 0.0 1.3 0 4 c3t24d0
0.0 30.0 0.0 2384.5 0.0 0.0 0.0 1.3 0 4 c3t24d0s0
0.0 37.3 0.0 2825.9 0.0 0.1 0.0 1.4 0 5 c3t21d0
0.0 37.3 0.0 2825.9 0.0 0.1 0.0 1.4 0 5 c3t21d0s0
0.0 33.3 0.0 2772.5 0.0 0.0 0.0 1.1 0 3 c3t18d0
0.0 33.3 0.0 2772.5 0.0 0.0 0.0 1.1 0 3 c3t18d0s0
0.0 27.0 0.0 2115.1 0.0 0.0 0.0 1.0 0 2 c3t40d0
0.0 27.0 0.0 2115.1 0.0 0.0 0.0 1.0 0 2 c3t40d0s0
0.0 28.3 0.0 2079.1 0.0 0.0 0.0 1.2 0 3 c3t36d0
0.0 28.3 0.0 2079.1 0.0 0.0 0.0 1.2 0 3 c3t36d0s0
0.0 28.0 0.0 2115.1 0.0 0.0 0.0 1.0 0 3 c3t26d0
0.0 28.0 0.0 2115.1 0.0 0.0 0.0 1.0 0 3 c3t26d0s0
0.0 28.3 0.0 2079.1 0.0 0.0 0.0 1.4 0 4 c3t37d0
0.0 28.3 0.0 2079.1 0.0 0.0 0.0 1.4 0 4 c3t37d0s0
0.0 27.0 0.0 2081.8 0.0 0.0 0.0 1.7 0 4 c3t35d0
0.0 27.0 0.0 2081.8 0.0 0.0 0.0 1.7 0 4 c3t35d0s0
0.0 28.0 0.0 2115.1 0.0 0.0 0.0 1.4 0 4 c3t27d0
0.0 28.0 0.0 2115.1 0.0 0.0 0.0 1.4 0 4 c3t27d0s0
0.0 27.0 0.0 2115.1 0.0 0.0 0.0 1.4 0 3 c3t41d0
0.0 27.0 0.0 2115.1 0.0 0.0 0.0 1.4 0 3 c3t41d0s0
0.0 37.0 0.0 2756.5 0.0 0.1 0.0 1.5 0 5 c3t22d0
0.0 37.0 0.0 2756.5 0.0 0.1 0.0 1.5 0 5 c3t22d0s0
0.0 25.0 0.0 2084.4 0.0 0.0 0.0 1.6 0 3 c3t33d0
0.0 25.0 0.0 2084.4 0.0 0.0 0.0 1.6 0 3 c3t33d0s0
0.0 37.0 0.0 2756.5 0.0 0.1 0.0 1.5 0 5 c3t23d0
0.0 37.0 0.0 2756.5 0.0 0.1 0.0 1.5 0 5 c3t23d0s0
0.0 25.0 0.0 2084.4 0.0 0.0 0.0 1.1 0 2 c3t32d0
0.0 25.0 0.0 2084.4 0.0 0.0 0.0 1.1 0 2 c3t32d0s0
0.0 27.0 0.0 2081.7 0.0 0.0 0.0 1.3 0 3 c3t34d0
0.0 27.0 0.0 2081.7 0.0 0.0 0.0 1.3 0 3 c3t34d0s0
cpu
us sy wt id
3 23 0 74
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 3.3 0.0 64.0 0.0 0.0 0.0 8.9 0 2 md3
0.0 3.3 0.0 64.0 0.0 0.0 0.0 7.5 0 2 md4
0.0 3.3 0.0 64.0 0.0 0.0 0.0 14.2 0 2 d5
0.0 3.7 0.0 64.2 0.0 0.0 0.0 8.9 0 2 c1t0d0
0.0 3.3 0.0 64.0 0.0 0.0 0.0 8.8 0 2 c1t0d0s3
0.0 0.3 0.0 0.2 0.0 0.0 0.0 10.0 0 0 c1t0d0s6
0.0 3.7 0.0 64.2 0.0 0.0 0.0 7.9 0 2 c1t1d0
0.0 3.3 0.0 64.0 0.0 0.0 0.0 7.4 0 2 c1t1d0s3
0.0 0.3 0.0 0.2 0.0 0.0 0.0 12.2 0 0 c1t1d0s6
0.0 33.7 0.0 2119.1 0.0 0.7 0.0 19.6 0 7 c3t19d0
0.0 33.7 0.0 2119.1 0.0 0.7 0.0 19.6 0 7 c3t19d0s0
0.0 30.3 0.0 2132.6 0.0 0.3 0.0 9.4 0 5 c3t42d0
0.0 30.3 0.0 2132.6 0.0 0.3 0.0 9.4 0 5 c3t42d0s0
0.0 33.7 0.0 2134.3 0.0 0.5 0.0 16.1 0 7 c3t20d0
0.0 33.7 0.0 2134.3 0.0 0.5 0.0 16.1 0 7 c3t20d0s0
0.0 31.7 0.0 2104.4 0.0 0.5 0.0 17.1 0 7 c3t16d0
0.0 31.7 0.0 2104.4 0.0 0.5 0.0 17.1 0 7 c3t16d0s0
0.0 31.7 0.0 2132.6 0.0 0.3 0.0 11.0 0 7 c3t43d0
0.0 31.7 0.0 2132.6 0.0 0.3 0.0 11.0 0 7 c3t43d0s0
0.0 65.7 0.0 2616.7 0.0 0.4 0.0 6.1 0 9 c3t25d0
0.0 65.7 0.0 2616.7 0.0 0.4 0.0 6.1 0 9 c3t25d0s0
0.0 31.0 0.0 2104.4 0.0 0.5 0.0 16.5 0 7 c3t17d0
0.0 31.0 0.0 2104.4 0.0 0.5 0.0 16.5 0 7 c3t17d0s0
0.0 28.3 0.0 2104.6 0.0 0.0 0.0 1.4 0 3 c3t39d0
0.0 28.3 0.0 2104.6 0.0 0.0 0.0 1.4 0 3 c3t39d0s0
0.0 28.3 0.0 2104.6 0.0 0.0 0.0 1.1 0 3 c3t38d0
0.0 28.3 0.0 2104.6 0.0 0.0 0.0 1.1 0 3 c3t38d0s0
0.0 58.7 0.0 2578.1 0.0 0.4 0.0 6.6 0 8 c3t24d0
0.0 58.7 0.0 2578.1 0.0 0.4 0.0 6.6 0 8 c3t24d0s0
0.0 34.0 0.0 2134.3 0.0 0.6 0.0 17.0 0 8 c3t21d0
0.0 34.0 0.0 2134.3 0.0 0.6 0.0 17.0 0 8 c3t21d0s0
0.0 33.7 0.0 2119.1 0.0 0.6 0.0 19.1 0 7 c3t18d0
0.0 33.7 0.0 2119.1 0.0 0.6 0.0 19.1 0 7 c3t18d0s0
0.0 28.0 0.0 2071.6 0.0 0.1 0.0 2.4 0 4 c3t40d0
0.0 28.0 0.0 2071.6 0.0 0.1 0.0 2.4 0 4 c3t40d0s0
0.0 29.0 0.0 2116.6 0.0 0.0 0.0 1.1 0 3 c3t36d0
0.0 29.0 0.0 2116.6 0.0 0.0 0.0 1.1 0 3 c3t36d0s0
0.0 50.0 0.0 2868.1 0.0 0.1 0.0 1.1 0 5 c3t26d0
0.0 50.0 0.0 2868.1 0.0 0.1 0.0 1.1 0 5 c3t26d0s0
0.0 29.0 0.0 2116.6 0.0 0.0 0.0 1.4 0 4 c3t37d0
0.0 29.0 0.0 2116.6 0.0 0.0 0.0 1.4 0 4 c3t37d0s0
0.0 38.0 0.0 2607.4 0.0 0.1 0.0 1.4 0 5 c3t35d0
0.0 38.0 0.0 2607.4 0.0 0.1 0.0 1.4 0 5 c3t35d0s0
0.0 49.7 0.0 2868.1 0.0 0.1 0.0 1.3 0 5 c3t27d0
0.0 49.7 0.0 2868.1 0.0 0.1 0.0 1.3 0 5 c3t27d0s0
0.0 28.0 0.0 2071.6 0.0 0.1 0.0 3.1 0 5 c3t41d0
0.0 28.0 0.0 2071.6 0.0 0.1 0.0 3.1 0 5 c3t41d0s0
0.0 50.0 0.0 2055.3 0.0 0.4 0.0 7.6 0 7 c3t22d0
0.0 50.0 0.0 2055.3 0.0 0.4 0.0 7.6 0 7 c3t22d0s0
0.0 41.0 0.0 2821.7 0.0 0.1 0.0 1.3 0 5 c3t33d0
0.0 41.0 0.0 2821.7 0.0 0.1 0.0 1.3 0 5 c3t33d0s0
0.0 44.3 0.0 2055.3 0.0 0.4 0.0 9.1 0 9 c3t23d0
0.0 44.3 0.0 2055.3 0.0 0.4 0.0 9.1 0 9 c3t23d0s0
0.0 41.0 0.0 2821.7 0.0 0.0 0.0 1.1 0 4 c3t32d0
0.0 41.0 0.0 2821.7 0.0 0.0 0.0 1.1 0 4 c3t32d0s0
0.0 38.3 0.0 2647.5 0.0 0.0 0.0 1.2 0 4 c3t34d0
0.0 38.3 0.0 2647.5 0.0 0.0 0.0 1.2 0 4 c3t34d0s0
cpu
us sy wt id
3 24 0 73
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 3.0 0.0 71.9 0.0 0.0 0.0 5.4 0 2 md3
0.0 3.0 0.0 71.9 0.0 0.0 0.0 5.1 0 1 md4
0.0 3.0 0.0 71.9 0.0 0.0 0.0 10.2 0 2 d5
0.0 3.0 0.0 71.9 0.0 0.0 0.0 5.3 0 2 c1t0d0
0.0 3.0 0.0 71.9 0.0 0.0 0.0 5.3 0 2 c1t0d0s3
0.0 3.0 0.0 71.9 0.0 0.0 0.0 5.0 0 1 c1t1d0
0.0 3.0 0.0 71.9 0.0 0.0 0.0 5.0 0 1 c1t1d0s3
0.0 29.0 0.0 2090.8 0.0 0.0 0.0 1.3 0 3 c3t19d0
0.0 29.0 0.0 2090.8 0.0 0.0 0.0 1.3 0 3 c3t19d0s0
0.0 39.3 0.0 2143.9 0.0 0.0 0.0 1.1 0 4 c3t42d0
0.0 39.3 0.0 2143.9 0.0 0.0 0.0 1.1 0 4 c3t42d0s0
0.0 28.3 0.0 2088.8 0.0 0.1 0.0 1.8 0 3 c3t20d0
0.0 28.3 0.0 2088.8 0.0 0.1 0.0 1.8 0 3 c3t20d0s0
0.0 29.0 0.0 2133.4 0.0 0.0 0.0 1.2 0 3 c3t16d0
0.0 29.0 0.0 2133.4 0.0 0.0 0.0 1.2 0 3 c3t16d0s0
0.0 39.3 0.0 2143.8 0.0 0.0 0.0 1.2 0 4 c3t43d0
0.0 39.3 0.0 2143.8 0.0 0.0 0.0 1.2 0 4 c3t43d0s0
0.0 30.6 0.0 2087.9 0.0 0.3 0.0 9.4 0 6 c3t25d0
0.0 30.6 0.0 2087.9 0.0 0.3 0.0 9.4 0 6 c3t25d0s0
0.0 29.0 0.0 2133.4 0.0 0.0 0.0 1.5 0 4 c3t17d0
0.0 29.0 0.0 2133.4 0.0 0.0 0.0 1.5 0 4 c3t17d0s0
0.0 56.9 0.0 2756.7 0.0 0.2 0.0 4.4 0 9 c3t39d0
0.0 56.9 0.0 2756.7 0.0 0.2 0.0 4.4 0 9 c3t39d0s0
0.0 54.6 0.0 2756.7 0.0 0.2 0.0 2.9 0 7 c3t38d0
0.0 54.6 0.0 2756.7 0.0 0.2 0.0 2.9 0 7 c3t38d0s0
0.0 30.6 0.0 2087.9 0.0 0.3 0.0 10.4 0 6 c3t24d0
0.0 30.6 0.0 2087.9 0.0 0.3 0.0 10.4 0 6 c3t24d0s0
0.0 28.3 0.0 2088.7 0.0 0.1 0.0 2.1 0 4 c3t21d0
0.0 28.3 0.0 2088.7 0.0 0.1 0.0 2.1 0 4 c3t21d0s0
0.0 29.0 0.0 2090.7 0.0 0.0 0.0 1.1 0 3 c3t18d0
0.0 29.0 0.0 2090.7 0.0 0.0 0.0 1.1 0 3 c3t18d0s0
0.0 52.3 0.0 2861.9 0.0 0.1 0.0 2.3 0 7 c3t40d0
0.0 52.3 0.0 2861.9 0.0 0.1 0.0 2.3 0 7 c3t40d0s0
0.0 40.3 0.0 2122.9 0.0 0.1 0.0 1.9 0 4 c3t36d0
0.0 40.3 0.0 2122.9 0.0 0.1 0.0 1.9 0 4 c3t36d0s0
0.0 29.3 0.0 2085.6 0.0 0.5 0.0 16.7 0 7 c3t26d0
0.0 29.3 0.0 2085.6 0.0 0.5 0.0 16.7 0 7 c3t26d0s0
0.0 42.6 0.0 2122.9 0.0 0.1 0.0 2.6 0 5 c3t37d0
0.0 42.6 0.0 2122.9 0.0 0.1 0.0 2.6 0 5 c3t37d0s0
0.0 47.6 0.0 2258.5 0.0 0.3 0.0 5.3 0 8 c3t35d0
0.0 47.6 0.0 2258.5 0.0 0.3 0.0 5.3 0 8 c3t35d0s0
0.0 29.3 0.0 2085.6 0.0 0.5 0.0 17.0 0 7 c3t27d0
0.0 29.3 0.0 2085.6 0.0 0.5 0.0 17.0 0 7 c3t27d0s0
0.0 52.3 0.0 2861.9 0.0 0.2 0.0 3.2 0 8 c3t41d0
0.0 52.3 0.0 2861.9 0.0 0.2 0.0 3.2 0 8 c3t41d0s0
0.0 31.3 0.0 2096.7 0.0 0.4 0.0 14.2 0 7 c3t22d0
0.0 31.3 0.0 2096.7 0.0 0.4 0.0 14.2 0 7 c3t22d0s0
0.0 34.6 0.0 2084.7 0.0 0.5 0.0 15.7 0 8 c3t33d0
0.0 34.6 0.0 2084.7 0.0 0.5 0.0 15.7 0 8 c3t33d0s0
0.0 31.0 0.0 2096.7 0.0 0.4 0.0 13.4 0 7 c3t23d0
0.0 31.0 0.0 2096.7 0.0 0.4 0.0 13.4 0 7 c3t23d0s0
0.0 34.6 0.0 2084.7 0.0 0.5 0.0 15.9 0 8 c3t32d0
0.0 34.6 0.0 2084.7 0.0 0.5 0.0 15.9 0 8 c3t32d0s0
0.0 48.3 0.0 2218.6 0.0 0.2 0.0 4.3 0 7 c3t34d0
0.0 48.3 0.0 2218.6 0.0 0.2 0.0 4.3 0 7 c3t34d0s0
cpu
us sy wt id
4 24 0 73
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 4.7 0.0 90.8 0.0 0.0 0.0 6.6 0 2 md3
0.0 4.7 0.0 90.8 0.0 0.0 0.0 7.3 0 3 md4
0.0 4.7 0.0 90.8 0.0 0.1 0.0 10.8 0 3 d5
0.0 4.7 0.0 90.8 0.0 0.0 0.0 6.6 0 2 c1t0d0
0.0 4.7 0.0 90.8 0.0 0.0 0.0 6.6 0 2 c1t0d0s3
0.0 4.7 0.0 90.8 0.0 0.0 0.0 7.3 0 3 c1t1d0
0.0 4.7 0.0 90.8 0.0 0.0 0.0 7.3 0 3 c1t1d0s3
0.0 36.0 0.0 2808.4 0.0 0.0 0.0 1.4 0 4 c3t19d0
0.0 36.0 0.0 2808.4 0.0 0.0 0.0 1.4 0 4 c3t19d0s0
0.0 36.0 0.0 2784.4 0.0 0.0 0.0 1.1 0 4 c3t42d0
0.0 36.0 0.0 2784.4 0.0 0.0 0.0 1.1 0 4 c3t42d0s0
0.0 37.0 0.0 2468.0 0.0 0.0 0.0 1.0 0 3 c3t20d0
0.0 37.0 0.0 2468.0 0.0 0.0 0.0 1.0 0 3 c3t20d0s0
0.0 36.0 0.0 2765.7 0.0 0.0 0.0 1.2 0 4 c3t16d0
0.0 36.0 0.0 2765.7 0.0 0.0 0.0 1.2 0 4 c3t16d0s0
0.0 36.0 0.0 2784.4 0.0 0.1 0.0 1.5 0 5 c3t43d0
0.0 36.0 0.0 2784.4 0.0 0.1 0.0 1.5 0 5 c3t43d0s0
0.0 26.7 0.0 2141.0 0.0 0.0 0.0 1.9 0 5 c3t25d0
0.0 26.7 0.0 2141.0 0.0 0.0 0.0 1.9 0 5 c3t25d0s0
0.0 36.0 0.0 2765.7 0.0 0.1 0.0 1.5 0 5 c3t17d0
0.0 36.0 0.0 2765.7 0.0 0.1 0.0 1.5 0 5 c3t17d0s0
0.0 29.7 0.0 2105.0 0.0 0.0 0.0 1.4 0 4 c3t39d0
0.0 29.7 0.0 2105.0 0.0 0.0 0.0 1.4 0 4 c3t39d0s0
0.0 29.7 0.0 2105.0 0.0 0.0 0.0 1.3 0 4 c3t38d0
0.0 29.7 0.0 2105.0 0.0 0.0 0.0 1.3 0 4 c3t38d0s0
0.0 26.7 0.0 2141.0 0.0 0.0 0.0 1.5 0 4 c3t24d0
0.0 26.7 0.0 2141.0 0.0 0.0 0.0 1.5 0 4 c3t24d0s0
0.0 37.4 0.0 2468.0 0.0 0.0 0.0 1.3 0 4 c3t21d0
0.0 37.4 0.0 2468.0 0.0 0.0 0.0 1.3 0 4 c3t21d0s0
0.0 36.0 0.0 2808.4 0.0 0.0 0.0 1.1 0 4 c3t18d0
0.0 36.0 0.0 2808.4 0.0 0.0 0.0 1.1 0 4 c3t18d0s0
0.0 25.7 0.0 2098.3 0.0 0.0 0.0 1.1 0 3 c3t40d0
0.0 25.7 0.0 2098.3 0.0 0.0 0.0 1.1 0 3 c3t40d0s0
0.0 30.4 0.0 2121.0 0.0 0.0 0.0 1.1 0 3 c3t36d0
0.0 30.4 0.0 2121.0 0.0 0.0 0.0 1.1 0 3 c3t36d0s0
0.0 28.4 0.0 2097.0 0.0 0.0 0.0 1.0 0 3 c3t26d0
0.0 28.4 0.0 2097.0 0.0 0.0 0.0 1.0 0 3 c3t26d0s0
0.0 30.4 0.0 2121.0 0.0 0.0 0.0 1.3 0 4 c3t37d0
0.0 30.4 0.0 2121.0 0.0 0.0 0.0 1.3 0 4 c3t37d0s0
0.0 26.7 0.0 2076.9 0.0 0.0 0.0 1.6 0 4 c3t35d0
0.0 26.7 0.0 2076.9 0.0 0.0 0.0 1.6 0 4 c3t35d0s0
0.0 28.4 0.0 2097.0 0.0 0.0 0.0 1.3 0 3 c3t27d0
0.0 28.4 0.0 2097.0 0.0 0.0 0.0 1.3 0 3 c3t27d0s0
0.0 25.7 0.0 2098.3 0.0 0.0 0.0 1.6 0 4 c3t41d0
0.0 25.7 0.0 2098.3 0.0 0.0 0.0 1.6 0 4 c3t41d0s0
0.0 28.0 0.0 2114.3 0.0 0.0 0.0 1.2 0 3 c3t22d0
0.0 28.0 0.0 2114.3 0.0 0.0 0.0 1.2 0 3 c3t22d0s0
0.0 25.0 0.0 2122.3 0.0 0.0 0.0 1.6 0 4 c3t33d0
0.0 25.0 0.0 2122.3 0.0 0.0 0.0 1.6 0 4 c3t33d0s0
0.0 28.0 0.0 2114.3 0.0 0.0 0.0 1.4 0 3 c3t23d0
0.0 28.0 0.0 2114.3 0.0 0.0 0.0 1.4 0 3 c3t23d0s0
0.0 25.0 0.0 2122.3 0.0 0.0 0.0 1.2 0 3 c3t32d0
0.0 25.0 0.0 2122.3 0.0 0.0 0.0 1.2 0 3 c3t32d0s0
0.0 26.7 0.0 2076.9 0.0 0.0 0.0 1.2 0 3 c3t34d0
0.0 26.7 0.0 2076.9 0.0 0.0 0.0 1.2 0 3 c3t34d0s0
cpu
us sy wt id
3 24 0 73
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.7 0.0 21.3 0.0 0.0 0.0 3.6 0 0 md3
0.0 1.7 0.0 21.3 0.0 0.0 0.0 5.5 0 1 md4
0.0 1.7 0.0 21.3 0.0 0.0 0.0 9.1 0 1 d5
0.0 2.0 0.0 21.5 0.0 0.0 0.0 4.4 0 1 c1t0d0
0.0 1.7 0.0 21.3 0.0 0.0 0.0 3.6 0 0 c1t0d0s3
0.0 0.3 0.0 0.2 0.0 0.0 0.0 8.6 0 0 c1t0d0s6
0.0 2.0 0.0 21.5 0.0 0.0 0.0 6.5 0 1 c1t1d0
0.0 1.7 0.0 21.3 0.0 0.0 0.0 5.5 0 1 c1t1d0s3
0.0 0.3 0.0 0.2 0.0 0.0 0.0 11.5 0 0 c1t1d0s6
0.0 35.0 0.0 2092.9 0.0 0.7 0.0 18.9 0 8 c3t19d0
0.0 35.0 0.0 2092.9 0.0 0.7 0.0 18.9 0 8 c3t19d0s0
0.0 30.0 0.0 2109.5 0.0 0.2 0.0 6.2 0 5 c3t42d0
0.0 30.0 0.0 2109.5 0.0 0.2 0.0 6.2 0 5 c3t42d0s0
0.0 33.3 0.0 2406.1 0.0 0.4 0.0 13.4 0 7 c3t20d0
0.0 33.3 0.0 2406.1 0.0 0.4 0.0 13.4 0 7 c3t20d0s0
0.0 33.3 0.0 2091.5 0.0 0.7 0.0 20.1 0 8 c3t16d0
0.0 33.3 0.0 2091.5 0.0 0.7 0.0 20.1 0 8 c3t16d0s0
0.0 29.7 0.0 2109.5 0.0 0.2 0.0 6.8 0 5 c3t43d0
0.0 29.7 0.0 2109.5 0.0 0.2 0.0 6.8 0 5 c3t43d0s0
0.0 38.0 0.0 2796.1 0.0 0.2 0.0 4.9 0 8 c3t25d0
0.0 38.0 0.0 2796.1 0.0 0.2 0.0 4.9 0 8 c3t25d0s0
0.0 33.7 0.0 2091.5 0.0 0.8 0.0 23.5 0 9 c3t17d0
0.0 33.7 0.0 2091.5 0.0 0.8 0.0 23.5 0 9 c3t17d0s0
0.0 27.7 0.0 2097.5 0.0 0.0 0.0 1.4 0 4 c3t39d0
0.0 27.7 0.0 2097.5 0.0 0.0 0.0 1.4 0 4 c3t39d0s0
0.0 27.7 0.0 2097.5 0.0 0.0 0.0 1.3 0 3 c3t38d0
0.0 27.7 0.0 2097.5 0.0 0.0 0.0 1.3 0 3 c3t38d0s0
0.0 38.0 0.0 2796.1 0.0 0.2 0.0 4.6 0 7 c3t24d0
0.0 38.0 0.0 2796.1 0.0 0.2 0.0 4.6 0 7 c3t24d0s0
0.0 32.7 0.0 2406.1 0.0 0.4 0.0 13.2 0 8 c3t21d0
0.0 32.7 0.0 2406.1 0.0 0.4 0.0 13.2 0 8 c3t21d0s0
0.0 35.0 0.0 2092.9 0.0 0.7 0.0 18.8 0 8 c3t18d0
0.0 35.0 0.0 2092.9 0.0 0.7 0.0 18.8 0 8 c3t18d0s0
0.0 28.0 0.0 2105.6 0.0 0.0 0.0 1.0 0 3 c3t40d0
0.0 28.0 0.0 2105.6 0.0 0.0 0.0 1.0 0 3 c3t40d0s0
0.0 32.0 0.0 2418.9 0.0 0.0 0.0 1.0 0 3 c3t36d0
0.0 32.0 0.0 2418.9 0.0 0.0 0.0 1.0 0 3 c3t36d0s0
0.0 57.7 0.0 2764.6 0.0 0.2 0.0 2.9 0 7 c3t26d0
0.0 57.7 0.0 2764.6 0.0 0.2 0.0 2.9 0 7 c3t26d0s0
0.0 32.0 0.0 2418.9 0.0 0.0 0.0 1.3 0 4 c3t37d0
0.0 32.0 0.0 2418.9 0.0 0.0 0.0 1.3 0 4 c3t37d0s0
0.0 44.7 0.0 2805.3 0.0 0.1 0.0 1.3 0 5 c3t35d0
0.0 44.7 0.0 2805.3 0.0 0.1 0.0 1.3 0 5 c3t35d0s0
0.0 60.3 0.0 2764.6 0.0 0.2 0.0 4.1 0 9 c3t27d0
0.0 60.3 0.0 2764.6 0.0 0.2 0.0 4.1 0 9 c3t27d0s0
0.0 28.0 0.0 2105.6 0.0 0.0 0.0 1.4 0 3 c3t41d0
0.0 28.0 0.0 2105.6 0.0 0.0 0.0 1.4 0 3 c3t41d0s0
0.0 39.3 0.0 2747.2 0.0 0.5 0.0 13.7 0 8 c3t22d0
0.0 39.3 0.0 2747.2 0.0 0.5 0.0 13.7 0 8 c3t22d0s0
0.0 60.3 0.0 2906.8 0.0 0.2 0.0 3.1 0 7 c3t33d0
0.0 60.3 0.0 2906.8 0.0 0.2 0.0 3.1 0 7 c3t33d0s0
0.0 39.3 0.0 2747.2 0.0 0.5 0.0 13.6 0 9 c3t23d0
0.0 39.3 0.0 2747.2 0.0 0.5 0.0 13.6 0 9 c3t23d0s0
0.0 58.7 0.0 2906.9 0.0 0.2 0.0 2.8 0 6 c3t32d0
0.0 58.7 0.0 2906.9 0.0 0.2 0.0 2.8 0 6 c3t32d0s0
0.0 44.7 0.0 2805.4 0.0 0.0 0.0 1.1 0 4 c3t34d0
0.0 44.7 0.0 2805.4 0.0 0.0 0.0 1.1 0 4 c3t34d0s0
cpu
us sy wt id
3 23 0 74
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 3.0 0.0 58.8 0.0 0.0 0.0 7.7 0 2 md3
0.0 3.0 0.0 58.8 0.0 0.0 0.0 7.1 0 2 md4
0.0 3.0 0.0 58.8 0.0 0.0 1.6 11.8 0 2 d5
0.0 3.3 0.0 59.0 0.0 0.0 0.0 8.2 0 2 c1t0d0
0.0 3.0 0.0 58.8 0.0 0.0 0.0 7.6 0 2 c1t0d0s3
0.0 0.3 0.0 0.2 0.0 0.0 0.0 13.8 0 0 c1t0d0s6
0.0 3.3 0.0 59.0 0.0 0.0 0.0 6.9 0 2 c1t1d0
0.0 3.0 0.0 58.8 0.0 0.0 0.0 7.1 0 2 c1t1d0s3
0.0 0.3 0.0 0.2 0.0 0.0 0.0 5.5 0 0 c1t1d0s6
0.0 28.0 0.0 2131.8 0.0 0.0 0.0 1.5 0 4 c3t19d0
0.0 28.0 0.0 2131.8 0.0 0.0 0.0 1.5 0 4 c3t19d0s0
0.0 32.3 0.0 2438.4 0.0 0.0 0.0 1.3 0 4 c3t42d0
0.0 32.3 0.0 2438.4 0.0 0.0 0.0 1.3 0 4 c3t42d0s0
0.0 26.3 0.0 2075.8 0.0 0.0 0.0 1.2 0 3 c3t20d0
0.0 26.3 0.0 2075.8 0.0 0.0 0.0 1.2 0 3 c3t20d0s0
0.0 24.7 0.0 2093.1 0.0 0.0 0.0 1.2 0 3 c3t16d0
0.0 24.7 0.0 2093.1 0.0 0.0 0.0 1.2 0 3 c3t16d0s0
0.0 32.3 0.0 2438.4 0.0 0.0 0.0 1.5 0 4 c3t43d0
0.0 32.3 0.0 2438.4 0.0 0.0 0.0 1.5 0 4 c3t43d0s0
0.0 26.3 0.0 2106.4 0.0 0.0 0.0 1.5 0 4 c3t25d0
0.0 26.3 0.0 2106.4 0.0 0.0 0.0 1.5 0 4 c3t25d0s0
0.0 24.7 0.0 2093.1 0.0 0.0 0.0 1.6 0 3 c3t17d0
0.0 24.7 0.0 2093.1 0.0 0.0 0.0 1.6 0 3 c3t17d0s0
0.0 41.7 0.0 2817.0 0.0 0.1 0.0 1.3 0 5 c3t39d0
0.0 41.7 0.0 2817.0 0.0 0.1 0.0 1.3 0 5 c3t39d0s0
0.0 41.7 0.0 2817.0 0.0 0.0 0.0 1.0 0 4 c3t38d0
0.0 41.7 0.0 2817.0 0.0 0.0 0.0 1.0 0 4 c3t38d0s0
0.0 26.3 0.0 2106.4 0.0 0.0 0.0 1.4 0 3 c3t24d0
0.0 26.3 0.0 2106.4 0.0 0.0 0.0 1.4 0 3 c3t24d0s0
0.0 26.3 0.0 2075.8 0.0 0.0 0.0 1.5 0 3 c3t21d0
0.0 26.3 0.0 2075.8 0.0 0.0 0.0 1.5 0 3 c3t21d0s0
0.0 28.0 0.0 2131.8 0.0 0.0 0.0 1.2 0 3 c3t18d0
0.0 28.0 0.0 2131.8 0.0 0.0 0.0 1.2 0 3 c3t18d0s0
0.0 40.7 0.0 2863.7 0.0 0.0 0.0 1.1 0 4 c3t40d0
0.0 40.7 0.0 2863.7 0.0 0.0 0.0 1.1 0 4 c3t40d0s0
0.0 37.7 0.0 2519.7 0.0 0.0 0.0 0.9 0 3 c3t36d0
0.0 37.7 0.0 2519.7 0.0 0.0 0.0 0.9 0 3 c3t36d0s0
0.0 27.3 0.0 2121.1 0.0 0.0 0.0 1.3 0 3 c3t26d0
0.0 27.3 0.0 2121.1 0.0 0.0 0.0 1.3 0 3 c3t26d0s0
0.0 37.7 0.0 2519.7 0.0 0.0 0.0 1.2 0 4 c3t37d0
0.0 37.7 0.0 2519.7 0.0 0.0 0.0 1.2 0 4 c3t37d0s0
0.0 27.3 0.0 2109.1 0.0 0.0 0.0 1.4 0 3 c3t35d0
0.0 27.3 0.0 2109.1 0.0 0.0 0.0 1.4 0 3 c3t35d0s0
0.0 27.3 0.0 2121.1 0.0 0.0 0.0 1.8 0 4 c3t27d0
0.0 27.3 0.0 2121.1 0.0 0.0 0.0 1.8 0 4 c3t27d0s0
0.0 40.7 0.0 2863.7 0.0 0.1 0.0 1.4 0 5 c3t41d0
0.0 40.7 0.0 2863.7 0.0 0.1 0.0 1.4 0 5 c3t41d0s0
0.0 25.7 0.0 2101.0 0.0 0.0 0.0 1.1 0 3 c3t22d0
0.0 25.7 0.0 2101.0 0.0 0.0 0.0 1.1 0 3 c3t22d0s0
0.0 27.0 0.0 2135.7 0.0 0.0 0.0 1.5 0 4 c3t33d0
0.0 27.0 0.0 2135.7 0.0 0.0 0.0 1.5 0 4 c3t33d0s0
0.0 25.7 0.0 2101.0 0.0 0.0 0.0 1.5 0 3 c3t23d0
0.0 25.7 0.0 2101.0 0.0 0.0 0.0 1.5 0 3 c3t23d0s0
0.0 27.0 0.0 2135.7 0.0 0.0 0.0 1.1 0 3 c3t32d0
0.0 27.0 0.0 2135.7 0.0 0.0 0.0 1.1 0 3 c3t32d0s0
0.0 27.3 0.0 2109.0 0.0 0.0 0.0 1.1 0 3 c3t34d0
0.0 27.3 0.0 2109.0 0.0 0.0 0.0 1.1 0 3 c3t34d0s0
^C
bash-3.00#
This message posted from opensolaris.org
bash-3.00# zpool status zfs_raid10_32disks
pool: zfs_raid10_32disks
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zfs_raid10_32disks ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t16d0 ONLINE 0 0 0
c3t17d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t18d0 ONLINE 0 0 0
c3t19d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t20d0 ONLINE 0 0 0
c3t21d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t22d0 ONLINE 0 0 0
c3t23d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t24d0 ONLINE 0 0 0
c3t25d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t26d0 ONLINE 0 0 0
c3t27d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t32d0 ONLINE 0 0 0
c3t33d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t34d0 ONLINE 0 0 0
c3t35d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t36d0 ONLINE 0 0 0
c3t37d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t38d0 ONLINE 0 0 0
c3t39d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t40d0 ONLINE 0 0 0
c3t41d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t42d0 ONLINE 0 0 0
c3t43d0 ONLINE 0 0 0
errors: No known data errors
bash-3.00#
This message posted from opensolaris.org
On Tue, Aug 08, 2006 at 10:42:41AM -0700, Robert Milkowski wrote:> filebench in varmail by default creates 16 threads - I configrm it > with prstat, 16 threrads are created and running.Ah, OK. Looking at these results, it doesn''t seem to be CPU bound, and the disks are not fully utilized either. However, because the test is doing so much synchronous writes (eg. by calling fsync()), we are continually writing out the intent log. Unfortunately, we are only able to issue a small number of concurrent i/os while doing the intent log writes. All the threads must wait for the intent log blocks to be written before they can enqueue more data. Therefore, we are essentially doing: many threads call fsync(). one of them will flush the intent log, issuing a few writes to the disks all of the threads wait for the writes to complete repeat. This test fundamentally requires waiting for lots of syncronous writes. Assuming no other activity on the system, the performance of syncronous writes does not scale with the number of drives, it scales with the drive''s write latency. If you were to alter the test to not require everything to be done synchronously, then you would see much different behavior. --matt
Hello Matthew, Tuesday, August 8, 2006, 8:08:39 PM, you wrote: MA> On Tue, Aug 08, 2006 at 10:42:41AM -0700, Robert Milkowski wrote:>> filebench in varmail by default creates 16 threads - I configrm it >> with prstat, 16 threrads are created and running.MA> Ah, OK. Looking at these results, it doesn''t seem to be CPU bound, and MA> the disks are not fully utilized either. However, because the test is MA> doing so much synchronous writes (eg. by calling fsync()), we are MA> continually writing out the intent log. MA> Unfortunately, we are only able to issue a small number of concurrent MA> i/os while doing the intent log writes. All the threads must wait for MA> the intent log blocks to be written before they can enqueue more data. MA> Therefore, we are essentially doing: MA> many threads call fsync(). MA> one of them will flush the intent log, issuing a few writes to the disks MA> all of the threads wait for the writes to complete MA> repeat. MA> This test fundamentally requires waiting for lots of syncronous writes. MA> Assuming no other activity on the system, the performance of syncronous MA> writes does not scale with the number of drives, it scales with the MA> drive''s write latency. MA> If you were to alter the test to not require everything to be done MA> synchronously, then you would see much different behavior. Well, with zil_disable I get about 35,000 application IOPS and then it looks like I''m CPU bound IIRC. Anyway it would be nice if such workloads would scale up better - especially it hurts NFS servers performance (with ZFS as local storage). -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Hello Matthew, Tuesday, August 8, 2006, 8:08:39 PM, you wrote: MA> On Tue, Aug 08, 2006 at 10:42:41AM -0700, Robert Milkowski wrote:>> filebench in varmail by default creates 16 threads - I configrm it >> with prstat, 16 threrads are created and running.MA> Ah, OK. Looking at these results, it doesn''t seem to be CPU bound, and MA> the disks are not fully utilized either. However, because the test is MA> doing so much synchronous writes (eg. by calling fsync()), we are MA> continually writing out the intent log. MA> Unfortunately, we are only able to issue a small number of concurrent MA> i/os while doing the intent log writes. All the threads must wait for MA> the intent log blocks to be written before they can enqueue more data. MA> Therefore, we are essentially doing: MA> many threads call fsync(). MA> one of them will flush the intent log, issuing a few writes to the disks MA> all of the threads wait for the writes to complete MA> repeat. MA> This test fundamentally requires waiting for lots of syncronous writes. MA> Assuming no other activity on the system, the performance of syncronous MA> writes does not scale with the number of drives, it scales with the MA> drive''s write latency. MA> If you were to alter the test to not require everything to be done MA> synchronously, then you would see much different behavior. Does it mean that instead of creating one pool if I create two pools with the same numbers of disks in summary, and run two tests at the same time each on its own pool I should observe better scaling than with one pool with all disks? -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
On Thu, Aug 10, 2006 at 11:48:09AM +0200, Robert Milkowski wrote:> MA> This test fundamentally requires waiting for lots of syncronous writes. > MA> Assuming no other activity on the system, the performance of syncronous > MA> writes does not scale with the number of drives, it scales with the > MA> drive''s write latency. > > MA> If you were to alter the test to not require everything to be done > MA> synchronously, then you would see much different behavior. > > Does it mean that instead of creating one pool if I create two pools > with the same numbers of disks in summary, and run two tests at the > same time each on its own pool I should observe better scaling than > with one pool with all disks?Yes, but a better solution would be to use one pool with multiple filesystems. The intent log is per filesystem, so if you can break up your workload into sub-loads that don''t depend on the others being on disk synchronously, then you can put each sub-load on a different filesystem, and it will scale approximately with the number of filesystems. --matt
Hello Matthew, Thursday, August 10, 2006, 4:50:31 PM, you wrote: MA> On Thu, Aug 10, 2006 at 11:48:09AM +0200, Robert Milkowski wrote:>> MA> This test fundamentally requires waiting for lots of syncronous writes. >> MA> Assuming no other activity on the system, the performance of syncronous >> MA> writes does not scale with the number of drives, it scales with the >> MA> drive''s write latency. >> >> MA> If you were to alter the test to not require everything to be done >> MA> synchronously, then you would see much different behavior. >> >> Does it mean that instead of creating one pool if I create two pools >> with the same numbers of disks in summary, and run two tests at the >> same time each on its own pool I should observe better scaling than >> with one pool with all disks?MA> Yes, but a better solution would be to use one pool with multiple MA> filesystems. MA> The intent log is per filesystem, so if you can break up your workload MA> into sub-loads that don''t depend on the others being on disk MA> synchronously, then you can put each sub-load on a different filesystem, MA> and it will scale approximately with the number of filesystems. That is great news as that is exactly what I''m doing right now (many fs''es in a single pool) and going to do. Thank you for info. btw: wouldn''t it be possible to write block only once (for synchronous IO) and than just point to that block instead of copying it again? IIRC raiserfs folks are trying to implement something like this. -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
On Thu, Aug 10, 2006 at 06:50:45PM +0200, Robert Milkowski wrote:> btw: wouldn''t it be possible to write block only once (for synchronous > IO) and than just point to that block instead of copying it again?We actually do exactly that for larger (>32k) blocks. --matt
Hello Matthew, Thursday, August 10, 2006, 6:55:41 PM, you wrote: MA> On Thu, Aug 10, 2006 at 06:50:45PM +0200, Robert Milkowski wrote:>> btw: wouldn''t it be possible to write block only once (for synchronous >> IO) and than just point to that block instead of copying it again?MA> We actually do exactly that for larger (>32k) blocks. Why such limit (32k)? -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Robert Milkowski wrote:> Hello Matthew, > > Thursday, August 10, 2006, 4:50:31 PM, you wrote: > > MA> On Thu, Aug 10, 2006 at 11:48:09AM +0200, Robert Milkowski wrote: > >>>MA> This test fundamentally requires waiting for lots of syncronous writes. >>>MA> Assuming no other activity on the system, the performance of syncronous >>>MA> writes does not scale with the number of drives, it scales with the >>>MA> drive''s write latency. >>> >>>MA> If you were to alter the test to not require everything to be done >>>MA> synchronously, then you would see much different behavior. >>> >>>Does it mean that instead of creating one pool if I create two pools >>>with the same numbers of disks in summary, and run two tests at the >>>same time each on its own pool I should observe better scaling than >>>with one pool with all disks? > > > MA> Yes, but a better solution would be to use one pool with multiple > MA> filesystems. > > MA> The intent log is per filesystem, so if you can break up your workload > MA> into sub-loads that don''t depend on the others being on disk > MA> synchronously, then you can put each sub-load on a different filesystem, > MA> and it will scale approximately with the number of filesystems. > > That is great news as that is exactly what I''m doing right now (many > fs''es in a single pool) and going to do. > > Thank you for info. > > > btw: wouldn''t it be possible to write block only once (for synchronous > IO) and than just point to that block instead of copying it again?We already do that if the block is sufficiently large (currently >32KB). We still have to write a log record as well though, but many of those records can get aggregated into the same log block if there are a lot of parallel write threads.> > IIRC raiserfs folks are trying to implement something like this. >-- Neil
Robert Milkowski wrote:> Hello Matthew, > > Thursday, August 10, 2006, 6:55:41 PM, you wrote: > > MA> On Thu, Aug 10, 2006 at 06:50:45PM +0200, Robert Milkowski wrote: > >>>btw: wouldn''t it be possible to write block only once (for synchronous >>>IO) and than just point to that block instead of copying it again? > > > MA> We actually do exactly that for larger (>32k) blocks. > > Why such limit (32k)?By experimentation that was the cutoff where it was found to be more efficient. It was recently reduced from 64K with a more efficient dmu-sync() implementaion. Feel free to experiment with the dynamically changable tunable: ssize_t zfs_immediate_write_sz = 32768; -- Neil
Hello Neil, Thursday, August 10, 2006, 7:02:58 PM, you wrote: NP> Robert Milkowski wrote:>> Hello Matthew, >> >> Thursday, August 10, 2006, 6:55:41 PM, you wrote: >> >> MA> On Thu, Aug 10, 2006 at 06:50:45PM +0200, Robert Milkowski wrote: >> >>>>btw: wouldn''t it be possible to write block only once (for synchronous >>>>IO) and than just point to that block instead of copying it again? >> >> >> MA> We actually do exactly that for larger (>32k) blocks. >> >> Why such limit (32k)?NP> By experimentation that was the cutoff where it was found to be NP> more efficient. It was recently reduced from 64K with a more NP> efficient dmu-sync() implementaion. I do not understand - why in some cases with smaller block writing block twice could be actually faster than doing it once every time? I definitely am missing something here... NP> Feel free to experiment with the dynamically changable tunable: NP> ssize_t zfs_immediate_write_sz = 32768; I will :) By dynamic you mean it works right a way without any need to umount/mount and export/import? -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Hello Neil, Thursday, August 10, 2006, 7:02:58 PM, you wrote: NP> Robert Milkowski wrote:>> Hello Matthew, >> >> Thursday, August 10, 2006, 6:55:41 PM, you wrote: >> >> MA> On Thu, Aug 10, 2006 at 06:50:45PM +0200, Robert Milkowski wrote: >> >>>>btw: wouldn''t it be possible to write block only once (for synchronous >>>>IO) and than just point to that block instead of copying it again? >> >> >> MA> We actually do exactly that for larger (>32k) blocks. >> >> Why such limit (32k)?NP> By experimentation that was the cutoff where it was found to be NP> more efficient. It was recently reduced from 64K with a more NP> efficient dmu-sync() implementaion. NP> Feel free to experiment with the dynamically changable tunable: NP> ssize_t zfs_immediate_write_sz = 32768; I''ve just checked using dtrace on one of production nfs servers that 90% of the time arg5 in zfs_log_write() is exactly 32768 and the rest is always smaller. With default 32768 value of 32768 it means that for NFS servers it will always copy data as I''ve just checked in the code and there is: 245 if (len > zfs_immediate_write_sz) { So in nfs server case above never will be true (with default nfs srv settings). Wouldn''t nfs server benefit from lowering zfs_immediate_write_sz to 32767? ps. and I checked in a source - changind zfs_immediate_write_sz should have immediate effect -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Robert Milkowski wrote:> Hello Neil, > > Thursday, August 10, 2006, 7:02:58 PM, you wrote: > > NP> Robert Milkowski wrote: > >>>Hello Matthew, >>> >>>Thursday, August 10, 2006, 6:55:41 PM, you wrote: >>> >>>MA> On Thu, Aug 10, 2006 at 06:50:45PM +0200, Robert Milkowski wrote: >>> >>> >>>>>btw: wouldn''t it be possible to write block only once (for synchronous >>>>>IO) and than just point to that block instead of copying it again? >>> >>> >>>MA> We actually do exactly that for larger (>32k) blocks. >>> >>>Why such limit (32k)? > > > NP> By experimentation that was the cutoff where it was found to be > NP> more efficient. It was recently reduced from 64K with a more > NP> efficient dmu-sync() implementaion. > > I do not understand - why in some cases with smaller block writing > block twice could be actually faster than doing it once every time? > I definitely am missing something here...Sorry, I wasn''t very clear. This was tuned with multiple threads doing writes (using O_DSYNC or fsync()). The ZIL dynamically allocates blocks of size up to 128K to suit future predicted needs. If the writes are small (say 8K) then we can fit a whole bunch (upto 15) of the write records and embedded data in one log block. Anyway we found for the moment that past 32K was the point at which we got better bandwidth to immediately write the data out and just write the block address into the log block record. Your milaege may vary... I suspect this won''t be the last time this variable will be tuned. I would like to re-design this area to do away with the tunable, which was not really intended for public tuning.> NP> Feel free to experiment with the dynamically changable tunable: > NP> ssize_t zfs_immediate_write_sz = 32768; > > I will :) > > By dynamic you mean it works right a way without any need to > umount/mount and export/import?Correct.
Robert Milkowski wrote:> Hello Neil, > > Thursday, August 10, 2006, 7:02:58 PM, you wrote: > > NP> Robert Milkowski wrote: > >>>Hello Matthew, >>> >>>Thursday, August 10, 2006, 6:55:41 PM, you wrote: >>> >>>MA> On Thu, Aug 10, 2006 at 06:50:45PM +0200, Robert Milkowski wrote: >>> >>> >>>>>btw: wouldn''t it be possible to write block only once (for synchronous >>>>>IO) and than just point to that block instead of copying it again? >>> >>> >>>MA> We actually do exactly that for larger (>32k) blocks. >>> >>>Why such limit (32k)? > > > NP> By experimentation that was the cutoff where it was found to be > NP> more efficient. It was recently reduced from 64K with a more > NP> efficient dmu-sync() implementaion. > NP> Feel free to experiment with the dynamically changable tunable: > > NP> ssize_t zfs_immediate_write_sz = 32768; > > > I''ve just checked using dtrace on one of production nfs servers that > 90% of the time arg5 in zfs_log_write() is exactly 32768 and the rest > is always smaller. > > With default 32768 value of 32768 it means that for NFS servers it > will always copy data as I''ve just checked in the code and there is: > > 245 if (len > zfs_immediate_write_sz) { > > So in nfs server case above never will be true (with default nfs srv > settings). > > Wouldn''t nfs server benefit from lowering zfs_immediate_write_sz to > 32767?Yes NFS (with default 32K max write sz) would benefit if WR_INDIRECT writes (using dmu_sync()) were faster, but that wasn''t the case when last benchmarked. I''m sure there are some cases currently where tuning zfs_immediate_write_sz will help certain workloads. Anyway, I think this whole area deserves more thought. If you experiment with tuning zfs_immediate_write_sz, then please share any performance data for your application/benchmark(s). Thanks: Neil
RM: > I do not understand - why in some cases with smaller block writing > block twice could be actually faster than doing it once every time? > I definitely am missing something here... In addition to what Neil said, I want to add that when an application O_DSYNC write cover only parts of a file record you have the choice of issuing a log I/O that contains only the newly written data or do a full record I/O (using the up-to-date cached record) along with a small log I/O to match. So if you do 8K writes to a file stored using 128K records, you truly want each 8K writes to go to the log and then every txg, take the state of a record and I/O that. You certainly don''t want to I/O 128K every 8K writes. But then if you do a 100K write, it''s not as clear a win. Should I cough up the full 128K I/O now, hoping that the record will not be modified further before the txg clock hits ? That''s part of what goes into zfs_immediate_write_sz. And even for full record writes, there are some block allocation issues that come into play and complicates things further. -r
Robert Milkowski writes: > Hello Neil, > > Thursday, August 10, 2006, 7:02:58 PM, you wrote: > > NP> Robert Milkowski wrote: > >> Hello Matthew, > >> > >> Thursday, August 10, 2006, 6:55:41 PM, you wrote: > >> > >> MA> On Thu, Aug 10, 2006 at 06:50:45PM +0200, Robert Milkowski wrote: > >> > >>>>btw: wouldn''t it be possible to write block only once (for synchronous > >>>>IO) and than just point to that block instead of copying it again? > >> > >> > >> MA> We actually do exactly that for larger (>32k) blocks. > >> > >> Why such limit (32k)? > > NP> By experimentation that was the cutoff where it was found to be > NP> more efficient. It was recently reduced from 64K with a more > NP> efficient dmu-sync() implementaion. > NP> Feel free to experiment with the dynamically changable tunable: > > NP> ssize_t zfs_immediate_write_sz = 32768; > > > I''ve just checked using dtrace on one of production nfs servers that > 90% of the time arg5 in zfs_log_write() is exactly 32768 and the rest > is always smaller. Those should not be O_DSYNC though. Are they ? The I/O should be deferred to a subsequent COMMIT but then I''m not sure how it''s handled then. -r