I am seeing a serious problem with performance on my zpool. It contains 4
5400ish RPM disks in a raidz1 config.
zpool iostat -v 4
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
medtank 2.42T 308G 387 68 3.01M 540K
raidz1 2.42T 308G 387 68 3.01M 540K
c5t4d0 - - 158 22 1.35M 162K
c5t3d0 - - 182 24 1.32M 159K
c5t5d0 - - 172 20 1.35M 161K
c5t2d0 - - 157 28 1.39M 184K
---------- ----- ----- ----- ----- ----- -----
rpool 16.8G 57.2G 0 0 0 0
mirror 16.8G 57.2G 0 0 0 0
c4d0s0 - - 0 0 0 0
c4d1s0 - - 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
iosnoop (for a split second)
UID PID D BLOCK SIZE COMM PATHNAME
0 310 R 1150839529 33792 zpool-medtank <none>
0 310 R 1150839711 53248 zpool-medtank <none>
0 310 R 1150839530 33792 zpool-medtank <none>
0 310 R 1150839711 53760 zpool-medtank <none>
0 310 R 1150839529 33792 zpool-medtank <none>
0 310 R 1150839711 58880 zpool-medtank <none>
0 310 R 943934838 5632 zpool-medtank <none>
0 310 R 1150839815 3072 zpool-medtank <none>
0 310 R 1150839843 2560 zpool-medtank <none>
0 310 R 943936173 2560 zpool-medtank <none>
0 310 R 1150839530 33792 zpool-medtank <none>
0 310 R 1150839711 53760 zpool-medtank <none>
0 310 R 1150839816 16896 zpool-medtank <none>
0 0 R 1150839832 3072 sched <none>
0 0 R 1150839838 2560 sched <none>
0 310 R 943934838 5632 zpool-medtank <none>
0 310 R 943934838 5632 zpool-medtank <none>
0 310 R 943934838 5632 zpool-medtank <none>
0 310 R 842754969 512 zpool-medtank <none>
0 310 R 943936173 2560 zpool-medtank <none>
0 0 R 1150839816 2560 sched <none>
0 0 R 1150839821 3072 sched <none>
0 310 R 1150839827 11264 zpool-medtank <none>
0 310 R 943936172 3072 zpool-medtank <none>
0 310 R 1150839826 3072 zpool-medtank <none>
0 0 R 1150839832 2560 sched <none>
0 310 R 943936172 3072 zpool-medtank <none>
0 310 R 1150839837 3072 zpool-medtank <none>
0 0 R 1150839815 3072 sched <none>
0 0 R 1150839821 2560 sched <none>
0 310 R 1150839826 11264 zpool-medtank <none>
0 310 R 842754969 512 zpool-medtank <none>
0 310 R 842754968 512 zpool-medtank <none>
0 0 R 1150839849 2560 sched <none>
0 0 R 1150839849 2560 sched <none>
0 0 R 1150839848 3072 sched <none>
0 0 R 1150839848 3072 sched <none>
0 0 R 1150839854 2560 sched <none>
0 0 R 1150839854 3072 sched <none>
0 0 R 1150839854 3072 sched <none>
0 0 R 1150839854 2560 sched <none>
0 0 R 1150839870 3072 sched <none>
0 0 R 1150839871 2560 sched <none>
0 0 R 1150839871 2560 sched <none>
0 0 R 1150839859 3072 sched <none>
0 0 R 1150839860 2560 sched <none>
0 0 R 1150839870 3072 sched <none>
0 310 R 1150839865 3072 zpool-medtank <none>
0 0 R 1150839860 2560 sched <none>
0 0 R 1150839859 3072 sched <none>
0 310 R 1150839865 3072 zpool-medtank <none>
0 310 R 1150839865 2560 zpool-medtank <none>
0 310 R 524524043 512 zpool-medtank <none>
0 310 R 1150839865 2560 zpool-medtank <none>
0 310 R 524524043 1024 zpool-medtank <none>
0 310 R 524524043 1024 zpool-medtank <none>
0 0 R 1150839882 2560 sched <none>
0 0 R 1150839881 3072 sched <none>
0 0 R 1150839882 2560 sched <none>
0 0 R 1150839876 2560 sched <none>
0 0 R 1150839876 3072 sched <none>
0 0 R 1150839881 3072 sched <none>
0 0 R 1150839887 2560 sched <none>
0 0 R 1150839887 3072 sched <none>
0 0 R 1150839876 3072 sched <none>
0 0 R 1150839892 3072 sched <none>
0 0 R 1150839893 2560 sched <none>
0 0 R 1150839876 2560 sched <none>
0 0 R 1150839887 3072 sched <none>
0 0 R 1150839887 2560 sched <none>
0 0 R 1150839893 2560 sched <none>
0 0 R 1150839892 3072 sched <none>
0 310 R 1151426165 42496 zpool-medtank <none>
0 310 R 1151426314 2560 zpool-medtank <none>
0 310 R 1153698655 24064 zpool-medtank <none>
0 310 R 1151426165 41984 zpool-medtank <none>
0 310 R 1151426313 3072 zpool-medtank <none>
0 310 R 1151426165 41984 zpool-medtank <none>
0 310 R 1151426313 3072 zpool-medtank <none>
0 0 R 1154466266 3072 sched <none>
0 310 R 1154466272 2560 zpool-medtank <none>
0 310 R 1151426165 42496 zpool-medtank <none>
0 310 R 1154530233 3072 zpool-medtank <none>
0 310 R 1150839596 16896 zpool-medtank <none>
0 310 R 1151426314 2560 zpool-medtank <none>
0 310 R 1151417272 5632 zpool-medtank <none>
0 310 R 1151426248 2560 zpool-medtank <none>
0 310 R 1150839629 25088 zpool-medtank <none>
0 0 R 1150839645 3072 sched <none>
0 310 R 1153698654 24576 zpool-medtank <none>
0 0 R 1150839651 2560 sched <none>
0 0 R 1150839656 3072 sched <none>
0 0 R 1150839662 2560 sched <none>
0 310 R 1150839777 3072 zpool-medtank <none>
0 310 R 1154466266 2560 zpool-medtank <none>
0 310 R 1154466271 3072 zpool-medtank <none>
0 310 R 1150839595 16896 zpool-medtank <none>
0 310 R 1150839628 25600 zpool-medtank <none>
0 310 R 1150839777 2560 zpool-medtank <none>
0 310 R 1150840179 11264 zpool-medtank <none>
0 310 R 1151417272 5632 zpool-medtank <none>
0 310 R 1151426247 3072 zpool-medtank <none>
0 310 R 1153698654 24576 zpool-medtank <none>
0 310 R 1153698655 24064 zpool-medtank <none>
0 310 R 1154466266 2560 zpool-medtank <none>
0 310 R 1154466271 3072 zpool-medtank <none>
0 310 R 1256989247 512 zpool-medtank <none>
0 310 R 1153597262 5632 zpool-medtank <none>
0 310 R 1154530233 2560 zpool-medtank <none>
0 0 R 1154466266 3072 sched <none>
0 310 R 1150839596 16896 zpool-medtank <none>
0 310 R 1150839629 25088 zpool-medtank <none>
0 310 R 1150839645 11264 zpool-medtank <none>
0 310 R 1150839777 3072 zpool-medtank <none>
0 310 R 1150840180 11264 zpool-medtank <none>
0 310 R 1151417272 5632 zpool-medtank <none>
0 310 R 1151426248 2560 zpool-medtank <none>
0 310 R 713103104 65536 zpool-medtank <none>
0 0 R 1150840185 3072 sched <none>
0 0 R 1150840196 3072 sched <none>
0 0 R 1150840180 2560 sched <none>
0 310 R 1150840191 2560 zpool-medtank <none>
0 310 R 1153597262 5632 zpool-medtank <none>
0 310 R 1154466272 2560 zpool-medtank <none>
0 310 R 713103104 65536 zpool-medtank <none>
0 310 R 1256989247 512 zpool-medtank <none>
0 310 R 1153597262 5632 zpool-medtank <none>
0 310 R 1154530233 3072 zpool-medtank <none>
0 310 R 713103104 65536 zpool-medtank <none>
0 310 R 1256989247 1024 zpool-medtank <none>
0 0 R 1225980852 2560 sched <none>
0 0 R 1225980851 3072 sched <none>
0 0 R 1225980857 2560 sched <none>
0 0 R 1225980857 3072 sched <none>
0 0 R 1155801798 2560 sched <none>
0 0 R 1155801803 3072 sched <none>
0 0 R 1155801817 2560 sched <none>
0 310 R 1155801822 3072 zpool-medtank <none>
0 0 R 1155801828 2560 sched <none>
0 310 R 1350666112 65536 zpool-medtank <none>
0 310 R 1150839595 16896 zpool-medtank <none>
0 310 R 1150839628 25600 zpool-medtank <none>
0 310 R 1150839777 2560 zpool-medtank <none>
0 310 R 1150840179 11264 zpool-medtank <none>
0 310 R 1151417272 5632 zpool-medtank <none>
0 310 R 1151426247 3072 zpool-medtank <none>
0 310 R 1153597262 5632 zpool-medtank <none>
0 310 R 713103104 65536 zpool-medtank <none>
0 0 R 1225980852 2560 sched <none>
0 0 R 1225980857 3072 sched <none>
0 0 R 1155801797 3072 sched <none>
0 0 R 1155801803 2560 sched <none>
0 0 R 1155801816 3072 sched <none>
0 0 R 1155801827 3072 sched <none>
0 310 R 780823211 512 zpool-medtank <none>
0 310 R 1154530233 2560 zpool-medtank <none>
0 310 R 1155801797 18432 zpool-medtank <none>
0 310 R 780823210 1024 zpool-medtank <none>
0 310 R 1155801822 2560 zpool-medtank <none>
0 0 R 1155801798 2560 sched <none>
0 310 R 1155801803 15360 zpool-medtank <none>
0 310 R 1225980851 5632 zpool-medtank <none>
0 310 R 780823211 1024 zpool-medtank <none>
0 0 R 1155801833 2560 sched <none>
0 0 R 1155801833 2560 sched <none>
0 0 R 1155801833 3072 sched <none>
0 0 R 1155801833 3072 sched <none>
0 0 R 1155801839 2560 sched <none>
0 0 R 1155801839 2560 sched <none>
0 0 R 1155801838 3072 sched <none>
0 0 R 1155801838 3072 sched <none>
0 0 R 1155802381 2560 sched <none>
0 0 R 1155801844 2560 sched <none>
0 0 R 1155801849 3072 sched <none>
0 0 R 1155801855 2560 sched <none>
0 0 R 1155802381 2560 sched <none>
0 0 R 1155801844 2560 sched <none>
0 0 R 1155801849 3072 sched <none>
0 0 R 1155801855 2560 sched <none>
0 310 R 1155801860 5632 zpool-medtank <none>
0 0 R 1155802381 3072 sched <none>
0 0 R 1155801844 3072 sched <none>
0 0 R 1155802381 3072 sched <none>
0 0 R 1155801850 2560 sched <none>
0 0 R 1155801844 3072 sched <none>
0 0 R 1155801855 3072 sched <none>
0 0 R 1155801850 2560 sched <none>
0 0 R 1155801855 3072 sched <none>
0 310 R 1155801861 5632 zpool-medtank <none>
0 310 R 1155801861 5632 zpool-medtank <none>
0 310 R 989956477 1024 zpool-medtank <none>
0 310 R 1155802301 36864 zpool-medtank <none>
0 310 R 1155802302 36352 zpool-medtank <none>
0 310 R 1155854822 5632 zpool-medtank <none>
0 310 R 1155854821 5632 zpool-medtank <none>
0 310 R 989956477 512 zpool-medtank <none>
0 310 R 1155802302 36352 zpool-medtank <none>
0 310 R 1155854822 5632 zpool-medtank <none>
0 310 R 989956477 512 zpool-medtank <none>
0 0 R 1155854833 2560 sched <none>
0 0 R 1155854833 2560 sched <none>
0 0 R 1155854832 3072 sched <none>
0 0 R 1155854838 3072 sched <none>
0 0 R 1155854838 3072 sched <none>
0 0 R 1155854838 2560 sched <none>
0 0 R 1155854844 2560 sched <none>
0 0 R 1155854844 2560 sched <none>
0 0 R 1155854849 3072 sched <none>
0 0 R 1155854843 3072 sched <none>
0 0 R 1155854855 2560 sched <none>
0 0 R 1155854849 3072 sched <none>
0 0 R 1155854849 2560 sched <none>
0 310 R 1155801860 5632 zpool-medtank <none>
0 310 R 1155802301 36864 zpool-medtank <none>
0 310 R 1151416923 16896 zpool-medtank <none>
0 310 R 1155854855 11264 zpool-medtank <none>
0 310 R 1155854965 3072 zpool-medtank <none>
0 310 R 1155854821 5632 zpool-medtank <none>
0 0 R 1155854832 3072 sched <none>
0 310 R 1155854838 19456 zpool-medtank <none>
0 310 R 1155854965 2560 zpool-medtank <none>
0 310 R 1151416923 16896 zpool-medtank <none>
0 310 R 1155854866 5632 zpool-medtank <none>
0 310 R 1155854965 3072 zpool-medtank <none>
This array has a few cifs volumes, that are not currently being accessed, and it
hosts an NFS VMware datastore. On my ESXi box, I show zero disk operations
because everything is idle. iotop shows the zpool-medtank process and sched as
using the most IO, which is consistent with iosnoop. The server has 3 gigs of
ram, compression is turned off, dedupe is turned on on a few volumes, that
again, are not being accessed currently.
--
This message posted from opensolaris.org