Hi,
I''m using OpenSolaris 127 from my previous posts to address CIFS
problems. I have a few zpools but lately (with an uptime of 32 days)
we''ve started to get CIFS issues and really bad IO performance.
I''ve been running scrubs on a nightly basis.
I''m not sure why its happenning either - I''m new to
OpenSolaris.
I ran fsstat whilst trying to unrar a 8.4Gb file with an ISO inside it:
fsstat zfs 1
new name name attr attr lookup rddir read read write write
file remov chng get set ops ops ops bytes ops bytes
3.29K 466 367 633K 1.50K 1.66M 8.66K 964K 15.6G 314K 9.38G zfs
0 0 0 4 0 8 0 135 5.13M 93 5.02M zfs
0 0 0 7 0 18 0 205 5.63M 137 5.64M zfs
0 0 0 4 0 8 0 90 3.92K 49 14.6K zfs
0 0 0 4 0 8 0 115 16.4K 65 27.5K zfs
0 0 0 8 0 13 0 153 8.36M 113 8.38M zfs
0 0 0 4 0 8 0 94 3.96K 53 19.1K zfs
0 0 0 7 0 18 0 80 800 42 1.13K zfs
0 0 0 4 0 8 0 90 3.92K 48 7.62K zfs
0 0 0 4 0 8 0 99 132K 53 7.14K zfs
0 0 0 4 0 8 0 188 5.99K 96 5.62K zfs
0 0 0 4 0 8 0 95 664K 52 420K zfs
0 0 0 9 0 22 0 164 7.97K 92 12.2K zfs
new name name attr attr lookup rddir read read write write
file remov chng get set ops ops ops bytes ops bytes
0 0 0 4 0 8 0 111 2.63M 70 2.63M zfs
0 0 0 4 0 8 0 262 6.63M 153 6.63M zfs
0 0 0 4 0 8 0 80 800 44 1.70K zfs
0 0 0 4 0 8 0 337 18.1M 247 18.1M zfs
0 0 0 7 0 18 0 127 5.75M 89 5.63M zfs
0 0 0 4 0 8 0 80 800 50 25.6K zfs
My iostat appears below this message (its quite long! to give you an idea).
I''m really not sure why the performance has really dropped all of a
sudden or how to diagnose it. CIFS shares occasionally drop out too.
Its a bit of a downer to be experiencing on the 31st of December. I hope
everyone has a Safe & Happy New Years :-)
I''m unable to upgrade to the latest release because of an issue with
python:
pfexec pkg image-update
Creating Plan /pkg: Cannot remove
''pkg://opensolaris.org/SUNWipkg-gui-l10n at
0.5.11,5.11-0.127:20091111T075414Z'' due to the following packages that
depend on it:
pkg://opensolaris.org/SUNWipkg-gui at 0.5.11,5.11-0.127:20091111T075333Z
So I''m stuck on 127 until I can rebuild this machine :(
Cheers,
Em
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.0 0.0 0.5 2.8 1.0 2815.9 1000.0 100 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 2.0 1.0 0.0 0.0 100 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 6.0 0.0 81.5 1.2 1.0 198.6 166.6 60 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 9.0 0.0 31.5 0.0 0.4 0.0 41.7 0 38 c7t3d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t4d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t5d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.1 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
56.0 122.0 5972.3 8592.2 0.0 0.5 0.0 2.9 0 25 c7t1d0
55.0 136.0 5998.3 8590.2 0.0 0.7 0.0 3.8 0 29 c7t2d0
0.0 111.0 0.0 4342.9 0.0 2.2 0.0 20.2 0 57 c7t3d0
103.0 153.0 5868.3 8590.7 0.0 0.4 0.0 1.7 0 21 c7t4d0
96.0 130.0 5946.8 8591.2 0.0 0.7 0.0 3.2 0 35 c7t5d0
56.0 120.0 5998.8 8592.7 0.0 0.5 0.0 2.7 0 24 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 4.0 0.0 257.0 0.0 1.8 0.0 437.8 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 22.0 0.0 1987.5 0.0 1.0 0.0 45.4 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
18.0 18.0 2050.5 2004.0 0.0 1.0 0.0 27.7 0 100 c7t3d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 10.6 0 1 c7t4d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.4 0 0 c7t5d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
35.0 0.0 3948.0 0.0 0.0 1.0 0.0 28.5 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 10.0 0.0 6.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 3.0 0.0 1.5 0.0 2.2 0.0 749.8 0 100 c7t3d0
0.0 8.0 0.0 6.5 0.0 0.1 0.0 7.3 0 2 c7t4d0
0.0 7.0 0.0 7.0 0.0 0.0 0.0 1.6 0 0 c7t5d0
0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 26.0 0.0 24.0 0.0 0.0 0.0 0.3 0 0 c7t1d0
0.0 26.0 0.0 22.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 9.0 0.0 8.5 0.0 4.0 0.0 443.9 0 100 c7t3d0
0.0 14.0 0.0 23.5 0.0 0.0 0.0 3.1 0 1 c7t4d0
0.0 25.0 0.0 23.5 0.0 0.0 0.0 0.2 0 0 c7t5d0
0.0 27.0 0.0 25.5 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 21.0 0.0 19.5 0.0 0.0 0.0 0.3 0 0 c7t1d0
0.0 21.0 0.0 17.5 0.0 0.0 0.0 0.4 0 0 c7t2d0
0.0 15.0 0.0 23.5 0.0 4.0 0.0 266.3 0 100 c7t3d0
0.0 17.0 0.0 16.5 0.0 0.0 0.0 0.3 0 0 c7t4d0
0.0 16.0 0.0 19.0 0.0 0.1 0.0 4.8 0 5 c7t5d0
0.0 21.0 0.0 21.5 0.0 0.0 0.0 0.3 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.0 0.0 0.5 0.0 4.0 0.0 3953.4 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t1d0
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t2d0
0.0 15.0 0.0 20.0 0.0 2.2 0.0 146.8 0 73 c7t3d0
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t4d0
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t5d0
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t3d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.4 0 0 c7t4d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t5d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 144.9 0.0 7431.8 0.0 0.4 0.0 2.6 0 14 c7t1d0
0.0 162.9 0.0 6404.2 0.0 0.4 0.0 2.3 0 14 c7t2d0
0.0 210.9 0.0 5983.8 0.0 0.5 0.0 2.5 0 15 c7t3d0
0.0 295.9 0.0 8585.9 0.0 0.2 0.0 0.5 0 6 c7t4d0
0.0 161.9 0.0 5710.9 0.0 0.3 0.0 1.7 0 15 c7t5d0
0.0 176.9 0.0 8588.4 0.0 0.3 0.0 1.9 0 14 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
56.0 11.0 5974.7 1157.9 0.0 0.2 0.0 3.7 0 13 c7t1d0
208.1 19.0 5481.0 2184.3 0.0 0.1 0.0 0.4 0 6 c7t2d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
211.1 0.0 5506.5 0.0 0.0 0.1 0.0 0.3 0 4 c7t4d0
117.0 25.0 5818.6 2877.5 0.0 0.2 0.0 1.3 0 10 c7t5d0
100.0 0.0 6000.7 0.0 0.0 0.1 0.0 1.4 0 4 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 4.0 0.0 231.0 0.0 3.0 0.0 762.5 0 100 c7t3d0
0.0 89.0 0.0 3859.3 0.0 0.1 0.2 0.8 1 7 c3t0d0
0.0 88.0 0.0 3859.3 0.0 0.1 0.2 0.8 1 7 c3t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 21.0 0.0 1961.5 0.0 1.0 0.0 47.5 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c7t1d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c7t2d0
31.0 4.0 3589.0 413.0 0.0 1.0 0.0 28.6 0 100 c7t3d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.4 0 0 c7t4d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 13.1 0 1 c7t5d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
21.0 0.0 2410.0 0.0 0.0 1.0 0.0 47.6 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 11.0 0.0 8.0 0.0 0.0 0.0 0.3 0 0 c7t1d0
0.0 9.0 0.0 6.5 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 3.0 0.0 1.5 0.0 1.1 0.0 354.8 0 100 c7t3d0
0.0 8.0 0.0 6.5 0.0 0.0 0.0 1.8 0 0 c7t4d0
0.0 7.0 0.0 6.0 0.0 0.1 0.0 9.2 0 2 c7t5d0
0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 41.0 0.0 25.5 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 34.0 0.0 21.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 9.0 0.0 9.5 0.0 4.0 0.0 443.9 0 100 c7t3d0
0.0 37.0 0.0 22.0 0.0 0.0 0.0 0.2 0 0 c7t4d0
0.0 31.0 0.0 20.0 0.0 0.0 0.0 0.2 0 0 c7t5d0
0.0 38.0 0.0 23.5 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 29.0 0.0 21.0 0.0 0.0 0.0 0.4 0 0 c7t1d0
0.0 26.0 0.0 16.5 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 22.0 0.0 27.0 0.0 4.0 0.0 181.5 0 100 c7t3d0
0.0 19.0 0.0 18.5 0.0 0.0 0.0 2.3 0 1 c7t4d0
0.0 24.0 0.0 16.0 0.0 0.0 0.0 0.3 0 0 c7t5d0
0.0 29.0 0.0 20.5 0.0 0.0 0.0 0.3 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 24.0 0.0 21.0 0.0 0.0 0.0 0.3 0 0 c7t1d0
0.0 21.0 0.0 17.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 16.0 0.0 18.0 0.0 4.0 0.0 249.6 0 100 c7t3d0
0.0 22.0 0.0 18.0 0.0 0.0 0.0 0.3 0 0 c7t4d0
0.0 18.0 0.0 17.0 0.0 0.1 0.0 3.7 0 6 c7t5d0
0.0 22.0 0.0 20.5 0.0 0.0 0.0 0.3 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 6.0 0.0 4.0 0.0 0.3 0.0 50.8 0 8 c7t2d0
0.0 24.0 0.0 25.5 0.0 0.4 0.0 16.6 0 10 c7t3d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t4d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t5d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
14.0 102.0 1256.0 6792.5 0.0 0.3 0.0 2.7 0 16 c7t1d0
15.0 146.0 1281.5 6791.5 0.0 0.3 0.0 1.9 0 10 c7t2d0
0.0 57.0 0.0 2028.0 0.0 3.1 0.0 54.1 0 77 c7t3d0
36.0 67.0 1204.0 6792.0 0.0 0.3 0.0 2.5 0 12 c7t4d0
37.0 128.0 1203.5 6792.0 0.0 0.4 0.0 2.6 0 17 c7t5d0
30.0 89.0 1281.5 6793.0 0.0 0.3 0.0 2.4 0 13 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
_________________________________________________________________
If It Exists, You''ll Find it on SEEK Australia''s #1 job site
http://clk.atdmt.com/NMN/go/157639755/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091231/13a4831a/attachment.html>
judging by a *very* quick glance it looks like you have an issue with c3t0d0 device which is responding very slowly. -- Robert Milkowski http://milek.blogspot.com On 31/12/2009 09:10, Emily Grettel wrote:> Hi, > > I''m using OpenSolaris 127 from my previous posts to address CIFS > problems. I have a few zpools but lately (with an uptime of 32 days) > we''ve started to get CIFS issues and really bad IO performance. I''ve > been running scrubs on a nightly basis. > > I''m not sure why its happenning either - I''m new to OpenSolaris. > > I ran fsstat whilst trying to unrar a 8.4Gb file with an ISO inside it: > > fsstat zfs 1 > new name name attr attr lookup rddir read read write write > file remov chng get set ops ops ops bytes ops bytes > 3.29K 466 367 633K 1.50K 1.66M 8.66K 964K 15.6G 314K 9.38G zfs > 0 0 0 4 0 8 0 135 5.13M 93 5.02M zfs > 0 0 0 7 0 18 0 205 5.63M 137 5.64M zfs > 0 0 0 4 0 8 0 90 3.92K 49 14.6K zfs > 0 0 0 4 0 8 0 115 16.4K 65 27.5K zfs > 0 0 0 8 0 13 0 153 8.36M 113 8.38M zfs > 0 0 0 4 0 8 0 94 3.96K 53 19.1K zfs > 0 0 0 7 0 18 0 80 800 42 1.13K zfs > 0 0 0 4 0 8 0 90 3.92K 48 7.62K zfs > 0 0 0 4 0 8 0 99 132K 53 7.14K zfs > 0 0 0 4 0 8 0 188 5.99K 96 5.62K zfs > 0 0 0 4 0 8 0 95 664K 52 420K zfs > 0 0 0 9 0 22 0 164 7.97K 92 12.2K zfs > new name name attr attr lookup rddir read read write write > file remov chng get set ops ops ops bytes ops bytes > 0 0 0 4 0 8 0 111 2.63M 70 2.63M zfs > 0 0 0 4 0 8 0 262 6.63M 153 6.63M zfs > 0 0 0 4 0 8 0 80 800 44 1.70K zfs > 0 0 0 4 0 8 0 337 18.1M 247 18.1M zfs > 0 0 0 7 0 18 0 127 5.75M 89 5.63M zfs > 0 0 0 4 0 8 0 80 800 50 25.6K zfs > > My iostat appears below this message (its quite long! to give you an > idea). I''m really not sure why the performance has really dropped all > of a sudden or how to diagnose it. CIFS shares occasionally drop out too. > > Its a bit of a downer to be experiencing on the 31st of December. I > hope everyone has a Safe & Happy New Years :-) > > I''m unable to upgrade to the latest release because of an issue with > python: > > pfexec pkg image-update > Creating Plan /pkg: Cannot remove > ''pkg://opensolaris.org/SUNWipkg-gui-l10n at 0.5.11,5.11-0.127:20091111T075414Z'' > due to the following packages that depend on it: > pkg://opensolaris.org/SUNWipkg-gui at 0.5.11,5.11-0.127:20091111T075333Z > > So I''m stuck on 127 until I can rebuild this machine :( > > Cheers, > Em > > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 1.0 0.0 0.5 2.8 1.0 2815.9 1000.0 100 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 2.0 1.0 0.0 0.0 100 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 6.0 0.0 81.5 1.2 1.0 198.6 166.6 60 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0 > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 0.0 9.0 0.0 31.5 0.0 0.4 0.0 41.7 0 38 c7t3d0 > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t4d0 > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t5d0 > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.1 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 56.0 122.0 5972.3 8592.2 0.0 0.5 0.0 2.9 0 25 c7t1d0 > 55.0 136.0 5998.3 8590.2 0.0 0.7 0.0 3.8 0 29 c7t2d0 > 0.0 111.0 0.0 4342.9 0.0 2.2 0.0 20.2 0 57 c7t3d0 > 103.0 153.0 5868.3 8590.7 0.0 0.4 0.0 1.7 0 21 c7t4d0 > 96.0 130.0 5946.8 8591.2 0.0 0.7 0.0 3.2 0 35 c7t5d0 > 56.0 120.0 5998.8 8592.7 0.0 0.5 0.0 2.7 0 24 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 4.0 0.0 257.0 0.0 1.8 0.0 437.8 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 22.0 0.0 1987.5 0.0 1.0 0.0 45.4 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.2 0 0 c7t1d0 > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 18.0 18.0 2050.5 2004.0 0.0 1.0 0.0 27.7 0 100 c7t3d0 > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 10.6 0 1 c7t4d0 > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.4 0 0 c7t5d0 > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 35.0 0.0 3948.0 0.0 0.0 1.0 0.0 28.5 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c7t1d0 > 0.0 10.0 0.0 6.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 0.0 3.0 0.0 1.5 0.0 2.2 0.0 749.8 0 100 c7t3d0 > 0.0 8.0 0.0 6.5 0.0 0.1 0.0 7.3 0 2 c7t4d0 > 0.0 7.0 0.0 7.0 0.0 0.0 0.0 1.6 0 0 c7t5d0 > 0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 26.0 0.0 24.0 0.0 0.0 0.0 0.3 0 0 c7t1d0 > 0.0 26.0 0.0 22.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 0.0 9.0 0.0 8.5 0.0 4.0 0.0 443.9 0 100 c7t3d0 > 0.0 14.0 0.0 23.5 0.0 0.0 0.0 3.1 0 1 c7t4d0 > 0.0 25.0 0.0 23.5 0.0 0.0 0.0 0.2 0 0 c7t5d0 > 0.0 27.0 0.0 25.5 0.0 0.0 0.0 0.2 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 21.0 0.0 19.5 0.0 0.0 0.0 0.3 0 0 c7t1d0 > 0.0 21.0 0.0 17.5 0.0 0.0 0.0 0.4 0 0 c7t2d0 > 0.0 15.0 0.0 23.5 0.0 4.0 0.0 266.3 0 100 c7t3d0 > 0.0 17.0 0.0 16.5 0.0 0.0 0.0 0.3 0 0 c7t4d0 > 0.0 16.0 0.0 19.0 0.0 0.1 0.0 4.8 0 5 c7t5d0 > 0.0 21.0 0.0 21.5 0.0 0.0 0.0 0.3 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 1.0 0.0 0.5 0.0 4.0 0.0 3953.4 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t1d0 > 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t2d0 > 0.0 15.0 0.0 20.0 0.0 2.2 0.0 146.8 0 73 c7t3d0 > 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t4d0 > 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t5d0 > 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0 > 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t3d0 > 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.4 0 0 c7t4d0 > 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t5d0 > 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 144.9 0.0 7431.8 0.0 0.4 0.0 2.6 0 14 c7t1d0 > 0.0 162.9 0.0 6404.2 0.0 0.4 0.0 2.3 0 14 c7t2d0 > 0.0 210.9 0.0 5983.8 0.0 0.5 0.0 2.5 0 15 c7t3d0 > 0.0 295.9 0.0 8585.9 0.0 0.2 0.0 0.5 0 6 c7t4d0 > 0.0 161.9 0.0 5710.9 0.0 0.3 0.0 1.7 0 15 c7t5d0 > 0.0 176.9 0.0 8588.4 0.0 0.3 0.0 1.9 0 14 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 56.0 11.0 5974.7 1157.9 0.0 0.2 0.0 3.7 0 13 c7t1d0 > 208.1 19.0 5481.0 2184.3 0.0 0.1 0.0 0.4 0 6 c7t2d0 > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > 211.1 0.0 5506.5 0.0 0.0 0.1 0.0 0.3 0 4 c7t4d0 > 117.0 25.0 5818.6 2877.5 0.0 0.2 0.0 1.3 0 10 c7t5d0 > 100.0 0.0 6000.7 0.0 0.0 0.1 0.0 1.4 0 4 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 4.0 0.0 231.0 0.0 3.0 0.0 762.5 0 100 c7t3d0 > 0.0 89.0 0.0 3859.3 0.0 0.1 0.2 0.8 1 7 c3t0d0 > 0.0 88.0 0.0 3859.3 0.0 0.1 0.2 0.8 1 7 c3t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 21.0 0.0 1961.5 0.0 1.0 0.0 47.5 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c7t1d0 > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c7t2d0 > 31.0 4.0 3589.0 413.0 0.0 1.0 0.0 28.6 0 100 c7t3d0 > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.4 0 0 c7t4d0 > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 13.1 0 1 c7t5d0 > 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 21.0 0.0 2410.0 0.0 0.0 1.0 0.0 47.6 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 11.0 0.0 8.0 0.0 0.0 0.0 0.3 0 0 c7t1d0 > 0.0 9.0 0.0 6.5 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 0.0 3.0 0.0 1.5 0.0 1.1 0.0 354.8 0 100 c7t3d0 > 0.0 8.0 0.0 6.5 0.0 0.0 0.0 1.8 0 0 c7t4d0 > 0.0 7.0 0.0 6.0 0.0 0.1 0.0 9.2 0 2 c7t5d0 > 0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 41.0 0.0 25.5 0.0 0.0 0.0 0.2 0 0 c7t1d0 > 0.0 34.0 0.0 21.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 0.0 9.0 0.0 9.5 0.0 4.0 0.0 443.9 0 100 c7t3d0 > 0.0 37.0 0.0 22.0 0.0 0.0 0.0 0.2 0 0 c7t4d0 > 0.0 31.0 0.0 20.0 0.0 0.0 0.0 0.2 0 0 c7t5d0 > 0.0 38.0 0.0 23.5 0.0 0.0 0.0 0.2 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 29.0 0.0 21.0 0.0 0.0 0.0 0.4 0 0 c7t1d0 > 0.0 26.0 0.0 16.5 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 0.0 22.0 0.0 27.0 0.0 4.0 0.0 181.5 0 100 c7t3d0 > 0.0 19.0 0.0 18.5 0.0 0.0 0.0 2.3 0 1 c7t4d0 > 0.0 24.0 0.0 16.0 0.0 0.0 0.0 0.3 0 0 c7t5d0 > 0.0 29.0 0.0 20.5 0.0 0.0 0.0 0.3 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 24.0 0.0 21.0 0.0 0.0 0.0 0.3 0 0 c7t1d0 > 0.0 21.0 0.0 17.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 > 0.0 16.0 0.0 18.0 0.0 4.0 0.0 249.6 0 100 c7t3d0 > 0.0 22.0 0.0 18.0 0.0 0.0 0.0 0.3 0 0 c7t4d0 > 0.0 18.0 0.0 17.0 0.0 0.1 0.0 3.7 0 6 c7t5d0 > 0.0 22.0 0.0 20.5 0.0 0.0 0.0 0.3 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0 > 0.0 6.0 0.0 4.0 0.0 0.3 0.0 50.8 0 8 c7t2d0 > 0.0 24.0 0.0 25.5 0.0 0.4 0.0 16.6 0 10 c7t3d0 > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t4d0 > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t5d0 > 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 14.0 102.0 1256.0 6792.5 0.0 0.3 0.0 2.7 0 16 c7t1d0 > 15.0 146.0 1281.5 6791.5 0.0 0.3 0.0 1.9 0 10 c7t2d0 > 0.0 57.0 0.0 2028.0 0.0 3.1 0.0 54.1 0 77 c7t3d0 > 36.0 67.0 1204.0 6792.0 0.0 0.3 0.0 2.5 0 12 c7t4d0 > 37.0 128.0 1203.5 6792.0 0.0 0.4 0.0 2.6 0 17 c7t5d0 > 30.0 89.0 1281.5 6793.0 0.0 0.3 0.0 2.4 0 13 c0t1d0 > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 > > > > > ------------------------------------------------------------------------ > Australia''s #1 job site If It Exists, You''ll Find it on SEEK > <http://clk.atdmt.com/NMN/go/157639755/direct/01/> > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091231/475ec5b3/attachment.html>
On Dec 31, 2009, at 2:49 AM, Robert Milkowski wrote:> > judging by a *very* quick glance it looks like you have an issue > with c3t0d0 device which is responding very slowly.Yes, there is an I/O stuck on the device which is not getting serviced. See below...> > -- > Robert Milkowski > http://milek.blogspot.com > > > > On 31/12/2009 09:10, Emily Grettel wrote: >> >> Hi, >> >> I''m using OpenSolaris 127 from my previous posts to address CIFS >> problems. I have a few zpools but lately (with an uptime of 32 >> days) we''ve started to get CIFS issues and really bad IO >> performance. I''ve been running scrubs on a nightly basis. >> >> I''m not sure why its happenning either - I''m new to OpenSolaris. >> >> I ran fsstat whilst trying to unrar a 8.4Gb file with an ISO inside >> it: >> >> fsstat zfs 1 >> new name name attr attr lookup rddir read read write write >> file remov chng get set ops ops ops bytes ops bytes >> 3.29K 466 367 633K 1.50K 1.66M 8.66K 964K 15.6G 314K 9.38G >> zfs >> 0 0 0 4 0 8 0 135 5.13M 93 5.02M >> zfs >> 0 0 0 7 0 18 0 205 5.63M 137 5.64M >> zfs >> 0 0 0 4 0 8 0 90 3.92K 49 14.6K >> zfs >> 0 0 0 4 0 8 0 115 16.4K 65 27.5K >> zfs >> 0 0 0 8 0 13 0 153 8.36M 113 8.38M >> zfs >> 0 0 0 4 0 8 0 94 3.96K 53 19.1K >> zfs >> 0 0 0 7 0 18 0 80 800 42 1.13K >> zfs >> 0 0 0 4 0 8 0 90 3.92K 48 7.62K >> zfs >> 0 0 0 4 0 8 0 99 132K 53 7.14K >> zfs >> 0 0 0 4 0 8 0 188 5.99K 96 5.62K >> zfs >> 0 0 0 4 0 8 0 95 664K 52 420K >> zfs >> 0 0 0 9 0 22 0 164 7.97K 92 12.2K >> zfs >> new name name attr attr lookup rddir read read write write >> file remov chng get set ops ops ops bytes ops bytes >> 0 0 0 4 0 8 0 111 2.63M 70 2.63M >> zfs >> 0 0 0 4 0 8 0 262 6.63M 153 6.63M >> zfs >> 0 0 0 4 0 8 0 80 800 44 1.70K >> zfs >> 0 0 0 4 0 8 0 337 18.1M 247 18.1M >> zfs >> 0 0 0 7 0 18 0 127 5.75M 89 5.63M >> zfs >> 0 0 0 4 0 8 0 80 800 50 25.6K >> zfs >> >> My iostat appears below this message (its quite long! to give you >> an idea). I''m really not sure why the performance has really >> dropped all of a sudden or how to diagnose it. CIFS shares >> occasionally drop out too. >> >> Its a bit of a downer to be experiencing on the 31st of December. I >> hope everyone has a Safe & Happy New Years :-) >> >> I''m unable to upgrade to the latest release because of an issue >> with python: >> >> pfexec pkg image-update >> Creating Plan /pkg: Cannot remove ''pkg://opensolaris.org/SUNWipkg-gui-l10n at 0.5.11 >> ,5.11-0.127:20091111T075414Z'' due to the following packages that >> depend on it: >> pkg://opensolaris.org/SUNWipkg- >> gui at 0.5.11,5.11-0.127:20091111T075333Z >> >> So I''m stuck on 127 until I can rebuild this machine :( >> >> Cheers, >> Em >> >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 1.0 0.0 0.5 2.8 1.0 2815.9 1000.0 100 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 2.0 1.0 0.0 0.0 100 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 6.0 0.0 81.5 1.2 1.0 198.6 166.6 60 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0 >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 0.0 9.0 0.0 31.5 0.0 0.4 0.0 41.7 0 38 c7t3d0 >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t4d0 >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t5d0 >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.1 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 56.0 122.0 5972.3 8592.2 0.0 0.5 0.0 2.9 0 25 c7t1d0 >> 55.0 136.0 5998.3 8590.2 0.0 0.7 0.0 3.8 0 29 c7t2d0 >> 0.0 111.0 0.0 4342.9 0.0 2.2 0.0 20.2 0 57 c7t3d0 >> 103.0 153.0 5868.3 8590.7 0.0 0.4 0.0 1.7 0 21 c7t4d0 >> 96.0 130.0 5946.8 8591.2 0.0 0.7 0.0 3.2 0 35 c7t5d0 >> 56.0 120.0 5998.8 8592.7 0.0 0.5 0.0 2.7 0 24 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 4.0 0.0 257.0 0.0 1.8 0.0 437.8 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0Here you can see that actv=1, which means that there is an I/O queued to the device. But there is no r/s or w/s, so the I/O is not being serviced. This could be a broken disk, or it could be some other hardware/software/firmware issue. Check the errors on the device with iostat -En You should also check the fma logs: fmadm faulty fmdump -eV -- richard>> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 22.0 0.0 1987.5 0.0 1.0 0.0 45.4 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.2 0 0 c7t1d0 >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 18.0 18.0 2050.5 2004.0 0.0 1.0 0.0 27.7 0 100 c7t3d0 >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 10.6 0 1 c7t4d0 >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.4 0 0 c7t5d0 >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 35.0 0.0 3948.0 0.0 0.0 1.0 0.0 28.5 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c7t1d0 >> 0.0 10.0 0.0 6.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 0.0 3.0 0.0 1.5 0.0 2.2 0.0 749.8 0 100 c7t3d0 >> 0.0 8.0 0.0 6.5 0.0 0.1 0.0 7.3 0 2 c7t4d0 >> 0.0 7.0 0.0 7.0 0.0 0.0 0.0 1.6 0 0 c7t5d0 >> 0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 26.0 0.0 24.0 0.0 0.0 0.0 0.3 0 0 c7t1d0 >> 0.0 26.0 0.0 22.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 0.0 9.0 0.0 8.5 0.0 4.0 0.0 443.9 0 100 c7t3d0 >> 0.0 14.0 0.0 23.5 0.0 0.0 0.0 3.1 0 1 c7t4d0 >> 0.0 25.0 0.0 23.5 0.0 0.0 0.0 0.2 0 0 c7t5d0 >> 0.0 27.0 0.0 25.5 0.0 0.0 0.0 0.2 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 21.0 0.0 19.5 0.0 0.0 0.0 0.3 0 0 c7t1d0 >> 0.0 21.0 0.0 17.5 0.0 0.0 0.0 0.4 0 0 c7t2d0 >> 0.0 15.0 0.0 23.5 0.0 4.0 0.0 266.3 0 100 c7t3d0 >> 0.0 17.0 0.0 16.5 0.0 0.0 0.0 0.3 0 0 c7t4d0 >> 0.0 16.0 0.0 19.0 0.0 0.1 0.0 4.8 0 5 c7t5d0 >> 0.0 21.0 0.0 21.5 0.0 0.0 0.0 0.3 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 1.0 0.0 0.5 0.0 4.0 0.0 3953.4 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t1d0 >> 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t2d0 >> 0.0 15.0 0.0 20.0 0.0 2.2 0.0 146.8 0 73 c7t3d0 >> 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t4d0 >> 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t5d0 >> 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0 >> 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t3d0 >> 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.4 0 0 c7t4d0 >> 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t5d0 >> 0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 144.9 0.0 7431.8 0.0 0.4 0.0 2.6 0 14 c7t1d0 >> 0.0 162.9 0.0 6404.2 0.0 0.4 0.0 2.3 0 14 c7t2d0 >> 0.0 210.9 0.0 5983.8 0.0 0.5 0.0 2.5 0 15 c7t3d0 >> 0.0 295.9 0.0 8585.9 0.0 0.2 0.0 0.5 0 6 c7t4d0 >> 0.0 161.9 0.0 5710.9 0.0 0.3 0.0 1.7 0 15 c7t5d0 >> 0.0 176.9 0.0 8588.4 0.0 0.3 0.0 1.9 0 14 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 56.0 11.0 5974.7 1157.9 0.0 0.2 0.0 3.7 0 13 c7t1d0 >> 208.1 19.0 5481.0 2184.3 0.0 0.1 0.0 0.4 0 6 c7t2d0 >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> 211.1 0.0 5506.5 0.0 0.0 0.1 0.0 0.3 0 4 c7t4d0 >> 117.0 25.0 5818.6 2877.5 0.0 0.2 0.0 1.3 0 10 c7t5d0 >> 100.0 0.0 6000.7 0.0 0.0 0.1 0.0 1.4 0 4 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 4.0 0.0 231.0 0.0 3.0 0.0 762.5 0 100 c7t3d0 >> 0.0 89.0 0.0 3859.3 0.0 0.1 0.2 0.8 1 7 c3t0d0 >> 0.0 88.0 0.0 3859.3 0.0 0.1 0.2 0.8 1 7 c3t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 21.0 0.0 1961.5 0.0 1.0 0.0 47.5 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c7t1d0 >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c7t2d0 >> 31.0 4.0 3589.0 413.0 0.0 1.0 0.0 28.6 0 100 c7t3d0 >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.4 0 0 c7t4d0 >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 13.1 0 1 c7t5d0 >> 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 21.0 0.0 2410.0 0.0 0.0 1.0 0.0 47.6 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 11.0 0.0 8.0 0.0 0.0 0.0 0.3 0 0 c7t1d0 >> 0.0 9.0 0.0 6.5 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 0.0 3.0 0.0 1.5 0.0 1.1 0.0 354.8 0 100 c7t3d0 >> 0.0 8.0 0.0 6.5 0.0 0.0 0.0 1.8 0 0 c7t4d0 >> 0.0 7.0 0.0 6.0 0.0 0.1 0.0 9.2 0 2 c7t5d0 >> 0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 41.0 0.0 25.5 0.0 0.0 0.0 0.2 0 0 c7t1d0 >> 0.0 34.0 0.0 21.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 0.0 9.0 0.0 9.5 0.0 4.0 0.0 443.9 0 100 c7t3d0 >> 0.0 37.0 0.0 22.0 0.0 0.0 0.0 0.2 0 0 c7t4d0 >> 0.0 31.0 0.0 20.0 0.0 0.0 0.0 0.2 0 0 c7t5d0 >> 0.0 38.0 0.0 23.5 0.0 0.0 0.0 0.2 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 29.0 0.0 21.0 0.0 0.0 0.0 0.4 0 0 c7t1d0 >> 0.0 26.0 0.0 16.5 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 0.0 22.0 0.0 27.0 0.0 4.0 0.0 181.5 0 100 c7t3d0 >> 0.0 19.0 0.0 18.5 0.0 0.0 0.0 2.3 0 1 c7t4d0 >> 0.0 24.0 0.0 16.0 0.0 0.0 0.0 0.3 0 0 c7t5d0 >> 0.0 29.0 0.0 20.5 0.0 0.0 0.0 0.3 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 24.0 0.0 21.0 0.0 0.0 0.0 0.3 0 0 c7t1d0 >> 0.0 21.0 0.0 17.0 0.0 0.0 0.0 0.2 0 0 c7t2d0 >> 0.0 16.0 0.0 18.0 0.0 4.0 0.0 249.6 0 100 c7t3d0 >> 0.0 22.0 0.0 18.0 0.0 0.0 0.0 0.3 0 0 c7t4d0 >> 0.0 18.0 0.0 17.0 0.0 0.1 0.0 3.7 0 6 c7t5d0 >> 0.0 22.0 0.0 20.5 0.0 0.0 0.0 0.3 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0 >> 0.0 6.0 0.0 4.0 0.0 0.3 0.0 50.8 0 8 c7t2d0 >> 0.0 24.0 0.0 25.5 0.0 0.4 0.0 16.6 0 10 c7t3d0 >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t4d0 >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t5d0 >> 0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 14.0 102.0 1256.0 6792.5 0.0 0.3 0.0 2.7 0 16 c7t1d0 >> 15.0 146.0 1281.5 6791.5 0.0 0.3 0.0 1.9 0 10 c7t2d0 >> 0.0 57.0 0.0 2028.0 0.0 3.1 0.0 54.1 0 77 c7t3d0 >> 36.0 67.0 1204.0 6792.0 0.0 0.3 0.0 2.5 0 12 c7t4d0 >> 37.0 128.0 1203.5 6792.0 0.0 0.4 0.0 2.6 0 17 c7t5d0 >> 30.0 89.0 1281.5 6793.0 0.0 0.3 0.0 2.4 0 13 c0t1d0 >> extended device statistics >> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device >> 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0 >> >> >> >> >> Australia''s #1 job site If It Exists, You''ll Find it on SEEK >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Thu, 31 Dec 2009, Emily Grettel wrote:> ? > I''m using OpenSolaris 127 from my previous posts to address CIFS problems. I have a few zpools but > lately (with an uptime of 32 days) we''ve started to get CIFS issues and really bad IO performance. > I''ve been running scrubs on a nightly basis. > ? > I''m not sure why its happenning either - I''m new to OpenSolaris.Without knowing anything about your pool, your c7t3d0 device seems possibly suspect. Notice that it often posts a very high asvc_t. What is the output from ''zpool status'' for this pool? Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Hello!> This could be a broken disk, or it could be some other > hardware/software/firmware issue. Check the errors on the > device with > iostat -EnHeres the output: c7t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD10EADS-00L Revision: 1A01 Serial No: Size: 1000.20GB <1000204886016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 4 Predictive Failure Analysis: 0 c7t2d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: Size: 1000.20GB <1000204886016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 4 Predictive Failure Analysis: 0 c7t3d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: Size: 1000.20GB <1000204886016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 4 Predictive Failure Analysis: 0 c7t4d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: Size: 1000.20GB <1000204886016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 4 Predictive Failure Analysis: 0 c7t5d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: Size: 1000.20GB <1000204886016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 4 Predictive Failure Analysis: 0 c7t0d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD740GD-00FL Revision: 8F33 Serial No: Size: 74.36GB <74355769344 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 6 Predictive Failure Analysis: 0 c0t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: Size: 1000.20GB <1000204886016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 5 Predictive Failure Analysis: 0 c3t0d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD7500AAKS-0 Revision: 4G30 Serial No: Size: 750.16GB <750156374016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 5 Predictive Failure Analysis: 0 c3t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD7500AAKS-0 Revision: 4G30 Serial No: Size: 750.16GB <750156374016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 5 Predictive Failure Analysis: 0> You should also check the fma logs: > fmadm faultyEmpty> fmdump -eV >This turned out to be huge. But they''re mostly something like this: Nov 13 2009 10:15:41.883716494 ereport.fs.zfs.checksum nvlist version: 0 class = ereport.fs.zfs.checksum ena = 0x7cfde552fd100401 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0xda1d003c03abad23 vdev = 0x4389ee65271b9187 (end detector) pool = tank pool_guid = 0xda1d003c03abad23 pool_context = 0 pool_failmode = wait vdev_guid = 0x4389ee65271b9187 vdev_type = replacing parent_guid = 0x79c2f2cf0b81ae5a parent_type = raidz zio_err = 0 zio_offset = 0xae9b3fa00 zio_size = 0x6600 zio_objset = 0x24 zio_object = 0x1b2 zio_level = 0 zio_blkid = 0x635 __ttl = 0x1 __tod = 0x4afc971d 0x34ac718e Thanks for helping and telling me about those commands :-) The scrub I started last night is still running, it usually takes about 8 hours. Will post the results. - Em> From: richard.elling at gmail.com > To: milek at task.gda.pl > Date: Thu, 31 Dec 2009 08:37:03 -0800 > CC: zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] ZFS extremely slow performance > > On Dec 31, 2009, at 2:49 AM, Robert Milkowski wrote: > > > > > judging by a *very* quick glance it looks like you have an issue > > with c3t0d0 device which is responding very slowly. > > Yes, there is an I/O stuck on the device which is not getting serviced. > See below... > > > > > -- > > Robert Milkowski > > http://milek.blogspot.com > > > >_________________________________________________________________ If It Exists, You''ll Find it on SEEK Australia''s #1 job site http://clk.atdmt.com/NMN/go/157639755/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100101/f9735813/attachment.html>
Hello again, I swapped out the PSU and replaced the cables and ran scrubs almost every day (after hours) with no reported faults. I also upgraded to SNV_130 thanks to Brock & changed cables and PSU after the suggestion from Richard. I owe you two both beers! We thought our troubles were resolved but I''m noticing alot of the messages above from my /var/adm/messages and I''m starting to worry. I tailed the log whilst we streamed some MPEG2 captures (it was about 12Gb) and the log went crazy! Jan 9 23:52:04 razor nwamd[34]: [ID 244715 daemon.error] scf_handle_destroy() failed: repository server unavailable Jan 9 23:52:04 razor smbd[13585]: [ID 354691 daemon.error] smb_nicmon_daemon: failed to refresh SMF instance svc:/network/smb/server:default Jan 9 23:52:04 razor last message repeated 11 times Jan 9 23:52:04 razor nwamd[34]: [ID 244715 daemon.error] scf_handle_destroy() failed: repository server unavailable Jan 9 23:52:04 razor smbd[13585]: [ID 354691 daemon.error] smb_nicmon_daemon: failed to refresh SMF instance svc:/network/smb/server:default Jan 9 23:52:05 razor last message repeated 4 times Jan 9 23:52:05 razor nwamd[34]: [ID 244715 daemon.error] scf_handle_destroy() failed: repository server unavailable Jan 9 23:52:05 razor smbd[13585]: [ID 354691 daemon.error] smb_nicmon_daemon: failed to refresh SMF instance svc:/network/smb/server:default Any ideas why this may be happenning? I''m really starting to worry, is it a ZFS issue or SMB again? Cheers, Emily> On Dec 31, 2009, at 11:38 PM, Emily Grettel wrote: > > > Hi Richard, > > > > This is my zpool status -v > > > > pool: tank > > state: ONLINE > > status: One or more devices has experienced an unrecoverable error. > > An > > attempt was made to correct the error. Applications are > > unaffected. > > action: Determine if the device needs to be replaced, and clear the > > errors > > using ''zpool clear'' or replace the device with ''zpool > > replace''. > > see: http://www.sun.com/msg/ZFS-8000-9P > > scrub: scrub completed after 5h15m with 0 errors on Fri Jan 1 > > 17:39:57 2010 > > config: > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 0 > > raidz1-0 ONLINE 0 0 0 > > c7t1d0 ONLINE 0 0 2 51.5K repaired > > c7t4d0 ONLINE 0 0 2 52K repaired > > c0t1d0 ONLINE 0 0 3 77.5K repaired > > c7t5d0 ONLINE 0 0 0 > > c7t3d0 ONLINE 0 0 1 26K repaired > > c7t2d0 ONLINE 0 0 0 > > errors: No known data errors > > > > I might swap the SATA cables to some better quality ones that are > > shielded I think (ACRyan has some) and see if its that. > > > > Cheers, > > Em > > > > > From: richard.elling at gmail.com > > > To: emilygrettelisnow at hotmail.com > > > Subject: Re: [zfs-discuss] ZFS extremely slow performance > > > Date: Thu, 31 Dec 2009 19:58:24 -0800 > > > > > > hmmm... might be something other than the disk, like cables or > > > vibration. > > > Let''s see what happens after the scrub completes. > > > -- richard > > > > > > On Dec 31, 2009, at 5:34 PM, Emily Grettel wrote: > > > > > > > Hello! > > > > > > > > > > > > > This could be a broken disk, or it could be some other > > > > > hardware/software/firmware issue. Check the errors on the > > > > > device with > > > > > iostat -En > > > > > > > > Heres the output: > > > > > > > > c7t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD10EADS-00L Revision: 1A01 Serial No: > > > > Size: 1000.20GB <1000204886016 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 4 Predictive Failure Analysis: 0 > > > > c7t2d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: > > > > Size: 1000.20GB <1000204886016 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 4 Predictive Failure Analysis: 0 > > > > c7t3d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: > > > > Size: 1000.20GB <1000204886016 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 4 Predictive Failure Analysis: 0 > > > > c7t4d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: > > > > Size: 1000.20GB <1000204886016 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 4 Predictive Failure Analysis: 0 > > > > c7t5d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: > > > > Size: 1000.20GB <1000204886016 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 4 Predictive Failure Analysis: 0 > > > > c7t0d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD740GD-00FL Revision: 8F33 Serial No: > > > > Size: 74.36GB <74355769344 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 6 Predictive Failure Analysis: 0 > > > > c0t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD10EADS-00P Revision: 0A01 Serial No: > > > > Size: 1000.20GB <1000204886016 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 5 Predictive Failure Analysis: 0 > > > > c3t0d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD7500AAKS-0 Revision: 4G30 Serial No: > > > > Size: 750.16GB <750156374016 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 5 Predictive Failure Analysis: 0 > > > > c3t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 > > > > Vendor: ATA Product: WDC WD7500AAKS-0 Revision: 4G30 Serial No: > > > > Size: 750.16GB <750156374016 bytes> > > > > Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 > > > > Illegal Request: 5 Predictive Failure Analysis: 0 > > > > > > > > > You should also check the fma logs: > > > > > fmadm faulty > > > > > > > > Empty > > > > > > > > > fmdump -eV > > > > > > > > > > > > > This turned out to be huge. But they''re mostly something like > > this: > > > > > > > > > > > > Nov 13 2009 10:15:41.883716494 ereport.fs.zfs.checksum > > > > nvlist version: 0 > > > > class = ereport.fs.zfs.checksum > > > > ena = 0x7cfde552fd100401 > > > > detector = (embedded nvlist) > > > > nvlist version: 0 > > > > version = 0x0 > > > > scheme = zfs > > > > pool = 0xda1d003c03abad23 > > > > vdev = 0x4389ee65271b9187 > > > > (end detector) > > > > pool = tank > > > > pool_guid = 0xda1d003c03abad23 > > > > pool_context = 0 > > > > pool_failmode = wait > > > > vdev_guid = 0x4389ee65271b9187 > > > > vdev_type = replacing > > > > parent_guid = 0x79c2f2cf0b81ae5a > > > > parent_type = raidz > > > > zio_err = 0 > > > > zio_offset = 0xae9b3fa00 > > > > zio_size = 0x6600 > > > > zio_objset = 0x24 > > > > zio_object = 0x1b2 > > > > zio_level = 0 > > > > zio_blkid = 0x635 > > > > __ttl = 0x1 > > > > __tod = 0x4afc971d 0x34ac718e > > > > > > > > > > > > Thanks for helping and telling me about those commands :-) > > > > > > > > The scrub I started last night is still running, it usually takes > > > > about 8 hours. Will post the results. > > > > > > > > - Em > > > > > > > > > > > > > From: richard.elling at gmail.com > > > > > To: milek at task.gda.pl > > > > > Date: Thu, 31 Dec 2009 08:37:03 -0800 > > > > > CC: zfs-discuss at opensolaris.org > > > > > Subject: Re: [zfs-discuss] ZFS extremely slow performance > > > > > > > > > > On Dec 31, 2009, at 2:49 AM, Robert Milkowski wrote: > > > > > > > > > > > > > > > > > judging by a *very* quick glance it looks like you have an > > issue > > > > > > with c3t0d0 device which is responding very slowly. > > > > > > > > > > Yes, there is an I/O stuck on the device which is not getting > > > > serviced. > > > > > See below... > > > > > > > > > > > > > > > > > -- > > > > > > Robert Milkowski > > > > > > http://milek.blogspot.com > > > > > > > > > > > > > > > > > > > > Australia''s #1 job site If It Exists, You''ll Find it on SEEK > > > > > > > Check out Domain Radar NOW! A world FIRST in property search has > > arrived! >Browse profiles for FREE View photos of singles in your area! _________________________________________________________________ If It Exists, You''ll Find it on SEEK Australia''s #1 job site http://clk.atdmt.com/NMN/go/157639755/direct/01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100110/b1076d48/attachment.html>