Sean Meighan
2006-Jul-15 19:20 UTC
[zfs-discuss] half duplex read/write operations to disk sometimes?
Hi Roch; thanks for the discussion yesterday. I said I would plot the
iostat data, here it is.
The strange thing is the seeming half duplex operation of ZFS. Here is a
45 minute plot of
%zpool iostat 1
The x-axis is data gathered every second. Note that when the write cycle
seems to start at 0,300 and 600 seconds that it kills all read ops to
the disk. Remember our Canary data? We receive 110,000 50-100kb gzipped
files per day. This means we on average are receiving one file every
second. We unzip this file, untar it into its original 10 files and then
read one of those 1000 line files to parse it. We are therefore writing
and reading all the time. I can see data being updated every second in
our directories, i assume that ZFS is caching this data. During the
600-800 second range below the system became very unresponsive (1 second
ls command turned into 45 second command, pwd takes 15 seconds).
A couple of questions is :
1) is it normal for all read operations to go to zero for 5 minutes why
only writes are being done?
2) is this something i can tune?
3) is it expected that this would slow down the user experience?
The previous graph shows read/write operations. Here is the same data
showing the bytes/second.
A question from this graph is why is read transfers around 20mb/second
and writes are around 2mb? Is there a reason there is a 10x difference.
For people inside of Sun contact me and i will give you the demiroot
login to itsm-mpk-2.sfbay (this is a T2000 32x1000mhz, 8 gig machine
running Solaris 10). We are using only the internal 2.5" disk drives to
store the 3.5 million files, 3 billion lines of ASCII. We use about
30gigs to store this in a sliding one month window. The internal drive
that we are on is at 100% busy all the time. You can see the write cycle
below followed by the big read cycle.
itsm-mpk-2% iostat -xn 1 999
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
206.2 199.8 10494.9 1286.4 14.1 13.4 34.7 33.1 76 93 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0 0 c0t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
itsm-mpk-2:vold(pid329)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
21.4 303.3 1263.9 3874.3 19.0 16.0 58.5 49.2 100 100 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
itsm-mpk-2:vold(pid329)
itsm-mpk-2% iostat -xn 1 999 | grep c1t0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
21.4 303.3 1263.9 3874.3 19.0 16.0 58.5 49.2 100 100 c1t0d0
1.0 502.0 8.0 1601.1 21.0 16.0 41.7 31.8 100 100 c1t0d0
1.0 442.0 56.0 1479.4 21.6 16.0 48.8 36.1 100 100 c1t0d0
2.0 521.0 128.0 1886.6 21.5 16.0 41.0 30.6 100 100 c1t0d0
0.0 697.0 0.0 2237.0 21.0 16.0 30.1 22.9 100 100 c1t0d0
1.0 736.0 64.0 2469.1 21.0 16.0 28.4 21.7 100 100 c1t0d0
1.0 696.0 64.0 2367.9 21.0 16.0 30.1 22.9 100 100 c1t0d0
0.0 642.0 0.0 2094.1 21.0 16.0 32.7 24.9 100 100 c1t0d0
2.0 661.9 128.0 2302.1 21.0 16.0 31.6 24.1 100 100 c1t0d0
1.0 674.1 64.0 2226.9 21.5 16.0 31.9 23.7 100 100 c1t0d0
0.0 684.1 0.0 2317.7 20.4 16.0 29.9 23.4 100 100 c1t0d0
0.0 677.0 0.0 2289.1 20.0 16.0 29.5 23.6 100 100 c1t0d0
0.0 679.7 0.0 2290.4 20.0 16.0 29.4 23.5 100 100 c1t0d0
0.0 679.5 0.0 2284.0 20.0 16.0 29.4 23.5 100 100 c1t0d0
0.0 675.5 0.0 2340.7 20.0 16.0 29.6 23.7 100 100 c1t0d0
0.0 714.0 0.0 2320.6 20.2 16.0 28.3 22.4 100 100 c1t0d0
0.0 658.9 0.0 2275.2 20.6 16.0 31.2 24.3 100 100 c1t0d0
0.0 696.0 0.0 2376.6 20.1 16.0 28.9 23.0 100 100 c1t0d0
0.0 642.0 0.0 2129.9 21.0 16.0 32.7 24.9 100 100 c1t0d0
0.0 642.0 0.0 2218.6 20.9 16.0 32.5 24.9 100 100 c1t0d0
0.0 658.0 0.0 2839.4 20.0 16.0 30.4 24.3 100 100 c1t0d0
1.0 613.0 8.0 1829.5 20.4 16.0 33.2 26.0 100 100 c1t0d0
0.0 639.0 0.0 1871.5 21.0 16.0 32.8 25.0 100 100 c1t0d0
1.0 645.0 56.0 2096.9 21.0 16.0 32.4 24.7 100 100 c1t0d0
10.0 598.0 208.0 2017.1 21.6 16.0 35.6 26.3 100 100 c1t0d0
1.0 636.0 8.0 2093.0 21.0 16.0 32.9 25.1 100 100 c1t0d0
2.0 657.0 16.0 2217.6 20.6 16.0 31.3 24.3 100 100 c1t0d0
0.0 652.0 0.0 2375.1 21.0 16.0 32.2 24.5 100 100 c1t0d0
0.0 596.0 0.0 2166.5 20.8 16.0 34.9 26.8 100 100 c1t0d0
0.0 645.0 0.0 2347.9 20.0 16.0 31.0 24.8 100 100 c1t0d0
0.0 718.9 0.0 2202.3 20.0 16.0 27.8 22.2 100 100 c1t0d0
0.0 690.0 0.0 2521.1 20.0 16.0 28.9 23.2 100 100 c1t0d0
0.0 703.0 0.0 2602.4 20.0 16.0 28.4 22.7 100 100 c1t0d0
0.0 737.0 0.0 2101.6 20.0 16.0 27.1 21.7 100 100 c1t0d0
1.0 671.9 8.0 2186.1 20.3 16.0 30.1 23.8 100 100 c1t0d0
0.0 639.1 0.0 2207.4 20.9 16.0 32.7 25.0 100 100 c1t0d0
223.2 213.2 13817.3 476.8 20.1 16.0 46.0 36.6 100 100 c1t0d0
103.4 448.6 6171.3 865.6 22.0 16.0 39.8 29.0 100 100 c1t0d0
23.0 677.0 1045.5 1169.0 22.5 16.0 32.2 22.8 100 100 c1t0d0
10.0 660.1 359.5 1151.1 21.2 16.0 31.7 23.9 100 100 c1t0d0
39.0 675.0 2340.5 987.0 21.2 16.0 29.8 22.4 100 100 c1t0d0
11.0 676.0 616.0 923.5 21.1 16.0 30.7 23.3 100 100 c1t0d0
4.0 705.0 160.0 1015.0 21.0 16.0 29.6 22.5 100 100 c1t0d0
4.0 727.0 168.0 1071.5 21.0 16.0 28.7 21.9 100 100 c1t0d0
297.0 79.0 18518.2 247.5 19.9 16.0 52.9 42.5 100 100 c1t0d0
25.0 613.0 1431.0 2088.0 22.4 16.0 35.0 25.1 100 100 c1t0d0
114.0 428.0 6756.1 1703.9 21.2 16.0 39.0 29.5 100 100 c1t0d0
207.9 217.9 12167.2 614.7 20.7 16.0 48.6 37.5 100 100 c1t0d0
363.2 11.0 21245.1 92.5 8.4 16.0 22.6 42.7 93 100 c1t0d0
351.0 7.0 21443.2 98.0 3.3 14.6 9.1 40.7 54 100 c1t0d0
332.7 5.0 20660.7 77.9 3.4 14.2 10.0 42.2 45 100 c1t0d0
354.3 3.0 22784.8 58.6 10.7 15.5 29.9 43.5 85 100 c1t0d0
349.0 4.0 21999.3 58.5 13.2 16.0 37.3 45.3 98 100 c1t0d0
353.0 3.0 22510.4 58.5 8.3 15.5 23.2 43.5 82 100 c1t0d0
344.9 0.0 21540.7 0.0 7.1 15.0 20.6 43.4 63 100 c1t0d0
386.0 0.0 22447.0 0.0 9.4 15.0 24.3 38.9 73 100 c1t0d0
373.1 0.0 20763.3 0.0 14.4 15.7 38.5 42.1 89 100 c1t0d0
364.9 0.0 23145.3 0.0 5.0 14.7 13.8 40.3 54 100 c1t0d0
363.8 0.0 22783.5 0.0 6.0 14.7 16.5 40.4 60 100 c1t0d0
357.3 0.0 22591.9 0.0 10.6 15.7 29.8 43.9 87 100 c1t0d0
369.6 0.0 22441.0 0.0 13.1 15.9 35.5 43.0 94 100 c1t0d0
344.4 1.0 22314.4 8.0 3.3 14.2 9.6 41.1 47 100 c1t0d0
344.0 0.0 22015.9 0.0 6.1 14.4 17.7 41.7 60 100 c1t0d0
372.0 0.0 22818.4 0.0 10.8 15.7 28.9 42.2 82 100 c1t0d0
376.0 2.0 21307.7 12.0 17.0 16.0 44.9 42.3 100 100 c1t0d0
372.1 0.0 23749.3 0.0 6.2 15.0 16.7 40.3 68 100 c1t0d0
347.9 0.0 22507.0 0.0 5.2 14.8 14.8 42.7 59 100 c1t0d0
357.0 0.0 22572.0 0.0 9.6 15.6 26.9 43.7 82 100 c1t0d0
365.0 0.0 21989.4 0.0 14.4 15.9 39.4 43.6 93 100 c1t0d0
355.0 0.0 22558.1 0.0 4.9 14.6 13.7 41.2 51 100 c1t0d0
website for the canary is located at http://canary.sfbay
thanks
sean
--
<http://www.sun.com> * Sean Meighan *
Mgr ITSM Engineering
*Sun Microsystems, Inc.*
US
Phone x32329 / +1 408 850-9537
Mobile 303-520-2024
Fax 408 850-9537
Email Sean.Meighan at Sun.COM
------------------------------------------------------------------------
NOTICE: This email message is for the sole use of the intended
recipient(s) and may contain confidential and privileged information.
Any unauthorized review, use, disclosure or distribution is prohibited.
If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060715/40d67247/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zpool_iostat_ops.png
Type: image/png
Size: 8644 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060715/40d67247/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zpool_iostat_bytes.png
Type: image/png
Size: 8883 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060715/40d67247/attachment-0001.png>
Roch
2006-Jul-17 15:45 UTC
[zfs-discuss] Re: half duplex read/write operations to disk sometimes?
Hi Sean, You suffer from an extreme bout of 6429205 each zpool needs to monitor it''s throughput and throttle heavy writers When this is fixed, your responsiveness will be better. Note to Mark, Sean is more than willing to test any fix we would have for this... -r