Tom Bird
2009-Jul-07 09:45 UTC
[zfs-discuss] poor performance / lots of disk activity with low throughput, seems prefetch related
Hi guys,
I''ve been having trouble with my archival kit, in the performance
department rather than data loss this time (phew!).
At the point when I took these stats where was about 250 mbit of traffic
outbound on an ixgb NIC on the thing, also about 100 mbit of new stuff
incoming.
As you can see, the amount of disk activity far exceeds the data being
served. I''ve disabled prefetching:
echo zfs_prefetch_disable/W0t1 | mdb -kw
and now activity looks more reasonable, the machine seems able to cope
with the load, so potentially the prefetch is a bit too aggressive.
The machine itself has 8GB of RAM and is serving lots of different large
files to users, so any cache will be fairly useless.
Stats: (with prefetch ENABLED)
With prefetch off most of the bandwidth figures are single / low double
digits.
# uname -a
SunOS cs0 5.11 snv_116 sun4v sparc SUNW,Sun-Fire-T200
# zpool iostat 5
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
content0 37.2T 2.85T 237 1 29.5M 9.54K
content1 28.5T 11.5T 300 53 37.4M 5.82M
content2 19.0T 1.04T 183 0 22.9M 4.27K
content3 3.85T 11.2T 312 65 39.0M 7.57M
content4 3.85T 11.2T 338 63 42.2M 7.47M
content5 3.84T 11.2T 312 65 39.0M 7.50M
content6 19.0T 1.05T 108 0 13.5M 4.12K
content7 14.0T 991G 115 0 14.4M 3.39K
---------- ----- ----- ----- ----- ----- -----
content0 37.2T 2.85T 129 0 16.1M 0
content1 28.5T 11.5T 564 57 70.5M 6.62M
content2 19.0T 1.04T 99 0 12.5M 0
content3 3.85T 11.2T 361 79 45.1M 8.08M
content4 3.85T 11.2T 995 0 124M 0
content5 3.84T 11.2T 243 46 30.3M 2.64M
content6 19.0T 1.05T 0 0 0 0
content7 14.0T 991G 203 0 25.4M 0
---------- ----- ----- ----- ----- ----- -----
content0 37.2T 2.85T 135 0 16.9M 0
content1 28.5T 11.5T 578 57 72.3M 5.79M
content2 19.0T 1.04T 96 0 12.0M 0
content3 3.85T 11.2T 377 100 47.1M 9.01M
content4 3.85T 11.2T 980 0 122M 0
content5 3.84T 11.2T 216 106 26.9M 10.4M
content6 19.0T 1.05T 0 0 0 0
content7 14.0T 991G 200 0 25.1M 0
---------- ----- ----- ----- ----- ----- -----
content0 37.2T 2.85T 131 0 16.4M 0
content1 28.5T 11.5T 578 50 72.3M 4.82M
content2 19.0T 1.04T 99 0 12.4M 0
content3 3.85T 11.2T 377 63 47.1M 6.40M
content4 3.85T 11.2T 944 0 118M 0
content5 3.84T 11.2T 243 61 30.4M 5.97M
content6 19.0T 1.05T 0 0 0 0
content7 14.0T 991G 209 0 26.1M 0
---------- ----- ----- ----- ----- ----- -----
^C
--
Tom
// www.portfast.co.uk -- internet services and consultancy
// hosting from 1.65 per domain
Richard Elling
2009-Jul-07 15:57 UTC
[zfs-discuss] poor performance / lots of disk activity with low throughput, seems prefetch related
Tom Bird wrote:> Hi guys, > > I''ve been having trouble with my archival kit, in the performance > department rather than data loss this time (phew!). > > At the point when I took these stats where was about 250 mbit of > traffic outbound on an ixgb NIC on the thing, also about 100 mbit of > new stuff incoming. > > As you can see, the amount of disk activity far exceeds the data being > served. I''ve disabled prefetching: > > echo zfs_prefetch_disable/W0t1 | mdb -kw > > and now activity looks more reasonable, the machine seems able to cope > with the load, so potentially the prefetch is a bit too aggressive.benr did a nice write-up on prefetching. http://www.cuddletech.com/blog/pivot/entry.php?id=1040 There he describes the adaptive DMU-level prefetching algorithm and includes a DTrace script which will help you see what is going on.> > The machine itself has 8GB of RAM and is serving lots of different > large files to users, so any cache will be fairly useless. > > > Stats: (with prefetch ENABLED) > > With prefetch off most of the bandwidth figures are single / low > double digits.Makes sense... doing less work. OTOH, if you''re seeing single digit bandwidth, I''d say the machine is loafing :-) -- richard> > # uname -a > SunOS cs0 5.11 snv_116 sun4v sparc SUNW,Sun-Fire-T200 > > # zpool iostat 5 > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > content0 37.2T 2.85T 237 1 29.5M 9.54K > content1 28.5T 11.5T 300 53 37.4M 5.82M > content2 19.0T 1.04T 183 0 22.9M 4.27K > content3 3.85T 11.2T 312 65 39.0M 7.57M > content4 3.85T 11.2T 338 63 42.2M 7.47M > content5 3.84T 11.2T 312 65 39.0M 7.50M > content6 19.0T 1.05T 108 0 13.5M 4.12K > content7 14.0T 991G 115 0 14.4M 3.39K > ---------- ----- ----- ----- ----- ----- ----- > content0 37.2T 2.85T 129 0 16.1M 0 > content1 28.5T 11.5T 564 57 70.5M 6.62M > content2 19.0T 1.04T 99 0 12.5M 0 > content3 3.85T 11.2T 361 79 45.1M 8.08M > content4 3.85T 11.2T 995 0 124M 0 > content5 3.84T 11.2T 243 46 30.3M 2.64M > content6 19.0T 1.05T 0 0 0 0 > content7 14.0T 991G 203 0 25.4M 0 > ---------- ----- ----- ----- ----- ----- ----- > content0 37.2T 2.85T 135 0 16.9M 0 > content1 28.5T 11.5T 578 57 72.3M 5.79M > content2 19.0T 1.04T 96 0 12.0M 0 > content3 3.85T 11.2T 377 100 47.1M 9.01M > content4 3.85T 11.2T 980 0 122M 0 > content5 3.84T 11.2T 216 106 26.9M 10.4M > content6 19.0T 1.05T 0 0 0 0 > content7 14.0T 991G 200 0 25.1M 0 > ---------- ----- ----- ----- ----- ----- ----- > content0 37.2T 2.85T 131 0 16.4M 0 > content1 28.5T 11.5T 578 50 72.3M 4.82M > content2 19.0T 1.04T 99 0 12.4M 0 > content3 3.85T 11.2T 377 63 47.1M 6.40M > content4 3.85T 11.2T 944 0 118M 0 > content5 3.84T 11.2T 243 61 30.4M 5.97M > content6 19.0T 1.05T 0 0 0 0 > content7 14.0T 991G 209 0 26.1M 0 > ---------- ----- ----- ----- ----- ----- ----- > ^C >