Jim Mauro
2009-Mar-31 14:28 UTC
[zfs-discuss] [Fwd: Re: [perf-discuss] ZFS performance issue - READ is slow as hell...]
Posting this back to zfs-discuss. Roland''s test case (below) is a single threaded sequential write followed by a single threaded sequential read. His bandwidth goes from horrible (~2MB/sec) to expected (~30MB/sec) when prefetch is disabled. This is with relatively recent nv bits (nv110). Roland - I''m wondering if you were tripping over CR6732803 ZFS prefetch creates performance issues for streaming workloads. It seems possible, but that CR is specific about multiple, concurrent IO streams, and your test case was only one. I think it''s more likely you were tripping over CR6412053 zfetch needs a whole lotta love. For both CR''s the workaround is disabling prefetch (echo "zfs_prefetch_disable/W 1" | mdb -kw) Any other theories on this test case? Thanks, /jim -------- Original Message -------- Subject: Re: [perf-discuss] ZFS performance issue - READ is slow as hell... Date: Tue, 31 Mar 2009 02:33:00 -0700 (PDT) From: roland <devzero at web.de> To: perf-discuss at opensolaris.org Hello Jim, i double checked again - but it`s like i told: echo zfs_prefetch_disable/W0t1 | mdb -kw fixes my problem. i did a reboot and only set this single param - which immediately makes the read troughput go up from ~2 MB/s to ~30 MB/s>I don''t understand why disabling ZFS prefetch solved this >problem. The test case was a single threaded sequential write, followed >by a single threaded sequential read.i did not even do a single write - after reboot i just did dd if=/zfs/TESTFILE of=/dev/null Solaris Express Community Edition snv_110 X86 FSC RX300 S2 4GB RAM LSI Logic MegaRaid 320 Onboard SCSI Raid Controller 1x Raid1 LUN 1x Raid5 LUN (3 Disks) (both LUN`s show same behaviour) before: extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 21.3 0.1 2717.6 0.1 0.7 0.0 31.8 1.7 2 4 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 16.0 0.0 2048.4 0.0 34.9 0.1 2181.8 4.8 100 3 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 28.0 0.0 3579.2 0.0 34.8 0.1 1246.2 4.9 100 5 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 45.0 0.0 5760.4 0.0 34.8 0.2 772.7 4.5 100 7 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 19.0 0.0 2431.9 0.0 34.9 0.1 1837.3 4.4 100 3 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 58.0 0.0 7421.1 0.0 34.6 0.3 597.4 5.8 100 12 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 after: extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 218.0 0.0 27842.3 0.0 0.0 0.4 0.1 1.8 1 40 c0t1d0 241.0 0.0 30848.0 0.0 0.0 0.4 0.0 1.6 0 38 c0t1d0 237.0 0.0 30340.1 0.0 0.0 0.4 0.0 1.6 0 38 c0t1d0 230.0 0.0 29434.7 0.0 0.0 0.4 0.0 1.8 0 40 c0t1d0 238.1 0.0 30471.3 0.0 0.0 0.4 0.0 1.5 0 37 c0t1d0 234.9 0.0 30001.9 0.0 0.0 0.4 0.0 1.6 1 37 c0t1d0 220.1 0.0 28171.4 0.0 0.0 0.4 0.2 1.6 5 35 c0t1d0 212.0 0.0 27137.2 0.0 0.0 0.4 0.2 1.8 4 39 c0t1d0 203.9 0.0 26103.5 0.0 0.0 0.4 0.2 1.9 5 39 c0t1d0 214.8 0.0 27489.8 0.0 0.0 0.4 0.2 1.7 5 37 c0t1d0 221.3 0.0 28327.6 0.0 0.0 0.4 0.2 1.6 5 36 c0t1d0 199.0 0.0 25407.9 0.0 0.0 0.4 0.2 2.0 4 39 c0t1d0 182.0 0.0 23297.1 0.0 0.0 0.4 0.2 2.4 4 44 c0t1d0 204.9 0.0 26230.2 0.0 0.0 0.4 0.2 1.8 5 36 c0t1d0 214.1 0.0 27399.9 0.0 0.0 0.4 0.2 1.7 5 37 c0t1d0 207.9 0.0 26611.5 0.0 0.0 0.4 0.2 1.9 4 39 c0t1d0 -- This message posted from opensolaris.org _______________________________________________ perf-discuss mailing list perf-discuss at opensolaris.org
Ben Rockwood
2009-Mar-31 16:45 UTC
[zfs-discuss] [Fwd: Re: [perf-discuss] ZFS performance issue - READ is slow as hell...]
Ya, I agree that we need some additional data and testing. The iostat data in itself doesn''t suggest to me that the process (dd) is slow but rather that most of the data is being retrieved elsewhere (ARC). An fsstat would be useful to correlate with the iostat data. One thing that also comes to mind with streaming write performance is the effects of the write throttle... curious if he''d have gotten more on the write side with that disabled. All these things don''t strike me particularly as bugs (although there is always improvement) but rather that ZFS is designed for real world environments, not antiquated benchmarks. benr. Jim Mauro wrote:> > Posting this back to zfs-discuss. > > Roland''s test case (below) is a single threaded sequential write > followed by a single threaded sequential read. His bandwidth > goes from horrible (~2MB/sec) to expected (~30MB/sec) > when prefetch is disabled. This is with relatively recent nv bits > (nv110). > > Roland - I''m wondering if you were tripping over > CR6732803 ZFS prefetch creates performance issues for streaming > workloads. > It seems possible, but that CR is specific about multiple, concurrent > IO streams, > and your test case was only one. > > I think it''s more likely you were tripping over > CR6412053 zfetch needs a whole lotta love. > > For both CR''s the workaround is disabling prefetch > (echo "zfs_prefetch_disable/W 1" | mdb -kw) > > Any other theories on this test case? > > Thanks, > /jim > > > -------- Original Message -------- > Subject: Re: [perf-discuss] ZFS performance issue - READ is slow > as hell... > Date: Tue, 31 Mar 2009 02:33:00 -0700 (PDT) > From: roland <devzero at web.de> > To: perf-discuss at opensolaris.org > > > > Hello Jim, > i double checked again - but it`s like i told: > > echo zfs_prefetch_disable/W0t1 | mdb -kw > fixes my problem. > > i did a reboot and only set this single param - which immediately > makes the read troughput go up from ~2 MB/s to ~30 MB/s > >> I don''t understand why disabling ZFS prefetch solved this >> problem. The test case was a single threaded sequential write, followed >> by a single threaded sequential read. > > i did not even do a single write - after reboot i just did > dd if=/zfs/TESTFILE of=/dev/null > > Solaris Express Community Edition snv_110 X86 > FSC RX300 S2 > 4GB RAM > LSI Logic MegaRaid 320 Onboard SCSI Raid Controller > 1x Raid1 LUN > 1x Raid5 LUN (3 Disks) > (both LUN`s show same behaviour) > > > before: > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 21.3 0.1 2717.6 0.1 0.7 0.0 31.8 1.7 2 4 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 16.0 0.0 2048.4 0.0 34.9 0.1 2181.8 4.8 100 3 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 28.0 0.0 3579.2 0.0 34.8 0.1 1246.2 4.9 100 5 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 45.0 0.0 5760.4 0.0 34.8 0.2 772.7 4.5 100 7 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 19.0 0.0 2431.9 0.0 34.9 0.1 1837.3 4.4 100 3 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 58.0 0.0 7421.1 0.0 34.6 0.3 597.4 5.8 100 12 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > 0.0 0.0 0.0 0.0 35.0 0.0 0.0 0.0 100 0 c0t1d0 > > > after: > extended device statistics > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device > 218.0 0.0 27842.3 0.0 0.0 0.4 0.1 1.8 1 40 c0t1d0 > 241.0 0.0 30848.0 0.0 0.0 0.4 0.0 1.6 0 38 c0t1d0 > 237.0 0.0 30340.1 0.0 0.0 0.4 0.0 1.6 0 38 c0t1d0 > 230.0 0.0 29434.7 0.0 0.0 0.4 0.0 1.8 0 40 c0t1d0 > 238.1 0.0 30471.3 0.0 0.0 0.4 0.0 1.5 0 37 c0t1d0 > 234.9 0.0 30001.9 0.0 0.0 0.4 0.0 1.6 1 37 c0t1d0 > 220.1 0.0 28171.4 0.0 0.0 0.4 0.2 1.6 5 35 c0t1d0 > 212.0 0.0 27137.2 0.0 0.0 0.4 0.2 1.8 4 39 c0t1d0 > 203.9 0.0 26103.5 0.0 0.0 0.4 0.2 1.9 5 39 c0t1d0 > 214.8 0.0 27489.8 0.0 0.0 0.4 0.2 1.7 5 37 c0t1d0 > 221.3 0.0 28327.6 0.0 0.0 0.4 0.2 1.6 5 36 c0t1d0 > 199.0 0.0 25407.9 0.0 0.0 0.4 0.2 2.0 4 39 c0t1d0 > 182.0 0.0 23297.1 0.0 0.0 0.4 0.2 2.4 4 44 c0t1d0 > 204.9 0.0 26230.2 0.0 0.0 0.4 0.2 1.8 5 36 c0t1d0 > 214.1 0.0 27399.9 0.0 0.0 0.4 0.2 1.7 5 37 c0t1d0 > 207.9 0.0 26611.5 0.0 0.0 0.4 0.2 1.9 4 39 c0t1d0