Erik Trimble
2010-Jul-29 08:41 UTC
[zfs-discuss] [osol-discuss] ZFS read performance terrible
On 7/28/2010 4:22 PM, Karol wrote:> I appear to be getting between 2-9MB/s reads from individual disks in my zpool as shown in iostat -v > I expect upwards of 100MBps per disk, or at least aggregate performance on par with the number of disks that I have. > > My configuration is as follows: > Two Quad-core 5520 processors > 48GB ECC/REG ram > 2x LSI 9200-8e SAS HBAs (2008 chipset) > Supermicro 846e2 enclosure with LSI sasx36 expander backplane > 20 seagate constellation 2TB SAS harddrives > 2x 8GB Qlogic dual-port FC adapters in target mode > 4x Intel X25-E 32GB SSDs available (attached via LSI sata-sas interposer) > mpt_sas driver > multipath enabled, all four LSI ports connected for 4 paths available: > f_sym, load-balance logical-block region size 11 on seagate drives > f_asym_sun, load-balance none, on intel ssd drives > > currently not using the SSDs in the pools since it seems I have a deeper issue here. > Pool configuration is four 2-drive mirror vdevs in one pool, and the same in another pool. 2 drives are for OS and 2 drives aren''t being used at the moment. > > Where should I go from here to figure out what''s wrong? > Thank you in advance - I''ve spent days reading and testing but I''m not getting anywhere. >This is a discussion for zfs-discuss at opensolaris.org. I''m moving replies to that list. We need to know specifics about what your usage pattern is. That is, what size of read/write are you doing to these mirrors? Also, are your SSDs configured as slog (ZIL) or as cache (L2ARC)? If the latter case, then ''iostat -v'' will not show performance of files which were in the L2ARC (or ARC). So, paradoxically, if you get really good performance by frequently reading files already resident in cache, then you won''t see much performance out of the disks, as they''re only being infrequently touched. This is good. :-) Also, I''m assuming you mean ''zpool iostat -v'', right? -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA
Hi Eric - thanks for your reply.
Yes, zpool iostat -v
I''ve re-configured the setup into two pools for a test:
1st pool: 8 disk stripe vdev
2nd pool: 8 disk stripe vdev
The SSDs are currently not in the pool since I am not even reaching what the
spinning rust is capable of - I believe I have a deeper issue and they would
only complicate things for me at this point.
I can reconfigure the pool however needed, since this server is not yet in
production.
My test is through 8gb FC target through comstar from a Windows Workstation.
The pool is currently configured with a default 128k recordsize.
Then I:
touch /pool/file
stmfadm create-lu -p wcd=false -s 10T /pool/file
stmfadm add-view <lu>
(The lu defaults to reporting a 512 blk size)
I formatted the volume NTFS cluster size default 4k
I do that twice (two seperate pools, two seperate LUNs, etc)
Then I copy a large file (700MB or so) to one of the LUNs from the local
workstation.
The read performance of my workstation harddrive is about 100+ MBps, and as such
the file copies at about that speed.
Then I make a few copies of the file on that LUN so that I have about 20+ GB of
that same file on one of the LUNs.
Then I reboot the opensolaris server (since the cache is nicely populated at
this point and everything is running fast)
Then I try copying the lot of those files from one lun to the other.
The read performance appears to be limiting my write performance.
I have tried matching recordsize to NTFS cluster size at 4k, 16k, 32 and 64k.
I have tried making NTFS clustersize a multiple of recordsize.
I have seen performance improvements as a result (I dont'' have numbers)
however, none of the cluster/block combinations brought me to where I should be
on reads.
I''ve tried many configurations - and I''ve seen my performance
fluctuate up and down here and there. However, it''s never on-par with
what it should be and the reads seem to be a limiting factor.
For clarity - here''s some ''zpool iostat -v 1'' output
from my current configuration directly following a reboot of the server
while copying 13GB of those files from LUN -> LUN:
capacity operations bandwidth
pool alloc free read write read write
----------------------- ----- ----- ----- ----- ----- -----
~snip~
edit1 13.8G 16.3T 773 0 96.5M 0
c0t5000C50020C7A44Bd0 1.54G 1.81T 75 0 9.38M 0
c0t5000C50020C7C9DFd0 1.54G 1.81T 89 0 11.2M 0
c0t5000C50020C7CE1Fd0 1.53G 1.81T 82 0 10.3M 0
c0t5000C50020C7D86Bd0 1.53G 1.81T 85 0 10.6M 0
c0t5000C50020C61ACBd0 1.55G 1.81T 83 0 10.4M 0
c0t5000C50020C79DEFd0 1.54G 1.81T 92 0 11.5M 0
c0t5000C50020CD3473d0 1.53G 1.81T 84 0 10.6M 0
c0t5000C50020CD5873d0 1.53G 1.81T 87 0 11.0M 0
c0t5000C500103F36BFd0 1.54G 1.81T 92 0 11.5M 0
----------------------- ----- ----- ----- ----- ----- -----
syspool 35.1G 1.78T 0 0 0 0
mirror 35.1G 1.78T 0 0 0 0
c0t5000C5001043D3BFd0s0 - - 0 0 0 0
c0t5000C500104473EFd0s0 - - 0 0 0 0
----------------------- ----- ----- ----- ----- ----- -----
test1 11.0G 16.3T 850 0 106M 0
c0t5000C500103F48FFd0 1.23G 1.81T 95 0 12.0M 0
c0t5000C500103F49ABd0 1.23G 1.81T 92 0 11.6M 0
c0t5000C500104A3CD7d0 1.22G 1.81T 92 0 11.6M 0
c0t5000C500104A5867d0 1.24G 1.81T 97 0 12.0M 0
c0t5000C500104A7723d0 1.22G 1.81T 95 0 11.9M 0
c0t5000C5001043A86Bd0 1.23G 1.81T 96 0 12.1M 0
c0t5000C5001043C1BFd0 1.22G 1.81T 91 0 11.3M 0
c0t5000C5001043D1A3d0 1.23G 1.81T 91 0 11.4M 0
c0t5000C5001046534Fd0 1.23G 1.81T 97 0 12.2M 0
----------------------- ----- ----- ----- ----- ----- -----
~snip~
Here''s some zpool iostat (no -v) output over the same time:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
~snip~
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 0 956 0 120M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 142 564 17.9M 52.8M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 723 0 90.3M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 1.37K 0 174M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 536 0 66.9M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 980 0 122M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 958 0 120M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 771 0 96.0M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 899 0 112M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 1.00K 0 128M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 968 0 121M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 256 0 32.0M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 305 690 38.2M 81.6M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 255 0 32.0M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 151 868 19.0M 109M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 47 933 5.99M 117M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 0 953 0 119M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 0 955 0 119M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 256 0 32.0M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 281 443 35.2M 46.4M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 537 0 67.0M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 780 0 97.4M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 458 0 56.9M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 837 0 105M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 1.67K 0 212M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 481 0 60.1M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 773 0 96.6M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 769 0 96.0M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 682 0 85.2M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 779 0 97.2M 0
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 237 0 29.7M 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 493 518 61.6M 56.6M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 55 788 6.99M 98.4M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 50 927 6.37M 116M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 0 969 0 121M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 0 965 0 121M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 25 707 3.25M 72.6M
---------- ----- ----- ----- ----- ----- -----
edit1 13.8G 16.3T 0 0 0 0
syspool 35.1G 1.78T 0 0 0 0
test1 11.9G 16.3T 108 11 13.6M 12.0K
~snip~
--
This message posted from opensolaris.org