I have a problem with my zfs system, it''s getting slower and slower over time. When the OpenSolaris machine is rebooted and just started I get about 30-35MB/s in read and write but after 4-8 hours I''m down to maybe 10MB/s and it varies between 4-18MB/s. Now, if i reboot the machine it''s all gone and I have perfect speed again. Does it have something to do with the cache? I use a separate SSD as a cache disk. Anyways, here''s my setup: OpenSolaris 1.34 dev C2D with 4GB ram 4x 1,5TB WD SATA drives and 1x Corsair 32GB SSD as cache Doesn''t seem to matter if I copy files locally on the computer or if I use CIFS, still getting the same degredation in speed. Last night I left my workstation copying files to/from the server for about 8 hours and you could see the performance dropping from about 28MB/s down to under 10MB/s after a couple of hours. Any suggestion on what to do? I''ve tried some tuning by setting the following variables in /etc/system: set zfs:zfs_txg_timeout = 1 set zfs:zfs_vdev_max_pending = 1 But it doesn''t seem to make any difference. Regards /Marcus Wilhelmsson, Kalmar, Sweden Message was edited by: tanngens -- This message posted from opensolaris.org
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Marcus Wilhelmsson > > I have a problem with my zfs system, it''s getting slower and slower > over time. When the OpenSolaris machine is rebooted and just started I > get about 30-35MB/s in read and write but after 4-8 hours I''m down to > maybe 10MB/s and it varies between 4-18MB/s. Now, if i reboot the > machine it''s all gone and I have perfect speed again. > > Does it have something to do with the cache? I use a separate SSD as a > cache disk.If it is somehow related to the cache, fortunately that''s easy to test for. Just remove your log device, or cache device, and see if that helps. But I doubt it.> Anyways, here''s my setup: > OpenSolaris 1.34 dev > C2D with 4GB ram > 4x 1,5TB WD SATA drives and 1x Corsair 32GB SSD as cacheNot knowing much, I''m going to suspect your RAM. 4G doesn''t sound like much to me. How large is your filesystem? I think the number of files is probably more relevant than the total number of Gb.> Doesn''t seem to matter if I copy files locally on the computer or if I > use CIFS, still getting the same degredation in speed. Last night I > left my workstation copying files to/from the server for about 8 hours > and you could see the performance dropping from about 28MB/s down to > under 10MB/s after a couple of hours.What are you using to measure the performance? Is it read, or write? Do you have compression or dedupe enabled? Please send your /etc/release file. At least the relevant parts. Also, please send your "zpool status"
> > From: zfs-discuss-bounces at opensolaris.org > [mailto:zfs-discuss- > > bounces at opensolaris.org] On Behalf Of Marcus > Wilhelmsson > > > > I have a problem with my zfs system, it''s getting > slower and slower > > over time. When the OpenSolaris machine is rebooted > and just started I > > get about 30-35MB/s in read and write but after 4-8 > hours I''m down to > > maybe 10MB/s and it varies between 4-18MB/s. Now, > if i reboot the > > machine it''s all gone and I have perfect speed > again. > > > > Does it have something to do with the cache? I use > a separate SSD as a > > cache disk. > > If it is somehow related to the cache, fortunately > that''s easy to test for. > Just remove your log device, or cache device, and see > if that helps. But I > doubt it.I doubt it as well, but it''s worth a try.> > > > Anyways, here''s my setup: > > OpenSolaris 1.34 dev > > C2D with 4GB ram > > 4x 1,5TB WD SATA drives and 1x Corsair 32GB SSD as > cache > > Not knowing much, I''m going to suspect your RAM. 4G > doesn''t sound like much > to me. How large is your filesystem? I think the > number of files is > probably more relevant than the total number of Gb. > > > > Doesn''t seem to matter if I copy files locally on > the computer or if I > > use CIFS, still getting the same degredation in > speed. Last night I > > left my workstation copying files to/from the > server for about 8 hours > > and you could see the performance dropping from > about 28MB/s down to > > under 10MB/s after a couple of hours. > > What are you using to measure the performance?Well, I''m comparing the transfer speed from my WinXP computer and my Mac and see how much it changes over five hours of constant copying (copying large iso-files of about 20GB). Since the system is used as a home NAS there won''t be lot''s of random I/O, but rather me copying stuff over the network. Any suggestions on how to do a proper performance test is welcome.> > Is it read, or write? Do you have compression or > dedupe enabled? >Both read and write, but mostly read. No compression or dedup.> Please send your /etc/release file. At least the > relevant parts.OpenSolaris Development snv_134 X86 Assembled 01 March 2010> Also, please send your "zpool status"pool: rpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 c3d0s0 ONLINE 0 0 0 errors: No known data errors pool: s1 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM s1 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c4t0d0 ONLINE 0 0 0 c4t1d0 ONLINE 0 0 0 c4t2d0 ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 cache c4t4d0 ONLINE 0 0 0 errors: No known data errors> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss >-- This message posted from opensolaris.org
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Marcus Wilhelmsson > pool: s1 > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > s1 ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > c4t0d0 ONLINE 0 0 0 > c4t1d0 ONLINE 0 0 0 > c4t2d0 ONLINE 0 0 0 > c4t3d0 ONLINE 0 0 0 > cache > c4t4d0 ONLINE 0 0 0With this configuration, you should be pretty good at reading large files, or repeat-reading random files that you''ve recently read. Your write performance could be lower. And you would have really poor sync write performance. If reading a large sequential file, you should be able to max out Gb Ethernet. But the system you''re receiving the file to can only go as fast as a single disk, unless you''ve got something like a hardware raid controller. Still, you should be able to get approx 60 Mbytes/sec across CIFS, where the bottleneck is your laptop hard drive, where you''re receiving the file. The test I would recommend, would be: "time cp /Volumes/somemount/somefile.iso ." on a mac, and in windows running cygwin, "time cp /cygdrive/someletter/somefile.iso ." That should be an apples-to-apples test, which would really give you some number you know is accurate.
Alright, I''ve made the benchmarks and there isn''t a difference worth mentioning except that i only get about 30MB/s (to my Mac, which has an SSD as system disk). I''ve also tried copying to a ram disk with slightly better results. Well, now that I''ve restarted the server I probably won''t see the 5 sec dips until later tonight. Thanks for the help, even though it doesn''t seem to make a difference. I''ll try to get a screen capture of the dips in performance next time they occur. -- This message posted from opensolaris.org
The dips are gone. I''ve run simple copy operations via CIFS for two days and the problem hasn''t appeared anymore. I''ll try to find out what caused it though, thanks for trying to help me. -- This message posted from opensolaris.org