hi , I''m using sun T1000 machines one machine is installed Solaris 10 with UFS and other system with ZFS file system , ZFS machine is performing slow . Running following commands on both systems shows Disk get busy immediatly to 100% ######################## ZFS MACHINE############ find / > /dev/null 2>&1 & iostat -xnmpz 5 [root at zfs-serv ktahir]# iostat -xnmpz 5 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.4 0.2 12.3 2.2 0.0 0.0 6.5 3.9 0 0 c0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.9 0 0 192.168.150.131:/export/home2 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 86.4 0.0 5527.4 0.0 0.0 1.0 0.0 11.2 0 97 c0d0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 87.4 0.0 5593.7 0.0 0.0 1.0 0.0 11.1 0 96 c0d0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 85.2 0.0 5452.8 0.0 0.0 1.0 0.0 11.3 0 96 c0d0 but on UFS file system averge busy is 50% , any idea why ZFS makes disk more busy ? any idea -- This message posted from opensolaris.org
On Wed, 17 Mar 2010, Kashif Mumtaz wrote:> > but on UFS file system averge busy is 50% , > > any idea why ZFS makes disk more busy ?Clearly there are many more reads per second occuring on the zfs filesystem than the ufs filesystem. Assuming that the application-level requests are really the same, this suggests that the system does not have enough RAM installed in order to cache the "working set". Another issue could be fileystem block size since zfs defaults the block size to 128K but some applications (e.g. database) work better with 4K, 8K, or 16K block size. Regardless, I suggest measuring the statistics with a 30 second interval rather than 5 seconds since zfs is assured to do whatever it does within 30 seconds. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Wed, Mar 17, 2010 at 10:15:53AM -0500, Bob Friesenhahn wrote:> Clearly there are many more reads per second occuring on the zfs > filesystem than the ufs filesystem.yes> Assuming that the application-level requests are really the sameFrom the OP, the workload is a "find /". So, ZFS makes the disks busier.. but is it find''ing faster as a result, or doing more reads per found file? The ZFS io pipeline will be able to use the cpu concurrency of the T1000 better than UFS, even for a single-threaded find, and may just be issuing IO faster. Count the number of lines printed and divide by the time taken to compare whether the extra work being done is producing extra output or not. However, it might also be worthwhile to look for a better / more representative benchmark and compare further using that. Also, to be clear, could you clarify whether the "problem" you see that the numbers in iostat are larger, or that find runs slower, or that other processes are more impacted by find?> this suggests that the system does not have > enough RAM installed in order to cache the "working set".Possibly, yes.> Another issue > could be fileystem block size since zfs defaults the block size to 128K > but some applications (e.g. database) work better with 4K, 8K, or 16K > block size.Unlikely to be relevant to fs metadata for find.> Regardless, I suggest measuring the statistics with a 30 second interval > rather than 5 seconds since zfs is assured to do whatever it does within > 30 seconds.Relevant for write benchmarks more so than read. -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100318/d2641147/attachment.bin>
Hi, I did another test on both machine. And write performance on ZFS extraordinary slow. I did the following test on both machines For write time dd if=/dev/zero of=test.dbf bs=8k count=1048576 For read time dd if=/testpool/test.dbf of=/dev/null bs=8k ZFS machine has 32GB memory UFS machine has 16GB memory #### UFS machine test ########### time dd if=/dev/zero of=test.dbf bs=8k count=1048576 1048576+0 records in 1048576+0 records out real 2m18.352s user 0m5.080s sys 1m44.388s #iostat -xnmpz 10 r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.6 107.9 4.8 62668.4 0.0 6.7 0.1 61.9 1 83 c0t0d0 0.0 0.2 0.0 0.2 0.0 0.0 0.0 0.8 0 0 c0t0d0s5 0.6 107.7 4.8 62668.2 0.0 6.7 0.1 62.0 1 83 c0t0d0s7 -------------------------------------------- For read # time dd if=test.dbf of=/dev/null bs=8k 1048576+0 records in 1048576+0 records out real 1m21.285s user 0m4.701s sys 1m15.322s For write it took 2.18 minutes and for read it took 1.21 minutes. ############## ZFS machine test ################## # time dd if=/dev/zero of=test.dbf bs=8k count=1048576 1048576+0 records in 1048576+0 records out real 140m33.590s user 0m5.182s sys 2m33.025s extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.0 8.2 0.0 1037.0 0.0 33.3 0.0 4062.3 0 100 c0t0d0 0.08.2 0.0 1037.0 0.0 33.3 0.0 4062.3 0 100 c0t0d0s0 ------------------------------------------------- For read #time dd if=test.dbf of=/dev/null bs=8k 1048576+0 records in 1048576+0 records out real 0m59.177s user 0m4.471s sys 0m54.723s For write it took 140 minutes and for read 59 seconds(less then UFS) ------------------------------------------------------------------------- In ZFS data was being write around 1037 kw/s while disk remain busy 100%. In UFS data was being written around 62668 kw/s while disk is busy at 83% Kindly help me how can I tune the writing performance on ZFS? -- This message posted from opensolaris.org
On 18/03/10 08:36 PM, Kashif Mumtaz wrote:> Hi, > I did another test on both machine. And write performance on ZFS extraordinary slow.Which build are you running? On snv_134, 2x dual-core cpus @ 3GHz and 8Gb ram (my desktop), I see these results: $ time dd if=/dev/zero of=test.dbf bs=8k count=1048576 1048576+0 records in 1048576+0 records out real 0m28.224s user 0m0.490s sys 0m19.061s This is a dataset on a straight mirrored pool, using two SATA2 drives (320Gb Seagate). $ time dd if=test.dbf bs=8k of=/dev/null 1048576+0 records in 1048576+0 records out real 0m5.749s user 0m0.458s sys 0m5.260s James C. McPherson -- Senior Software Engineer, Solaris Sun Microsystems http://www.jmcp.homeunix.com/blog
Hi, Thanks for your reply BOTH are Sun Sparc T1000 machines. Hard disk 1 TB sata on both ZFS system Memory 32 GB , Processor 1GH 6 core os Solaris 10 10/09 s10s_u8wos_08a SPARC PatchCluster level 142900-02(Dec 09 ) UFS machine Hard disk 1 TB sata Memory 16 GB Processor Processor 1GH 6 core Solaris 10 8/07 s10s_u4wos_12b SPARC -- This message posted from opensolaris.org
On Thu, Mar 18, 2010 at 03:36:22AM -0700, Kashif Mumtaz wrote:> I did another test on both machine. And write performance on ZFS extraordinary slow. > ------------------------------------------------------------------------- > In ZFS data was being write around 1037 kw/s while disk remain busy 100%.That is, as you say, such an extraordinarily slow number that we have to start at the very basics and eliminate fundamental problems. I have seen disks go bad in a way that they simply become very very slow. You need to be sure that this isn''t your problem. Or perhaps there''s some hardware issue when the disks are used in parallel? Check all the cables and connectors. Check logs for any errors. Do you have the opportunity to try testing write speed with dd to the raw disks? If the pool is mirrored, can you detach one side at a time? Test the detached disk with dd, and the pool with the other disk, one at a time and then concurrently. One slow disk will slow down the mirror (but I don''t recall seeing such an imbalance in your iostat output either). Do you have some spare disks to try other tests with? Try a ZFS install on those, and see they also have the problem. Try a UFS install on the current disks, and see if they still have the problem. Can you swap the disks between the T1000s and see if the problem stays with the disks or the chassis? You have a gremlin to hunt... -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100319/1731d024/attachment.bin>
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18.03.2010 21:31, Daniel Carosone wrote:> You have a gremlin to hunt...Wouldn''t Sun help here? ;) (sorry couldn''t help myself, I''ve spent a week hunting gremlins until I hit the brick wall of the MPT problem) //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuikAIACgkQSBMQn1jNM7ZHpQCgn15+EsQzafhJw1HnhBWlTW9X STUAoPvVS4bfq3E3N3Vg7JCuQ3M5+Am6 =YSRa -----END PGP SIGNATURE-----
On 18/03/10 10:05 PM, Kashif Mumtaz wrote:> Hi, Thanks for your reply > > BOTH are Sun Sparc T1000 machines. > > Hard disk 1 TB sata on both > > ZFS system Memory 32 GB , Processor 1GH 6 core > os Solaris 10 10/09 s10s_u8wos_08a SPARC > PatchCluster level 142900-02(Dec 09 ) > > > UFS machine > Hard disk 1 TB sata > Memory 16 GB > Processor Processor 1GH 6 core > > Solaris 10 8/07 s10s_u4wos_12b SPARCSince you are seeing this on a Solaris 10 update release, you should log a call with your support provider to get this investigated. James C. McPherson -- Senior Software Engineer, Solaris Sun Microsystems http://www.jmcp.homeunix.com/blog
James C. McPherson wrote:> On 18/03/10 10:05 PM, Kashif Mumtaz wrote: >> Hi, Thanks for your reply >> >> BOTH are Sun Sparc T1000 machines. >> >> Hard disk 1 TB sata on both >> >> ZFS system Memory 32 GB , Processor 1GH 6 core >> os Solaris 10 10/09 s10s_u8wos_08a SPARC >> PatchCluster level 142900-02(Dec 09 ) >> >> >> UFS machine >> Hard disk 1 TB sata >> Memory 16 GB >> Processor Processor 1GH 6 core >> >> Solaris 10 8/07 s10s_u4wos_12b SPARC > > Since you are seeing this on a Solaris 10 update > release, you should log a call with your support > provider to get this investigated. > > > James C. McPherson > -- > Senior Software Engineer, Solaris > Sun Microsystems > http://www.jmcp.homeunix.com/blogI would generally agree with James, with the caveaut that you could try to update to something a bit latter than Update 4. That''s pretty early-on in the ZFS deployment in Solaris 10. At the minimum, grab the latest Recommended Patch set and apply that, then see what your issues are. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800)
Erik Trimble wrote:> James C. McPherson wrote: >> On 18/03/10 10:05 PM, Kashif Mumtaz wrote: >>> Hi, Thanks for your reply >>> >>> BOTH are Sun Sparc T1000 machines. >>> >>> Hard disk 1 TB sata on both >>> >>> ZFS system Memory 32 GB , Processor 1GH 6 core >>> os Solaris 10 10/09 s10s_u8wos_08a SPARC >>> PatchCluster level 142900-02(Dec 09 ) >>> >>> >>> UFS machine >>> Hard disk 1 TB sata >>> Memory 16 GB >>> Processor Processor 1GH 6 core >>> >>> Solaris 10 8/07 s10s_u4wos_12b SPARC >> >> Since you are seeing this on a Solaris 10 update >> release, you should log a call with your support >> provider to get this investigated. >> >> >> James C. McPherson >> -- >> Senior Software Engineer, Solaris >> Sun Microsystems >> http://www.jmcp.homeunix.com/blog > I would generally agree with James, with the caveaut that you could > try to update to something a bit latter than Update 4. That''s pretty > early-on in the ZFS deployment in Solaris 10. > > At the minimum, grab the latest Recommended Patch set and apply that, > then see what your issues are. > > >Oh, nevermind. I''m an idiot. I was looking at the UFS machine. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800)
hi, Thanks for all the reply. I have found the real culprit. Hard disk was faulty. I changed the hard disk.And now ZFS performance is much better. -- This message posted from opensolaris.org
> On 18/03/10 08:36 PM, Kashif Mumtaz wrote: > > Hi, > > I did another test on both machine. And write > performance on ZFS extraordinary slow. > > Which build are you running? > > On snv_134, 2x dual-core cpus @ 3GHz and 8Gb ram (my > desktop), I > see these results: > > > $ time dd if=/dev/zero of=test.dbf bs=8k > count=1048576 > 1048576+0 records in > 1048576+0 records out > > real 0m28.224s > user 0m0.490s > sys 0m19.061s > > This is a dataset on a straight mirrored pool, using > two SATA2 > drives (320Gb Seagate).On my Ultra24 with two mirrored 1Tb WD drives 8gb memory and snv_125 I only get :- rich: ptime dd if=/dev/zero of=test.dbf bs=8k count=1048576 1048576+0 records in 1048576+0 records out real 1:44.352133699 user 0.444280089 sys 13.526079085 rich: uname -a SunOS ultra24 5.11 snv_125 i86pc i386 i86pc rich: zpool status tank pool: tank state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using ''zpool upgrade''. Once this is done, the pool will no longer be accessible on older software versions. scrub: scrub completed after 0h30m with 0 errors on Mon Apr 19 02:36:08 2010 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 errors: No known data errors rich: ipstat -En c1t3d0 ipstat: Command not found. rich: iostat -En c1t3d0 c1t3d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: ATA Product: WDC WD1001FALS-0 Revision: 0K05 Serial No: Size: 1000.20GB <1000204886016 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 4264 Predictive Failure Analysis: 0 rich: psrinfo -v Status of virtual processor 0 as of: 04/19/2010 14:23:42 on-line since 12/16/2009 21:56:59. The i386 processor operates at 3000 MHz, and has an i387 compatible floating point processor. Status of virtual processor 1 as of: 04/19/2010 14:23:42 on-line since 12/16/2009 21:57:03. The i386 processor operates at 3000 MHz, and has an i387 compatible floating point processor. Status of virtual processor 2 as of: 04/19/2010 14:23:42 on-line since 12/16/2009 21:57:03. The i386 processor operates at 3000 MHz, and has an i387 compatible floating point processor. Status of virtual processor 3 as of: 04/19/2010 14:23:42 on-line since 12/16/2009 21:57:03. The i386 processor operates at 3000 MHz, and has an i387 compatible floating point processor. Why are my drives so slow?> > $ time dd if=test.dbf bs=8k of=/dev/null > 1048576+0 records in > 1048576+0 records out > > real 0m5.749s > user 0m0.458s > sys 0m5.260s > > > James C. McPherson > -- > Senior Software Engineer, Solaris > Sun Microsystems > http://www.jmcp.homeunix.com/blog > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss >-- This message posted from opensolaris.org