Hi,
This test-run number 2 using LVM/UFS.
In test-run numer 1 I was using raw devices on LVM, (Ref More on ZFS and
performance)
The hardware I am using is a 2 procs Sun450 with 2G of memmory.
I am using build 27 (snv_27)
And I have 8 internal 18G drives to play with.
I wanted to see how zfs is performing on seq database writes so I am using a
Sybase 12.5.3
to load a sybase database with the bcp command.
The database consists of 73 tables and the total number of rows in the tables
are 1082731 rows.
TEST-CASES
1) Load Sybase db on UFS using LVM raid 0 ( 4 disks)
2) Load Sybase db on ZFS with no raid pool (4 disks)
3) Load Sybase db on UFS using LVM raid 1+0 ( 8 disks)
4) Load Sybase db on ZFS with mirrored pool (8 disks)
5) Load Sybase db on UFS using LVM raid 5 ( 4 disks)
6) Load Sybase db on ZFS with raidz pool (4 disks)
7) Load Sybase db on ZFS mirrored pool (8 disks) (New mirrored config)
TEST RESULTS
1) real 21m47.816s user 4m59.034s sys 1m13.231s
2) real 22m24.896s user 4m33.678s sys 1m26.798s
3) real 24m41.917s user 5m16.934s sys 1m17.835s
4) real 21m53.277s user 4m42.986s sys 1m30.321s
7) real 23m12.916s user 4m48.234s sys 1m30.259s
5) real 79m44.429s user 5m28.397s sys 1m20.859s
6) real 27m31.590s user 4m58.835s sys 1m32.611s
CONCLUSIONS
Shouldn''t ZFS be faster ?
Hope it is, but I am more and more convinced that ZFS is lot
more CPU intensive then LVM/UFS and that was hitting my slow procs
in the E450.
I will have a go later in december when I vill have V210 to play with.
DETAILED INFO ABOUT THE TESTS CONFIG
Test 1)
Sybase-binaries is installed on UFS (/opt).
Sybase database is on UFS filesystem.
LVM Layout:
d10 1 4 c3t0d0s0 c3t1d0s0 c3t2d0s0 c2t2d0s0 -i 16b
Note: LVM didn''t allow to set stripe size below 8k.
UFS:
Filesystem is created with newfs -m 1 -i 131072 -C 128 /dev/md/rdsk/d10
and mounted in /data/sybdevices
MOUNT OPTIONS:
mount -F ufs -o logging,forcedirectio /dev/md/dsk/d10 /data/sybdevices
Test 2)
Sybase binaries is installed on zfs (/data/sybase)
Sybase databases is in /data/sybdevices using no-raid pool.
ZFS-CONFIG
pool: data
state: ONLINE
scrub: scrub completed with 0 errors on Mon Dec 5 14:41:39 2005
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
NAME USED AVAIL REFER MOUNTPOINT
data 4.66G 61.8G 9.5K /data
data/sybase 680M 61.8G 680M /data/sybase
data/sybdevices 3.24G 62.6G 3.24G /data/sybdevices
recordsize is set to 2K, because Sybase in this case in reading/writing in 2K.
reservation on data/sybdevices is set to 4G
Test 3)
Sybase-binaries is installed on UFS (/opt).
Sybase database is on UFS filesystem.
LVM Layout:
d10 -m d11 d12 1
d11 1 4 c3t0d0s0 c3t1d0s0 c3t2d0s0 c3t3d0s0 -i 16b
d12 1 4 c2t0d0s0 c2t1d0s0 c2t2d0s0 c2t3d0s0 -i 16b
Note: LVM didn''t allow to set stripe size below 8k.
UFS:
Filesystem is created with newfs -m 1 -i 131072 -C 128 /dev/md/rdsk/d10
and mounted in /data/sybdevices
MOUNT OPTIONS:
mount -F ufs -o logging,forcedirectio /dev/md/dsk/d10 /data/sybdevices
Test 4)
Sybase binaries is installed on zfs (/data/sybase)
Sybase databases is in /data/sybdevices using 4-way mirrored pool.
ZFS-CONFIG
pool: data
state: ONLINE
scrub: scrub completed with 0 errors on Tue Dec 6 07:42:08 2005
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
c3t1d0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
NAME USED AVAIL REFER MOUNTPOINT
data 4.66G 28.6G 9.5K /data
data/sybase 680M 28.6G 680M /data/sybase
data/sybdevices 3.24G 29.3G 3.24G /data/sybdevices
recordsize is set to 2K, because Sybase in this case in reading/writing in 2K.
reservation on data/sybdevices is set to 4G
Test 5)
Sybase-binaries is installed on UFS (/opt).
Sybase database is on UFS filesystem.
LVM Layout:
d10 -r c2t0d0s0 c2t1d0s0 c2t2d0s0 c2t3d0s0 -i 16b
Note: LVM didn''t allow to set stripe size below 8k.
UFS:
Filesystem is created with newfs -m 1 -i 131072 -C 128 /dev/md/rdsk/d10
and mounted in /data/sybdevices
MOUNT OPTIONS:
mount -F ufs -o logging,forcedirectio /dev/md/dsk/d10 /data/sybdevices
Test 6)
Sybase binaries is installed on zfs (/data/sybase)
Sybase databases is on emulated volumes using raidz pool.
ZFS-CONFIG
pool: data
state: ONLINE
scrub: scrub completed with 0 errors on Tue Dec 6 13:41:19 2005
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz ONLINE 0 0 0
c3t0d0s0 ONLINE 0 0 0
c3t1d0s0 ONLINE 0 0 0
c3t2d0s0 ONLINE 0 0 0
c3t3d0s0 ONLINE 0 0 0
NAME USED AVAIL REFER MOUNTPOINT
data 5.75G 60.7G 18.0K /data
data/sybase 910M 60.7G 910M /data/sybase
data/sybdevices 4.86G 60.7G 4.86G /data/sybdevices
Test 7)
Sybase binaries is installed on zfs (/data/sybase)
Sybase databases is in /data/sybdevices mirrored pool.
recordsize is set to 2K, because Sybase in this case in reading/writing in 2K.
reservation on data/sybdevices is set to 4G
ZFS-CONFIG
pool: data
state: ONLINE
scrub: scrub completed with 0 errors on Tue Dec 6 08:44:08 2005
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c3t1d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
c3t2d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
NAME USED AVAIL REFER MOUNTPOINT
data 4.66G 61.8G 9.5K /data
data/sybase 680M 61.8G 680M /data/sybase
data/sybdevices 3.24G 62.6G 3.24G /data/sybdevices
recordsize is set to 2K, because Sybase in this case in reading/writing in 2K.
reservation on data/sybdevices is set to 4G
This message posted from opensolaris.org
I''m puzzled by your puzzlement. ;-) In summary, this is what you reported: With no replication: UFS/SVM = 22 minutes ZFS = 22 minutes With two-way mirroring: UFS/SVM = 24 minutes ZFS = 23 minutes With RAID (RAID-5 for SVM, RAID-Z for ZFS): UFS/SVM = 80 minutes ZFS = 27 minutes You then asked: Shouldn''t ZFS be faster? Er... it appears that it is, no? It looks like you break even with simple stripes or mirrors, but RAID-Z seriously kicks RAID-5''s butt. This makes sense because RAID-Z never has to do read-modify-write. As these numbers indicate, that''s a very big deal. I''d have to check the SVM code or run dtrace to confirm this, but I believe that when you write a single block synchronously, the combination of UFS logging plus SVM RAID-5 will entail three dependent I/Os, whereas ZFS using the ZIL on RAID-Z will have only one. That would explain the almost-exact 3x performance win. Jeff
Darren Reed - Sun Microsystems
2005-Dec-07 20:10 UTC
[zfs-discuss] More on ZFS performance vs LVM/UFS
Jeff Bonwick wrote:>I''m puzzled by your puzzlement. ;-) > >In summary, this is what you reported: > >... >You then asked: > > Shouldn''t ZFS be faster? >With the way the results are presented, I find there''s a tendency to concentrate on the right hand column, which in all cases shows ZFS using more system CPU than UFS. The shift in numbers from user to system can be a little deceptive... Also, I think there are some other interesting ways to look at these results... In (1) and (2), total cpu time is 6m12.265s (UFS) vs 6m00.476 (ZFS) For (3) and (4), it is 6m34.769s (UFS) and 6m13.307s (ZFS) So while the "real time" is approximately the same, the combined CPU time in both cases in less. So running ZFS instead of UFS could be seen to be "cooler" :) If you were being billed for CPU time, ZFS would be more economical than UFS. I also wonder if there are less IOs from ZFS than from UFS? :) Darren
Patrik Gustavsson wrote:> >CONCLUSIONS > >Shouldn''t ZFS be faster ? > > >I think another way of looking at this is that you are hitting the limits of the SCSI subsystem. The big winner is ZFS vs RAID5. RAID5 has never been viable where performance is an issue, hence the proliferation of hardware RAID5. My experience is hardware RAID5 gives similar sequential write performance to SVM RAID1, so your results point to ZFS offering similar performance to hardware RAID5. This is something we have never had before, hardware level performance with added data integrity. I don''t think I''ll be buying any more RAID cards :) Ian
> I also wonder if there are less IOs from ZFS than from UFS? :) > > DarrenIt depends! However, it''s quite likely that ZFS would do less IO. It''s not a simple calculation for database applications. For UFS, if the database file is pre-written then there are no meta data changes and no log, directory, indirect block writes etc. So assuming a random distribution of record writes we have one write per record. For ZFS, it''s more complex. If records are written in parallel (ie multi-threaded) we can aggregate the writes into one log block write, up to a limit of 128KB. Then later the record updates will also be written out in the background as part of the transaction group commit. However, these also get coalesced, and written sequentially (apart from the uberblock updates) and as larger writes. Neil. This message posted from opensolaris.org