jiniusatwork-zfs at yahoo.com
2008-Feb-01 15:02 UTC
[zfs-discuss] Un/Expected ZFS performance?
I''m running Postgresql (v8.1.10) on Solaris 10 (Sparc) from within a
non-global zone. I originally had the database "storage" in the
non-global zone (e.g. /var/local/pgsql/data on a UFS filesystem) and was getting
performance of "X" (e.g. from a TPC-like application:
http://www.tpc.org). I then wanted to try relocating the database storage from
the zone (UFS filesystem) over to a ZFS-based filesystem (where I could do
things like set quotas, etc.). When I do this, I get roughly half the
performance (X/2) I did on the UFS system.
I
ran
some
low-level
I/O
tests
(from
http://iozone.org/)
on
my
setup
and
have
listed
a
sampling
below
for
an
8k
file
and
8k
record
size:
[Hopefully
the
table
formatting
survives]
UFS
filesystem
[on
local
disk]
=====================Run
KB
reclen
write
rewrite
read
reread
--------------------------------------------------------------------
1
8
8
40632
156938
199960
222501
[./iozone
-i
0
-i
1
-r
8
-s
8
->
no
fsync
include]
2
8
8
4517
5434
11997
11052
[./iozone
-i
0
-i
1
-r
8
-s
8
-e
->
fsync
included]
3
8
8
4570
5578
199960
215360
[./iozone
-i
0
-i
1
-r
8
-s
8
-o
->
usig
O_SYNC]
ZFS
filesystem
[on
StorageTek
2530
Array
in
RAID
1+0
configuration
with
a
512K
segment
size]
=====================================================================Run
KB
reclen
write
rewrite
read
reread
--------------------------------------------------------------------------
3
8
8
52281
95107
142902
142902
[./iozone
-i
0
-i
1
-r
8
-s
8
->
no
fsync
include]
4
8
8
996
1013
129152
114206
[./iozone
-i
0
-i
1
-r
8
-s
8
-e
->
fsync
included]
5
8
8
925
1007
145379
170495
[./iozone
-i
0
-i
1
-r
8
-s
8
-o
->
usig
O_SYNC]
Comparing run 1 and 3 shows that ZFS is roughly 20% faster on (unsynchronized)
writes versus UFS. What''s really surprising, to me at least, is that in
cases 3 and 5, for example, ZFS becomes almost 400% slower on synchronized
writes versus UFS. I realize that the ZFS-on-RAID setup has a "safety"
penalty, but should it really be 400% slower than UFS? If not, then I''m
hoping for suggestions on how to get some better ZFS performance from this
setup.
Thanks,
Bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080201/28c6bc5d/attachment.html>
jiniusatwork-zfs at yahoo.com said:> . . . > ZFS filesystem [on StorageTek 2530 Array in RAID 1+0 configuration > with a 512K segment size] > . . . > Comparing run 1 and 3 shows that ZFS is roughly 20% faster on > (unsynchronized) writes versus UFS. What''s really surprising, to me at least, > is that in cases 3 and 5, for example, ZFS becomes almost 400% slower on > synchronized writes versus UFS. I realize that the ZFS-on-RAID setup has a > "safety" penalty, but should it really be 400% slower than UFS? If not, then > I''m hoping for suggestions on how to get some better ZFS performance from > this setup.I don''t think there is any "safety penalty" for ZFS on RAID, unless you''re comparing it to ZFS on JBOD. On RAID without ZFS-level redundancy, you only give up ZFS-level self-healing. The sync-write issue here is likely similar to that of an NFS server. If all of your ZFS pools on this system are on battery-backed cache RAID (e.g. the 2530 array), then you could safely set zfs_nocacheflush=1. If not, then there should be a way to set the 2530 to ignore the ZFS sync-cache requests. Give it a try and let us all know how it affects your tests. We''ve got a 2530 here doing Oracle duty, but it''s so much faster than the storage it replaced that we haven''t bothered doing any performance tuning. Regards, Marion
jiniusatwork-zfs at yahoo.com
2008-Feb-05 02:03 UTC
[zfs-discuss] Un/Expected ZFS performance?
----- Original Message ----> From: Marion Hakanson <hakansom at ohsu.edu> > To: jiniusatwork-zfs at yahoo.com > Cc: zfs-discuss at opensolaris.org > Sent: Friday, February 1, 2008 1:01:46 PM > Subject: Re: [zfs-discuss] Un/Expected ZFS performance? > > jiniusatwork-zfs at yahoo.com said: > > . . . > > ZFS filesystem [on StorageTek 2530 Array in RAID 1+0 configuration > > with a 512K segment size] > > . . . > > Comparing run 1 and 3 shows that ZFS is roughly 20% faster on > > (unsynchronized) writes versus UFS. What''s really surprising, to me at least, > > is that in cases 3 and 5, for example, ZFS becomes almost 400% slower on > > synchronized writes versus UFS. I realize that the ZFS-on-RAID setup has a > > "safety" penalty, but should it really be 400% slower than UFS? If not, then > > I''m hoping for suggestions on how to get some better ZFS performance from > > this setup. > > > I don''t think there is any "safety penalty" for ZFS on RAID, unless you''re > comparing it to ZFS on JBOD. On RAID without ZFS-level redundancy, you only > give up ZFS-level self-healing. > > The sync-write issue here is likely similar to that of an NFS server. If all > of your ZFS pools on this system are on battery-backed cache RAID (e.g. the > 2530 array), then you could safely set zfs_nocacheflush=1. If not, then > there should be a way to set the 2530 to ignore the ZFS sync-cache requests. > > Give it a try and let us all know how it affects your tests. We''ve got > a 2530 here doing Oracle duty, but it''s so much faster than the storage > it replaced that we haven''t bothered doing any performance tuning. >Marion, Thanks for the reply. As it turns out, I think I''m going to try a different strategy. Apparently, I can make the 2530 acknowledge the "sync" requests back to the OS without causing the write cache to get flushed (which was happening by default). Setting zfs_nocacheflush=1 would, I think, accomplish the same thing (i.e. ignore sync requests) from the OS side, but this would end up being a "global" ZFS setting on the OS. The problem with the latter solution is that if I ever decided to add non-intelligent storage to the system, it wouldn''t get the sync calls either. Thanks again, Bob