zfs discuss - Dec 2005 - Read performance ZFS vs. UFS

I ran a simple test to compare read performance of ZFS and UFS while
there''s a simultaneous heavy write load.  (I wanted to demonstrate that
the I/O reordering works.)  I first took a baseline without the write load, and
UFS performed considerably better than ZFS.  Is there a good explanation for
this, other than that ZFS tuning is still in progress?  Or that the way
I''ve created the test file is essentially optimal for UFS but
sub-optimal for ZFS?

The server has three SCSI drives on which I''ve created equivalently
sized slices 6 and 7.  I''ve set up UFS/SVM RAID 5 on slices 6 and ZFS
raidz on slices 7.

I first created a 10 GB file on each filesystem with mkfile.  I then ran this to
get a baseline:

ZFS:

cs86:/export/home# /bin/time sh -c "cat 10g.txt | cat > /dev/null"

real    10:37.2
user        8.9
sys     12:21.0
cs86:/export/home# 

UFS:

cs86:/mnt# /bin/time sh -c "cat 10g.txt | cat > /dev/null"

real     3:04.5
user        5.5
sys      3:48.9
cs86:/mnt# 

As expected, ZFS performed much better than UFS with the write load, but
I''m still curious why there''s such a large discrepancy in the
unloaded read performance.  Is it just the case that ZFS is by its nature more
optimized for writes and that UFS has had years of tuning to get good read
performance?

Thanks,
Chad
This message posted from opensolaris.org

To reply to my own question:

I note this from Bill Moore in the thread "Old-style sequential read
performance":
> To answer your question, the real advantage of ZFS is that a random
> write workload becomes sequential writes on disk (really fast). As you
> point out, a sequntial read then becomes random reads. The thing to
> note, however, is that when you''re doing a sequntial access to a
file,
> we can easily predict the next N blocks that are accessed and send all
> the I/O requests down to our scheduler to pull them off disk in an
> efficient and non-random fashion. So it''s not as bad as you think.
But the test that I ran was about the best it gets for sequential access to a
file, so I''m still a little bewildered by the disparity.

Thanks,
Chad
This message posted from opensolaris.org

And again I''ll reply to my own thread.  In another thread, Jonathan
Adams suggests running this:

cs86:/export/home# /usr/ccs/bin/what /usr/sbin/zfs
/usr/sbin/zfs:
        SunOS 5.11 opensol-20051116 October 2007
        SunOS Internal Development:  stevel 2005-11-11 [tonic-split.27]
cs86:/export/home# 

So, given that I''m running DEBUG bits, performance comparisons
won''t be accurate.

Thanks,
Chad
This message posted from opensolaris.org

>And again I''ll reply to my own thread.  In another thread, Jonathan
Adams suggests running this:
>
>cs86:/export/home# /usr/ccs/bin/what /usr/sbin/zfs
>/usr/sbin/zfs:
>        SunOS 5.11 opensol-20051116 October 2007
>        SunOS Internal Development:  stevel 2005-11-11 [tonic-split.27]
>cs86:/export/home# 
>

Contrast, the non-debug (Express) bits look like:

/usr/sbin/zfs:
        SunOS 5.11 snv_27 October 2007

Casper

I worked with 2G files (no raidz) and did not get that big a
delta : 36s(UFS) vs 44s(ZFS).

A  thing  to note here is  that  cat(1) mmaps the  file then
fault''s it in. UFS  does implement a faultahead scheme which
worked with 1M blocks  on my system and  kept the  process busy.

For ZFS I noticed there are a lot of  calls to zio_wait() in
the context  of the cat(1)  command which could  account for
the performance  delta I got.  I''d be interested to see what
you  get with  a  straight pool.  Also  of interest   is the
performance   of  going   through write(2)   as   apposed to
pagefaulting-in the file.

-r



Chad Mynhier writes:
 > I ran a simple test to compare read performance of ZFS and UFS while
 > there''s a simultaneous heavy write load.  (I wanted to
demonstrate
 > that the I/O reordering works.)  I first took a baseline without the
 > write load, and UFS performed considerably better than ZFS.  Is there
 > a good explanation for this, other than that ZFS tuning is still in
 > progress?  Or that the way I''ve created the test file is
essentially
 > optimal for UFS but sub-optimal for ZFS? 
 > 
 > The server has three SCSI drives on which I''ve created
equivalently
 > sized slices 6 and 7.  I''ve set up UFS/SVM RAID 5 on slices 6 and
ZFS
 > raidz on slices 7. 
 > 
 > I first created a 10 GB file on each filesystem with mkfile.  I then
 > ran this to get a baseline: 
 > 
 > ZFS:
 > 
 > cs86:/export/home# /bin/time sh -c "cat 10g.txt | cat >
/dev/null"
 > 
 > real    10:37.2
 > user        8.9
 > sys     12:21.0
 > cs86:/export/home# 
 > 
 > UFS:
 > 
 > cs86:/mnt# /bin/time sh -c "cat 10g.txt | cat > /dev/null"
 > 
 > real     3:04.5
 > user        5.5
 > sys      3:48.9
 > cs86:/mnt# 
 > 
 > As expected, ZFS performed much better than UFS with the write load,
 > but I''m still curious why there''s such a large
discrepancy in the
 > unloaded read performance.  Is it just the case that ZFS is by its
 > nature more optimized for writes and that UFS has had years of tuning
 > to get good read performance? 
 > 
 > Thanks,
 > Chad
 > This message posted from opensolaris.org
 > _______________________________________________
 > zfs-discuss mailing list
 > zfs-discuss at opensolaris.org
 > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

To give some closure to this anecdote:

Here are the numbers using the non-DEBUG bits using cat:

UFS:

real     2:49.4
user        3.4
sys        16.1

ZFS:

real     4:37.4
user        8.3
sys      4:54.2


Still not ideal, but much better than the 3:1 ratio I was seeing before.  And to
avoid the behavior Roch points out with cat, here''s an example using
digest:

UFS:

# /bin/time digest -a md5 10g.txt
2dd26c4d4799ebd29fa31e48d49e8e53

real     2:52.6
user       34.9
sys      1:05.0
#

ZFS:

# /bin/time digest -a md5 10g.txt
2dd26c4d4799ebd29fa31e48d49e8e53

real     1:38.5
user       36.4
sys        49.0
# 


This is the kind of performance I was hoping to see.

Thanks to Casper and Roch for the responses.

Chad Mynhier
This message posted from opensolaris.org

>Still not ideal, but much better than the 3:1 ratio I was seeing before.
>And to avoid the behavior Roch points out with cat, here''s an
example using digest:

Looks like we need to either change our programs not to use page in
on ZFS (which was only a marginal improvement before anyway)

Casper