zfs discuss - May 2007 - Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.

Hi.

I''m all set for doing performance comparsion between Solaris/ZFS and
FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I
think I''m ready. The machine is 1xQuad-core DELL PowerEdge 1950, 2GB
RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the
links to disks are the bottleneck, so I''m going to use not more than 4
disks, probably.

I do know how to tune FreeBSD properly, but I don''t know much about
Solaris tunning. I just upgraded Solaris to:

	SunOS lab14.wheel.pl 5.11 opensol-20070521 i86pc i386 i86pc

I took upgrades from:

	http://dlc.sun.com/osol/on/downloads/current/

I believe this is a version with some debugging options turned on. How
can I turn debug off? Can I or do I need to install something else?
What other tunnings should I apply?

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070523/ce5f8dae/attachment.bin>

Pawel Jakub Dawidek wrote:
> Hi.
> 
> I''m all set for doing performance comparsion between Solaris/ZFS
and
> FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I
> think I''m ready. The machine is 1xQuad-core DELL PowerEdge 1950,
2GB
> RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the
> links to disks are the bottleneck, so I''m going to use not more
than 4
> disks, probably.
> 
> I do know how to tune FreeBSD properly, but I don''t know much
about
> Solaris tunning. I just upgraded Solaris to:
> 
> 	SunOS lab14.wheel.pl 5.11 opensol-20070521 i86pc i386 i86pc
> 
> I took upgrades from:
> 
> 	http://dlc.sun.com/osol/on/downloads/current/
> 
> I believe this is a version with some debugging options turned on. How
> can I turn debug off? Can I or do I need to install something else?
> What other tunnings should I apply?
Don''t install from bfu archives instead install a Solaris Express 
directly from a DVD image.

Or if you do want to use bfu because you really want to match your 
source code revisions up to a given day then you will need to build the 
ON consolidation yourself and you an the install the non debug bfu 
archives (note you will need to download the non debug closed bins to do 
that).

Easiest way is to just use a DVD install.

-- 
Darren J Moffat

> Or if you do want to use bfu because you really want to match your
> source code revisions up to a given day then you will need to build the
> ON consolidation yourself and you an the install the non debug bfu
> archives (note you will need to download the non debug closed bins to do
> that).
The README.opensolaris
(http://dlc.sun.com/osol/on/downloads/current/README.opensolaris)
still states:
2. Non-DEBUG kernel builds have not been tested.  Systems that require
   the ata driver are known not to work with non-DEBUG builds.

Are debug builds now know to work?

James

On Thu, May 24, 2007 at 11:20:44AM +0100, Darren J Moffat
wrote:
> Pawel Jakub Dawidek wrote:
> >Hi.
> >I''m all set for doing performance comparsion between
Solaris/ZFS and
> >FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I
> >think I''m ready. The machine is 1xQuad-core DELL PowerEdge
1950, 2GB
> >RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the
> >links to disks are the bottleneck, so I''m going to use not
more than 4
> >disks, probably.
> >I do know how to tune FreeBSD properly, but I don''t know much
about
> >Solaris tunning. I just upgraded Solaris to:
> >	SunOS lab14.wheel.pl 5.11 opensol-20070521 i86pc i386 i86pc
> >I took upgrades from:
> >	http://dlc.sun.com/osol/on/downloads/current/
> >I believe this is a version with some debugging options turned on. How
> >can I turn debug off? Can I or do I need to install something else?
> >What other tunnings should I apply?
> 
> Don''t install from bfu archives instead install a Solaris Express
directly from a DVD image.
> 
> Or if you do want to use bfu because you really want to match your source
code revisions up to a given day then you will need to build the ON
consolidation yourself and you
> an the install the non debug bfu archives (note you will need to download
the non debug closed bins to do that).
> 
> Easiest way is to just use a DVD install.
Ha, I originally installed from sol-nv-b55b-x86-dvd-iso-[a-e].zip, but
then upgraded to OpenSolaris.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070524/68467063/attachment.bin>

> I''m all set for doing performance comparsion between Solaris/ZFS
and
> FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I
> think I''m ready. The machine is 1xQuad-core DELL PowerEdge 1950,
2GB
> RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the
> links to disks are the bottleneck, so I''m going to use not more
than 4
> disks, probably.
>
> I do know how to tune FreeBSD properly, but I don''t know much
about
> Solaris tunning. I just upgraded Solaris to:
I have just (re)installed FreeBSD amd64 current with gcc 4.2 with src
from May. 21''st on a dual Dell PE 2850.  Does the post-gcc-4-2 current
include all your zfs-optimizations?

I have commented out INVARIANTS, INVARIANTS_SUPPORT, WITNESS and
WITNESS_SKIPSPIN in my kernel and recompiled with CPUTYPE=nocona.

A few weeks ago I installed FreeBSD but it panicked when I used
iozone. So I installed solaris 10 on this box and wanted to keep it
that way. But solaris lacks FreeBSD ports ;-) so when current upgraded
gcc to 4.2 I re-installed FreeBSD and the box is so far very stable.

I have imported a 3.9 GB compressed postgresql dump five times to tune
io-performance, have copied 66 GB of data from another server using
nfs, installed 117 packages from the ports-collection and it''s *very*
stable.

A default install solaris fares better io-wise compared to a default
FreeBSD where writes could pass 100 MB/s (zpool iostat 1) and FreeBSD
would write 30-40 MB/s. After adding the following to
/boot/loader.conf writes peak at 90-95 MB/s:

vm.kmem_size_max=2147483648
vfs.zfs.arc_max=1610612736

Now FreeBSD seems to perfom almost as good as solaris io-wise although
I don''t have any numbers to justify my statement. I did not import
postgresql in solaris as one thing.

Copying the 3.9 GB dump from $HOME to a subdir takes 1 min. 13 secs.
which is approx. 55 MB/s. Reads peaked at 115 MB/s.

The storage is a atabeast with two raid-controllers connected via two
qlogic 2300 hba''s. Each controller have four raid5-arrays with five
400 GB disks each.

zetta~#>zpool status
  pool: disk1
 state: ONLINE
 scrub: scrub completed with 0 errors on Thu May 24 21:39:46 2007
config:

        NAME        STATE     READ WRITE CKSUM
        disk1       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0

errors: No known data errors


The atabeast is not the fastest storage-provider around but on this
machine will primarily be a file- and mail-server.

Are there any other tunables on FreeBSD I can look at?

-- 
regards
Claus

When lenity and cruelty play for a kingdom,
the gentlest gamester is the soonest winner.

Shakespeare

On Thu, May 24, 2007 at 01:16:32PM +0200, Claus Guttesen
wrote:
> >I''m all set for doing performance comparsion between
Solaris/ZFS and
> >FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I
> >think I''m ready. The machine is 1xQuad-core DELL PowerEdge
1950, 2GB
> >RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the
> >links to disks are the bottleneck, so I''m going to use not
more than 4
> >disks, probably.
> >
> >I do know how to tune FreeBSD properly, but I don''t know much
about
> >Solaris tunning. I just upgraded Solaris to:
> 
> I have just (re)installed FreeBSD amd64 current with gcc 4.2 with src
> from May. 21''st on a dual Dell PE 2850.  Does the post-gcc-4-2
current
> include all your zfs-optimizations?
> 
> I have commented out INVARIANTS, INVARIANTS_SUPPORT, WITNESS and
> WITNESS_SKIPSPIN in my kernel and recompiled with CPUTYPE=nocona.
> 
> A few weeks ago I installed FreeBSD but it panicked when I used
> iozone. So I installed solaris 10 on this box and wanted to keep it
> that way. But solaris lacks FreeBSD ports ;-) so when current upgraded
> gcc to 4.2 I re-installed FreeBSD and the box is so far very stable.
> 
> I have imported a 3.9 GB compressed postgresql dump five times to tune
> io-performance, have copied 66 GB of data from another server using
> nfs, installed 117 packages from the ports-collection and it''s
*very*
> stable.
> 
> A default install solaris fares better io-wise compared to a default
> FreeBSD where writes could pass 100 MB/s (zpool iostat 1) and FreeBSD
> would write 30-40 MB/s. After adding the following to
> /boot/loader.conf writes peak at 90-95 MB/s:
> 
> vm.kmem_size_max=2147483648
> vfs.zfs.arc_max=1610612736
> 
> Now FreeBSD seems to perfom almost as good as solaris io-wise although
> I don''t have any numbers to justify my statement. I did not import
> postgresql in solaris as one thing.
> 
> Copying the 3.9 GB dump from $HOME to a subdir takes 1 min. 13 secs.
> which is approx. 55 MB/s. Reads peaked at 115 MB/s.
> 
> The storage is a atabeast with two raid-controllers connected via two
> qlogic 2300 hba''s. Each controller have four raid5-arrays with
five
> 400 GB disks each.
> 
> zetta~#>zpool status
>  pool: disk1
> state: ONLINE
> scrub: scrub completed with 0 errors on Thu May 24 21:39:46 2007
> config:
> 
>        NAME        STATE     READ WRITE CKSUM
>        disk1       ONLINE       0     0     0
>          raidz1    ONLINE       0     0     0
>            da0     ONLINE       0     0     0
>            da1     ONLINE       0     0     0
>            da4     ONLINE       0     0     0
>            da5     ONLINE       0     0     0
>          raidz1    ONLINE       0     0     0
>            da2     ONLINE       0     0     0
>            da3     ONLINE       0     0     0
>            da6     ONLINE       0     0     0
>            da7     ONLINE       0     0     0
> 
> errors: No known data errors
> 
> 
> The atabeast is not the fastest storage-provider around but on this
> machine will primarily be a file- and mail-server.
> 
> Are there any other tunables on FreeBSD I can look at?
There is probably not much you can do to tune sequential I/Os. I''d
suggest starting investigation from benchmarking drivers on both
systems, by using raw disks (without ZFS).
There are some other things you could try to improve different
workloads.

To improve concurrency you should use shared locks for VFS lookups:

	# sysctl vfs.lookup_shared=1

This patch also improve concurrency in VFS:

	http://people.freebsd.org/~pjd/patches/vfs_shared.patch

When you want to operate on mmap(2)ed files, you should disable ZIL and
remote file systems:

	# sysctl vfs.zfs.zil_disable=1
	# zpool export <name>
	# zpool import <name>

I think ZIL should be dataset property, as differences depending on the
workload are huge. For example fsx test is like 15 _times_ faster when
ZIL is disabled.

There are still some things to optimize, like using UMA for memory
allocations, but we run out of KVA too fast then.

Benchmarking file system is not easy, as there are other subsystems
involved, like namecache or VM. fsstress test, which mostly operates on
metadata (creates, removes files, directories, renames them, etc.) is 3
times faster on FreeBSD/ZFS than on Solaris/ZFS, but I believe it''s
mostly because of namecache implementation. Solaris guys should
seriously look at improving DNLC or replacing it. Another possibility is
VFS, but Solaris VFS is much cleaner, and I somehow don''t believe
it''s
slower. fsx is about 20% faster on FreeBSD, this could be VM''s fault.
Don''t take this numbers too seriously - those were only first tries to
see where my port is and I was using OpenSolaris for comparsion, which
has debugging turned on.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070524/0b4e7e19/attachment.bin>

On Thu, May 24, 2007 at 01:16:32PM +0200, Claus Guttesen
wrote:
> 
> iozone. So I installed solaris 10 on this box and wanted to keep it
> that way. But solaris lacks FreeBSD ports ;-) so when current upgraded
Not entirely.  :)

I don''t know about FreeBSD PORTS, but NetBSD''s ports system
works very
well on solaris.  The only thing I didn''t like about it is it considers
gcc a dependency to certain things, so even though I have Studio 11
installed, it would insist on installing gcc, which kinda irritated me. :)

-brian
-- 
"Perl can be fast and elegant as much as J2EE can be fast and elegant.
In the hands of a skilled artisan, it can and does happen; it''s just
that most of the shit out there is built by people who''d be better
suited to making sure that my burger is cooked thoroughly."  -- Jonathan
Patschke

> > iozone. So I installed solaris 10 on this box and wanted to keep it
> > that way. But solaris lacks FreeBSD ports ;-) so when current upgraded
>
> Not entirely.  :)
>
> I don''t know about FreeBSD PORTS, but NetBSD''s ports
system works very
> well on solaris.  The only thing I didn''t like about it is it
considers
> gcc a dependency to certain things, so even though I have Studio 11
> installed, it would insist on installing gcc, which kinda irritated me. :)
I actually tried pkgsrc 6-8 months ago but had problems building ruby
1.8.4 and juggled between sun studio and gcc and never had it compile
completely. And I wanted zfs.

Now I get zfs with a package-system I very comfortable with :-)

-- 
regards
Claus

When lenity and cruelty play for a kingdom,
the gentlest gamester is the soonest winner.

Shakespeare

On 24/05/07, Brian Hechinger <wonko at 4amlunch.net> wrote:
> I don''t know about FreeBSD PORTS, but NetBSD''s ports
system works very
> well on solaris.  The only thing I didn''t like about it is it
considers
> gcc a dependency to certain things, so even though I have Studio 11
> installed, it would insist on installing gcc, which kinda irritated me. :)
pkgsrc is ok, but I found it got very messy very quickly once you get
a reasonably
sized dependency tree. portugrade. FreeBSD portupgrade (a layer on top of ports,
but easy to install) handled that much better.

To be fair I haven''t used NetBSD in a couple of years - but pkgsrc was
the reason
I left :)

-- 
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

> Don''t take this numbers too seriously - those were only first
tries to
> see where my port is and I was using OpenSolaris for comparsion, which
> has debugging turned on.
>
Yeah, ZFS does a lot of extra work with debugging on (such as  
verifying checksums in the ARC), so always do serious performance  
comparisons with non-debug bits.

Nice to see ZFS performing quite well on FreeBSD!

eric

> > I have just (re)installed FreeBSD amd64 current with gcc 4.2 with src
> > from May. 21''st on a dual Dell PE 2850.  Does the
post-gcc-4-2 current
> > include all your zfs-optimizations?
> >
> > I have commented out INVARIANTS, INVARIANTS_SUPPORT, WITNESS and
> > WITNESS_SKIPSPIN in my kernel and recompiled with CPUTYPE=nocona.
> >
> > A default install solaris fares better io-wise compared to a default
> > FreeBSD where writes could pass 100 MB/s (zpool iostat 1) and FreeBSD
> > would write 30-40 MB/s. After adding the following to
> > /boot/loader.conf writes peak at 90-95 MB/s:
> >
> > vm.kmem_size_max=2147483648
> > vfs.zfs.arc_max=1610612736
> >
> > Now FreeBSD seems to perfom almost as good as solaris io-wise although
> > I don''t have any numbers to justify my statement. I did not
import
> > postgresql in solaris as one thing.
> >
> This patch also improve concurrency in VFS:
>
>         http://people.freebsd.org/~pjd/patches/vfs_shared.patch
I applied the patch and it seems to speed up my reads and writes.
Watching zpool iostat I saw reads at 155 MB/s and writes at 111 MB/s.
But it also seems to introduce some minor complete stops accessing the
zpool lasting for 10-20 secs. I apologize I''m not very specific but I
only had time to test disk-io but not digg into the issues.
> When you want to operate on mmap(2)ed files, you should disable ZIL and
> remote file systems:
>
>         # sysctl vfs.zfs.zil_disable=1
>         # zpool export <name>
>         # zpool import <name>
Won''t disabling ZIL minimize the chance of a consistent zfs-filesystem
if - for some reason - the server did an unplanned reboot?

-- 
regards
Claus

When lenity and cruelty play for a kingdom,
the gentlest gamester is the soonest winner.

Shakespeare

On Fri, May 25, 2007 at 09:41:05AM +0200, Claus Guttesen
wrote:
> >> I have just (re)installed FreeBSD amd64 current with gcc 4.2 with
src
> >> from May. 21''st on a dual Dell PE 2850.  Does the
post-gcc-4-2 current
> >> include all your zfs-optimizations?
> >>
> >> I have commented out INVARIANTS, INVARIANTS_SUPPORT, WITNESS and
> >> WITNESS_SKIPSPIN in my kernel and recompiled with CPUTYPE=nocona.
> >>
> >> A default install solaris fares better io-wise compared to a
default
> >> FreeBSD where writes could pass 100 MB/s (zpool iostat 1) and
FreeBSD
> >> would write 30-40 MB/s. After adding the following to
> >> /boot/loader.conf writes peak at 90-95 MB/s:
> >>
> >> vm.kmem_size_max=2147483648
> >> vfs.zfs.arc_max=1610612736
> >>
> >> Now FreeBSD seems to perfom almost as good as solaris io-wise
although
> >> I don''t have any numbers to justify my statement. I did
not import
> >> postgresql in solaris as one thing.
> >>
> >This patch also improve concurrency in VFS:
> >
> >        http://people.freebsd.org/~pjd/patches/vfs_shared.patch
> 
> I applied the patch and it seems to speed up my reads and writes.
> Watching zpool iostat I saw reads at 155 MB/s and writes at 111 MB/s.
> But it also seems to introduce some minor complete stops accessing the
> zpool lasting for 10-20 secs. I apologize I''m not very specific
but I
> only had time to test disk-io but not digg into the issues.
> 
> >When you want to operate on mmap(2)ed files, you should disable ZIL and
> >remote file systems:
> >
> >        # sysctl vfs.zfs.zil_disable=1
> >        # zpool export <name>
> >        # zpool import <name>
> 
> Won''t disabling ZIL minimize the chance of a consistent
zfs-filesystem
> if - for some reason - the server did an unplanned reboot?
ZIL in ZFS is only used to speed-up various workloads, it has nothing to
do with file system consistency. ZFS is always consistent on disk no
matter if you use ZIL or not.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070525/b2ffe7f0/attachment.bin>

>>
>> Won''t disabling ZIL minimize the chance of a consistent zfs- 
>> filesystem
>> if - for some reason - the server did an unplanned reboot?
>
> ZIL in ZFS is only used to speed-up various workloads, it has  
> nothing to
> do with file system consistency. ZFS is always consistent on disk no
> matter if you use ZIL or not.
But it can cause NFS client corruption and you no longer gets  
synchronous write semantics (see if your app depend on that):
http://blogs.sun.com/erickustarz/entry/zil_disable

I highly recommend *against* setting zil_disable.

eric

> >> Won''t disabling ZIL minimize the chance of a consistent
zfs-
> >> filesystem
> >> if - for some reason - the server did an unplanned reboot?
> >
> > ZIL in ZFS is only used to speed-up various workloads, it has
> > nothing to
> > do with file system consistency. ZFS is always consistent on disk no
> > matter if you use ZIL or not.
>
> But it can cause NFS client corruption and you no longer gets
> synchronous write semantics (see if your app depend on that):
> http://blogs.sun.com/erickustarz/entry/zil_disable
>
> I highly recommend *against* setting zil_disable.
Disabling zil did improve the postgresql-import from 1 h. 45 min. to
1. h. 35 min. I get a 10 min. speedup but as the link point out
disabling has it''s disadvantages. So I''ll revert to the old
setting.

-- 
regards
Claus

When lenity and cruelty play for a kingdom,
the gentlest gamester is the soonest winner.

Shakespeare