zfs discuss - Dec 2008 - zfs create performance degrades dramatically with increasing number of file systems

I am attempting to create approx 10600 zfs file systems across two
pools.  The devices underlying the pools are mirrored iscsi volumes
shared over a dedicated gigabit Ethernet with jumbo frames enabled
(MTU 9000) from a Linux Openfiler 2.3 system. I have added a couple of
4GByte  zvols from the root pool to use as zil devices for the two
iscsi backed pools.

The solaris system is running snv_101b, has 8 Gbyte RAM and dual 64
bit Xeon processors.

Initially I was able to create zfs file systems at a rate of around 1
every 6 seconds (wall clock time), now three days later I have created
9000 zfs file systems and the creation rate has dropped to approx  1
per minute, an order of magnitude slower.

I attempted an experiment on a system with half the memory and using
looped back zvols and saw similar performance.

I find it hard to believe that such a performance degradation is
expected.  Is there some parameters I should be tuning for using large
numbers of file systems?

Regards, Alastair

Alastair Neil writes:
 > I am attempting to create approx 10600 zfs file systems across two
 > pools.  The devices underlying the pools are mirrored iscsi volumes
 > shared over a dedicated gigabit Ethernet with jumbo frames enabled
 > (MTU 9000) from a Linux Openfiler 2.3 system. I have added a couple of
 > 4GByte  zvols from the root pool to use as zil devices for the two
 > iscsi backed pools.
 > 
 > The solaris system is running snv_101b, has 8 Gbyte RAM and dual 64
 > bit Xeon processors.
 > 
 > Initially I was able to create zfs file systems at a rate of around 1
 > every 6 seconds (wall clock time), 
 > now three days later I have created
 > 9000 zfs file systems and the creation rate has dropped to approx  1
 > per minute, an order of magnitude slower.
 > 
 > I attempted an experiment on a system with half the memory and using
 > looped back zvols and saw similar performance.
 > 
 > I find it hard to believe that such a performance degradation is
 > expected.  Is there some parameters I should be tuning for using large
 > numbers of file systems?
 > 
 > Regards, Alastair
 > _______________________________________________
 > zfs-discuss mailing list
 > zfs-discuss at opensolaris.org
 > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Sounds like 

6763592 : creating zfs filesystems gets slower as the number of zfs filesystems
increase
6572357 : libzfs should do more to avoid mnttab lookups

Which  just integrated in SNV 105.

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6763592
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6572357

-r

On Mon, Jan 5, 2009 at 5:27 AM, Roch <Roch.Bourbonnais at sun.com>
wrote:
> Alastair Neil writes:
>  > I am attempting to create approx 10600 zfs file systems across two
>  > pools.  The devices underlying the pools are mirrored iscsi volumes
>  > shared over a dedicated gigabit Ethernet with jumbo frames enabled
>  > (MTU 9000) from a Linux Openfiler 2.3 system. I have added a couple
of
>  > 4GByte  zvols from the root pool to use as zil devices for the two
>  > iscsi backed pools.
>  >
>  > The solaris system is running snv_101b, has 8 Gbyte RAM and dual 64
>  > bit Xeon processors.
>  >
>  > Initially I was able to create zfs file systems at a rate of around 1
>  > every 6 seconds (wall clock time),
>  > now three days later I have created
>  > 9000 zfs file systems and the creation rate has dropped to approx  1
>  > per minute, an order of magnitude slower.
>  >
>  > I attempted an experiment on a system with half the memory and using
>  > looped back zvols and saw similar performance.
>  >
>  > I find it hard to believe that such a performance degradation is
>  > expected.  Is there some parameters I should be tuning for using
large
>  > numbers of file systems?
>  >
>  > Regards, Alastair
>  > _______________________________________________
>  > zfs-discuss mailing list
>  > zfs-discuss at opensolaris.org
>  > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
>
> Sounds like
>
> 6763592 : creating zfs filesystems gets slower as the number of zfs
filesystems increase
> 6572357 : libzfs should do more to avoid mnttab lookups
>
> Which  just integrated in SNV 105.
>
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6763592
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6572357
>
> -r
>
>
Roch

thanks for the information, I replied to you directly by accident,
sorry about that.  I am curious what is the update process for
opensolaris.  I installed my machines almost the instant 2008.11 was
available and yet so far there have been no released updates - unless
I am doing something wrong.  Will updates from the SNV releases be
periodically rolled into 2008.11?  Am I out of luck with 2008.11 and
have to wait for 2009.04 for these fixes?  I guess I could wait and
upgrade to a Developers edition release?