zfs discuss - Oct 2008 - SXCE snv_97 ZFS lots of filesystems test

I did some testing of a couple of x4500 servers to see how well they scaled
in terms of number of filesystems. Both were running snv_97 with the
following config:

# zpool create -f export mirror c0t0d0 c1t0d0 mirror c4t0d0 c6t0d0 mirror
c0t1d0 c1t1d0 mirror c4t1d0 c5t1d0 mirror c6t1d0 c7t1d0 mirror c0t2d0
c1t2d0 mirror c4t2d0 c5t2d0 mirror c6t2d0 c7t2d0 mirror c0t3d0 c1t3d0
mirror c4t3d0 c5t3d0 mirror c6t3d0 c7t3d0 mirror c0t4d0 c1t4d0 mirror
c4t4d0 c6t4d0 mirror c0t5d0 c1t5d0 mirror c4t5d0 c5t5d0 mirror c6t5d0
c7t5d0 mirror c0t6d0 c1t6d0 mirror c4t6d0 c5t6d0 mirror c6t6d0 c7t6d0
mirror c0t7d0 c1t7d0 mirror c4t7d0 c5t7d0 mirror c6t7d0 c7t7d0 spare c7t0d0
c7t4d0

# zfs set devices=off export
# zfs set setuid=off export
# zfs set aclmode=passthrough export
# zfs set aclinherit=passthrough export
# zfs create export/user

On only one of them I turned on NFS:

# zfs set sharenfs=nosuid,sec=krb5i:krb5p export

On each of them I then created 6000 filesystems, 1000 at a time, measuring
how long it took to create each batch of 1000, and how long the reboot
cycle was for increasing filesystem counts.

No NFS:

#fs,create time,reboot time
1000,369,471
2000,1063,268
3000,2340,318
4000,4622,570
5000,6918,832
6000,9416,960

NFS:

#fs,create time,reboot time
1000,670,492
2000,2981,446
3000,7556,934
4000,15423,1406
5000,25222,2035
6000,38322,2899

Performance started out great, taking less than .4 seconds on average per
filesystem, and less than .7 when shared, with a reboot cycle of only 7-8
minutes.

Once 6000 filesystems were created, things were a *lot* slower, taking
almost 10 seconds to create a new filesystem, and almost 40 seconds if
sharing was enabled. I could live with a 16 minute boot cycle, but with
NFS enabled it was over 40 minutes.

I ran a similar test way back under U4:

#FS     create/avg
1000     589/.59
2000     709/.71
3000     783/.78
4000     836/.84
5000     968/.97
6000     930/.93
7000     961/.96
8000     1045/1.05
9000     1098/1.10
10000    1165/1.17

Creation time was actually considerably better under U4 than snv_97.

I was playing with SXCE to get a feel for the soon to be released U6.
Performance wise, I''m hoping U6 will be better, hopefully some new code
in
SXCE was introduced that hasn''t quite been optimized yet... Last date I
heard was Nov 10, if I''m lucky I''ll be able to start testing
with actual U6
soon.


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768

Paul B. Henson wrote:
> I was playing with SXCE to get a feel for the soon to be released U6.
> Performance wise, I''m hoping U6 will be better, hopefully some new
code in
> SXCE was introduced that hasn''t quite been optimized yet... Last
date I
> heard was Nov 10, if I''m lucky I''ll be able to start
testing with actual U6
> soon.
>
>   
Solaris 10 update 6 is based on NV b88.
 -- richard