zfs discuss - Oct 2006 - zpool import takes to long with large numbers of file systems

To have user quota with ZFS you have to have a file system per user. However
this leads to a very large number of file systems on a large server.  I
understand there is work already in hand to make sharing a large number of file
systems faster, however even mounting a large number of file systems takes a
very considerable amount of time.

Is this being worked on?

The example I was given was for a system with 30,000 users. So I tried this on a
Thumper runnig  Solaris 10 6/06  first with out the file systems being shared:

root at va64-x4500a-gmp03 # time zpool import zpool1

real    37m32.98s
user    0m2.42s
sys     1m50.40s
root at va64-x4500a-gmp03 # 

I need to investigate where all the time is being lost.

If you share the file systems the time increases even further but as I
understand it that issue is being worked:

root at va64-x4500a-gmp03 # time zpool import zpool1

real    7h6m28.62s
user    14m55.28s
sys     5h58m13.24s
root at va64-x4500a-gmp03 #
 
 
This message posted from opensolaris.org

On Thu, Oct 26, 2006 at 02:38:32PM -0700, Chris Gerhard
wrote:
> To have user quota with ZFS you have to have a file system per user.
> However this leads to a very large number of file systems on a large
> server.  I understand there is work already in hand to make sharing a
> large number of file systems faster, however even mounting a large
> number of file systems takes a very considerable amount of time. 
> 
> Is this being worked on?
> 
> The example I was given was for a system with 30,000 users. So I tried
> this on a Thumper runnig  Solaris 10 6/06  first with out the file
> systems being shared:
> 
> root at va64-x4500a-gmp03 # time zpool import zpool1
> 
> real    37m32.98s
> user    0m2.42s
> sys     1m50.40s
> root at va64-x4500a-gmp03 # 
> 
> I need to investigate where all the time is being lost.
Most of this time is spent traversing the datasets (notice the low
user/sys time, which indicates we''re waiting on I/O).  There is no
prefetching in the DSL, so this can be slow.  See:

6386929 initial "zfs list" is slow
> If you share the file systems the time increases even further but as I
> understand it that issue is being worked:
> 
> root at va64-x4500a-gmp03 # time zpool import zpool1
> 
> real    7h6m28.62s
> user    14m55.28s
> sys     5h58m13.24s
> root at va64-x4500a-gmp03 #
Yes, this is a limitation of the antiquated NFS share subsystem.  This is being
worked on as part of the sharemgr work.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

>> If you share the file systems the time increases even further but as I
>> understand it that issue is being worked:
>>
>> root at va64-x4500a-gmp03 # time zpool import zpool1
>>
>> real 7h6m28.62s
>> user 14m55.28s
>> sys 5h58m13.24s
>> root at va64-x4500a-gmp03 #
>
> Yes, this is a limitation of the antiquated NFS share subsystem. This is
being
> worked on as part of the sharemgr work.
The initial putback of sharemgr won''t help this much but we understand
the issues.
A good part of the time is in the popen() used to run the share command. The
plan
is to use the sharemgr interfaces to avoid the two forks and two execs for every
share being started.

Doug
 
 
This message posted from opensolaris.org

thank you Eric, Doug.

Is there anymore information about the sharemgr project out in opensolaris.org? 
Searching for it just finds this thread.
 
 
This message posted from opensolaris.org

as an alternative,  I thaught this  would be relevant to the
discussion:

	Bug ID: 6478980
	Synopsis: zfs should support automount property

In other words, do we really need to mount 10000 FS in a
snap, or do we just need to system to be up quickly then
mount on demand....

-r


Chris Gerhard writes:
 > thank you Eric, Doug.
 > 
 > Is there anymore information about the sharemgr project out in
opensolaris.org?  Searching for it just finds this thread.
 >  
 >  
 > This message posted from opensolaris.org
 > _______________________________________________
 > zfs-discuss mailing list
 > zfs-discuss at opensolaris.org
 > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

I, too, experienced a long delay while importing a zpool on a second machine. I
do not have any filesystems in the pool. Just the Solaris 10 Operating system,
Emulex 10002DC HBA, and a 4884 LSI array (dual attached).

I don''t have any file systems created but when STMS(mpxio) is enabled I
see

# time zpool import testpool
real 6m41.01s
user 0m.30s
sys 0m0.14s

When I disable STMS(mpxio), the times are much better but still not that great? 

# time zpool import testpool
real 1m15.01s
user 0m.15s
sys 0m0.35s

Are these normal symproms??

Can anyone explain why I too see delays even though I don''t have any
file systems in the zpool?
 
 
This message posted from opensolaris.org

Hi Luke,

Is the 4884 using two or four ports? Also, how many FSs are involved?

Best Regards,
Jason

On 12/6/06, Luke Schwab <lukeschwab at yahoo.com>
wrote:
> I, too, experienced a long delay while importing a zpool on a second
machine. I do not have any filesystems in the pool. Just the Solaris 10
Operating system, Emulex 10002DC HBA, and a 4884 LSI array (dual attached).
>
> I don''t have any file systems created but when STMS(mpxio) is
enabled I see
>
> # time zpool import testpool
> real 6m41.01s
> user 0m.30s
> sys 0m0.14s
>
> When I disable STMS(mpxio), the times are much better but still not that
great?
>
> # time zpool import testpool
> real 1m15.01s
> user 0m.15s
> sys 0m0.35s
>
> Are these normal symproms??
>
> Can anyone explain why I too see delays even though I don''t have
any file systems in the zpool?
>
>
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>