zfs discuss - May 2010 - Problems (bug?) with slow bulk ZFS filesystem creation

Hi,

This thread refers to Solaris 10, but it was suggested that I post it here as
ZFS developers may well be more likely to respond.

http://forums.sun.com/thread.jspa?threadID=5438393&messageID=10986502#10986502

Basically after about ZFS 1000 filesystem creations the creation time slows down
to around 4 seconds, and gets progressively worse.

This is not the case for normal mkdir which creates thousands of directories
very quickly.

I wanted users home directories (60,000 of them) all to be individual ZFS file
systems, but there seems to be a bug/limitation due to the prohibitive creation
time.
-- 
This message posted from opensolaris.org

On 10 May, 2010 - charles sent me these 0,8K bytes:
> Hi,
> 
> This thread refers to Solaris 10, but it was suggested that I post it here
as ZFS developers may well be more likely to respond.
> 
>
http://forums.sun.com/thread.jspa?threadID=5438393&messageID=10986502#10986502
> 
> Basically after about ZFS 1000 filesystem creations the creation time slows
down to around 4 seconds, and gets progressively worse.
> 
> This is not the case for normal mkdir which creates thousands of
directories very quickly.
> 
> I wanted users home directories (60,000 of them) all to be individual ZFS
file systems, but there seems to be a bug/limitation due to the prohibitive
creation time.
If you''re going to share them over nfs, you''ll be looking at
even worse
times.

In my experience, you don''t want to go over 1-2k filesystems due to
various scalability problems, esp if you''re doing NFS as well. It will
be slow to create and slow when (re)booting, but other than that it
might be ok..

Look into the zfs userquota/groupquota instead.. That''s what I did, and
it''s partly because of these issues that the userquota/groupquota got
implemented I guess.

/Tomas
-- 
Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Ume?
`- Sysadmin at {cs,acc}.umu.se

charles wrote:
> 
> Basically after about ZFS 1000 filesystem creations the creation time slows
down to around 4 seconds, and gets progressively worse.
> 
You can speed up the process by initially setting the mountpoint to
''legacy''.
It''s not the creation that takes that much time, it''s mounting
and sharing.
But, as said earlier, before taking it into production you should test if
the reboot time is within limits.
Also I think there is a hard limit of 64k mounts, at least for NFS.

-Arne

Yes, I have recently tried the userquota option, (one ZFS filesystem with 60,000
quotas and 60,000 ordinary ''mkdir'' home directories within),
and this works fine....but you end up with less granularity of snapshots.

It does seem odd that after only 1000 ZFS filesystems there is a slow down. It
does sound like a bug rather than a limitation. I know that the slow boot
problem you mentioned did get resolved in more recent versions of Solaris,
although I cannot test it with 60,000 filesystems, as life is too short to wait
for them to create!
-- 
This message posted from opensolaris.org

----- "charles" <cems2 at cam.ac.uk> skrev:
> Hi,
> 
> This thread refers to Solaris 10, but it was suggested that I post it
> here as ZFS developers may well be more likely to respond.
> 
>
http://forums.sun.com/thread.jspa?threadID=5438393&messageID=10986502#10986502
> 
> Basically after about ZFS 1000 filesystem creations the creation time
> slows down to around 4 seconds, and gets progressively worse.
> 
> This is not the case for normal mkdir which creates thousands of
> directories very quickly.
> 
> I wanted users home directories (60,000 of them) all to be individual
> ZFS file systems, but there seems to be a bug/limitation due to the
> prohibitive creation time.
Is there a chance of you running out of memory here? If ZFS runs out of memory,
it''ll read indicies from disk instead of keeping them in memory,
something that can almost kill a system.

Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

----- "Roy Sigurd Karlsbakk" <roy at karlsbakk.net> skrev:
> ----- "charles" <cems2 at cam.ac.uk> skrev:
> 
> > Hi,
> > 
> > This thread refers to Solaris 10, but it was suggested that I post
> it
> > here as ZFS developers may well be more likely to respond.
> > 
> >
>
http://forums.sun.com/thread.jspa?threadID=5438393&messageID=10986502#10986502
> > 
> > Basically after about ZFS 1000 filesystem creations the creation
> time
> > slows down to around 4 seconds, and gets progressively worse.
> > 
> > This is not the case for normal mkdir which creates thousands of
> > directories very quickly.
> > 
> > I wanted users home directories (60,000 of them) all to be
> individual
> > ZFS file systems, but there seems to be a bug/limitation due to the
> > prohibitive creation time.
> 
> Is there a chance of you running out of memory here? If ZFS runs out
> of memory, it''ll read indicies from disk instead of keeping them
in
> memory, something that can almost kill a system.
Try to monitor the disk utilisation with iostat -xd 2 or something and compare
the numbers with low/high dataset count. If disk usage increases, it''s
likely you''re down on RAM. Adding more RAM or L2ARC might help.

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.