zfs discuss - Jun 2007 - ZFS Performance with Thousands of File Systems

I''m investigating the feasibility of migrating from UFS to ZFS for a
mail-store supporting 20K users. I need separate quotas for all of my users,
which forces me to create separate ZFS file systems for each user. I had read
that there were issues with slow boot times with thousands of ZFS file systems,
but it seemed that a fix had been released with Solaris 10 U3.

I just recently setup OpenSolaris b66 on a T2000 Logical Domain (4 VCPU, 4GB
RAM, SAS-backed virtual storage) and created 50K file systems as a test, but
I''ve found boot times to be very slow (> 5 hours) and operations on
ZFS (ex., zfs set) to also be pretty slow. Has there been a regression, or do I
need to set tuning factors (ex., in /etc/system)?

We''re currently have ZFS deployed for a near-line backup system with
tens of millions of files, and it''s working great, so we''d
also like to move some of our other datasets to it.
 
 
This message posted from opensolaris.org

On Jun 29, 2007, at 20:51, Stephen Le wrote:
> I''m investigating the feasibility of migrating from UFS to ZFS for
> a mail-store supporting 20K users. I need separate quotas for all  
> of my users, which forces me to create separate ZFS file systems  
> for each user.
Does each and every user have a different quota, or does each user  
have (say) a 200 MB limit? Or perhaps there are groups of users  
(under grad, staff, grad, faculty)?

David Magda wrote:
> On Jun 29, 2007, at 20:51, Stephen Le wrote:
> 
>> I''m investigating the feasibility of migrating from UFS to ZFS
for a
>> mail-store supporting 20K users. I need separate quotas for all of my 
>> users, which forces me to create separate ZFS file systems for each
user.
> 
> Does each and every user have a different quota, or does each user have 
> (say) a 200 MB limit? Or perhaps there are groups of users (under grad, 
> staff, grad, faculty)?
Excellent question.  The problem with using file system quotas for
a service such as mail store is that you have very little control
over implementation of policies.  The only thing the mail service
knows is that a write a mailbox fails.  This doesn''t give the mail
service much option to do something else with the message.  By
contrast, a mail service which implements quotas, such as the Sun
Java Messaging Server, implements quotas with various policies for
handling the full-quota case.
	http://docs.sun.com/app/docs/doc/819-4428/6n6j4267q?a=view

Here you can implement user-friendly policies such as send the user
a message warning them that they are approaching their quota limit.

I think you will find that managing quotas for services is better
when implemented by the service, rather than the file system.
  -- richard

(accidentally replied off-list via email, posting message here)

We''ve already considered pooling user quotas together, if
that''s what you''re going to suggest. Pooling user quotas would
present a few problems for us, as we have a fair number of users with unique
quotas and the general user quota is pretty small (10 MB), so we''d be
worried about one account maxing out the pooled quota and denying mail for
everyone else on the same file system.
 
 
This message posted from opensolaris.org

> I think you will find that managing quotas for services is better
> when implemented by the service, rather than the file system.
Thanks for the suggestion, Richard, but we''re very happy with our
current mail software, and we''d rather use file system quotas to
control inbox sizes (our mail administrators are a bit hesitant to rely just
upon the mail software).

Is having a few thousand file systems still an issue with ZFS? I was under the
impression this issue was resolved awhile ago
(http://www.opensolaris.org/jive/message.jspa?messageID=68020#68020). If so,
we''ll just stick to UFS for now.

Thanks.
 
 
This message posted from opensolaris.org

On Jun 30, 2007, at 17:08, Richard Elling wrote:
> Excellent question.  The problem with using file system quotas for
> a service such as mail store is that you have very little control
> over implementation of policies.  The only thing the mail service
> knows is that a write a mailbox fails.
FWIW I believe that maildir++ has added support for quotas.

Stephen Le wrote:
>> I think you will find that managing quotas for services is better
>> when implemented by the service, rather than the file system.
> 
> Thanks for the suggestion, Richard, but we''re very happy with our
current mail software, and we''d rather use file system quotas to
control inbox sizes (our mail administrators are a bit hesitant to rely just
upon the mail software).
> 
> Is having a few thousand file systems still an issue with ZFS? I was under
the impression this issue was resolved awhile ago
(http://www.opensolaris.org/jive/message.jspa?messageID=68020#68020). If so,
we''ll just stick to UFS for now.
A few thousand shouldn''t be much of a problem -- just a bit longer
reboot
times.  And we''re working on making it even faster (so that tens and
hundreds
of thousands of filesystems works well too).

--matt