zfs discuss - Aug 2006 - user quotas vs filesystem quotas?

Hi,

I''m looking at moving two UFS quota-ed filesystems to ZFS under
Solaris 10 release 6/06, and the quota issue is gnarly.

One filesystem is user home directories and I''m aiming towards the
"one zfs filesystem per user" model, attempting to use Casper
Dik''s auto_home script for on-the-fly zfs filesystem creation.
I''m having problems there, but that is an automounter issue, not
ZFS.

The other filesystem is /var/mail on my mail server.  I''ve
traditionally
run (big) user quotas in mailboxes just to keep some malicious
emailer from filling up /var/mail, maybe.   The notion of having
one zfs filesystem per mailbox seems unwieldy, just to run quotas
per user.

Are there any plans/schemes for per-user quotas within a ZFS filesystem,
akin to the UFS quotaon(1M) mechanism?  I take it that quotaon won''t
work with a ZFS filesystem, right?  Suggestions please?  My notion 
right now is to drop quotas for /var/mail.

Jeff Earickson
Colby College

On August 11, 2006 10:31:50 AM -0400 "Jeff A. Earickson" <jaearick
at colby.edu> wrote:
> Suggestions please?
Ideally you''d be able to move to mailboxes in $HOME instead of
/var/mail.

-frank

> The other filesystem is /var/mail on my mail server.
>  I''ve traditionally
> un (big) user quotas in mailboxes just to keep some
> malicious
> emailer from filling up /var/mail, maybe.   The
> notion of having
> one zfs filesystem per mailbox seems unwieldy, just
> to run quotas
> per user.
It shouldn''t be that bad really though. Especially since you can nest
the filesystems. So, I believe you should be able to do something like:

zfs create home/username/mail
zfs set mountpoint=/var/mail/username

Perhaps I''m just naieve, but other than the other poster''s
suggestion to switch mail software, immediate solutions don''t come to
mind.

With zfs new filesystems are so cheap it is unbelievable. If managing so many
filesystems seems unwieldly, then please, by all means post about it in detail!
Improving the management and user experience is something SUN (as well as the
community) appears to be very interested in.
>From everything I''ve read, because of the "cheapness" of
filesystems, there doesn''t seem to be any interest within SUN to
support traditional quotas on ZFS.
-Shawn
 
 
This message posted from opensolaris.org

Jeff A. Earickson wrote:
> Are there any plans/schemes for per-user quotas within a ZFS filesystem,
> akin to the UFS quotaon(1M) mechanism?  I take it that quotaon
won''t
> work with a ZFS filesystem, right?  Suggestions please?  My notion right 
> now is to drop quotas for /var/mail.
An alternative might be to switch to using mail server software that 
does this internally.  For example the Messaging Server software that 
Sun produces has this functionality.

Having the mail quota in the mail server allows you to do much more 
interesting things, especially if you have single instance storage for 
attachments supported by your mail server.  It also often gives you auto 
mail expiry and clean up on the server side.

-- 
Darren J Moffat

One problem with this approach is that software expects /var/mail to be full of
files, not directories, for each user.  I don''t think you can get the
right semantics out of ZFS for this yet (loopback mounting a file comes to mind,
but breaks down if something tries to delete the user''s mailbox when it
goes empty).
 
 
This message posted from opensolaris.org

Anton B. Rang wrote:
> One problem with this approach is that software expects /var/mail to be
full of files,
> not directories, for each user.  I don''t think you can get the
right semantics out of
> ZFS for this yet (loopback mounting a file comes to mind, but breaks down
if something
> tries to delete the user''s mailbox when it goes empty).
This can be configured with the local mail delivery agent.  You could even
put incoming mail in someone''s $HOME, however that isn''t
always a good idea.
In other words, by default, /usr/bin/mail delivers to /var/mail/$USER
  -- richard

On 8/15/06, Richard Elling - PAE <Richard.Elling at sun.com> wrote:
> This can be configured with the local mail delivery agent.  You could even
> put incoming mail in someone''s $HOME, however that isn''t
always a good idea.
I''ve run across systems that don''t deliver into $HOME, and it
always
ends up leading to trouble (running out of disk).  Why do people set
systems up that way?
-- 
David Dyer-Bennet, <mailto:dd-b at dd-b.net>,
<http://www.dd-b.net/dd-b/>
RKBA: <http://www.dd-b.net/carry/>
Pics: <http://www.dd-b.net/dd-b/SnapshotAlbum/>
Dragaera/Steven Brust: <http://dragaera.info/>

On August 15, 2006 12:03:17 PM -0500 David Dyer-Bennet <dd-b at dd-b.net>
wrote:
> On 8/15/06, Richard Elling - PAE <Richard.Elling at sun.com> wrote:
>
>> This can be configured with the local mail delivery agent.  You could
even
>> put incoming mail in someone''s $HOME, however that
isn''t always a good idea.
>
> I''ve run across systems that don''t deliver into $HOME,
and it always
> ends up leading to trouble (running out of disk).  Why do people set
> systems up that way?
It''s the default on many systems (e.g. Solaris), or mail was setup a
long
time ago.

-frank

Delivering into $HOME raises some new failure modes if the home directory
servers are NFS mounted, but otherwise often works OK. However, in some cases
it''s simply impossible--for instance, in a secure NFS environment where
the home directory can''t be mounted without a Kerberos ticket.

I think the main reason systems are set up to deliver into /var/mail, though, is
simply because that''s how they ship by default. Few customers seem
motivated to change the defaults unless they''re convinced of a very
good reason for it.
 
 
This message posted from opensolaris.org

On 8/15/06, David Dyer-Bennet <dd-b at dd-b.net>
wrote:
> On 8/15/06, Richard Elling - PAE <Richard.Elling at sun.com> wrote:
>
> > This can be configured with the local mail delivery agent.  You could
even
> > put incoming mail in someone''s $HOME, however that
isn''t always a good idea.
>
> I''ve run across systems that don''t deliver into $HOME,
and it always
> ends up leading to trouble (running out of disk).  Why do people set
> systems up that way?
Because it leads to LESS trouble than putting it in $HOME. $HOME could
be on separate servers for diffrent user groups and/or it could be on
a server(s) that is less stable than the mailhost.

-- 
Peter Bortas

On 8/15/06, Anton B. Rang <Anton.Rang at sun.com>
wrote:
> Delivering into $HOME raises some new failure modes if the home directory
servers are NFS mounted, but otherwise often works OK. However, in some cases
it''s simply impossible--for instance, in a secure NFS environment where
the home directory can''t be mounted without a Kerberos ticket.
Oy, Kerberos.  Never heard of any place that actually *uses* it, and
it sure has caused lots of work over the years.  So does this mean
that in some sense the mail directory is less well protected than the
home directory in that situation?

Right, I forgot people might still deliver into something other than a
Maildir (which is NFS-safe).  I suspect that was an important issue
for a lot of sites.
> I think the main reason systems are set up to deliver into /var/mail,
though, is simply because that''s how they ship by default. Few
customers seem motivated to change the defaults unless they''re
convinced of a very good reason for it.
After the third time one or the other partition filled up while the
other one had lots of space, it seemed like a no-brainer to me.
-- 
David Dyer-Bennet, <mailto:dd-b at dd-b.net>,
<http://www.dd-b.net/dd-b/>
RKBA: <http://www.dd-b.net/carry/>
Pics: <http://www.dd-b.net/dd-b/SnapshotAlbum/>
Dragaera/Steven Brust: <http://dragaera.info/>

On 8/15/06, Peter Bortas <bortas at gmail.com>
wrote:
> On 8/15/06, David Dyer-Bennet <dd-b at dd-b.net> wrote:
> > On 8/15/06, Richard Elling - PAE <Richard.Elling at sun.com>
wrote:
> >
> > > This can be configured with the local mail delivery agent.  You
could even
> > > put incoming mail in someone''s $HOME, however that
isn''t always a good idea.
> >
> > I''ve run across systems that don''t deliver into
$HOME, and it always
> > ends up leading to trouble (running out of disk).  Why do people set
> > systems up that way?
>
> Because it leads to LESS trouble than putting it in $HOME. $HOME could
> be on separate servers for diffrent user groups and/or it could be on
> a server(s) that is less stable than the mailhost.
My experience is very much "more trouble".  For example, having the
mailhost up while the home directories are down just confuses people,
and doens''t provide any useful service.  The server $HOME is on
shouldn''t matter to much of anything.
-- 
David Dyer-Bennet, <mailto:dd-b at dd-b.net>,
<http://www.dd-b.net/dd-b/>
RKBA: <http://www.dd-b.net/carry/>
Pics: <http://www.dd-b.net/dd-b/SnapshotAlbum/>
Dragaera/Steven Brust: <http://dragaera.info/>

Getting way off-topic, but here goes ...

On August 15, 2006 12:33:01 PM -0700 "Anton B. Rang" <Anton.Rang at
Sun.COM> wrote:
> Delivering into $HOME raises some new failure modes if the home directory
servers are NFS
> mounted, but otherwise often works OK. However, in some cases it''s
simply impossible--for
> instance, in a secure NFS environment where the home directory
can''t be mounted without a
> Kerberos ticket.
This can be fairly easily done if Solaris is your NFS server; if you can
put the server that delivers mail on a secure subnet (relative to the
NFS servers) then you can export homedirs using unix auth to that server
(and krb5 everywhere else).  Or you can arrange for the MDA to have a
kerberos ticket.  I''ve done it both ways.

If you are using Netapp you''re more limited.

Now with zfs there is more reason to go with Solaris as an NFS server,
although it''s not a clear win ... yet.

On August 15, 2006 2:37:27 PM -0500 David Dyer-Bennet <dd-b at dd-b.net>
wrote:
> Oy, Kerberos.  Never heard of any place that actually *uses* it, and
Lots of places use it.

-frank

On 8/15/06, Frank Cusack <fcusack at fcusack.com>
wrote:
> On August 15, 2006 2:37:27 PM -0500 David Dyer-Bennet <dd-b at
dd-b.net> wrote:
> > Oy, Kerberos.  Never heard of any place that actually *uses* it, and
>
> Lots of places use it.
Intellectually, I''m sure that''s true -- because that much work
wouldn''t have been spent, and *continue* to be spent, making it work
if nobody cared.  However, my statement is factually true, I don''t
know of any place that actually uses it.

Then again, *huge* amounts of work were spent for years making OSI
networking work, too, so maybe I shouldn''t take that as so meaningful.
-- 
David Dyer-Bennet, <mailto:dd-b at dd-b.net>,
<http://www.dd-b.net/dd-b/>
RKBA: <http://www.dd-b.net/carry/>
Pics: <http://www.dd-b.net/dd-b/SnapshotAlbum/>
Dragaera/Steven Brust: <http://dragaera.info/>

Peter Bortas wrote:
> On 8/15/06, David Dyer-Bennet <dd-b at dd-b.net> wrote:
>> On 8/15/06, Richard Elling - PAE <Richard.Elling at sun.com>
wrote:
>>
>> > This can be configured with the local mail delivery agent.  You 
>> could even
>> > put incoming mail in someone''s $HOME, however that
isn''t always a
>> good idea.
>>
>> I''ve run across systems that don''t deliver into
$HOME, and it always
>> ends up leading to trouble (running out of disk).  Why do people set
>> systems up that way?
> 
> Because it leads to LESS trouble than putting it in $HOME. $HOME could
> be on separate servers for diffrent user groups and/or it could be on
> a server(s) that is less stable than the mailhost.
It is more difficult to put in $HOME, especially in sites where you
have many home directory servers and mail servers.  The trend seems to
be towards centralized mail services, ala gmail.

In any case, I think the answer to "why?" dates back to the late 1970s
or early 1980s.  Perhaps Eric Allman remembers.
  -- richard