zfs discuss - Jun 2008 - Filesystem for each home dir - 10,000 users?

Hi All,

I''m new to ZFS but I''m intrigued by the possibilities it
presents.

I''m told one of the greatest benefits is that, instead of setting 
quotas, each user can have their own ''filesystem'' under a
single pool.

This is obviously great if you''ve got 10 users but what if you have 
10,000?  Are the overheads too great and do they outweigh the potential 
benefits?

I''ve got a test system running with 5,000 dummy users which seems to 
perform fine, even if my ''df'' output is a little sluggish :-)
.

Any advice or experiences would be greatly appreciated.

> Hi All,
> 
> I''m new to ZFS but I''m intrigued by the possibilities
> it presents.
> 
> I''m told one of the greatest benefits is that,
> instead of setting 
> quotas, each user can have their own ''filesystem''
> under a single pool.
> 
> This is obviously great if you''ve got 10 users but
> what if you have 
> 10,000?  Are the overheads too great and do they
> outweigh the potential 
> benefits?
> 
> I''ve got a test system running with 5,000 dummy users
> which seems to 
> perform fine, even if my ''df'' output is a little
> sluggish :-) .
> 
> Any advice or experiences would be greatly
> appreciated.
I think sharemgr was created to speed up the case of sharing out very
high numbers of filesystems on NFS servers, which otherwise took
quite a long time.

That''s not to say that there might not be other problems with scaling
to
thousands of filesystems.  But you''re certainly not the first one to
test it.

For cases where a single filesystem must contain files owned by
multiple users (/var/mail being one example), old fashioned
UFS quotas still solve the problem where the alternative approach
with ZFS doesn''t.
 
 
This message posted from opensolaris.org

Richard L. Hamilton wrote:
>> Hi All,
>>
>> I''m new to ZFS but I''m intrigued by the possibilities
>> it presents.
>>
>> I''m told one of the greatest benefits is that,
>> instead of setting 
>> quotas, each user can have their own ''filesystem''
>> under a single pool.
>>
>> This is obviously great if you''ve got 10 users but
>> what if you have 
>> 10,000?  Are the overheads too great and do they
>> outweigh the potential 
>> benefits?
>>
>> I''ve got a test system running with 5,000 dummy users
>> which seems to 
>> perform fine, even if my ''df'' output is a little
>> sluggish :-) .
>>
>> Any advice or experiences would be greatly
>> appreciated.
>>     
>
> I think sharemgr was created to speed up the case of sharing out very
> high numbers of filesystems on NFS servers, which otherwise took
> quite a long time.
>
> That''s not to say that there might not be other problems with
scaling to
> thousands of filesystems.  But you''re certainly not the first one
to test it.
>
> For cases where a single filesystem must contain files owned by
> multiple users (/var/mail being one example), old fashioned
> UFS quotas still solve the problem where the alternative approach
> with ZFS doesn''t.
>  
>   
This seems to come up over and over, and while I haven''t had a need to 
implement it yet I probably will eventually.
I don''t like the 1ZFS/User idea either, and I''ve just been
thinking that
user/group quotas would eventually be available, probably before I 
needed them.

But even with the demand shown by these posts, I don''t see mention of 
anyone working on this.

Is there some technical difference with ZFS that makes a normal 
user/group quota impossible?
Or is there a general feeling that it''s an icky idea and (for some 
reason?) something to be avoided at all costs?
Or is it on the list of things to add, and we just haven''t made it to
it
yet?

I''m ok with the last option. It''s the first 2 that I
don''t like. :) It
just seems that a lot of effort has gone into making performance 
improvements, and new utilities like the ''sharemgr'' that
Richard
mentions, all to work around the lack of user/group quotas?? It''s 
starting to look like it might have just been easier to implement the 
quotas from the beginning.

   -Kyle
>  
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

Kyle McDonald wrote:
> Richard L. Hamilton wrote: 
>   
>> I think sharemgr was created to speed up the case of sharing out very
>> high numbers of filesystems on NFS servers, which otherwise took
>> quite a long time.
>>
>> That''s not to say that there might not be other problems with
scaling to
>> thousands of filesystems.  But you''re certainly not the first
one to test it.
>>
>> For cases where a single filesystem must contain files owned by
>> multiple users (/var/mail being one example), old fashioned
>> UFS quotas still solve the problem where the alternative approach
>> with ZFS doesn''t.
>>  
>>   
>>     
> This seems to come up over and over, and while I haven''t had a
need to
> implement it yet I probably will eventually.
> I don''t like the 1ZFS/User idea either, and I''ve just
been thinking that
> user/group quotas would eventually be available, probably before I 
> needed them.
>
> But even with the demand shown by these posts, I don''t see mention
of
> anyone working on this.
>
> Is there some technical difference with ZFS that makes a normal 
> user/group quota impossible?
> Or is there a general feeling that it''s an icky idea and (for some
> reason?) something to be avoided at all costs?
> Or is it on the list of things to add, and we just haven''t made it
to it
> yet?
>
> I''m ok with the last option. It''s the first 2 that I
don''t like. :) It
> just seems that a lot of effort has gone into making performance 
> improvements, and new utilities like the ''sharemgr'' that
Richard
> mentions, all to work around the lack of user/group quotas?? It''s 
> starting to look like it might have just been easier to implement the 
> quotas from the beginning.
>
>    -Kyle
One thing I think we should seriously consider is rather than just 
demanding ''we want quotas'' (and, I''d like them too),
is think about what
we want.  Good old UFS quotas had some _major_ drawbacks, and I don''t 
think we just want that re-implemented.

Quotas are great when, for administrative purposes, you want a large 
number of users on a single filesystem, but to restrict the amount of 
space for each.  The primary place I can think of this being useful is 
/var/mail  , or for users creating large block files (say each user 
needs a DB back-end file).

The ZFS filesystem approach is actually better than quotas for User and 
Shared directories, since the purpose is to limit the amount of space 
taken up *under that directory tree*.   Quotas do a miserable job with 
Shared directories, and get damned confusing if there is the ability of 
anyone else to write to your User directory (or vice versa).

Remember, that quotas aren''t free, and while we have seen some 
performance problems with the ''10,000 ZFS filesystems''
approach, there
are performance issues to be had when trying to keep track of 10,000 
user quotas on a file system, as well. I can''t say they are equal, but 
don''t think that quotas are just there for the implementing.
There''s a
penalty for them, too.

As far as implementing quotas go, I can''t say how hard it is. But, 
here''s a least something to think about:  where do we count? At the
Pool
level? At the filesystem level?  What about nested filesystems?  How 
hard does it get to count (and manage) nested quotas?

For instance, if I have a nested  ZFS filesystem setup, and assuming a 
simple restriction that the quota on any nested filesystem can be equal 
to or less than its parent,  what about this example:

filesystem      quota
foo               10 mb
foo/bar          6 mb
foo/bar/baz   6 mb
foo/quux       6mb

how do I easily report these quotas (i.e. can I even see all the 
filesystems)?  how confusing is it if I have 6MB in foo/bar/baz, and 
then can''t put more than 4mb in foo/quux?  Does "quota foo"
report all
the nested quotas, also? Should "quota foo/bar/baz" also include the 
quotas of its parents (i.e. foo and foo/bar )?

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

Richard L. Hamilton wrote:
>> Hi All,
>>
>> I''m new to ZFS but I''m intrigued by the possibilities
>> it presents.
>>
>> I''m told one of the greatest benefits is that,
>> instead of setting 
>> quotas, each user can have their own ''filesystem''
>> under a single pool.
>>
>> This is obviously great if you''ve got 10 users but
>> what if you have 
>> 10,000?  Are the overheads too great and do they
>> outweigh the potential 
>> benefits?
>>
>> I''ve got a test system running with 5,000 dummy users
>> which seems to 
>> perform fine, even if my ''df'' output is a little
>> sluggish :-) .
>>
>> Any advice or experiences would be greatly
>> appreciated.
>>     
>
> I think sharemgr was created to speed up the case of sharing out very
> high numbers of filesystems on NFS servers, which otherwise took
> quite a long time.
>
> That''s not to say that there might not be other problems with
scaling to
> thousands of filesystems.  But you''re certainly not the first one
to test it.
>
> For cases where a single filesystem must contain files owned by
> multiple users (/var/mail being one example), old fashioned
> UFS quotas still solve the problem where the alternative approach
> with ZFS doesn''t.
>   
A single /var/mail doesn''t work well for 10,000 users either.  When you
start getting into that scale of service provisioning, you might look at
how the big boys do it... Apple, Verizon, Google, Amazon, etc.  You
should also look at e-mail systems designed to scale to large numbers of 
users
which implement limits without resorting to file system quotas.  Such
e-mail systems actually tell users that their mailbox is too full rather 
than
just failing to deliver mail.  So please, when we start having this 
conversation
again, lets leave /var/mail out.
 -- richard

| The ZFS filesystem approach is actually better than quotas for User
| and Shared directories, since the purpose is to limit the amount of
| space taken up *under that directory tree*.

 Speaking only for myself, I would find ZFS filesystems somewhat more
useful if they were more like directory trees and less like actual
filesystems. Right now, their filesystem nature creates several
limitations:

- you cannot semi-transparently convert an existing directory tree into
  a new ZFS filesystem; you need to move the directory tree aside, make
  a new filesystem, and copy all the data over.
- as separate filesystems, they have to be separately NFS mounted
- you cannot hardlink between separate filesystems, which is a problem
  if you want to use a lot of ZFS filesystems for fine-grained management
  of things like NFS permissions.

	- cks

> A single /var/mail doesn''t work well for 10,000 users either. 
When you
> start getting into that scale of service provisioning, you might look at
> how the big boys do it... Apple, Verizon, Google, Amazon, etc.  You
pantzer at gepetto /var/mail >echo *|wc
       1   20632  185597

pantzer at gepetto /var/mail >/usr/platform/sun4u/sbin/prtdiag
System Configuration:  Sun Microsystems  sun4u Sun Enterprise 220R (2 X UltraSPA
RC-II 450MHz)
System clock frequency: 113 MHz
Memory size: 2048 Megabytes

So, 10,000 mailaccounts on a new server is not a problem. Of course
depending on usage patterns.

On Thu, Jun 05, 2008 at 12:02:42PM -0400, Chris Siebenmann
wrote:
> 
> - as separate filesystems, they have to be separately NFS mounted
I think this is the one that gets under my skin.  If there would be a
way to "merge" a filesystem into a parent filesystem for the purposes
of NFS, that would be simply amazing.  I want to have the fine-grained
control over my NFS server that multiple ZFS filesystems gives me, but
I don''t want the client systems to have to know anything about it.

Maybe a pipe dream?  ;)

-brian
-- 
"Coding in C is like sending a 3 year old to do groceries. You gotta
tell them exactly what you want or you''ll end up with a cupboard full
of
pop tarts and pancake mix." -- IRC User (http://www.bash.org/?841435)

[...]
> > That''s not to say that there might not be other
> problems with scaling to
> > thousands of filesystems.  But you''re certainly not
> the first one to test it.
> >
> > For cases where a single filesystem must contain
> files owned by
> > multiple users (/var/mail being one example), old
> fashioned
> > UFS quotas still solve the problem where the
> alternative approach
> > with ZFS doesn''t.
> >   
> 
> A single /var/mail doesn''t work well for 10,000 users
> either.  When you
> start getting into that scale of service
> provisioning, you might look at
> how the big boys do it... Apple, Verizon, Google,
> Amazon, etc.  You
> should also look at e-mail systems designed to scale
> to large numbers of 
> users
> which implement limits without resorting to file
> system quotas.  Such
> e-mail systems actually tell users that their mailbox
> is too full rather 
> than
> just failing to deliver mail.  So please, when we
> start having this 
> conversation
> again, lets leave /var/mail out.
I''m not recommending such a configuration; I quite agree that it is
neither
scalable nor robust.

It''s only merit is that it''s an obvious example of where one
would have
potentially large files owned by many users necessarily on one filesystem,
inasmuch as they were in one common directory.  But there must  be
other examples where the ufs quota model is a better fit than the
zfs quota model with potentially one filesystem per user.

In terms of the limitations they can provide, zfs filesystem quotas remind me
of DG/UX control point directories (presumably a relic of AOS/VS) - like regular
directories except they could have a quota bound to them restricting the sum of
the space of the subtree rooted there (the native filesystem on DG/UX
didn''t
have UID-based quotas).

Given restricted chown (non-root can''t give files away),
per-UID*filesystem
quotas IMO make just as much sense as per-filesystem quotas themselves
do on zfs, save only that per-UID*filesystem quotas make the filesystem less
lightweight.  For zfs, perhaps an answer might be if it were possible to
have per-zpool uid/gid/projid/zoneid/sid quotas too?
 
 
This message posted from opensolaris.org

Brian Hechinger wrote:
> On Thu, Jun 05, 2008 at 12:02:42PM -0400, Chris Siebenmann wrote:
>> - as separate filesystems, they have to be separately NFS mounted
> 
> I think this is the one that gets under my skin.  If there would be a
> way to "merge" a filesystem into a parent filesystem for the
purposes
> of NFS, that would be simply amazing.  I want to have the fine-grained
> control over my NFS server that multiple ZFS filesystems gives me, but
> I don''t want the client systems to have to know anything about it.
> 
> Maybe a pipe dream?  ;)
As has been pointed out before when this topic came up this is really an 
NFS client or client automounter problem not an NFS server or ZFS problem.

Current Solaris/OpenSolaris NFS clients don''t have any issues with the 
number of NFS filesystems or even traversing the server side mountpoints.

-- 
Darren J Moffat

On Fri, 6 Jun 2008, Brian Hechinger wrote:
> On Thu, Jun 05, 2008 at 12:02:42PM -0400, Chris Siebenmann wrote:
>>
>> - as separate filesystems, they have to be separately NFS mounted
>
> I think this is the one that gets under my skin.  If there would be a
> way to "merge" a filesystem into a parent filesystem for the
purposes
> of NFS, that would be simply amazing.  I want to have the fine-grained
> control over my NFS server that multiple ZFS filesystems gives me, but
> I don''t want the client systems to have to know anything about it.
Solaris 10 clients already do that.  The problem is that non-Solaris 
clients do not.  Without per-filesystem mounts, ''df'' on the
client
will not report correct data though.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Richard L. Hamilton wrote:
>> A single /var/mail doesn''t work well for 10,000 users
>> either.  When you
>> start getting into that scale of service
>> provisioning, you might look at
>> how the big boys do it... Apple, Verizon, Google,
>> Amazon, etc.  You
>> should also look at e-mail systems designed to scale
>> to large numbers of 
>> users
>> which implement limits without resorting to file
>> system quotas.  Such
>> e-mail systems actually tell users that their mailbox
>> is too full rather 
>> than
>> just failing to deliver mail.  So please, when we
>> start having this 
>> conversation
>> again, lets leave /var/mail out.
>>     
>
> I''m not recommending such a configuration; I quite agree that it
is neither
> scalable nor robust.
>   
I was going to post some history of scaling mail, but I blogged it instead.
http://blogs.sun.com/relling/entry/on_var_mail_and_quotas
 -- richard

2008/6/6 Richard Elling <Richard.Elling at
sun.com>:
> Richard L. Hamilton wrote:
>>> A single /var/mail doesn''t work well for 10,000 users
>>> either.  When you
>>> start getting into that scale of service
>>> provisioning, you might look at
>>> how the big boys do it... Apple, Verizon, Google,
>>> Amazon, etc.  You
>>> should also look at e-mail systems designed to scale
>>> to large numbers of
>>> users
>>> which implement limits without resorting to file
>>> system quotas.  Such
>>> e-mail systems actually tell users that their mailbox
>>> is too full rather
>>> than
>>> just failing to deliver mail.  So please, when we
>>> start having this
>>> conversation
>>> again, lets leave /var/mail out.
>>>
>>
>> I''m not recommending such a configuration; I quite agree that
it is neither
>> scalable nor robust.
>>
>
> I was going to post some history of scaling mail, but I blogged it instead.
> http://blogs.sun.com/relling/entry/on_var_mail_and_quotas
>  -- richard
>
The problem with that argument is that 10.000 users on one vxfs or UFS
filesystem is no problem at all, be it /var/mail or home directories.
You don''t even need a fast server for that. 10.000 zfs file systems is
a problem.

So, if it makes you happier, substitute mail with home directories.

On Thu, Jun 5, 2008 at 2:11 PM, Erik Trimble <Erik.Trimble at sun.com>
wrote:
>
> Quotas are great when, for administrative purposes, you want a large
> number of users on a single filesystem, but to restrict the amount of
> space for each.  The primary place I can think of this being useful is
> /var/mail
Not really. Controlling mail usage needs to be done by the mail app -
simply failing a write is a terrible way to implement policy.
> The ZFS filesystem approach is actually better than quotas for User and
> Shared directories, since the purpose is to limit the amount of space
> taken up *under that directory tree*.   Quotas do a miserable job with
> Shared directories, and get damned confusing if there is the ability of
> anyone else to write to your User directory (or vice versa).
Erm, that''s backwards. You want quotas to control shared directories.
In particular, you can''t use ZFS filesystem to control usage in a
single directory (essentially by definition). What we use quotas for there
is to make sure a bad user (or rogue application) is controlled and
can''t
fill up a filesystem, thereby impacting other users.
> Remember, that quotas aren''t free, and while we have seen some
> performance problems with the ''10,000 ZFS filesystems''
approach, there
> are performance issues to be had when trying to keep track of 10,000
> user quotas on a file system, as well. I can''t say they are equal,
but
> don''t think that quotas are just there for the implementing.
There''s a
> penalty for them, too.
But 5-10,000 users with quotas worked just fine on a supersparc based
machine in the last millenium. Even on a decent modern machine that
number of filesystems could best be described as painful. The reality is
that there are something like 3 orders of magnitude difference in cost
between traditional ufs quotas and using zfs filesystems to try and emulate
the same thing.

(Although I have to say that, in a previous job, scrapping user quotas entirely
not only resulted in happier users, much less work for the helpdesk, and -
paradoxically - largely eliminated systems running out of space.)

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

On Fri, Jun 06, 2008 at 10:42:45AM -0500, Bob Friesenhahn
wrote:
> On Fri, 6 Jun 2008, Brian Hechinger wrote:
> 
> > On Thu, Jun 05, 2008 at 12:02:42PM -0400, Chris Siebenmann wrote:
> >>
> >> - as separate filesystems, they have to be separately NFS mounted
> >
> > I think this is the one that gets under my skin.  If there would be a
> > way to "merge" a filesystem into a parent filesystem for the
purposes
> > of NFS, that would be simply amazing.  I want to have the fine-grained
> > control over my NFS server that multiple ZFS filesystems gives me, but
> > I don''t want the client systems to have to know anything
about it.
> 
> Solaris 10 clients already do that.  The problem is that non-Solaris 
> clients do not.  Without per-filesystem mounts, ''df'' on
the client
> will not report correct data though.
I expect that mirror mounts will be coming Linux''s way too.

On Fri, Jun 06, 2008 at 07:37:18AM -0400, Brian Hechinger
wrote:
> On Thu, Jun 05, 2008 at 12:02:42PM -0400, Chris Siebenmann wrote:
> > 
> > - as separate filesystems, they have to be separately NFS mounted
> 
> I think this is the one that gets under my skin.  If there would be a
> way to "merge" a filesystem into a parent filesystem for the
purposes
> of NFS, that would be simply amazing.  I want to have the fine-grained
> control over my NFS server that multiple ZFS filesystems gives me, but
> I don''t want the client systems to have to know anything about it.
> 
> Maybe a pipe dream?  ;)
Mirror mounts take care of the NFS problem (with NFSv4).

NFSv3 automounters could be made more responsive to server-side changes
is share lists, but hey, NFSv4 is the future.

On Fri, Jun 06, 2008 at 08:51:13PM +0200, Mattias Pantzare
wrote:
> 2008/6/6 Richard Elling <Richard.Elling at sun.com>:
> > I was going to post some history of scaling mail, but I blogged it
instead.
> > http://blogs.sun.com/relling/entry/on_var_mail_and_quotas
> 
> The problem with that argument is that 10.000 users on one vxfs or UFS
> filesystem is no problem at all, be it /var/mail or home directories.
> You don''t even need a fast server for that. 10.000 zfs file
systems is
> a problem.
10k one-file filesystems, all with the same parent dataset would be
silly as you''d be spreading the overhead of each dataset over one file
and listing the parent would require listing child datasets, and ...

10k home directories is entirely different.
> So, if it makes you happier, substitute mail with home directories.
I don''t think comparing /var/mail and home directories is comparing
apples to apples.

Also, don''t use /var/mail.  Use IMAP.

On Jun 6, 2008, at 2:50 PM, Nicolas Williams wrote:
> On Fri, Jun 06, 2008 at 10:42:45AM -0500, Bob Friesenhahn wrote:
>> On Fri, 6 Jun 2008, Brian Hechinger wrote:
>>
>>> On Thu, Jun 05, 2008 at 12:02:42PM -0400, Chris Siebenmann wrote:
>>>>
>>>> - as separate filesystems, they have to be separately NFS
mounted
>>>
>>> I think this is the one that gets under my skin.  If there would  
>>> be a
>>> way to "merge" a filesystem into a parent filesystem for
the
>>> purposes
>>> of NFS, that would be simply amazing.  I want to have the fine- 
>>> grained
>>> control over my NFS server that multiple ZFS filesystems gives me,
>>> but
>>> I don''t want the client systems to have to know anything
about it.
>>
>> Solaris 10 clients already do that.  The problem is that non-Solaris
>> clients do not.  Without per-filesystem mounts, ''df''
on the client
>> will not report correct data though.
>
> I expect that mirror mounts will be coming Linux''s way too.
The should already have them:
http://blogs.sun.com/erickustarz/en_US/entry/linux_support_for_mirror_mounts

eric

On Fri, Jun 06, 2008 at 02:58:09PM -0700, eric kustarz
wrote:
> >I expect that mirror mounts will be coming Linux''s way too.
> 
> The should already have them:
>
http://blogs.sun.com/erickustarz/en_US/entry/linux_support_for_mirror_mounts
Even better.

On Fri, Jun 06, 2008 at 02:58:09PM -0700, eric kustarz
wrote:
> 
> >> clients do not.  Without per-filesystem mounts,
''df'' on the client
> >> will not report correct data though.
> >
> > I expect that mirror mounts will be coming Linux''s way too.
> 
> The should already have them:
>
http://blogs.sun.com/erickustarz/en_US/entry/linux_support_for_mirror_mounts
Where does that leave those of us who need to deal with OSX clients?  Does apple
have any plans to get in on this?

-brian
-- 
"Coding in C is like sending a 3 year old to do groceries. You gotta
tell them exactly what you want or you''ll end up with a cupboard full
of
pop tarts and pancake mix." -- IRC User (http://www.bash.org/?841435)

On Fri, Jun 06, 2008 at 04:52:45PM -0500, Nicolas Williams
wrote:
> 
> Mirror mounts take care of the NFS problem (with NFSv4).
> 
> NFSv3 automounters could be made more responsive to server-side changes
> is share lists, but hey, NFSv4 is the future.
So basically it''s just a waiting game at this point?  I guess I can
live
with that.

-brian
-- 
"Coding in C is like sending a 3 year old to do groceries. You gotta
tell them exactly what you want or you''ll end up with a cupboard full
of
pop tarts and pancake mix." -- IRC User (http://www.bash.org/?841435)

On Jun 6, 2008, at 3:27 PM, Brian Hechinger wrote:
> On Fri, Jun 06, 2008 at 02:58:09PM -0700, eric kustarz wrote:
>>
>>>> clients do not.  Without per-filesystem mounts,
''df'' on the client
>>>> will not report correct data though.
>>>
>>> I expect that mirror mounts will be coming Linux''s way
too.
>>
>> The should already have them:
>>
http://blogs.sun.com/erickustarz/en_US/entry/linux_support_for_mirror_mounts
>
> Where does that leave those of us who need to deal with OSX  
> clients?  Does apple
> have any plans to get in on this?
They need to implement NFSv4 in general first :)  But you''d have to  
ask them on their lists what the status of that is... i know i would  
like it...

eric

On Fri, Jun 06, 2008 at 06:27:01PM -0400, Brian Hechinger
wrote:
> On Fri, Jun 06, 2008 at 02:58:09PM -0700, eric kustarz wrote:
> > 
> > >> clients do not.  Without per-filesystem mounts,
''df'' on the client
> > >> will not report correct data though.
> > >
> > > I expect that mirror mounts will be coming Linux''s way
too.
> > 
> > The should already have them:
> >
http://blogs.sun.com/erickustarz/en_US/entry/linux_support_for_mirror_mounts
> 
> Where does that leave those of us who need to deal with OSX clients?  Does
apple
> have any plans to get in on this?
Apple plans on supporting NFSv4... including "mirror mounts" (barring
any
unseen, insurmountable hurdles).

HTH
--macko
Not speaking "officially" for Apple, but just as an engineer who works
on this stuff.

Mattias Pantzare wrote:
> 2008/6/6 Richard Elling <Richard.Elling at sun.com>:
>   
>> Richard L. Hamilton wrote:
>>     
>>>> A single /var/mail doesn''t work well for 10,000 users
>>>> either.  When you
>>>> start getting into that scale of service
>>>> provisioning, you might look at
>>>> how the big boys do it... Apple, Verizon, Google,
>>>> Amazon, etc.  You
>>>> should also look at e-mail systems designed to scale
>>>> to large numbers of
>>>> users
>>>> which implement limits without resorting to file
>>>> system quotas.  Such
>>>> e-mail systems actually tell users that their mailbox
>>>> is too full rather
>>>> than
>>>> just failing to deliver mail.  So please, when we
>>>> start having this
>>>> conversation
>>>> again, lets leave /var/mail out.
>>>>
>>>>         
>>> I''m not recommending such a configuration; I quite agree
that it is neither
>>> scalable nor robust.
>>>
>>>       
>> I was going to post some history of scaling mail, but I blogged it
instead.
>> http://blogs.sun.com/relling/entry/on_var_mail_and_quotas
>>  -- richard
>>
>>     
>
> The problem with that argument is that 10.000 users on one vxfs or UFS
> filesystem is no problem at all, be it /var/mail or home directories.
> You don''t even need a fast server for that. 10.000 zfs file
systems is
> a problem.
>
> So, if it makes you happier, substitute mail with home directories.
>   
If you feel strongly, please pile onto CR 6557894
http://bugs.opensolaris.org/view_bug.do?bug_id=6557894
If we continue to talk about it on the alias, we will just end up
finding ways to solve the business problem using available
technologies.

A single file system serving 10,000 home directories doesn''t scale
either, unless the vast majority are unused -- in which case it is a
practical problem for much less than 10,000 home directories.
I think you will find that the people who scale out have a better
long-term strategy.

The limitations of UFS do become apparent as you try to scale
to the size permitted with ZFS.  For example, the largest UFS
file system supported is 16 TBytes, or 1/4 of a thumper.  So if you
are telling me that you are serving 10,000 home directories in
a 16 TByte UFS file system with quotas (1.6 GBytes/user?  I''ve
got 16 GBytes in my phone :-), then I will definitely buy you a
beer.  And aspirin.  I''ll bring a calendar so we can measure the
fsck time when the log can''t be replayed.  Actually, you''d
probably run out of inodes long before you filled it up.  I wonder
how long it would take to run quotacheck?  But I digress.  Let''s
just agree that UFS won''t scale well and the people who do
serve UFS as home directories for large populations tend to use
multiple file systems.

For ZFS, there are some features which conflict with the
notion of user quotas: compression, copies, and snapshots come
immediately to mind.  UFS (and perhaps VxFS?) do not have
these features, so accounting space to users is much simpler.
Indeed, if was was easy to add to ZFS, then CR 6557894
would have been closed long ago.  Surely we can describe the
business problems previously solved by user-quotas and then
proceed to solve them?  Mail is already solved.

 -- richard

On Fri, Jun 06, 2008 at 03:43:29PM -0700, eric kustarz
wrote:
> 
> On Jun 6, 2008, at 3:27 PM, Brian Hechinger wrote:
> 
> > On Fri, Jun 06, 2008 at 02:58:09PM -0700, eric kustarz wrote:
> >>
> >>>> clients do not.  Without per-filesystem mounts,
''df'' on the client
> >>>> will not report correct data though.
> >>>
> >>> I expect that mirror mounts will be coming Linux''s
way too.
> >>
> >> The should already have them:
> >>
http://blogs.sun.com/erickustarz/en_US/entry/linux_support_for_mirror_mounts
> >
> > Where does that leave those of us who need to deal with OSX  
> > clients?  Does apple
> > have any plans to get in on this?
> 
> They need to implement NFSv4 in general first :)
Technically, Mac OS X 10.5 "Leopard" has some basic NFSv4.0 support in
it.
But just enough to make it look like it passes all the Connectathon tests.
Not enough to warrant use by anyone but the terminally curious (or masochistic).
This is mentioned briefly in the mount_nfs(8) man page.

It would be reasonable to expect that future MacOSX releases will include
increasing levels of functionality and that NFSv4 will eventually be made
the default NFS version.
> But you''d have to  
> ask them on their lists what the status of that is... i know i would  
> like it...
Or get lucky and happen to have one of their engineers catch the question
on this list and reply...  ;-)

--macko

>>
>> The problem with that argument is that 10.000 users on one vxfs or UFS
>> filesystem is no problem at all, be it /var/mail or home directories.
>> You don''t even need a fast server for that. 10.000 zfs file
systems is
>> a problem.
>>
>> So, if it makes you happier, substitute mail with home directories.
>>
>
> If you feel strongly, please pile onto CR 6557894
> http://bugs.opensolaris.org/view_bug.do?bug_id=6557894
> If we continue to talk about it on the alias, we will just end up
> finding ways to solve the business problem using available
> technologies.
If I need to count useage I can use du. But if you can implement space
usage info on a per-uid basis you are not far from quota per uid...
>
> A single file system serving 10,000 home directories doesn''t scale
> either, unless the vast majority are unused -- in which case it is a
> practical problem for much less than 10,000 home directories.
> I think you will find that the people who scale out have a better
> long-term strategy.
We have a file system (vxfs) that is serving 30,000 home directories.
Yes, most of those are unused, but we still have to have them as we
don''t know when the student will use it.

If this where zfs we would have to create 30,000 filesystem. Every
file system has a cost in RAM and in performance.

So, in ufs or vxfs unused home directories costs close to nothing. In
zfs they have a very real cost.

>
> The limitations of UFS do become apparent as you try to scale
> to the size permitted with ZFS.  For example, the largest UFS
> file system supported is 16 TBytes, or 1/4 of a thumper.  So if you
> are telling me that you are serving 10,000 home directories in
> a 16 TByte UFS file system with quotas (1.6 GBytes/user?  I''ve
> got 16 GBytes in my phone :-), then I will definitely buy you a
> beer.  And aspirin.  I''ll bring a calendar so we can measure the
> fsck time when the log can''t be replayed.  Actually,
you''d
> probably run out of inodes long before you filled it up.  I wonder
> how long it would take to run quotacheck?  But I digress.  Let''s
> just agree that UFS won''t scale well and the people who do
> serve UFS as home directories for large populations tend to use
> multiple file systems.
We have 30,000 accounts on a 1TByte file system. If we need to we
could make 16 1Tb file systems, no problem. But 30,000 file systems on
one server? Maybe not so good...

If we could lower the cost of a zfs file system to zero all would be
good for my usages.

The best thing to do is probably AFS on ZFS. AFS can handle many
volumes (file systems) and ZFS is very good at the storage.

On Fri, Jun 6, 2008 at 4:14 PM, Peter Tribble <peter.tribble at gmail.com>
wrote:
... very big snip ...
> (Although I have to say that, in a previous job, scrapping user quotas
entirely
> not only resulted in happier users, much less work for the helpdesk, and -
> paradoxically - largely eliminated systems running out of space.)
[Hi Peter]

Agreed.

So, one has to re-evaluate "legacy" thinking in the context of
inexpensive storage offered by ZFS in combination with cost effective
disk drives and ask the question: what lowers the total cost of
ownership and provides the best user experience?

Option a) A complex quota based system implemented on top of providing
the "correct" required system storage capacity.

Option b) A ZFS based storage system with x2 or  x4 (etc) times the
"correct" required storage capacity with a once a day cron job to
remind the storage hogs (users) to trim their disk space, or face the
"or else" option (the stick approach).  And perhaps a few quotas on
filesystems used by applications or users know to be problematic.

I would submit that Option b) will provide a lower cost, in terms of
total system cost, over time - expecially given the price performance
of modern disk drives in combination with high performance log and
cache devices (if required).

Every time I''ve come across a usage scenario where the submitter asks
for per user quotas, its usually a university type scenario where
univeristies are notorious for providing lots of CPU horsepower (many,
many servers) attached to a simply dismal amount of back-end storage.
Where users are, and continue to be, miffed at the dismal amount of
storage they are offered.  This is legacy thinking looking for a
"legacy-thinking-compliant" solution to a problem that has already
been solved by ZFS and the current generation of high capacity, ultra
low-cost per terabyte, offered by modern hardware.  IOW - it''s a
people issue, rather than a technological issue.

Regards,

-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

For the cifs side of the house, I think it would be in Sun''s best
interest
to work with a third party vendor like NTP software.  The quota
functionality they provide is far more robust than anything I expect
we''ll
ever see come directly with zfs.  And rightly so... it''s what they
specialize in.

http://www.ntpsoftware.com/products/qfs/?adrid


On Sat, Jun 7, 2008 at 2:20 PM, Al Hopper <al at logical-approach.com>
wrote:
> On Fri, Jun 6, 2008 at 4:14 PM, Peter Tribble <peter.tribble at
gmail.com>
> wrote:
> ... very big snip ...
> > (Although I have to say that, in a previous job, scrapping user quotas
> entirely
> > not only resulted in happier users, much less work for the helpdesk,
and
> -
> > paradoxically - largely eliminated systems running out of space.)
>
> [Hi Peter]
>
> Agreed.
>
> So, one has to re-evaluate "legacy" thinking in the context of
> inexpensive storage offered by ZFS in combination with cost effective
> disk drives and ask the question: what lowers the total cost of
> ownership and provides the best user experience?
>
> Option a) A complex quota based system implemented on top of providing
> the "correct" required system storage capacity.
>
> Option b) A ZFS based storage system with x2 or  x4 (etc) times the
> "correct" required storage capacity with a once a day cron job to
> remind the storage hogs (users) to trim their disk space, or face the
> "or else" option (the stick approach).  And perhaps a few quotas
on
> filesystems used by applications or users know to be problematic.
>
> I would submit that Option b) will provide a lower cost, in terms of
> total system cost, over time - expecially given the price performance
> of modern disk drives in combination with high performance log and
> cache devices (if required).
>
> Every time I''ve come across a usage scenario where the submitter
asks
> for per user quotas, its usually a university type scenario where
> univeristies are notorious for providing lots of CPU horsepower (many,
> many servers) attached to a simply dismal amount of back-end storage.
> Where users are, and continue to be, miffed at the dismal amount of
> storage they are offered.  This is legacy thinking looking for a
> "legacy-thinking-compliant" solution to a problem that has
already
> been solved by ZFS and the current generation of high capacity, ultra
> low-cost per terabyte, offered by modern hardware.  IOW - it''s a
> people issue, rather than a technological issue.
>
> Regards,
>
> --
> Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
>                   Voice: 972.379.2133 Timezone: US CDT
> OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
> http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080607/4935701f/attachment.html>

On Sat, 7 Jun 2008, Mattias Pantzare wrote:
>
> If I need to count useage I can use du. But if you can implement space
> usage info on a per-uid basis you are not far from quota per uid...
That sounds like quite a challenge.  UIDs are just numbers and new 
ones can appear at any time.  Files with existing UIDs can have their 
UIDs switched from one to another at any time.  The space used per UID 
needs to be tallied continuously and needs to track every change, 
including real-time file growth and truncation.  We are ultimately 
talking about 128 bit counters here.  Instead of having one counter 
per filesystem we now have potentially hundreds of thousands, which 
represents substantial memory.

Multicore systems have the additional challenge that this complex 
information needs to be effectively shared between cores.  Imagine if 
you have 512 CPU cores, all of which are running some of the ZFS code 
and have their own caches which become invalidated whenever one of 
those counters is updated.  This sounds like a no-go for an almost 
infinite-sized pooled "last word" filesystem like ZFS.

ZFS is already quite lazy at evaluating space consumption.  With ZFS, 
''du'' does not always reflect true usage since updates are
delayed.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Richard Elling wrote:
> For ZFS, there are some features which conflict with the
> notion of user quotas: compression, copies, and snapshots come
> immediately to mind.  UFS (and perhaps VxFS?) do not have
> these features, so accounting space to users is much simpler.
> Indeed, if was was easy to add to ZFS, then CR 6557894
> would have been closed long ago.  Surely we can describe the
> business problems previously solved by user-quotas and then
> proceed to solve them?  Mail is already solved.
I just find it ironic that before ZFS I kept hearing of people wanting 
group quotas rather than user quotas.  Now that we have ZFS group quotas 
are easy - quota the filesystem and ensure only that group can write to 
it - but now the focus is back on user quotas again ;-)

-- 
Darren J Moffat

> Richard Elling wrote:
> > For ZFS, there are some features which conflict
> with the
> > notion of user quotas: compression, copies, and
> snapshots come
> > immediately to mind.  UFS (and perhaps VxFS?) do
> not have
> > these features, so accounting space to users is
> much simpler.
> > Indeed, if was was easy to add to ZFS, then CR
> 6557894
> > would have been closed long ago.  Surely we can
> describe the
> > business problems previously solved by user-quotas
> and then
> > proceed to solve them?  Mail is already solved.
> 
> I just find it ironic that before ZFS I kept hearing
> of people wanting 
> group quotas rather than user quotas.  Now that we
> have ZFS group quotas 
> are easy - quota the filesystem and ensure only that
> group can write to 
> it - but now the focus is back on user quotas again
> ;-)
> 
I hate to say that, but most of us didn''t expect zfs takes possibility
of user quotas away.. I am not sure "trading" one capability with the
other is desired.

Back to zfs'' "filesystem is cheap" paradigm, it
won''t be complained so much if other facilities provided by OS (for
example, automounter) scales equally well with zfs.

Making other fs related facility fitting into the new paradigm would help much,
at least I think..

Ivan.

> -- 
> Darren J Moffat
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discu
> ss
 
 
This message posted from opensolaris.org

> On Sat, 7 Jun 2008, Mattias Pantzare wrote:
> >
> > If I need to count useage I can use du. But if you
> can implement space
> > usage info on a per-uid basis you are not far from
> quota per uid...
> 
> That sounds like quite a challenge.  UIDs are just
> numbers and new 
> ones can appear at any time.  Files with existing
> UIDs can have their 
> UIDs switched from one to another at any time.  The
> space used per UID 
> needs to be tallied continuously and needs to track
> every change, 
> including real-time file growth and truncation.  We
> are ultimately 
> talking about 128 bit counters here.  Instead of
> having one counter 
> per filesystem we now have potentially hundreds of
> thousands, which 
> represents substantial memory.
But if you already have the ZAP code, you ought to be able to do
quick lookups of arbitrary byte sequences, right?  Just assume that
a value not stored is zero (or infinity, or uninitialized, as applicable),
and you have the same functionality as  the sparse quota file on ufs,
without the problems.

Besides, uid/gid/sid quotas would usually make more sense at the zpool level
than
at the individual filesystem level, so perhaps it''s not _that_ bad. 
Which is to
say, you want user X to have an n GB quota over the whole zpool, and you
probably don''t so much care whether the filesystem within the zpool
corresponds to his home directory or to some shared directory.
> Multicore systems have the additional challenge that
> this complex 
> information needs to be effectively shared between
> cores.  Imagine if 
> you have 512 CPU cores, all of which are running some
> of the ZFS code 
> and have their own caches which become invalidated
> whenever one of 
> those counters is updated.  This sounds like a no-go
> for an almost 
> infinite-sized pooled "last word" filesystem like
> ZFS.
> 
> ZFS is already quite lazy at evaluating space
> consumption.  With ZFS, 
> ''du'' does not always reflect true usage since updates
> are delayed.
Whatever mechanism can check at block allocation/deallocation time
to keep track of per-filesystem space (vs a filesystem quota, if there is one)
could surely also do something similar against per-uid/gid/sid quotas.  I
suspect
a lot of existing functions and data structures could be reused or adapted for
most of it.  Just one more piece of metadata to update, right?  Not as if ufs
quotas had zero runtime penalty if enabled.   And you only need counters and
quotas in-core for identifiers applicable to in-core znodes, not for every
identifier used on the zpool.

Maybe I''m off base on the details.  But in any event, I expect that
it''s entirely
possible to make it happen, scalably.  Just a question of whether it''s
worth the
cost of designing, coding, testing, documenting.  I suspect there may be enough
scenarios for sites with really high numbers of accounts (particularly
universities, which are not only customers in their own right, but a chance
for future mindshare) that it might be worthwhile, but I don''t know
that to
be the case.

IMO, even if no one sort of site using existing deployment architectures would
justify it, given the future blurring of server, SAN, and NAS (think recent
SSD announcement + COMSTAR + iSCSI initiator + separate device for zfs
zil & cache + in-kernel CIFS + enterprise authentication with Windows
interoperability + Thumper + ...), the ability to manage all that storage in all
sorts of as-yet unforseen deployment configurations _by user or other identity_
may well be important across a broad base of customers.  Maybe identity-based,
as well as filesystem-based quotas, should be part of that.
 
 
This message posted from opensolaris.org

Richard L. Hamilton wrote:
> Whatever mechanism can check at block allocation/deallocation time
> to keep track of per-filesystem space (vs a filesystem quota, if there is
one)
> could surely also do something similar against per-uid/gid/sid quotas.  I
suspect
> a lot of existing functions and data structures could be reused or adapted
for
> most of it.  Just one more piece of metadata to update, right?  Not as if
ufs
> quotas had zero runtime penalty if enabled.   And you only need counters
and
> quotas in-core for identifiers applicable to in-core znodes, not for every
> identifier used on the zpool.
The current quota system does its checking of quota constraints in the 
DSL (dsl_sync_task_group_sync ends up getting the quota check made)

A user based quota system would I believe need to be in the ZPL because 
that is where we understand what users are.  I suspect this means that 
quotas would probably be easiest implemented per dataset rather than per 
pool.


-- 
Darren J Moffat

On Wed, 11 Jun 2008, Richard L. Hamilton wrote:
>
> But if you already have the ZAP code, you ought to be able to do
> quick lookups of arbitrary byte sequences, right?  Just assume that
> a value not stored is zero (or infinity, or uninitialized, as applicable),
> and you have the same functionality as  the sparse quota file on ufs,
> without the problems.
I don''t know anything about "ZAP code" but I do know that CPU
caches
are only so large and there can be many caches for the same data since 
each CPU has its own cache.  Some of us do actual computing using 
these same CPUs so it would be nice if they weren''t entirely consumed 
by the filesystem.

Current application performance on today''s hardware absolutely sucks 
as compared to its theroretical potential.  Let''s try to improve that 
performance rather than adding more cache thrashing, leading to more 
wait states.

Bob
=====================================Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

This is one of those issues, where the developers generally seem to think that
old-style quotas is legacy baggage.  And that people running large
home-directory sort of servers with 10,000+ users are a minority that can safely
be ignored.

I can understand their thinking.    However it does represent a problem here at
the University of California Davis.  I would love to replace our Solaris 9
home-directory server with one running Solaris 10 and ZFS.  A past issue with
UFS corruption keeps us all nervous.  But there is no other alternative to
UFS+quotas as yet it seems.
 
 
This message posted from opensolaris.org

| Every time I''ve come across a usage scenario where the submitter asks
| for per user quotas, its usually a university type scenario where
| univeristies are notorious for providing lots of CPU horsepower (many,
| many servers) attached to a simply dismal amount of back-end storage.

 Speaking as one of those pesky university people (although we don''t
use
quotas): one of the reasons this happens is that servers are a lot less
expensive than disk space. With disk space you have to factor in the
cost of backups and ongoing maintenance, wheras another server is just N
thousand dollars in one time costs and some rack space.

(This assumes that you are not rack space, heat, or power constrained,
which I think most university environments generally are not.)

 Or to put it another way: disk space is a permanent commitment,
servers are not.

	- cks

On Jun 12, 2008, at 12:46 PM, Chris Siebenmann wrote:
>
>  Or to put it another way: disk space is a permanent commitment,
> servers are not.

In the olden times (e.g. 1980s) on various CDC and Univac timesharing  
services, I recall there being two kinds of storage ... "dayfiles"  
and permanent files. The former could (and as a matter of policy did)  
be removed at the end of the day.

It was typically cheaper to move the fraction of one''s dayfile output  
to tape, and have it rolled back in the next day ... but that was an  
optimization (or pessimization if the true costs were calculated).

I could easily imagine providing two tiers of storage for a  
university environment ... one which wasn''t backed up, and
doesn''t
come with any serious promises ... which could be pretty inexpensive  
and the second tier which has the kind of commitments you suggest are  
required.

Tier 2 should be better than storing things in /tmp, but could  
approach consumer pricing ... and still be "good enough" for a lot of
uses.
-- 
Keith H. Bierman   khbkhb at gmail.com      | AIM kbiermank
5430 Nassau Circle East                  |
Cherry Hills Village, CO 80113           | 303-997-2749
<speaking for myself*> Copyright 2008

Chris Siebenmann wrote:
|  Speaking as one of those pesky university people (although we don''t
use
| quotas): one of the reasons this happens is that servers are a lot less
| expensive than disk space. With disk space you have to factor in the
| cost of backups and ongoing maintenance, wheras another server is just N
| thousand dollars in one time costs and some rack space.

This is also common in organizations where IT is a cost center,
including some *very* large ones I''ve encountered in the past
and several which are just, well, conservative.

--dave
-- 
David Collier-Brown            | Always do right. This will gratify
Sun Microsystems, Toronto      | some people and astonish the rest
davecb at sun.com                 |                      -- Mark Twain
(905) 943-1983, cell: (647) 833-9377, (800) 555-9786 x56583
bridge: (877) 385-4099 code: 506 9191#

On 6/13/08 12:25 AM, "Keith Bierman" <khbkhb at gmail.com>
wrote:
> I could easily imagine providing two tiers of storage for a
> university environment ... one which wasn''t backed up, and
doesn''t
> come with any serious promises ... which could be pretty inexpensive
> and the second tier which has the kind of commitments you suggest are
> required.
> 
> Tier 2 should be better than storing things in /tmp, but could
> approach consumer pricing ... and still be "good enough" for a
lot of
> uses.
We have provided multiple "tiers" of storage for years.  However, this
usually didn''t involve different "tiers" of hardware. 
Rather, it
represented how we treated the files.  We have everything from "staging
pools" where everything is transient (no backups, no real SLA, wild west
rules) to snapshots, disaster recovery replication and backup.

What''s really going to change everything is SAMFS.  We''re able
to take
advantage of $.60/GB disk on X4500, $5/GB disk on SAN and hundreds of TB of
tape "backing store" that also provides real-time backup (our
traditional
backup windows are untenable).  Most importantly, we''re not tied to a
specific vendor''s solutions (though I''m very happy with our
closed SAN''s
capabilities).

"ILM" is essentially a necessity.  You can''t manage storage
beyond the "home
server" without it.  I hope that all storage technologies take a holistic
view of the storage management picture.  While ZFS goes a long way to
eliminating distinctions between volume and filesystem management, it is
still a niche player.  As much hype as ZFS snapshots get, that''s barely
tiptoeing into the managed storage envelope.  However, I do appreciate the
focus on data integrity.  Without that at every tier, ILM cannot properly do
its job.

Charles

-----

Charles Soto                        charles.soto at austin.utexas.edu
Director, Information Technology? ? ? ??         TEL: 512-740-1888
The University of Texas at Austin?? ?? ?         FAX: 512-475-9711
College of Communication, CMA 5.150G
1 University Station A0900, Austin, TX 78712
http://communication.utexas.edu/technology/

On 6/12/08 1:46 PM, "Chris Siebenmann" <cks at cs.toronto.edu>
wrote:
> | Every time I''ve come across a usage scenario where the submitter
asks
> | for per user quotas, its usually a university type scenario where
> | univeristies are notorious for providing lots of CPU horsepower (many,
> | many servers) attached to a simply dismal amount of back-end storage.
> 
>  Speaking as one of those pesky university people (although we
don''t use
> quotas): one of the reasons this happens is that servers are a lot less
> expensive than disk space. With disk space you have to factor in the
> cost of backups and ongoing maintenance, wheras another server is just N
> thousand dollars in one time costs and some rack space.
> 
> (This assumes that you are not rack space, heat, or power constrained,
> which I think most university environments generally are not.)
> 
>  Or to put it another way: disk space is a permanent commitment,
> servers are not.
> 
> - cks
Well, servers have a "running cost," as keeping them up (e.g. running
and
still under your control!) requires a certain commitment of resources.  But,
I think the resource emphasis on storage is quite appropriate.  The DATA are
the valuable things, not the servers or applications.  Appropriately,
servers reached commodity status before storage.  But storage hardware will
go that way, and the focus will be on data (storage) management, where it
rightfully belongs.

Charles

-----

Charles Soto                        charles.soto at austin.utexas.edu
Director, Information Technology? ? ? ??         TEL: 512-740-1888
The University of Texas at Austin?? ?? ?         FAX: 512-475-9711
College of Communication, CMA 5.150G
1 University Station A0900, Austin, TX 78712
http://communication.utexas.edu/technology/

Charles Soto wrote:
> On 6/13/08 12:25 AM, "Keith Bierman" <khbkhb at gmail.com>
wrote:
>
>   
>> I could easily imagine providing two tiers of storage for a
>> university environment ... one which wasn''t backed up, and
doesn''t
>> come with any serious promises ... which could be pretty inexpensive
>> and the second tier which has the kind of commitments you suggest are
>> required.
>>
>> Tier 2 should be better than storing things in /tmp, but could
>> approach consumer pricing ... and still be "good enough" for
a lot of
>> uses.
>>     
>
> We have provided multiple "tiers" of storage for years.  However,
this
> usually didn''t involve different "tiers" of hardware. 
Rather, it
> represented how we treated the files.  We have everything from
"staging
> pools" where everything is transient (no backups, no real SLA, wild
west
> rules) to snapshots, disaster recovery replication and backup.
>
> What''s really going to change everything is SAMFS.  We''re
able to take
> advantage of $.60/GB disk on X4500, $5/GB disk on SAN and hundreds of TB of
> tape "backing store" that also provides real-time backup (our
traditional
> backup windows are untenable).  Most importantly, we''re not tied
to a
> specific vendor''s solutions (though I''m very happy with
our closed SAN''s
> capabilities).
>
> "ILM" is essentially a necessity.  You can''t manage
storage beyond the "home
> server" without it.  I hope that all storage technologies take a
holistic
> view of the storage management picture.  While ZFS goes a long way to
> eliminating distinctions between volume and filesystem management, it is
> still a niche player.  As much hype as ZFS snapshots get, that''s
barely
> tiptoeing into the managed storage envelope.  However, I do appreciate the
> focus on data integrity.  Without that at every tier, ILM cannot properly
do
> its job.
>   
Well said, Charles.  It is worth mentioning that ZFS is just one part of
such solutions.  To see how everything fits together, go to the storage
community
    http://opensolaris.org/os/community/storage/projects/

a very interesting project to follow is the Automatic Data Migration (ADM)
project at
    http://opensolaris.org/os/project/adm/

 -- richard