zfs discuss - Apr 2012 - [developer] Setting default user/group quotas[usage accounting]?

On Apr 24, 2012, at 2:50 PM, Fred Liu wrote:


Yes.

Thanks.

I am not aware of anyone looking into this.

I don''t think it is very hard, per se. But such quotas don''t
fit well with the
notion of many file systems. There might be some restricted use cases
where it makes good sense, but I''m not convinced it will scale well --
user
quotas never scale well.
 -- richard

>OK. I see. And I agree such quotas will scale well. From users''
side, they always
> ask for more space or even no quotas at all. One of the  main purposes
behind such quotas
> is that we can account usage and get the statistics. Is it possible to do
it without setting
> such quotas?
>
>Thanks.
>
>Fred
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120425/9d49e80e/attachment.html>

Missing an important ?NOT?:
>OK. I see. And I agree such quotas will **NOT** scale well. From
users'' side, they always
> ask for more space or even no quotas at all. One of the  main purposes
behind such quotas
> is that we can account usage and get the statistics. Is it possible to do
it without setting
> such quotas?
>
>Thanks.
>
>Fred
>

_____________________________________________
From: Fred Liu
Sent: ???, ?? 25, 2012 20:05
To: developer at lists.illumos.org
Cc: ''zfs-discuss at opensolaris.org''
Subject: RE: [developer] Setting default user/group quotas[usage accounting]?





On Apr 24, 2012, at 2:50 PM, Fred Liu wrote:


Yes.

Thanks.

I am not aware of anyone looking into this.

I don''t think it is very hard, per se. But such quotas don''t
fit well with the
notion of many file systems. There might be some restricted use cases
where it makes good sense, but I''m not convinced it will scale well --
user
quotas never scale well.
 -- richard

>OK. I see. And I agree such quotas will scale well. From users''
side, they always
> ask for more space or even no quotas at all. One of the  main purposes
behind such quotas
> is that we can account usage and get the statistics. Is it possible to do
it without setting
> such quotas?
>
>Thanks.
>
>Fred
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120425/5d989b62/attachment.html>

ZFS will always track per-user usage information even in the absence of
quotas. See the the zfs ''userused@'' properties and
''zfs userspace'' command.

- Eric

2012/4/25 Fred Liu <Fred_Liu at issi.com>
> Missing an important ?NOT?:
>
> >OK. I see. And I agree such quotas will **NOT** scale well. From
users''
> side, they always
> > ask for more space or even no quotas at all. One of the  main purposes
> behind such quotas
> > is that we can account usage and get the statistics. Is it possible to
> do it without setting
> > such quotas?
> >
> >Thanks.
> >
> >Fred
> >
>
>
> _____________________________________________
> *From:* Fred Liu
> *Sent:* ???, ?? 25, 2012 20:05
> *To:* developer at lists.illumos.org
> *Cc:* ''zfs-discuss at opensolaris.org''
> *Subject:* RE: [developer] Setting default user/group quotas[usage
> accounting]?
>
>
>
>
>
> On Apr 24, 2012, at 2:50 PM, Fred Liu wrote:
>
>
> Yes.
>
> Thanks.
>
> I am not aware of anyone looking into this.
>
> I don''t think it is very hard, per se. But such quotas
don''t fit well with
> the
> notion of many file systems. There might be some restricted use cases
> where it makes good sense, but I''m not convinced it will scale
well -- user
> quotas never scale well.
>  -- richard
>
>
> >OK. I see. And I agree such quotas will scale well. From
users'' side,
> they always
> > ask for more space or even no quotas at all. One of the  main purposes
> behind such quotas
> > is that we can account usage and get the statistics. Is it possible to
> do it without setting
> > such quotas?
> >
> >Thanks.
> >
> >Fred
> >
>
>   *illumos-developer* |
Archives<https://www.listbox.com/member/archive/182179/=now>
> <https://www.listbox.com/member/archive/rss/182179/21175057-f8151d0d>
|
>
Modify<https://www.listbox.com/member/?member_id=21175057&id_secret=21175057-02786781>Your
Subscription
> <http://www.listbox.com>
>


-- 
Eric Schrock
Delphix
http://blog.delphix.com/eschrock

275 Middlefield Road, Suite 50
Menlo Park, CA 94025
http://www.delphix.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120425/5a222387/attachment-0001.html>

On Apr 25, 2012, at 8:14 AM, Eric Schrock wrote:
> ZFS will always track per-user usage information even in the absence of
quotas. See the the zfs ''userused@'' properties and
''zfs userspace'' command.
tip: zfs get -H -o value -p userused at username filesystem

Yes, and this is the logical size, not physical size. Some ZFS features increase
logical size
(copies) while others decrease physical size (compression, dedup)
 -- richard

--
ZFS Performance and Training
Richard.Elling at RichardElling.com
+1-760-896-4422







-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120425/cb155fee/attachment.html>

?zfs ''userused@'' properties? and ?''zfs
userspace'' command? are good enough to gather usage statistics.
I think I mix that with NetApp. If my memory is correct, we have to set quotas
to get usage statistics under DataOnTAP.
Further, if we can add an ILM-like feature to poll the time-related
info(atime,mtime,ctime,etc) with that statistic from ZFS, that will be
really cool.

Since no one is focusing on enabling default user/group quota now, the
temporarily remedy could be a script which traverse all the users/groups
in the directory tree. Tough it is not so decent.

Currently, dedup/compression is pool-based right now, they don?t have the
granularity on file system or user or group level. There is also a lot of
improving  space in this aspect.


Thanks.

Fred

From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at
opensolaris.org] On Behalf Of Richard Elling
Sent: ???, ?? 26, 2012 0:48
To: Eric Schrock
Cc: zfs-discuss at opensolaris.org; developer at lists.illumos.org
Subject: Re: [zfs-discuss] [developer] Setting default user/group quotas[usage
accounting]?

On Apr 25, 2012, at 8:14 AM, Eric Schrock wrote:


ZFS will always track per-user usage information even in the absence of quotas.
See the the zfs ''userused@'' properties and ''zfs
userspace'' command.

tip: zfs get -H -o value -p userused at username filesystem

Yes, and this is the logical size, not physical size. Some ZFS features increase
logical size
(copies) while others decrease physical size (compression, dedup)
 -- richard

--
ZFS Performance and Training
Richard.Elling at RichardElling.com<mailto:Richard.Elling at
RichardElling.com>
+1-760-896-4422







-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120426/99477ca4/attachment.html>

On 2012-04-26 11:27, Fred Liu wrote:
> ?zfs ''userused@'' properties? and ?''zfs
userspace'' command? are good
> enough to gather usage statistics.
...
> Since no one is focusing on enabling default user/group quota now, the
> temporarily remedy could be a script which traverse all the users/groups
> in the directory tree. Tough it is not so decent.
find /export/home -type f -uid 12345 -exec du -ks ''{}'' \; |
summing-script

I think you could use some prefetch of dirtree traversal,
like a "slocate" database, or roll your own (perl script).
But yes, it does seem like stone age compared to ZFS ;)
> Currently, dedup/compression is pool-based right now,
Dedup is pool-wide, compression is dataset-wide, applied to
individual blocks. Even deeper, both settings apply to new
writes after the corresponding dataset''s property was set
(i.e. a dataset can have files with mixed compression levels,
as well as both deduped and unique files).

> they don?t have
> the granularity on file system or user or group level.
> There is also a lot of improving space in this aspect.
This particular problem was discussed a number of times back
on OpenSolaris forum. It boiled down to what you actually
want to have accounted and perhaps billed - the raw resources
spent by storage system, or the logical resources accessed
and used by its users?

Say, you provide VMs with 100Gb of disk space, but your dedup
is lucky enough to use 1TB overall for say 100 VMs. You can
bill 100 users for full 100Gb each, but your operations budget
(and further planning, etc.) has only been hit for 1Tb.

HTH,
//Jim

2012/4/26 Fred Liu <Fred_Liu at issi.com>
>
> ** **
>
> Currently, dedup/compression is pool-based right now, they don?t have the
> granularity on file system or user or group level. There is also a lot of
> improving  space in this aspect.
>
Compression is not pool-based, you can control it with the
''compression''
property on a per-filesystem level, and is fundamentally per-block. Dedup
is also controlled per-filesystem, though the DDT is global to the pool.

If you think there are compelling features lurking here, then by all means
grab the code and run with it :-)

- Eric

-- 
Eric Schrock
Delphix
http://blog.delphix.com/eschrock

275 Middlefield Road, Suite 50
Menlo Park, CA 94025
http://www.delphix.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120426/dda239a0/attachment.html>

On Apr 26, 2012, at 12:27 AM, Fred Liu wrote:
> ?zfs ''userused@'' properties? and ?''zfs
userspace'' command? are good enough to gather usage statistics.
> I think I mix that with NetApp. If my memory is correct, we have to set
quotas to get usage statistics under DataOnTAP.
> Further, if we can add an ILM-like feature to poll the time-related
info(atime,mtime,ctime,etc) with that statistic from ZFS, that will be
> really cool.
In general, file-based ILM has limitations that cause all sorts of issues for
things like
operating systems, where files might only be needed infrequently, but when they
are
needed, they are needed "right now"

Have you looked at "zfs diff"  for changed files?
> Since no one is focusing on enabling default user/group quota now, the
temporarily remedy could be a script which traverse all the users/groups
> in the directory tree. Tough it is not so decent.
The largest market for user/group quotas is .edu. But they represent only a
small market
when measured by $.

There are also many corner cases in this problem space. One might pine for the
days of
VMS and its file resource management features, those features don''t
scale well to company-
wide LDAP and thousands of file systems.

So, for now, the fastest method to solve the problem might be to script some
walkers.
 -- richard

--
ZFS Performance and Training
Richard.Elling at RichardElling.com
+1-760-896-4422







-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120426/571fc00d/attachment-0001.html>

>
>2012/4/26 Fred Liu <Fred_Liu at issi.com>
>
>Currently, dedup/compression is pool-based right now, they don''t
have the
>granularity on file system or user or group level. There is also a lot of
>improving ?space in this aspect.
>
>Compression is not pool-based, you can control it with the
''compression''
>property on a per-filesystem level, and is fundamentally per-block. Dedup is
>also controlled per-filesystem, though the DDT is global to the pool.
>
>If you think there are compelling features lurking here, then by all means
grab
>the code and run with it :-)
>
>- Eric
>
>--
Thanks for correcting me. I will have a try and see how far I can go.


Thanks.

Fred

>On 2012-04-26 11:27, Fred Liu wrote:
>> "zfs ''userused@'' properties" and
"''zfs userspace'' command" are good
>> enough to gather usage statistics.
>...
>> Since no one is focusing on enabling default user/group quota now, the
>> temporarily remedy could be a script which traverse all the
>> users/groups in the directory tree. Tough it is not so decent.
>
>find /export/home -type f -uid 12345 -exec du -ks ''{}'' \;
| summing-script
>
>I think you could use some prefetch of dirtree traversal, like a
"slocate"
>database, or roll your own (perl script).
>But yes, it does seem like stone age compared to ZFS ;)
>
Thanks for the hint. I mean " traverse all the users/groups in the
directory tree"
as "getting all user/group info from naming service like nis/ldap" for
a specific file system.
And for each found item, we can use "zfs set userquota@/groupquota@"
to set the default value.
As for usage accounting, "zfs ''userused@'' properties and
''zfs userspace'' command" are good enough.
We can also use a script to do the summing jobs via traversing all the
pools/filesystems.
>> Currently, dedup/compression is pool-based right now,
>
>Dedup is pool-wide, compression is dataset-wide, applied to individual
blocks.
>Even deeper, both settings apply to new writes after the corresponding
>dataset''s property was set (i.e. a dataset can have files with
mixed
>compression levels, as well as both deduped and unique files).
>
>
>> they don''t have
>> the granularity on file system or user or group level.
>> There is also a lot of improving space in this aspect.
>
>This particular problem was discussed a number of times back on OpenSolaris
>forum. It boiled down to what you actually want to have accounted and
>perhaps billed - the raw resources spent by storage system, or the logical
>resources accessed and used by its users?
>
>Say, you provide VMs with 100Gb of disk space, but your dedup is lucky
>enough to use 1TB overall for say 100 VMs. You can bill 100 users for full
>100Gb each, but your operations budget (and further planning, etc.) has only
>been hit for 1Tb.
>
The ideal situation is we know exactly both the logical usage and the physical
usage per user/group.
But that is not applicable for now. And assuming even we know it, we still
cannot estimate the physical
usage for dedup/compression varies by the using pattern.

Yes. We do get bonus from dedup/compression. But there is no good way to make it
fit into budget plan from my side.
>HTH,
>//Jim
>
>_______________________________________________
>zfs-discuss mailing list
>zfs-discuss at opensolaris.org
>http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

>On Apr 26, 2012, at 12:27 AM, Fred Liu wrote:
>"zfs ''userused@'' properties" and
"''zfs userspace'' command" are good enough to gather
usage statistics.
>I think I mix that with NetApp. If my memory is correct, we have to set
quotas to get usage statistics under DataOnTAP.
>Further, if we can add an ILM-like feature to poll the time-related
info(atime,mtime,ctime,etc) with that statistic from ZFS, that will be
>really cool.
>In general, file-based ILM has limitations that cause all sorts of issues
for things like
>operating systems, where files might only be needed infrequently, but when
they are
>needed, they are needed "right now"
>Have you looked at "zfs diff"  for changed files?
                   Here "ILM-like feature", I mean we know how the
data is distributed by time per pool/filesystem like how many data are
modified/accessed
       before mm/dd/yyyy. And we don''t need to do the actual
storage-tying operations immediately(moving the infrequently-used data to tie-2
       storage). The time-related usage statistics are very useful reference for
us.
       "zfs diff" will show the delta but not come with the time info.

>Since no one is focusing on enabling default user/group quota now, the
temporarily remedy could be a script which traverse all the users/groups
>in the directory tree. Tough it is not so decent.
>The largest market for user/group quotas is .edu. But they represent only a
small market
>when measured by $.
>There are also many corner cases in this problem space. One might pine for
the days of
>VMS and its file resource management features, those features don''t
scale well to company-
>wide LDAP and thousands of file systems.
                My understanding is the quota management is needed as long as
zfs storage is used in NAS way(shared by multi-users).

>So, for now, the fastest method to solve the problem might be to script some
walkers.
Yes. That is ture.

Thanks.


Fred

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120429/b6599408/attachment.html>

On Apr 29, 2012, at 7:59 PM, Fred Liu wrote:
>  
> >On Apr 26, 2012, at 12:27 AM, Fred Liu wrote:
> 
> >?zfs ''userused@'' properties? and ?''zfs
userspace'' command? are good enough to gather usage statistics.
> >I think I mix that with NetApp. If my memory is correct, we have to set
quotas to get usage statistics under DataOnTAP.
> >Further, if we can add an ILM-like feature to poll the time-related
info(atime,mtime,ctime,etc) with that statistic from ZFS, that will be
> >really cool.
>  
> >In general, file-based ILM has limitations that cause all sorts of
issues for things like
> >operating systems, where files might only be needed infrequently, but
when they are
> >needed, they are needed "right now"
>  
> >Have you looked at "zfs diff"  for changed files?
>  
>                    Here ?ILM-like feature?, I mean we know how the data is
distributed by time per pool/filesystem like how many data are modified/accessed
>        before mm/dd/yyyy. And we don?t need to do the actual storage-tying
operations immediately(moving the infrequently-used data to tie-2
>        storage). The time-related usage statistics are very useful
reference for us.
>        ?zfs diff? will show the delta but not come with the time info.
The time is the creation time of the snapshots.
 -- richard

--
ZFS Performance and Training
Richard.Elling at RichardElling.com
+1-760-896-4422







-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120430/9458590a/attachment-0002.html>

2012/4/25 Richard Elling <richard.elling at
gmail.com>:
> On Apr 25, 2012, at 8:14 AM, Eric Schrock wrote:
>
> ZFS will always track per-user usage information even in the absence of
> quotas. See the the zfs ''userused@'' properties and
''zfs userspace'' command.
>
>
> tip: zfs get -H -o value -p userused at username filesystem
>
> Yes, and this is the logical size, not physical size. Some ZFS features
> increase logical size
> (copies) while others decrease physical size (compression, dedup)
The size accounted for by the userused@ and groupused@ properties is
the "referenced" space, which is used as the basis for many other
space accounting values in ZFS (e.g. "du" / "ls -s" /
stat(2), and the
zfs accounting properties "referenced", "refquota",
"refreservation",
"refcompressratio", "written").  It includes changes local
to the
dataset (compression, the "copies" property, file-specific metadata
such as indirect blocks), but ignores pool-wide or cross-dataset
changes (space shared between a clone and its origin, mirroring,
raid-z, dedup[*]).

--matt

[*] Although dedup can be turned on and off per-dataset, the data is
deduplicated against all dedup-enabled data in the pool.  Ie,
identical data in different datasets will be stored only once, if
dedup is enabled for both datasets.

>The time is the creation time of the snapshots.
Yes. That is true.

Thanks.

Fred


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120502/1736d2ba/attachment.html>

>
>The size accounted for by the userused@ and groupused@ properties is the
>"referenced" space, which is used as the basis for many other
space
>accounting values in ZFS (e.g. "du" / "ls -s" / stat(2),
and the zfs accounting
>properties "referenced", "refquota",
"refreservation", "refcompressratio",
>"written").  It includes changes local to the dataset
(compression, the
>"copies" property, file-specific metadata such as indirect
blocks), but ignores
>pool-wide or cross-dataset changes (space shared between a clone and its
>origin, mirroring, raid-z, dedup[*]).
>
>--matt
>
>[*] Although dedup can be turned on and off per-dataset, the data is
>deduplicated against all dedup-enabled data in the pool.  Ie, identical data
in
>different datasets will be stored only once, if dedup is enabled for both
>datasets.
>
>
Can we also get the *ignored* space accounted?

Thanks.

Fred

On Wed, May 2, 2012 at 3:28 PM, Fred Liu <Fred_Liu at issi.com>
wrote:
>>
>>The size accounted for by the userused@ and groupused@ properties is the
>>"referenced" space, which is used as the basis for many other
space
>>accounting values in ZFS (e.g. "du" / "ls -s" /
stat(2), and the zfs accounting
>>properties "referenced", "refquota",
"refreservation", "refcompressratio",
>>"written"). ?It includes changes local to the dataset
(compression, the
>>"copies" property, file-specific metadata such as indirect
blocks), but ignores
>>pool-wide or cross-dataset changes (space shared between a clone and its
>>origin, mirroring, raid-z, dedup[*]).
> Can we also get the *ignored* space accounted?
What problem are you trying to solve?  How would you want referenced
or userused at ... to work?

To be more clear:  space shared between a clone and its origin is
"referenced" by both the clone and the origin, so it is charged to
both the clone''s and origin''s userused at ... properties.  The
additional
space used by mirroring and raid-z applies to all blocks in the
pool[*], and is not charged anywhere (except by /sbin/zpool).

--matt

[*] Assuming you are using the recommended configuration of all the
same type of top-level vdevs; if you are not then there''s no control
over which blocks go to which vdevs.

>
>What problem are you trying to solve?  How would you want referenced or
>userused at ... to work?
>
>To be more clear:  space shared between a clone and its origin is
>"referenced" by both the clone and the origin, so it is charged to
both the
>clone''s and origin''s userused at ... properties.  The
additional space used by
>mirroring and raid-z applies to all blocks in the pool[*], and is not
charged
>anywhere (except by /sbin/zpool).
>
>--matt
>
>[*] Assuming you are using the recommended configuration of all the same
>type of top-level vdevs; if you are not then there''s no control
over which
>blocks go to which vdevs.
>
There is no specific problem to resolve. Just want to get sort of accurate
equation between
the "raw storage size" and the "usable storage size"
although the *meta file* size is trivial.
If you do mass storage budget, this equation is meaningful.

Thanks.

Fred

2012-05-03 3:07, Fred Liu wrote:
> There is no specific problem to resolve. Just want to get sort of accurate
equation between
> the "raw storage size" and the "usable storage size"
although the *meta file* size is trivial.
> If you do mass storage budget, this equation is meaningful.
I don''t think "accurate equations" are applicable in this
case.
You can have estimates like "no more/no less than X" based on,
basically, level of redundancy and its overhead. ZFS metadata
overhead can also be smaller or bigger, depending on your data''s
typical block size (fixed for zvols at creation time, variable
for files); i.e. if your data is expected to be in very small
pieces (comparable to single sector size), you''d have big
overhead due to required redundancy and metadata. For data
in large chunks overheads would be smaller.

This gives you something like "available space won''t be smaller
than M disks from my M+N redundant raidzN arrays minus O percent
for metadata."

You can also constrain these estimates'' range by other
assumptions like expected dedup or compression ratios,
and hope that your end-users would be able to stuff even
more of their addressable data into the pool (because it
would be sparse, compressable, and/or not unique), but
in the end that''s unpredictable from the start.

HTH,
//Jim

> 
> I don''t think "accurate equations" are applicable in
this case.
> You can have estimates like "no more/no less than X" based on,
> basically, level of redundancy and its overhead. ZFS metadata
> overhead can also be smaller or bigger, depending on your data''s
> typical block size (fixed for zvols at creation time, variable
> for files); i.e. if your data is expected to be in very small
> pieces (comparable to single sector size), you''d have big
> overhead due to required redundancy and metadata. For data
> in large chunks overheads would be smaller.
> 
> This gives you something like "available space won''t be
smaller
> than M disks from my M+N redundant raidzN arrays minus O percent
> for metadata."
> 
> You can also constrain these estimates'' range by other
> assumptions like expected dedup or compression ratios,
> and hope that your end-users would be able to stuff even
> more of their addressable data into the pool (because it
> would be sparse, compressable, and/or not unique), but
> in the end that''s unpredictable from the start.
> 
Totally agree. 
We have the similar experimental practice.
It varies case by case.

Thanks.

Fred