zfs discuss - Jan 2009 - ZFS+NFS+refquota: full filesystems still return EDQUOT for unlink()

We have been using ZFS for user home directories for a good while now.
 When we discovered the problem with full filesystems not allowing
deletes over NFS, we became very anxious to fix this; our users fill
their quotas on a fairly regular basis, so it''s important that they
have a simple recourse to fix this (e.g., rm).  I played around with
this on my OpenSolaris box at home, read around on mailing lists, and
concluded that the ''refquota'' property would solve this.  With
some
trepidation, we decided at work that we would ignore the problem and
wait for 10u6, at which point we would put the value of the quota
property in the refquota property, and set quota=none.  We did this a
week or so ago, and we''re still having the problem.  Here''s an
example:
(on the client workstation)
willm1 at chasca:~$ dd if=/dev/urandom of=bigfile
dd: closing output file `bigfile'': Disk quota exceeded
willm1 at chasca:~$ rm bigfile
rm: cannot remove `bigfile'': Disk quota exceeded
willm1 at chasca:~$ strace rm bigfile
execve("/bin/rm", ["rm", "bigfile"], [/* 57 vars
*/]) = 0
(...)
access("bigfile", W_OK)                 = 0
unlink("bigfile")                       = -1 EDQUOT (Disk quota
exceeded)
(on the NFS server)
willm1 at c64:~$ rm bigfile
(no error)

This is a big problem.  We don''t want to allow (or force) users to log
in to the NFS server to delete files.  Why is the behavior different
over NFSv3/4 (I tried both, same problem both times) versus locally?
In case it matters, here are the properties of the filesystem above:
# zfs get all home1/willm1
NAME          PROPERTY         VALUE                  SOURCE
home1/willm1  type             filesystem             -
home1/willm1  creation         Mon Jun  2 14:37 2008  -
home1/willm1  used             3.47G                  -
home1/willm1  available        136M                   -
home1/willm1  referenced       3.47G                  -
home1/willm1  compressratio    1.00x                  -
home1/willm1  mounted          yes                    -
home1/willm1  quota            none                   default
home1/willm1  reservation      none                   default
home1/willm1  recordsize       128K                   default
home1/willm1  mountpoint       /export/home1/willm1   inherited from home1
home1/willm1  sharenfs         rw=cmscnet             inherited from home1
home1/willm1  checksum         on                     default
home1/willm1  compression      off                    default
home1/willm1  atime            on                     default
home1/willm1  devices          on                     default
home1/willm1  exec             on                     default
home1/willm1  setuid           on                     default
home1/willm1  readonly         off                    default
home1/willm1  zoned            off                    default
home1/willm1  snapdir          hidden                 default
home1/willm1  aclmode          groupmask              default
home1/willm1  aclinherit       restricted             default
home1/willm1  canmount         on                     default
home1/willm1  shareiscsi       off                    default
home1/willm1  xattr            on                     default
home1/willm1  copies           1                      default
home1/willm1  version          1                      -
home1/willm1  utf8only         off                    -
home1/willm1  normalization    none                   -
home1/willm1  casesensitivity  sensitive              -
home1/willm1  vscan            off                    default
home1/willm1  nbmand           off                    default
home1/willm1  sharesmb         off                    default
home1/willm1  refquota         3.60G                  local
home1/willm1  refreservation   none                   default

Any suggestions are welcome.  If we can''t resolve this we''ll
have to
investigate other options for our home directories; going without
quotas is unacceptable for administrative reasons, and other options
don''t have this problem.

Thanks,
Will

On Jan 28, 2009, at 11:49 AM, Will Murnane wrote:
>
> (on the client workstation)
> willm1 at chasca:~$ dd if=/dev/urandom of=bigfile
> dd: closing output file `bigfile'': Disk quota exceeded
> willm1 at chasca:~$ rm bigfile
> rm: cannot remove `bigfile'': Disk quota exceeded
Will,

   I filed a CR on this (6798878), fix on the way for OpenSolaris.
Can you continue using regular quotas (instead of refquotas)?  Those
don''t suffer from the same issue, although of course you''ll
lose the
refquota functionality.

-Chris

On Wed, Jan 28, 2009 at 19:04, Chris Kirby <chris.kirby at sun.com>
wrote:
> On Jan 28, 2009, at 11:49 AM, Will Murnane wrote:
>
>>
>> (on the client workstation)
>> willm1 at chasca:~$ dd if=/dev/urandom of=bigfile
>> dd: closing output file `bigfile'': Disk quota exceeded
>> willm1 at chasca:~$ rm bigfile
>> rm: cannot remove `bigfile'': Disk quota exceeded
>
> Will,
>
>  I filed a CR on this (6798878), fix on the way for OpenSolaris.
> Can you continue using regular quotas (instead of refquotas)?  Those
> don''t suffer from the same issue
Weird, you''re right.  I was under the impression that quotas suffered
the same issue, and refquota was the *fix* to this issue (c.f. [1]).
Well, live and learn; I''ll switch them back.  Thanks for the quick
fix.

I didn''t see anything about this in the 10u6 release notes [2]; where
could I have found the notification that quotas were behaving nicely
WRT unlink()?
> although of course you''ll lose the
> refquota functionality.
That''s okay, at least for the time being.  I''ll track the CR
for
further details.

Will

[1]: https://kerneltrap.org/mailarchive/freebsd-bugs/2008/3/11/1134694
[2]: http://docs.sun.com/app/docs/doc/817-0547/ghgdx