zfs discuss - Apr 2006 - Can''t remove directory when over quota

I was running some tests on ZFS (what happens when you create a lot of
subdirectories) and filled quota.

[...]
887000
888000
mkdir failed at 888348.  Disc quota exceeded
/rootz/test:%

Fantastic.  Now however, I cannot seem to clean up after myself:

/rootz:% rmdir test/888347
rmdir: directory "test/888347": Disc quota exceeded
/rootz:% rm -rf test
rm: Unable to remove directory test/1: Disc quota exceeded
rm: Unable to remove directory test/116131: Disc quota exceeded
rm: Unable to remove directory test/474688: Disc quota exceeded
[...]

(Yes, the only thing in the filesystem is one directory (test) and 800K
subdirs.)

/rootz:% zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
rootz                 1000M      0   999M  /rootz

Obviously I can increase the quota temporarily to get around this, but what if I
had no quota and had really filled the pool?  Is there something I can do to
clean this up without touching the quota?

This is on x86 with snv_33.

Thanks,

-- 
Darren
 
 
This message posted from opensolaris.org

That seems to be directly related to ZFS''s copy-on-write (COW) scheme.

Do you have any snapshots? If so, it means it''s not deleting that old
data, and is having trouble writing the new block.
 
 
This message posted from opensolaris.org

> Do you have any snapshots? If so, it means it''s not
> deleting that old data, and is having trouble writing
> the new block.
Nope.  Basically a clean pool.  I created the pool, chowned the root-level mount
to a non-root user and ran the script.

Once running, I realized filling the entire pool was going to take *way* too
long, so I set a quota of 1000M.  No snapshots.  The pool itself still has
space.

NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
rootz                  11.7G   1000M   10.7G     8%  ONLINE     -

Maybe there needs to be an overhead buffer?  I''d hate for it to require
administrative intervention for a user that fills a filesystem in this manner.

-- 
Darren
 
 
This message posted from opensolaris.org

Here is what I did:

# zfs create tank/t2
# zfs set quota=1m tank/t2
# cd /tank/t2
# perl -e ''for (1..10000) { mkdir "$_"; }''
# zfs list tank/t2
NAME                   USED  AVAIL  REFER  MOUNTPOINT
tank/t2               1.00M      0  1.00M  /tank/t2
# mkdir 10001
mkdir: Failed to make directory "10001"; Disc quota exceeded
# rmdir 1
# mkdir 1

Everything looks fine on my machine, which is build 36.

Can you provide an example that is a little easier to test than 1000M of
directories?

When I tried with a snapshot on the fs it did indeed fail when removing the
directory. That is somewhat of a "gotcha", but I can''t think
of a better way for ZFS to handle it.
 
 
This message posted from opensolaris.org

> Here is what I did:
[snip]
> # rmdir 1
> # mkdir 1
> 
> Everything looks fine on my machine, which is build
> 36.
> Can you provide an example that is a little easier to
> test than 1000M of directories? 
What, a gig and about 10 minutes?  :-)

No problem, I''ll try that when I get back to the system.  I''ve
left the existing filesystem alone in case there''s any more
investigation that can be done on it.

-- 
Darren
 
 
This message posted from opensolaris.org

On Tue, Apr 25, 2006 at 10:59:41PM -0700, Jeff Davis
wrote:
> When I tried with a snapshot on the fs it did indeed fail when
> removing the directory. That is somewhat of a "gotcha", but I
can''t
> think of a better way for ZFS to handle it.
There can''t be any other way to handle it: with the snapshot in the
picture removing the directory consumes more space, not less, since the
old and new parent directory blocks and the removed directory''s dnode
have to exist on disk and be charged to the filesystem.  To try to fudge
accounting to allow this would provide users with a way to consume much
more storage than their quota allows.

Nico
--

> > Everything looks fine on my machine, which is
> build
> > 36.
 
> > Can you provide an example that is a little easier
> to
> > test than 1000M of directories? 
Same behavior...

center:~> sudo zfs create zfs_test_pool/t2
center:~> sudo zfs set quota=1m zfs_test_pool/t2
center:~> cd /zfs_test_pool/t2/
center:/zfs_test_pool/t2> sudo perl /var/tmp/maxsubdir 
0
mkdir failed at 879.  Disc quota exceeded
center:/zfs_test_pool/t2> sudo rmdir 878
rmdir: directory "878": Disc quota exceeded
center:/zfs_test_pool/t2> sudo rmdir 0
rmdir: directory "0": Disc quota exceeded

I guess the next step would be to see when I can upgrade this to a later build
unless someone can reproduce this on 36....

-- 
Darren
 
 
This message posted from opensolaris.org

> I guess the next step would be to see when I can upgrade this to a later
build unless someone can reproduce this on 36....
I tried this on a system running snv_38.

With a 1m quota (on a 5-wide raidz):

# mkdir $(jot 500)
mkdir: Failed to make directory "422"; Disc quota exceeded
mkdir: Failed to make directory "423"; Disc quota exceeded
...

I could remove directories just fine without needing to up the quota
any.

Did see some slightly surprising behavior when trying to recreate a
successfully deleted directory:

# rmdir 1
# mkdir 1
mkdir: Failed to make directory "1"; Disc quota exceeded

but no doubt that''s an artifact of being so close to the edge..

						- Bill

Hey Darren,
It seems you may be running into this bug:
6364685 freed space is not immediately available
a fix for this was putback into snv_36, so updating your system to  
build 36 or later should make your machine much happier.

Noel :-)


************************************************************************ 
**

"Question all the answers"
On Apr 27, 2006, at 6:06 AM, Bill Sommerfeld wrote:
>> I guess the next step would be to see when I can upgrade this to a  
>> later build unless someone can reproduce this on 36....
>
> I tried this on a system running snv_38.
>
> With a 1m quota (on a 5-wide raidz):
>
> # mkdir $(jot 500)
> mkdir: Failed to make directory "422"; Disc quota exceeded
> mkdir: Failed to make directory "423"; Disc quota exceeded
> ...
>
> I could remove directories just fine without needing to up the quota
> any.
>
> Did see some slightly surprising behavior when trying to recreate a
> successfully deleted directory:
>
> # rmdir 1
> # mkdir 1
> mkdir: Failed to make directory "1"; Disc quota exceeded
>
> but no doubt that''s an artifact of being so close to the edge..
>
> 						- Bill
>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

copy paste has defeated me (thanks Nico!).  Please note the bug number  
should have been
6381209 freed space is not immediately available


Noel :-)


************************************************************************ 
**

"Question all the answers"
On Apr 27, 2006, at 6:06 AM, Bill Sommerfeld wrote:
>> I guess the next step would be to see when I can upgrade this to a  
>> later build unless someone can reproduce this on 36....
>
> I tried this on a system running snv_38.
>
> With a 1m quota (on a 5-wide raidz):
>
> # mkdir $(jot 500)
> mkdir: Failed to make directory "422"; Disc quota exceeded
> mkdir: Failed to make directory "423"; Disc quota exceeded
> ...
>
> I could remove directories just fine without needing to up the quota
> any.
>
> Did see some slightly surprising behavior when trying to recreate a
> successfully deleted directory:
>
> # rmdir 1
> # mkdir 1
> mkdir: Failed to make directory "1"; Disc quota exceeded
>
> but no doubt that''s an artifact of being so close to the edge..
>
> 						- Bill
>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Why does making a snapshot cause the problem still on build 36?

# zfs create tank/t2
# zfs set quota=100k tank/t2
# zfs snapshot foo/bar at snap1
# cd /tank/t2
# perl -e ''for (1..10000) { mkdir "$_"; }''
(above command fills fs up to quota, the last directory created is
"140")
# rmdir 1
rmdir: directory "1": Disc quota exceeded

Since there is no snapshot between the operations "mkdir 140" and
"rmdir 1", why is the quota violated in this case? It
shouldn''t take up more space because it doesn''t hold more
information as far as I can tell: after "rmdir 1" the state should be
just as if the directory "1" never existed. Or, if it does take up
more space temporarily, what does that have to do with the snapshot created at
the very beginning? If I don''t make any snapshots, or if I destroy the
snapshots, it works as expected.

I would think that a snapshot of an empty filesystem would be pretty much
entirely independent in the on-disk tree after the filesystem filled up. Nearly
all of the blocks would be already copied and no longer shared, and therefore
not subject to COW making more copies, right?

Regards,
    Jeff Davis
 
 
This message posted from opensolaris.org

On Fri, Apr 28, 2006 at 07:11:21PM -0700, Jeff Davis
wrote:
> Why does making a snapshot cause the problem still on build 36?
Having a snapshot causes a problem because of the details of how we
implement file (and directory) deletion in ZFS.  In particular, we might
have to put the file (or directory) onto an on-disk "delete queue" for
later processing (eg. if the file is currently open).  The delete queue
data structure *is* part of the snapshot, so we will end up using more
net space, at least temporarily.  Currently, we use the delete queue
even when it is not strictly necessary, like in ''rmdir'' case
that you
are testing, causing this unexpected behavior.

You should note that even with no snapshots, you will not be able to
rmdir a directory on a full filesystem if that filesystem is in a
multi-block[*] directory.  In multi-block directories, it is nontrivial
to determine which blocks will need to be modified to record the removal
of a given directory entry, so we don''t currently attempt that.

In general, once you are out of space (due to a quota or pool-wide space
exaustion), you can only remove files that are larger than some minimum
size.  Smaller files and metadata operations will fail because we make
some overly-conservative estimates of the amount of space that will be
needed.

We''re working on these problems. The excessive delete queue usage, and
a
great number of these estimates will be tightened down as part of bug id
6362156, which is expected to be fixed in the next few months.

--matt

[*] Multi-block directories are implemented using the "fat ZAP" (as
opposed to the "micro ZAP" which uses only a single block).  The fat
zap
will be used if there are more than 2000 directory entries, or if any
entry name is longer than 49 bytes.

On Mon, May 01, 2006 at 02:43:00PM -0700, Matthew Ahrens
wrote:
> On Fri, Apr 28, 2006 at 07:11:21PM -0700, Jeff Davis wrote:
> > Why does making a snapshot cause the problem still on build 36?
> > 
> > # zfs create tank/t2
> > # zfs set quota=100k tank/t2
> > # zfs snapshot foo/bar at snap1
> > # cd /tank/t2
> > # perl -e ''for (1..10000) { mkdir "$_"; }''
> Having a snapshot causes a problem because of the details of how we
> implement file (and directory) deletion in ZFS.  In particular, we might
> have to put the file (or directory) onto an on-disk "delete
queue" for
> later processing
Actually, I just noticed that in your particular test case, we will also
have to modify the filesystem''s root directory''s link count
when you do
the ''rmdir''.  Since the root directory is part of the
snapshot, this
will result in a net increase in the amount of space used.

Since your particular test does in fact use more space after the
''rmdir''
(and this is not due to an unnecessary artifact of ZFS''s
implementaion),
it is correct for it to fail.

To isolate the delete queue problem, you can populate a subdirectory of
the filesystem''s root directory:

	# zfs create tank/t2
	# zfs set quota=100k tank/t2
	# zfs snapshot foo/bar at snap1
	# mkdir /tank/t2/dir
	# cd /tank/t2/dir
	# perl -e ''for (1..10000) { mkdir "$_"; }''

--matt