freebsd stable - Sep 2008 - snapshots and disk usage

Hi,

I am using ufs snapshots on RELENG_7 for some time now, and am generally
happy with it. I have noticed a strange behaviour when removing large
amount of files, and wanted to ask if this is expected.

Before starting, we check the free space on /usr:

| ice:/usr>df -h .
| Filesystem         Size    Used   Avail Capacity  Mounted on
| /dev/ad4s2.elid    9.7G    7.6G    1.3G    64%    /usr

Then delete /usr/obj and run df again:

| ice:/usr>sudo rm -rf obj 2>/dev/null
| ice:/usr>df -h .
| Filesystem         Size    Used   Avail Capacity  Mounted on
| /dev/ad4s2.elid    9.7G    5.7G    3.2G    64%    /usr

This is unexpected. With snapshots, removing something should not
release space.

Sure enough, in the course of the next minute, the fake free space
vanishes....

| ice:/usr>df -h .
| Filesystem         Size    Used   Avail Capacity  Mounted on
| /dev/ad4s2.elid    9.7G    5.9G    3.0G    66%    /usr
| ice:/usr>df -h .
| Filesystem         Size    Used   Avail Capacity  Mounted on
| /dev/ad4s2.elid    9.7G    6.6G    2.3G    74%    /usr
| ice:/usr>df -h .
| Filesystem         Size    Used   Avail Capacity  Mounted on
| /dev/ad4s2.elid    9.7G    8.6G    269M    97%    /usr

and all the free space is allocated in the snapshot:

| ice:~>sudo snapshot list
| Filesystem          User   User%     Snap   Snap%  Snapshot
| /usr                 8GB   89.3%      2GB   21.5%  daily.1
| /usr                 8GB   89.3%    344MB    3.5%  daily.0
| /usr                 8GB   89.3%    344MB    3.5%  weekly.0
| /usr                 8GB   89.3%    344MB    3.5%  hourly.1
| /usr                 8GB   89.3%      7MB    0.1%  hourly.0

My understanding so far was that df may underreport free space, but i
find overreporting it a bit troublesome. -- What would happen if I tried
to use that space before it was allocated to the snapshot?

Case in point: I created a few unkillable hung process on /usr a few
weeks ago, by running make world which was running out of diskspace, and
deleting files from another windows. -- At leat thats what I think has
happened. 

CU,
    Sec
-- 
Consider the need for having to type "www.domain.name" a little IQ
test
that you have to take before you can access my web site.'     -- Wietse

In the last episode (Sep 07), Stefan `Sec` Zehl said:
> Hi,
> 
> I am using ufs snapshots on RELENG_7 for some time now, and am generally
> happy with it. I have noticed a strange behaviour when removing large
> amount of files, and wanted to ask if this is expected.
> 
> Before starting, we check the free space on /usr:
> 
> | ice:/usr>df -h .
> | Filesystem         Size    Used   Avail Capacity  Mounted on
> | /dev/ad4s2.elid    9.7G    7.6G    1.3G    64%    /usr
> 
> Then delete /usr/obj and run df again:
> 
> | ice:/usr>sudo rm -rf obj 2>/dev/null
> | ice:/usr>df -h .
> | Filesystem         Size    Used   Avail Capacity  Mounted on
> | /dev/ad4s2.elid    9.7G    5.7G    3.2G    64%    /usr
> 
> This is unexpected. With snapshots, removing something should not
> release space.
> 
> Sure enough, in the course of the next minute, the fake free space
> vanishes....
> 
> | ice:/usr>df -h .
> | Filesystem         Size    Used   Avail Capacity  Mounted on
> | /dev/ad4s2.elid    9.7G    5.9G    3.0G    66%    /usr
> | ice:/usr>df -h .
> | Filesystem         Size    Used   Avail Capacity  Mounted on
> | /dev/ad4s2.elid    9.7G    6.6G    2.3G    74%    /usr
> | ice:/usr>df -h .
> | Filesystem         Size    Used   Avail Capacity  Mounted on
> | /dev/ad4s2.elid    9.7G    8.6G    269M    97%    /usr
> 
> and all the free space is allocated in the snapshot:
> 
> | ice:~>sudo snapshot list
> | Filesystem          User   User%     Snap   Snap%  Snapshot
> | /usr                 8GB   89.3%      2GB   21.5%  daily.1
> | /usr                 8GB   89.3%    344MB    3.5%  daily.0
> | /usr                 8GB   89.3%    344MB    3.5%  weekly.0
> | /usr                 8GB   89.3%    344MB    3.5%  hourly.1
> | /usr                 8GB   89.3%      7MB    0.1%  hourly.0
> 
> My understanding so far was that df may underreport free space, but i
> find overreporting it a bit troublesome. -- What would happen if I tried
> to use that space before it was allocated to the snapshot?
I think you're running into the softupdates delay.  When you delete a
file on a SU-enabled filessytem, the space isn't actually freed until
sync.  But applications expect that statfs() info is updated
immediately, so the kernel pretends that the space is available.  That
doesn't really work with a snapshot, since if you delete a file that
existed in the snapshot, no space will free up.  So you see a jump in
freespace as the kernel fakes the f_bfree statfs amount, then it slowly
drops to the correct value as the deletions actually sync to disk.

-- 
	Dan Nelson
	dnelson@allantgroup.com