thr3ads.net - zfs discuss - [zfs-discuss] Disk space usage of zfs snapshots and filesystems

If this information is useful, please help other people find it:
Share via:

John.Stewart at emerson.com

2008-Nov-06 16:52 UTC

[zfs-discuss] Disk space usage of zfs snapshots and filesystems - my math doesn''t add up

We''re running a Thumper with Solaris 10_u4 with 127112-11 kernel patch
in production as our mail CIFS/NFS file server. We have a big zpool
consisting of 6 raidz2 groups.
 
We have quite a few filesystems underneath that. We use TIm Foster''s
automatic snapshot service, version 0.10, to do regular snapshots of all
filesystems.
 
Over the weekend, we hit 90% space usage (which, it seemed also caused
some serious NFS and performance issues, but that''s not why
I''m
writing)... the reason I''m writing is that It seems we''re
missing some
terabytes of storage.
 
Here''s how I gathered my data:
 
First, I gathered the used and referenced for all snapshots and
filesystems:

	zfs get -Hpr used,referenced vol0 > vol0_all_used_referenced

Then I chunked this into four seperate files:

	grep snap vol0_all_used_referenced | grep used > vol0_snap_used
	grep snap vol0_all_used_referenced | grep referenced >
vol0_snap_referenced
	grep -v snap vol0_all_used_referenced | grep used > vol0_fs_used
	grep -v snap vol0_all_used_referenced | grep referenced >
vol0_fs_referenced

I pulled these files into Excel. I summed all of the snapshot "used"
fields, and get 0.75TB. The sum of all of the filesystem "referenced"
fields is 2.51TB, for a total of 3.26TB. However, the vol0 "used"
numer
shows 5.04TB.
 
Snapshot Used Total (GB) == 766.73
Filesystem Referenced Total (GB) == 2570.04
Total of Snap Used + FS Ref (GB) == 3336.76

Vol0 filesystem Used (GB) == 5159.35

Where is my missing disk space? Or am I misunderstanding something as
far as how this data is reported?
 
I''ve got an open case with Sun on this... but so far the tech has not
been able to explain either where this space has gone, or what I''ve
done
wrong in my methodology to build this data.
 
Can anyone here shed light?
 
thank you!
 
johnS

Mark J Musante

2008-Nov-06 19:50 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems - my math doesn''t add up

Hi John,
> Snapshot Used Total (GB) == 766.73
> Filesystem Referenced Total (GB) == 2570.04
> Total of Snap Used + FS Ref (GB) == 3336.76
>
> Vol0 filesystem Used (GB) == 5159.35
The sums don''t really work that way.

Consider this scenario:
- I create a dataset and copy 1g into it
- I take a snapshot of it, @snap1
- I copy 1g more data into the datset
- I take another snapshot, @snap2
- I delete the two 1g files.

What''s left on the system?  The first file is available via both the 
snapshots, and the second file will be available via the second snapshot 
only.  In other words, @snap1 will have a ''refer'' value of 1g,
and @snap2
will have a ''refer'' value of 2g.  The dataset itself will only
''refer'' to
the overhead, but will have a ''used'' value of 2g.

However, the ''used'' values of @snap1 and @snap2 will only
contain the
deltas for the snapshots.  @snap1 will contain just the filesystem 
metadata for the ''used'' - around 16-20k, and @snap2 will
contain the
metadata plus the second 1g file.

So, crunching the numbers using the method outlined above, the snapshot 
used total is approx 1g, and the filesystem refer total is 16-20k.  These 
don''t add up to the amount of data still being consumed by the pool
(the
two 1g files), because used & refer are tracking different pieces of 
information.


Regards,
markm

John.Stewart at emerson.com

2008-Nov-06 21:46 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems -my math doesn''t add up

Mark - thank you very much for your explanation and example. I really
appreciate. Any ranting below is directed at ZFS, not you. =)
> > Snapshot Used Total (GB) == 766.73
> > Filesystem Referenced Total (GB) == 2570.04 Total of Snap Used + FS 
> > Ref (GB) == 3336.76
> >
> > Vol0 filesystem Used (GB) == 5159.35
> 
> The sums don''t really work that way.
I followed your recipe, except using a 70MB tarfile instead of a 1GB
file:
> Consider this scenario:
> - I create a dataset and copy 1g into it
titan[root]:/> zfs create vol0/testfs

titan[root]:/> zfs list vol0/testfs
NAME          USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs  52.3K  1.58T  52.3K  /vol0/testfs

titan[root]:/> cd /vol0/testfs
titan[root]:/vol0/testfs> tar cvf test.tar /groups/eng_staff

titan[root]:/vol0/testfs> zfs list vol0/testfs
NAME          USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs  72.4M  1.58T  72.4M  /vol0/testfs
> - I take a snapshot of it, @snap1
titan[root]:/vol0/testfs> zfs snapshot vol0/testfs at snap1

titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs        72.4M  1.58T  72.4M  /vol0/testfs
vol0/testfs at snap1      0      -  72.4M  -
> - I copy 1g more data into the datset
titan[root]:/vol0/testfs> tar cvf test2.tar /groups/eng_staff

titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs         145M  1.58T   145M  /vol0/testfs
vol0/testfs at snap1  48.0K      -  72.4M  -

titan[root]:/vol0/testfs> du -sh .
 145M   .
> - I take another snapshot, @snap2
titan[root]:/vol0/testfs> zfs snapshot vol0/testfs at snap2

titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs         145M  1.58T   145M  /vol0/testfs
vol0/testfs at snap1  48.0K      -  72.4M  -
vol0/testfs at snap2      0      -   145M  -
> - I delete the two 1g files.
titan[root]:/vol0/testfs> rm test.tar test2.tar
titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs         145M  1.58T   145M  /vol0/testfs
vol0/testfs at snap1  48.0K      -  72.4M  -
vol0/testfs at snap2      0      -   145M  -

After a minute:

titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs         145M  1.58T  52.3K  /vol0/testfs
vol0/testfs at snap1  48.0K      -  72.4M  -
vol0/testfs at snap2  72.4M      -   145M  -

 > What''s left on the system?  The first file is available via 
> both the snapshots, and the second file will be available via 
> the second snapshot only.  In other words, @snap1 will have a 
> ''refer'' value of 1g, and @snap2 will have a
''refer'' value of
> 2g.  The dataset itself will only ''refer'' to the
overhead,
> but will have a ''used'' value of 2g.
And you''re completely correct!
> However, the ''used'' values of @snap1 and @snap2 will only
> contain the deltas for the snapshots.  @snap1 will contain 
> just the filesystem metadata for the ''used'' - around
16-20k,
> and @snap2 will contain the metadata plus the second 1g file.
> 
> So, crunching the numbers using the method outlined above, 
> the snapshot used total is approx 1g, and the filesystem 
> refer total is 16-20k.  These don''t add up to the amount of 
> data still being consumed by the pool (the two 1g files), 
> because used & refer are tracking different pieces of information.
Right, so I follow everything you have said here... but I guess I''m
still at a loss.

I think based on this, there really isn''t a good way to know how much
space is being consumed by snapshots? 

The only way I can think of based on this is to use "zfs list" or
"zfs
get" to get the "referenced" numbers, ONLY for filesystems
(annoyingly,
"zfs get" doesn''t seem to have a -t flag). Then I have to add
up all of
the referenced numbers.

Only THEN can I subtract that number from the "used" number for the
entire volume to find out how much is being consumed by snapshots.

Is there any other way?

Then back to the issue at hand... which was the point of this whole
exercise:

How do I figure out how much disk space is being used by a given
snapshot, or group of snapshots?

This all of a sudden feels like a huge limitation of ZFS. I''m already
dealing with the annoyance of not having quotas... now I feel more at
sea without a rudder given that I can''t even tell what disk space is
being used by my snapshots!

thanks

johnS

Romain Chatelain

2008-Nov-06 23:02 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

Hi John,

You should take a look at this :

http://www.opensolaris.org/os/community/zfs/version/13/

There is some improve on zfs v13 on this.

(Please, Excuse my poor English)

Regards

-----Message d''origine-----
De?: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at
opensolaris.org] De la part de John.Stewart at emerson.com
Envoy??: jeudi 6 novembre 2008 22:47
??: zfs-discuss at opensolaris.org
Cc?: Mark.Musante at Sun.COM
Objet?: Re: [zfs-discuss] Disk space usage of zfs snapshots and filesystems-my
math doesn''t add up


Mark - thank you very much for your explanation and example. I really
appreciate. Any ranting below is directed at ZFS, not you. =)
> > Snapshot Used Total (GB) == 766.73
> > Filesystem Referenced Total (GB) == 2570.04 Total of Snap Used + FS 
> > Ref (GB) == 3336.76
> >
> > Vol0 filesystem Used (GB) == 5159.35
> 
> The sums don''t really work that way.
I followed your recipe, except using a 70MB tarfile instead of a 1GB
file:
> Consider this scenario:
> - I create a dataset and copy 1g into it
titan[root]:/> zfs create vol0/testfs

titan[root]:/> zfs list vol0/testfs
NAME          USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs  52.3K  1.58T  52.3K  /vol0/testfs

titan[root]:/> cd /vol0/testfs
titan[root]:/vol0/testfs> tar cvf test.tar /groups/eng_staff

titan[root]:/vol0/testfs> zfs list vol0/testfs
NAME          USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs  72.4M  1.58T  72.4M  /vol0/testfs
> - I take a snapshot of it, @snap1
titan[root]:/vol0/testfs> zfs snapshot vol0/testfs at snap1

titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs        72.4M  1.58T  72.4M  /vol0/testfs
vol0/testfs at snap1      0      -  72.4M  -
> - I copy 1g more data into the datset
titan[root]:/vol0/testfs> tar cvf test2.tar /groups/eng_staff

titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs         145M  1.58T   145M  /vol0/testfs
vol0/testfs at snap1  48.0K      -  72.4M  -

titan[root]:/vol0/testfs> du -sh .
 145M   .
> - I take another snapshot, @snap2
titan[root]:/vol0/testfs> zfs snapshot vol0/testfs at snap2

titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs         145M  1.58T   145M  /vol0/testfs
vol0/testfs at snap1  48.0K      -  72.4M  -
vol0/testfs at snap2      0      -   145M  -
> - I delete the two 1g files.
titan[root]:/vol0/testfs> rm test.tar test2.tar
titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs         145M  1.58T   145M  /vol0/testfs
vol0/testfs at snap1  48.0K      -  72.4M  -
vol0/testfs at snap2      0      -   145M  -

After a minute:

titan[root]:/vol0/testfs> zfs list -r vol0/testfs
NAME                USED  AVAIL  REFER  MOUNTPOINT
vol0/testfs         145M  1.58T  52.3K  /vol0/testfs
vol0/testfs at snap1  48.0K      -  72.4M  -
vol0/testfs at snap2  72.4M      -   145M  -

 > What''s left on the system?  The first file is available via 
> both the snapshots, and the second file will be available via 
> the second snapshot only.  In other words, @snap1 will have a 
> ''refer'' value of 1g, and @snap2 will have a
''refer'' value of
> 2g.  The dataset itself will only ''refer'' to the
overhead,
> but will have a ''used'' value of 2g.
And you''re completely correct!
> However, the ''used'' values of @snap1 and @snap2 will only
> contain the deltas for the snapshots.  @snap1 will contain 
> just the filesystem metadata for the ''used'' - around
16-20k,
> and @snap2 will contain the metadata plus the second 1g file.
> 
> So, crunching the numbers using the method outlined above, 
> the snapshot used total is approx 1g, and the filesystem 
> refer total is 16-20k.  These don''t add up to the amount of 
> data still being consumed by the pool (the two 1g files), 
> because used & refer are tracking different pieces of information.
Right, so I follow everything you have said here... but I guess I''m
still at a loss.

I think based on this, there really isn''t a good way to know how much
space is being consumed by snapshots? 

The only way I can think of based on this is to use "zfs list" or
"zfs
get" to get the "referenced" numbers, ONLY for filesystems
(annoyingly,
"zfs get" doesn''t seem to have a -t flag). Then I have to add
up all of
the referenced numbers.

Only THEN can I subtract that number from the "used" number for the
entire volume to find out how much is being consumed by snapshots.

Is there any other way?

Then back to the issue at hand... which was the point of this whole
exercise:

How do I figure out how much disk space is being used by a given
snapshot, or group of snapshots?

This all of a sudden feels like a huge limitation of ZFS. I''m already
dealing with the annoyance of not having quotas... now I feel more at
sea without a rudder given that I can''t even tell what disk space is
being used by my snapshots!

thanks

johnS
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

none

2008-Nov-06 23:02 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems -my math doesn''t add up

Hi,
My problem is very similar. I have a bunch of snapshots referring to old data
which has been deleted from the current filesystem and I''d like to find
out which snapshots refer to how much data, and how they are shared amongst the
snapshots and the current filesystem. This is so I can work out which snapshots
I''d need to delete to regain free disk space. Ie, trying not to delete
snapshots that would only free up a few GB since they share most of their data
with the current filesystem. The documentation doesn''t seem to say
anything about querying this information.

Thanks,
Mark
-- 
This message posted from opensolaris.org

John.Stewart at emerson.com

2008-Nov-06 23:03 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

> I think based on this, there really isn''t a good way to know 
> how much space is being consumed by snapshots? 
This perl script hacked together by a colleague, given a zfs filesystem,
will tell you how much space is used by the files, and how much by the
snapshots. This has proven helpful in tracking down some hogs.

It''s still not useful for my core issue: How do I tell how much disk
space would be freed if I nuke a snapshot?

But I thought I''d post it in case others find it useful.

Example outputs:

titan[root]:/> snapsize vol0

Space utilization for volume vol0

   4421278 MB Total Used
   2119551 MB Used by Files
   2301727 MB Used by Snapshots



titan[root]:/> snapsize vol0/groups

Space utilization for volume vol0/groups

   2127068 MB Total Used
    623498 MB Used by Files
   1503570 MB Used by Snapshots

johnS



#!/bin/perl
########################################################################
#############################
# snapsize
#
# This script compares the disk space used by files & overhead and the
total space used
# to calculate the space used by snapshots for a given volume.
#
# usage: snapsize <volname>
#
# History
#
------------------------------------------------------------------------
--------------------------
# 06-Nov-2008 Adam Bodette - Initial creation
# 06-Nov-2008 John Stewart - Change output to megabytes
# 06-Nov-2008 Adam Bodette - Changed to accept volume name from input
and removed hardcodes
# 06-Nov-2008 Adam Bodette - Cleaned up output and finished commenting
code
#
########################################################################
#############################

###################################
# read in the volume to report on #
###################################

$vol = $ARGV[0];

#############################
# verify a volume was given #
#############################

if ("$ARGV[0]" eq "") {
  die "\nusage: snapsize <volume>\n\n";
}

#######################################
# grab total space used by the volume #
#######################################

$total = int(`zfs get -Hp -o value used $vol` / 1024) or die "\n$vol is
not a valid zfs volume\n\n";

#####################################################################
# grab the total space used by files and overhead as reported by df #
#####################################################################

$fs = `df -k \| grep $vol \| awk \''\{s \+\=\$3\} END \{print
s\}\''`;

##############################
# Convert sizes to Megabytes #
##############################

$total = int($total / 1024);
$fs = int($fs / 1024);

########################################################################
##########
# subtract space used by files from total space used to calculate
snapshot space #
########################################################################
##########

$snap = $total - $fs;

#################################
# print out the storage results #
#################################

print "\nSpace utilization for volume $vol\n\n";
printf(''%10s'',$total);
print " MB Total Used\n";
printf(''%10s'',$fs);
print " MB Used by Files\n";
printf(''%10s'',$snap);
print " MB Used by Snapshots\n\n";

################
# exit happily #
################

exit;

none

2008-Nov-06 23:59 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

Hi,
Had a look at the new additions from zpool version 13 but I still don''t
think it will give me the information I was after.
Here is what I have gathered, can someone correct me if I''m wrong?

1. The USED property for a snapshot only includes the data that is UNIQUE to
that snapshot.
2. The USED property for a filesystem includes all data that is SHARED amongst
two or more of either the filesystem itself or its child snapshots.
3. There is no way to tell how much data would be freed by deleting a particular
set of (one or more) snapshots, without actually deleting that set of snapshots.
Or in other words, there is no way to tell how much data is shared amongst a
particular set of snapshots but not shared with the current filesystem.

Maybe an incremental zfs send could be used by counting the number of bytes
produced but that is only between two snapshots and is awkward.
-- 
This message posted from opensolaris.org

Miles Nordin

2008-Nov-07 18:46 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems -my math doesn''t add up

>>>>> "n" == none  <mark16829 at gmail.com>
writes:
     n> snapshots referring to old data which has been deleted from
     n> the current filesystem and I''d like to find out which
     n> snapshots refer to how much data

Imagine you have a filesystem containing ten 1MB files,

  zfs create root/export/home/carton/t
  cd t
  n=0; while [ $n -lt 10 ]; do mkfile 1m $n; n=$(( $n + 1 )); done

and you change nothing on the filesystem except to slowly delete one
file at a time.  After you delete each of them, you take another
snapshot.

  n=0; while [ $n -lt 10 ]; do 
     zfs snapshot root/export/home/carton/t@$n; 
     rm $n; 
     n=$(( $n + 1 )); 
  done

When they''re all gone, you stop. Now you want to know, destroying
which snapshot can save you space.

If you think about how you got the filesystem to this state, the ideal
answer is, only deleting the oldest snapshot will save space.
Deleting any middle snapshot saves no space.  Here''s what ''zfs
list''
says:

bash-3.2# zfs list -r root/export/home/carton/t
NAME                          USED  AVAIL  REFER  MOUNTPOINT
root/export/home/carton/t    10.2M  26.6G    18K  /export/home/carton/t
root/export/home/carton/t at 0  1.02M      -  10.0M  -
root/export/home/carton/t at 1    18K      -  9.04M  -
root/export/home/carton/t at 2    18K      -  8.04M  -
root/export/home/carton/t at 3    18K      -  7.03M  -
root/export/home/carton/t at 4    17K      -  6.03M  -
root/export/home/carton/t at 5    17K      -  5.03M  -
root/export/home/carton/t at 6    17K      -  4.03M  -
root/export/home/carton/t at 7    17K      -  3.02M  -
root/export/home/carton/t at 8    17K      -  2.02M  -
root/export/home/carton/t at 9    17K      -  1.02M  -

now, uselessly destroy a middle snapshot:

bash-3.2# zfs destroy root/export/home/carton/t at 3
bash-3.2# zfs list -r root/export/home/carton/t
NAME                          USED  AVAIL  REFER  MOUNTPOINT
root/export/home/carton/t    10.2M  26.6G    18K  /export/home/carton/t
root/export/home/carton/t at 0  1.02M      -  10.0M  -
root/export/home/carton/t at 1    18K      -  9.04M  -
root/export/home/carton/t at 2    18K      -  8.04M  -
root/export/home/carton/t at 4    17K      -  6.03M  -
root/export/home/carton/t at 5    17K      -  5.03M  -
root/export/home/carton/t at 6    17K      -  4.03M  -
root/export/home/carton/t at 7    17K      -  3.02M  -
root/export/home/carton/t at 8    17K      -  2.02M  -
root/export/home/carton/t at 9    17K      -  1.02M  -

still using 10MB.  but if you destroy the oldest snapshot:

bash-3.2# zfs destroy root/export/home/carton/t at 0
bash-3.2# zfs list -r root/export/home/carton/t
NAME                          USED  AVAIL  REFER  MOUNTPOINT
root/export/home/carton/t    9.17M  26.6G    18K  /export/home/carton/t
root/export/home/carton/t at 1  1.02M      -  9.04M  -
root/export/home/carton/t at 2    18K      -  8.04M  -
root/export/home/carton/t at 4    17K      -  6.03M  -
root/export/home/carton/t at 5    17K      -  5.03M  -
root/export/home/carton/t at 6    17K      -  4.03M  -
root/export/home/carton/t at 7    17K      -  3.02M  -
root/export/home/carton/t at 8    17K      -  2.02M  -
root/export/home/carton/t at 9    17K      -  1.02M  -

space is freed.  However, there are other cases where deleting a
middle snapshot will save space:

bash-3.2# mkfile 1m 0
bash-3.2# zfs snapshot root/export/home/carton/t at 0
bash-3.2# mkfile 1m 1
bash-3.2# zfs snapshot root/export/home/carton/t at 1
bash-3.2# rm 1
bash-3.2# zfs snapshot root/export/home/carton/t at 2
bash-3.2# rm 0
bash-3.2# zfs list -r root/export/home/carton/t
NAME                          USED  AVAIL  REFER  MOUNTPOINT
root/export/home/carton/t    2.07M  26.6G    18K  /export/home/carton/t
root/export/home/carton/t at 0    17K      -  1.02M  -
root/export/home/carton/t at 1  1.02M      -  2.02M  -
root/export/home/carton/t at 2    17K      -  1.02M  -
bash-3.2# zfs destroy root/export/home/carton/t at 1
bash-3.2# zfs list -r root/export/home/carton/t
NAME                          USED  AVAIL  REFER  MOUNTPOINT
root/export/home/carton/t    1.05M  26.6G    18K  /export/home/carton/t
root/export/home/carton/t at 0    17K      -  1.02M  -
root/export/home/carton/t at 2    17K      -  1.02M  -

To me, it looks as though the USED column tells you ``how much space
would be freed if I destroyed the thing on this row,'''' and the
REFER
column tells you ``about what would ''du -s'' report if I ran it
on the
root of this dataset.''''  The answer to your question is then
to look
at the USED column, but with one caveat: after you delete something,
the USED column will reshuffle, so it''s useful for deciding the next
single thing you want to destroy, not for choosing a set of things for
a multiple-thing destruction-plan.  In the first example, USED
couldn''t tell you before you''ve destroyed anything that
``deleting
snapshots @0 - @4 will save 5MB,'''' but the column will keep
guiding
you to the oldest snapshot as the next helpful one to destroy.

Obviously I can''t prove that the columns always behave this way, but
they behaved consistently and reasonably in these two simple tests.
Also I did not look at changes in pool-wide free space.  I trusted the
USED column of ''t''.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081107/6899abb5/attachment.bin>

Adam.Bodette at emerson.com

2008-Nov-07 20:20 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

I really think there is something wrong with how space is being reported
by zfs list in terms of snapshots.

Stealing for the example earlier where a new file system was created, 10
1MB files were created and then do snap, remove a file, snap, remove a
file, until they are all gone and you are left with:
 > bash-3.2# zfs list -r root/export/home/carton/t
> NAME                          USED  AVAIL  REFER  MOUNTPOINT
> root/export/home/carton/t    10.2M  26.6G    18K  
> /export/home/carton/t
> root/export/home/carton/t at 0  1.02M      -  10.0M  -
> root/export/home/carton/t at 1    18K      -  9.04M  -
> root/export/home/carton/t at 2    18K      -  8.04M  -
> root/export/home/carton/t at 3    18K      -  7.03M  -
> root/export/home/carton/t at 4    17K      -  6.03M  -
> root/export/home/carton/t at 5    17K      -  5.03M  -
> root/export/home/carton/t at 6    17K      -  4.03M  -
> root/export/home/carton/t at 7    17K      -  3.02M  -
> root/export/home/carton/t at 8    17K      -  2.02M  -
> root/export/home/carton/t at 9    17K      -  1.02M  -
So the file system itself is now empty of files (18K refer for overhead)
but still using 10MB because of the snapshots still holding onto all 10
1MB files.

By how I understand snapshots the oldest one actually holds the file
after it is deleted and any newer snapshot just points to what that
oldest one is holding.

So because the 0 snapshot was taken first it knows about all 10 files,
snap 1 only knows about 9, etc.

The refer numbers all match up correctly as that is how much data
existed at the time of the snapshot.

But the used seems wrong.

The 0 snapshot should be holding onto all 10 files so I would expect it
to be 10MB "Used" when it''s only reporting 1MB used.  Where
is the other
9MB hiding?  It only exists because a snapshot is holding it so that
space should be charged to a snapshot.  Since snapshot 1-9 should only
be pointing at the data held by 0 their numbers are correct.

To take the idea further you can delete snapshots 1-9 and snapshot 0
will still say it has 1MB "Used", so where again is the other 9MB?

Adding up the total "used" by snapshots and the "refer" by
the file
system *should* add up to the "used" for the file system for it all to
make sense right?

Another way to look at it if you have all 10 snapshots and you delete 0
I would expect snapshot 1 to change from 18K used (overhead) to 9MB used
since it would now be the oldest snapshot and official holder of the
data with snapshots 2-9 now pointing at the data it is holding.  The
first 1MB file delete would now be gone forever.

Am I missing something or is the math to account for snapshot space just
not working right in zfs list/get?

Adam

Adam.Bodette at emerson.com

2008-Nov-07 20:59 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots andfilesystems-my math doesn''t add up

I decided to do some more test situations to try to figure out how
adding/removing snapshots changes the space used reporting.

First I setup a test area, a new zfs file system and created some test
files and then created snapshots removing the files one by one.
> mkfile 1m 0
> mkfile 1m 1
> mkfile 1m 2
> mkfile 1m 3
> zfs snapshot u01/foo at 0
> rm 0
> zfs snapshot u01/foo at 1
> rm 1                  
> zfs snapshot u01/foo at 2
> rm 2
> zfs snapshot u01/foo at 3
> rm 3
> zfs list -r u01/foo	NAME        USED  AVAIL  REFER  MOUNTPOINT
	u01/foo    4.76M  1.13T  55.0K  /u01/foo
	u01/foo at 0  1.18M      -  4.56M  -
	u01/foo at 1  50.5K      -  3.43M  -
	u01/foo at 2  50.5K      -  2.31M  -
	u01/foo at 3  50.5K      -  1.18M  -


So 4M used on the file system but only 1M used by the snapshots so they
claim.

If I delete @1
> zfs destroy u01/foo at 1
> zfs list -r u01/foo  	NAME        USED  AVAIL  REFER  MOUNTPOINT
	u01/foo    4.71M  1.13T  55.0K  /u01/foo
	u01/foo at 0  2.30M      -  4.56M  -
	u01/foo at 2  50.5K      -  2.31M  -
	u01/foo at 3  50.5K      -  1.18M  -

now suddenly the @0 snapshot is claiming to use more space?

If I delete the newest of the snapshots @3
> zfs destroy u01/foo at 3
> zfs list -r u01/foo	NAME        USED  AVAIL  REFER  MOUNTPOINT
	u01/foo    4.66M  1.13T  55.0K  /u01/foo
	u01/foo at 0  2.30M      -  4.56M  -
	u01/foo at 2  50.5K      -  2.31M  -

No change in the claimed used space by the @0 snapshot!

Now I delete the @2 snapshot
> zfs destroy u01/foo at 2
> zfs list -r u01/foo  	NAME        USED  AVAIL  REFER  MOUNTPOINT
	u01/foo    4.61M  1.13T  55.0K  /u01/foo
	u01/foo at 0  4.56M      -  4.56M  -

The @0 snapshot finally claims all the space it''s really been holding
all along.

Up until that point other than subtracting space used (as reported by
df) from the used for the file system as reported by zfs there was no
way to know how much space the snapshots were really using.

Something is not right in the space accounting for snapshots.

---
Adam 
> -----Original Message-----
> From: zfs-discuss-bounces at opensolaris.org 
> [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of 
> Adam.Bodette at emerson.com
> Sent: Friday, November 07, 2008 14:21
> To: carton at Ivy.NET; zfs-discuss at opensolaris.org
> Subject: Re: [zfs-discuss] Disk space usage of zfs snapshots 
> andfilesystems-my math doesn''t add up
> 
> I really think there is something wrong with how space is 
> being reported
> by zfs list in terms of snapshots.
> 
> Stealing for the example earlier where a new file system was 
> created, 10
> 1MB files were created and then do snap, remove a file, snap, remove a
> file, until they are all gone and you are left with:
>  
> > bash-3.2# zfs list -r root/export/home/carton/t
> > NAME                          USED  AVAIL  REFER  MOUNTPOINT
> > root/export/home/carton/t    10.2M  26.6G    18K  
> > /export/home/carton/t
> > root/export/home/carton/t at 0  1.02M      -  10.0M  -
> > root/export/home/carton/t at 1    18K      -  9.04M  -
> > root/export/home/carton/t at 2    18K      -  8.04M  -
> > root/export/home/carton/t at 3    18K      -  7.03M  -
> > root/export/home/carton/t at 4    17K      -  6.03M  -
> > root/export/home/carton/t at 5    17K      -  5.03M  -
> > root/export/home/carton/t at 6    17K      -  4.03M  -
> > root/export/home/carton/t at 7    17K      -  3.02M  -
> > root/export/home/carton/t at 8    17K      -  2.02M  -
> > root/export/home/carton/t at 9    17K      -  1.02M  -
> 
> So the file system itself is now empty of files (18K refer 
> for overhead)
> but still using 10MB because of the snapshots still holding 
> onto all 10
> 1MB files.
> 
> By how I understand snapshots the oldest one actually holds the file
> after it is deleted and any newer snapshot just points to what that
> oldest one is holding.
> 
> So because the 0 snapshot was taken first it knows about all 10 files,
> snap 1 only knows about 9, etc.
> 
> The refer numbers all match up correctly as that is how much data
> existed at the time of the snapshot.
> 
> But the used seems wrong.
> 
> The 0 snapshot should be holding onto all 10 files so I would 
> expect it
> to be 10MB "Used" when it''s only reporting 1MB used. 
Where
> is the other
> 9MB hiding?  It only exists because a snapshot is holding it so that
> space should be charged to a snapshot.  Since snapshot 1-9 should only
> be pointing at the data held by 0 their numbers are correct.
> 
> To take the idea further you can delete snapshots 1-9 and snapshot 0
> will still say it has 1MB "Used", so where again is the other
9MB?
> 
> Adding up the total "used" by snapshots and the "refer"
by the file
> system *should* add up to the "used" for the file system for it
all to
> make sense right?
> 
> Another way to look at it if you have all 10 snapshots and 
> you delete 0
> I would expect snapshot 1 to change from 18K used (overhead) 
> to 9MB used
> since it would now be the oldest snapshot and official holder of the
> data with snapshots 2-9 now pointing at the data it is holding.  The
> first 1MB file delete would now be gone forever.
> 
> Am I missing something or is the math to account for snapshot 
> space just
> not working right in zfs list/get?
> 
> Adam
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

none

2008-Nov-07 23:06 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems -my math doesn''t add up

Hi Miles,
Thanks for your reply.
My zfs situation is a little different from your test. I have a few early
snapshots which I believe would still be sharing most of their data with the
current filesystem. Then later I have snapshots which would still be holding
onto data that is now deleted.

So I''d like to get more detailed reports on just what and how much of
the data is shared between the snapshots and current filesystem, rather than
relying on my memory. For example, a report showing for all data blocks where
they are shared would be what I really need:
10% unique to storage/fs
20% shared by storage/fs, storagefs at 1, storagefs at 2
40% shared by storagefs at 1, storagefs at 3
10% unique to storagefs at 1
10% unique to storagefs at 2
10% unique to storagefs at 3


Unfortunately the USED column is of little help since it only shows you the data
unique to that snapshot. In my case almost all data is shared amongst the
snapshots so it only shows up in the USED of the whole fs.
-- 
This message posted from opensolaris.org

Miles Nordin

2008-Nov-07 23:58 UTC

head link

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

>>>>> "a" == <Adam.Bodette at emerson.com>
writes:
>>>>> "c" == Miles Nordin <carton at Ivy.NET>
writes:
>>>>> "n" == none <mark16829 at gmail.com>
writes:
n> Unfortunately the USED column is of little help since it only
n> shows you the data unique to that snapshot. In my case almost
n> all data is shared amongst the snapshots so it only shows up
n> in the USED of the whole fs.

Unfortunately sounds right to me.

bash-3.2# mkfile 1m 0
bash-3.2# zfs snapshot root/export/home/carton/t at 0
bash-3.2# zfs snapshot root/export/home/carton/t at 1
bash-3.2# rm 0
bash-3.2# zfs list -r root/export/home/carton/t
NAME USED AVAIL REFER MOUNTPOINT
root/export/home/carton/t 1.04M 26.6G 18K /export/home/carton/t
root/export/home/carton/t at 0 0 - 1.02M -
root/export/home/carton/t at 1 0 - 1.02M -

It seems like taking a lot of snapshots can make ''zfs list''
output
less useful.

I''ve trouble imagining an interface to expose the information you
want. The obvious extension of what we have now is ``show
hypothetical ''zfs list'' output after I''d deleted this
dataset,'''' but I
think that''d be clumsy to use and inefficient to implement.

Maybe btrfs will think of something, since AIUI their model is to have
mandatory infinite snapshots, and instead of explicitly requesting
point-in-time snapshots, your manual action is to free up space by
specifying ranges of snapshot you''d like to delete. They''ll
likely
have some interface to answer questions like ``how much space is used
by the log between 2008-01-25 and 2008-02-22?'''', and they will
have
background log-quantitization daemons that delete snapshot ranges
analagous to our daemons that take snapshots.

Maybe imagining infinitely-granular snapshots is the key to a better
interface: ``show me the USED value for snapshots t at 2 - t at 8
inclusive,'''' and you must always give a contiguous range.
That''s
analagous to btrfs-world''s hypothetical ``show me the space consumed
by the log between these two timestamps.''''

the btrfs guy in here seemed to be saying there''s dedup across clone
branches, which ZFS does not do. I suppose that would complicate
space reporting and their interface to do it.

a> I really think there is something wrong with how space is
a> being reported by zfs list in terms of snapshots.

I also had trouble understanding it.

right:

conduct tests and use them to form a rational explanation of the
numbers'' meanings. If you can''t determine one, ask for help
and try
harder. Create experiments to debunk a series of proposed, wrong
explanations. If a right explanation doesn''t emerge, complain
it''s
broken chaos.

wrong:

use the one-word column titles to daydream about what you THINK the
numbers ought to mean if you''d designed it, and act obstinately
surprised when the actual meaning doesn''t match your daydream.

One word isn''t enough to get you and the designers on the same page.
You must mentally prepare yourself to accept column-meanings other
than your assumption.

Imagine the columns were given to you without headings. Or, reread
your ''zfs list'' output assuming the column headings are
ORANGE,
TASMANIAN_DEVIL, and BARBIE, then try to understand what each means
through experiment rather than assumption.

a> But the used seems wrong.

%$"#$%! But I can only repeat myself:

c> USED column tells you ``how much
c> space would be freed if I destroyed the thing on this
row,''''

AND

c> after you delete something, the USED column will
c> reshuffle,

I''d been staring at the USED column for over a year, and did not know
either of those two things until I ran the experiment in my post.

I''m not saying it''s necessarily working properly, but I see no
evidence of a flaw from the numbers in the mail I posted, and in your
writeup I see a bunch of assumptions that are not worth
untangling---paragraphs-long explanations don''t work for everyone,
including me. Adam, suggest you keep doing tests until you understand
the actual behavior, because at least the behavior seen in my post is
understandable and sane to me.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081107/715531f6/attachment.bin>

zfs discuss - Nov 2008 - Disk space usage of zfs snapshots and filesystems - my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems - my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems - my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems -my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems -my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems -my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots andfilesystems-my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems -my math doesn''t add up

[zfs-discuss] Disk space usage of zfs snapshots and filesystems-my math doesn''t add up