zfs discuss - Apr 2010 - Large size variations - what is canonical method

Seems like you can get some pretty large discrepancies in sizes of
pools. and directories.

zpool list
zfs list
du 

All have different readings for each and every pool.  
Then zfs and du will differ on each fs inside the pool.

I don''t understand why this late in the game there is not some
canonical way to determine size.  Do no other administrators feel the
need to know accurate sizes?

Or is there a canonical method I have yet to discover?

On Sun, Apr 18, 2010 at 20:08, Harry Putnam <reader at newsguy.com>
wrote:
> Seems like you can get some pretty large discrepancies in sizes of
> pools. and directories.
They all answer different things, sure, but they''re all things that an
administrator might want to know.
> zpool list
"How many bytes are in use on the storage device?  How many
unallocated bytes are there?"
> zfs list
"If I have to ship this filesystem to another box (uncompressed and
not deduped) how many bytes is that?"
> du
"How many bytes are used to store the contents of the files in this
directory?"

and "ls -l":
"How many bytes are addressable in this file?"
> Do no other administrators feel the
> need to know accurate sizes?
It''s important to consider what you want this data for.  Considering
upgrading your storage to get more room?  Check out "zpool list".
Need to know whether accounting or engineering is using more space?
Look at "zfs list".  Looking at a sparse or compressed file, and want
to know how many bytes are allocated to it?  "du" does the trick.
Planning to email someone a file, and want to know if it''ll fit in
their 10MB quota?  "ls -l" is the relevant command.

In short, there are many commands because there are many answers, and
many questions.  No single tool has all the information available to
it.

Will

Will Murnane <will.murnane at gmail.com> writes:
> In short, there are many commands because there are many answers, and
> many questions.  No single tool has all the information available to
> it.
Thanks for such a complete answer... and nicely put too.

Will Murnane <will.murnane at gmail.com> writes:
> It''s important to consider what you want this data for. 
Considering
> upgrading your storage to get more room?  Check out "zpool list".
> Need to know whether accounting or engineering is using more space?
> Look at "zfs list".  Looking at a sparse or compressed file, and
want
> to know how many bytes are allocated to it?  "du" does the trick.
> Planning to email someone a file, and want to know if it''ll fit in
> their 10MB quota?  "ls -l" is the relevant command.
>
> In short, there are many commands because there are many answers, and
> many questions.  No single tool has all the information available to
> it.
I''m still confused, sorry I thought I got it for a minute there.

I''m seeing a really big (to big to be excused lightly) difference with
the 2 zfs native methods  zpool and rpool
compared to 2 native unix methods, du and /bin/df

I''m seeing 100s of GB in use, that /bin/df... doesn''t know
about, and
du cannot find.

Something isn''t adding up here:

  zpool list

  NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
  rpool   466G  50.7G   415G    10%  1.00x  ONLINE  -
  z2      464G   366G  97.9G    78%  1.00x  ONLINE  -
  z3      696G   124G   572G    17%  1.00x  ONLINE  -
 
You see z2 shows 366G allocated and 97.9 free

  zfs list -r z2

  NAME                                USED  AVAIL  REFER  MOUNTPOINT
  z2                                  366G  90.7G    19K  /z2
  z2/pub                             4.98G  90.7G  4.97G  /pub
  z2/rhosts                           326G  90.7G    22K  /rhosts
  z2/rhosts/imgs                      299G  90.7G    23K  /rhosts/imgs
  z2/rhosts/imgs/bjp                 83.6G  90.7G  28.4G  /rhosts/imgs/bjp
  z2/rhosts/imgs/harvey               150G  90.7G  17.6G  /rhosts/imgs/harvey
  z2/rhosts/imgs/harvey/ImagesMusic   132G  90.7G  28.3G 
/rhosts/imgs/harvey/ImagesMusic
  z2/rhosts/imgs/mob1                65.9G  90.7G  27.3G  /rhosts/imgs/mob1
  z2/rhosts/misc                     27.1G  90.7G    23K  /rhosts/misc
  z2/rhosts/misc/bjp                   20K  90.7G    20K  /rhosts/misc/bjp
  z2/rhosts/misc/harvey              27.1G  90.7G  27.1G  /rhosts/misc/harvey
  z2/rhosts/misc/mob1                  20K  90.7G    20K  /rhosts/misc/mob1
  z2/win                             34.8G  90.7G  34.6G  /win

You see  366G Used and 90.7G Available

  /bin/df -h 

  z2/pub                 457G   5.0G    91G     6%    /pub
  z2/rhosts              457G    22K    91G     1%    /rhosts
  z2/rhosts/imgs         457G    23K    91G     1%    /rhosts/imgs
  z2/rhosts/imgs/bjp     457G    28G    91G    24%    /rhosts/imgs/bjp
  z2/rhosts/imgs/harvey   457G    18G    91G    17%    /rhosts/imgs/harvey
  z2/rhosts/imgs/harvey/ImagesMusic   457G    28G    91G    24%   
/rhosts/imgs/harvey/ImagesMusic
  z2/rhosts/imgs/mob1    457G    27G    91G    24%    /rhosts/imgs/mob1
  z2/rhosts/misc         457G    23K    91G     1%    /rhosts/misc
  z2/rhosts/misc/bjp     457G    20K    91G     1%    /rhosts/misc/bjp
  z2/rhosts/misc/harvey   457G    27G    91G    24%    /rhosts/misc/harvey
  z2/rhosts/misc/mob1    457G    20K    91G     1%    /rhosts/misc/mob1
  z2/win                 457G    35G    91G    28%    /win
  z2                     457G    19K    91G     1%    /z2

Looks similar, but look again.  If you add up the `used'' column
(column 3) it only adds up to - 163 GB 127 KB

  du -sh /pub /win /rhosts
  5.0G    /pub
  35G     /win
  129G    /rhosts
====  169G

du almost agrees with /bin/df

But both du and /bin/df are around 200G smaller than either zpool or
zfs list.

Hi Harry,

Both du and df are pre-ZFS commands and don''t really understand ZFS
space issues, which are described in the ZFS FAQ here:

http://hub.opensolaris.org/bin/view/Community+Group+zfs/faq

Why does du(1) report different file sizes for ZFS and UFS? Why doesn''t
the space consumption that is reported by the df command and the zfs
list command match?

Will''s advice is good:

Use zpool list and zfs list to determine how much space is available for 
your ZFS file systems and use du or ls -l to review file sizes. Don''t
use du or df to look at ZFS file systems sizes.

Thanks,

Cindy

On 04/19/10 16:18, Harry Putnam wrote:
> Will Murnane <will.murnane at gmail.com> writes:
> 
>> It''s important to consider what you want this data for. 
Considering
>> upgrading your storage to get more room?  Check out "zpool
list".
>> Need to know whether accounting or engineering is using more space?
>> Look at "zfs list".  Looking at a sparse or compressed file,
and want
>> to know how many bytes are allocated to it?  "du" does the
trick.
>> Planning to email someone a file, and want to know if it''ll
fit in
>> their 10MB quota?  "ls -l" is the relevant command.
>>
>> In short, there are many commands because there are many answers, and
>> many questions.  No single tool has all the information available to
>> it.
> 
> I''m still confused, sorry I thought I got it for a minute there.
> 
> I''m seeing a really big (to big to be excused lightly) difference
with
> the 2 zfs native methods  zpool and rpool
> compared to 2 native unix methods, du and /bin/df
> 
> I''m seeing 100s of GB in use, that /bin/df... doesn''t
know about, and
> du cannot find.
> 
> Something isn''t adding up here:
> 
>   zpool list
> 
>   NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
>   rpool   466G  50.7G   415G    10%  1.00x  ONLINE  -
>   z2      464G   366G  97.9G    78%  1.00x  ONLINE  -
>   z3      696G   124G   572G    17%  1.00x  ONLINE  -
>  
> You see z2 shows 366G allocated and 97.9 free
> 
>   zfs list -r z2
> 
>   NAME                                USED  AVAIL  REFER  MOUNTPOINT
>   z2                                  366G  90.7G    19K  /z2
>   z2/pub                             4.98G  90.7G  4.97G  /pub
>   z2/rhosts                           326G  90.7G    22K  /rhosts
>   z2/rhosts/imgs                      299G  90.7G    23K  /rhosts/imgs
>   z2/rhosts/imgs/bjp                 83.6G  90.7G  28.4G  /rhosts/imgs/bjp
>   z2/rhosts/imgs/harvey               150G  90.7G  17.6G 
/rhosts/imgs/harvey
>   z2/rhosts/imgs/harvey/ImagesMusic   132G  90.7G  28.3G 
/rhosts/imgs/harvey/ImagesMusic
>   z2/rhosts/imgs/mob1                65.9G  90.7G  27.3G  /rhosts/imgs/mob1
>   z2/rhosts/misc                     27.1G  90.7G    23K  /rhosts/misc
>   z2/rhosts/misc/bjp                   20K  90.7G    20K  /rhosts/misc/bjp
>   z2/rhosts/misc/harvey              27.1G  90.7G  27.1G 
/rhosts/misc/harvey
>   z2/rhosts/misc/mob1                  20K  90.7G    20K  /rhosts/misc/mob1
>   z2/win                             34.8G  90.7G  34.6G  /win
> 
> You see  366G Used and 90.7G Available
> 
>   /bin/df -h 
> 
>   z2/pub                 457G   5.0G    91G     6%    /pub
>   z2/rhosts              457G    22K    91G     1%    /rhosts
>   z2/rhosts/imgs         457G    23K    91G     1%    /rhosts/imgs
>   z2/rhosts/imgs/bjp     457G    28G    91G    24%    /rhosts/imgs/bjp
>   z2/rhosts/imgs/harvey   457G    18G    91G    17%    /rhosts/imgs/harvey
>   z2/rhosts/imgs/harvey/ImagesMusic   457G    28G    91G    24%   
/rhosts/imgs/harvey/ImagesMusic
>   z2/rhosts/imgs/mob1    457G    27G    91G    24%    /rhosts/imgs/mob1
>   z2/rhosts/misc         457G    23K    91G     1%    /rhosts/misc
>   z2/rhosts/misc/bjp     457G    20K    91G     1%    /rhosts/misc/bjp
>   z2/rhosts/misc/harvey   457G    27G    91G    24%    /rhosts/misc/harvey
>   z2/rhosts/misc/mob1    457G    20K    91G     1%    /rhosts/misc/mob1
>   z2/win                 457G    35G    91G    28%    /win
>   z2                     457G    19K    91G     1%    /z2
> 
> Looks similar, but look again.  If you add up the `used'' column
> (column 3) it only adds up to - 163 GB 127 KB
> 
>   du -sh /pub /win /rhosts
>   5.0G    /pub
>   35G     /win
>   129G    /rhosts
> ====>   169G
> 
> du almost agrees with /bin/df
> 
> But both du and /bin/df are around 200G smaller than either zpool or
> zfs list.
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Harry Putnam <reader at newsguy.com> writes:
> I''m seeing a really big (to big to be excused lightly) difference
with
> the 2 zfs native methods  zpool and rpool
         
  Typo alert: The above line should have read:
  the 2 zfs native methods   ZPOOL list and  ZFS list
    
> compared to 2 native unix methods, du and /bin/df

Cindy Swearingen <cindy.swearingen at oracle.com> writes:
> Hi Harry,
>
> Both du and df are pre-ZFS commands and don''t really understand
ZFS
> space issues, which are described in the ZFS FAQ here:
>
> http://hub.opensolaris.org/bin/view/Community+Group+zfs/faq
>
> Why does du(1) report different file sizes for ZFS and UFS? Why
doesn''t
> the space consumption that is reported by the df command and the zfs
> list command match?
>
> Will''s advice is good:
>
> Use zpool list and zfs list to determine how much space is available
> for your ZFS file systems and use du or ls -l to review file
> sizes. Don''t
> use du or df to look at ZFS file systems sizes.
>
I think I''m beginning to see how this goes.  And at risk of sounding
like a total idiot I notice the information below:

  zfs list z2/rhosts/imgs/harvey

  NAME                    USED  AVAIL  REFER  MOUNTPOINT
  z2/rhosts/imgs/harvey   150G  90.7G  17.6G  /rhosts/imgs/harvey

Its wasn''t clear (to me) really what REFER means... according to
`man zfs'':
 
    [...]  
    referenced

         The amount of data that is accessible by  this  dataset,
         which  may  or  may not be shared with other datasets in
         the pool. When a snapshot or clone is created,  it  ini-
         tially  references  the same amount of space as the file
         system or snapshot it was created from, since  its  con-
         tents are identical.

         This property can also be referred to by  its  shortened
         column name, refer.

So apparently it means that even though 150 GB are used, only 17.6G
can be accessed.

Now du -sh on the same data:

# /bin/du -sh /rhosts/imgs/harvey
  46G   /rhosts/imgs/harvey

That is 300+ % less.

But then if you examine the [...].zfs/snapshot directory... you find
the data... or I guess its really the possible data.

  pwd
  /rhosts/imgs/harvey/.zfs/snapshot

du -sh `ls`
  18G     zfs-auto-snap:frequent-2010-04-18-03:15
  18G     zfs-auto-snap:frequent-2010-04-18-16:30
  18G     zfs-auto-snap:frequent-2010-04-18-18:45
  18G     zfs-auto-snap:frequent-2010-04-20-07:15
  18G     zfs-auto-snap:monthly-2010-04-01-00:00
  18G     zfs-auto-snap:weekly-2010-03-22-00:00
  18G     zfs-auto-snap:weekly-2010-03-29-00:00
  18G     zfs-auto-snap:weekly-2010-04-08-00:00
  18G     zfs-auto-snap:weekly-2010-04-15-00:00
 ====  162GB

-------        ---------       ---=---       ---------      -------- 

So then it seems some careful attention must be paid to the snapshots,
especially when you''ve removed quite a lot of data from the zfs
filesystem above them.

If you really want the space back now you''ll need to follow up by
removing the data from the snapshots too.  Instead of letting the
rollover of snapshots, eventually square with the data above them.

Or I guess if you feel lucky and don''t think there will be a need for
that removed data you could remove all the snapshots and let the first
new one happen.