zfs discuss - Feb 2013 - how to know available disk space

I have a bunch of VM''s, and some samba shares, etc, on a pool.  I
created the VM''s using zvol''s, specifically so they would have
an appropriate refreservation and never run out of disk space, even with
snapshots.  Today, I ran out of disk space, and all the VM''s died.  So
obviously it didn''t work.

When I used "zpool status" after the system crashed, I saw this:
NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
storage   928G   568G   360G         -    61%  1.00x  ONLINE  -

I did some cleanup, so I could turn things back on ... Freed up about 4G.

Now, when I use "zpool status" I see this:
NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
storage   928G   564G   364G         -    60%  1.00x  ONLINE  -

When I use "zfs list storage" I see this:
NAME      USED  AVAIL  REFER  MOUNTPOINT
storage   909G  4.01G  32.5K  /storage

So I guess the lesson is (a) refreservation and zvol alone aren''t
enough to ensure your VM''s will stay up.  and (b) if you want to know
how much room is *actually* available, as in "usable," as in,
"how much can I write before I run out of space," you should use
"zfs list" and not "zpool status"

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20130206/1c95e666/attachment-0001.html>

This is one of the greatest annoyances of ZFS.  I don''t really
understand how, a zvol''s space can not be accurately enumerated from
top to bottom of the tree in ''df'' output etc.  Why does a
"zvol" divorce the space used from the root of the volume?

Gregg Wonderly

On Feb 6, 2013, at 5:26 PM, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris) <opensolarisisdeadlongliveopensolaris
at nedharvey.com> wrote:
> I have a bunch of VM''s, and some samba shares, etc, on a pool.  I
created the VM''s using zvol''s, specifically so they would have
an appropriaterefreservation and never run out of disk space, even with
snapshots.  Today, I ran out of disk space, and all the VM''s died.  So
obviously it didn''t work.
>  
> When I used "zpool status" after the system crashed, I saw this:
> NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
> storage   928G   568G   360G         -    61%  1.00x  ONLINE  -
>  
> I did some cleanup, so I could turn things back on ... Freed up about 4G.
>  
> Now, when I use "zpool status" I see this:
> NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
> storage   928G   564G   364G         -    60%  1.00x  ONLINE  -
>  
> When I use "zfs list storage" I see this:
> NAME      USED  AVAIL  REFER  MOUNTPOINT
> storage   909G  4.01G  32.5K  /storage
>  
> So I guess the lesson is (a) refreservation and zvol alone aren''t
enough to ensure your VM''s will stay up.  and (b) if you want to know
how much room is *actually* available, as in "usable," as in,
"how much can I write before I run out of space," you should use
"zfs list" and not "zpoolstatus"
>  
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20130206/6f9450e8/attachment.html>

> From: Gregg Wonderly [mailto:greggwon at gmail.com]
> 
> This is one of the greatest annoyances of ZFS. ?I don''t really
understand how,
> a zvol''s space can not be accurately enumerated from top to bottom
of the
> tree in ''df'' output etc. ?Why does a "zvol"
divorce the space used from the
> root of the volume?
The way I would say that is:  Intuitively, I think people expect reservations to
count against Alloc and Used.

On Wed, Feb 6, 2013 at 4:26 PM, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris)
<opensolarisisdeadlongliveopensolaris at nedharvey.com>
wrote:
>
> When I used "zpool status" after the system crashed, I saw this:
> NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
> storage   928G   568G   360G         -    61%  1.00x  ONLINE  -
>
> I did some cleanup, so I could turn things back on ... Freed up about 4G.
>
> Now, when I use "zpool status" I see this:
> NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
> storage   928G   564G   364G         -    60%  1.00x  ONLINE  -
>
> When I use "zfs list storage" I see this:
> NAME      USED  AVAIL  REFER  MOUNTPOINT
> storage   909G  4.01G  32.5K  /storage
>
> So I guess the lesson is (a) refreservation and zvol alone aren''t
enough to
> ensure your VM''s will stay up.  and (b) if you want to know how
much room is
> *actually* available, as in "usable," as in, "how much can I
write before I
> run out of space," you should use "zfs list" and not
"zpool status"
Could you run "zfs list -o space storage"? It will show how much is
used by the data, the snapshots, refreservation, and children (if
any). I read somewhere that one should always use "zfs list" to
determine how much space is actually available to be written on a
given filesystem.

I have an idea, but it''s a long shot. If you created more than one zfs
on that pool, and added a reservation to each one, then that space is
still technically unallocated as far as "zpool list" is concerned, but
is not available to writing when you do "zfs list". I would imagine
you have one or more of your VMs that grew outside of their
"refreservation" and now crashed for lack of free space on their zfs.
Some of the other VMs aren''t using their refreservation (yet), so they
could, between them, still write 360GB of stuff to the drive.

Jan

On Feb 6, 2013, at 5:17 PM, Gregg Wonderly <greggwon at gmail.com> wrote:
> This is one of the greatest annoyances of ZFS.  I don''t really
understand how, a zvol''s space can not be accurately enumerated from
top to bottom of the tree in ''df'' output etc.  Why does a
"zvol" divorce the space used from the root of the volume?
Thick (with reservation) or thin provisioning behave differently. Also,
depending on
how you created the reservation, it might or might not account for the metadata 
overhead needed. By default, space for metadata is reserved, but if you use the
-s (sparse aka thin provisioning) option, then later reservation changes are set
as
absolute.

Also, metadata space, compression, copies, and deduplication must be accounted
for.
The old notions of free/available don''t match very well with these
modern features.
> 
> Gregg Wonderly
> 
> On Feb 6, 2013, at 5:26 PM, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris) <opensolarisisdeadlongliveopensolaris
at nedharvey.com> wrote:
> 
>> I have a bunch of VM''s, and some samba shares, etc, on a pool.
I created the VM''s using zvol''s, specifically so they would
have an appropriaterefreservation and never run out of disk space, even with
snapshots.  Today, I ran out of disk space, and all the VM''s died.  So
obviously it didn''t work.
>>  
>> When I used "zpool status" after the system crashed, I saw
this:
>> NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
>> storage   928G   568G   360G         -    61%  1.00x  ONLINE  -
>>  
>> I did some cleanup, so I could turn things back on ... Freed up about
4G.
>>  
>> Now, when I use "zpool status" I see this:
>> NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
>> storage   928G   564G   364G         -    60%  1.00x  ONLINE  -
>>  
>> When I use "zfs list storage" I see this:
>> NAME      USED  AVAIL  REFER  MOUNTPOINT
>> storage   909G  4.01G  32.5K  /storage
>>  
>> So I guess the lesson is (a) refreservation and zvol alone
aren''t enough to ensure your VM''s will stay up.  and (b) if
you want to know how much room is *actually* available, as in
"usable," as in, "how much can I write before I run out of
space," you should use "zfs list" and not "zpoolstatus"
Correct. zpool status does not show dataset space available.
 -- richard

--

Richard.Elling at RichardElling.com
+1-760-896-4422









-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20130207/fca3a2c4/attachment.html>

On Wed, Feb 06, 2013 at 08:03:13PM -0700, Jan Owoc
wrote:
> On Wed, Feb 6, 2013 at 4:26 PM, Edward Ned Harvey
> (opensolarisisdeadlongliveopensolaris)
> <opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:
> >
> > When I used "zpool status" after the system crashed, I saw
this:
> > NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
> > storage   928G   568G   360G         -    61%  1.00x  ONLINE  -
> >
> > I did some cleanup, so I could turn things back on ... Freed up about
4G.
> >
> > Now, when I use "zpool status" I see this:
> > NAME      SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
> > storage   928G   564G   364G         -    60%  1.00x  ONLINE  -
> >
> > When I use "zfs list storage" I see this:
> > NAME      USED  AVAIL  REFER  MOUNTPOINT
> > storage   909G  4.01G  32.5K  /storage
> >
> > So I guess the lesson is (a) refreservation and zvol alone
aren''t enough to
> > ensure your VM''s will stay up.  and (b) if you want to know
how much room is
> > *actually* available, as in "usable," as in, "how much
can I write before I
> > run out of space," you should use "zfs list" and not
"zpool status"
> 
> Could you run "zfs list -o space storage"? It will show how much
is
> used by the data, the snapshots, refreservation, and children (if
> any). I read somewhere that one should always use "zfs list" to
> determine how much space is actually available to be written on a
> given filesystem.
> 
> I have an idea, but it''s a long shot. If you created more than one
zfs
> on that pool, and added a reservation to each one, then that space is
> still technically unallocated as far as "zpool list" is
concerned, but
> is not available to writing when you do "zfs list". I would
imagine
> you have one or more of your VMs that grew outside of their
> "refreservation" and now crashed for lack of free space on their
zfs.
> Some of the other VMs aren''t using their refreservation (yet), so
they
> could, between them, still write 360GB of stuff to the drive.
> 
I''m seeing weird output aswell:

# zpool list foo
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
foo      5.44T  4.44T  1023G    81%  14.49x  ONLINE  -

# zfs list | grep foo
foo                          62.9T      0   250G  /volumes/foo
foo/.nza-reserve               31K   100M    31K  none
foo/foo                      62.6T      0  62.6T  /volumes/foo/foo

# zfs list -o space foo
NAME     AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
foo          0  62.9T         0    250G              0      62.7T

# zfs list -o space foo/foo
NAME             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
foo/foo              0  62.6T         0   62.6T              0          0


What''s the correct way of finding out what actually uses/reserves that
1023G of FREE in the zpool?

At this point the filesystems are full, and it''s not possible to write
to them anymore.
Also creating new filesystems to the pool fail:

"Operation completed with error: cannot create
''foo/Test'': out of space"

So the zpool is full for real.

I''d like to better understand what actually uses that 1023G of FREE
space reported by zpool..
1023G out of 4.32T is around 22% overhead..
zpool "foo" consists of 3x mirror vdevs, so there''s no raidz
involved.

62.6T / 14.49x dedup-ratio = 4.32T 
Which is pretty close to the ALLOC value reported by zpool.. 

Data on the filesystem is VM images written over NFS.


Thanks,

-- Pasi

> From: Pasi K?rkk?inen [mailto:pasik at iki.fi]
> 
> What''s the correct way of finding out what actually uses/reserves
that 1023G
> of FREE in the zpool?
Maybe this isn''t exactly what you need, but maybe:

for fs in `zfs list -H -o name` ; do echo $fs ; zfs get
reservation,refreservation,usedbyrefreservation $fs ; done

> At this point the filesystems are full, and it''s not possible to
write to them
> anymore.
You''ll have to either reduce your reservations, or destroy old
snapshots.  Or add more disks.

On Fri, Feb 08, 2013 at 09:47:38PM +0000, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris) wrote:
> > From: Pasi K?rkk?inen [mailto:pasik at iki.fi]
> > 
> > What''s the correct way of finding out what actually
uses/reserves that 1023G
> > of FREE in the zpool?
> 
> Maybe this isn''t exactly what you need, but maybe:
> 
> for fs in `zfs list -H -o name` ; do echo $fs ; zfs get
reservation,refreservation,usedbyrefreservation $fs ; done
>
I checked this and there are no reservations configured (or well, there are the
100MB defaults, but not more than that). So reservations don''t explain
this..
 
> 
> > At this point the filesystems are full, and it''s not possible
to write to them
> > anymore.
> 
> You''ll have to either reduce your reservations, or destroy old
snapshots.  Or add more disks.
>
There aren''t any snapshots either.. I know adding disks will fix the
problem,
but I''d like to understand why zpool says there is almost 1TB of FREE
space when clearly there isn''t..


Thanks for the reply!

-- Pasi

On 2013-02-08 22:47, Edward Ned Harvey 
(opensolarisisdeadlongliveopensolaris) wrote:
> Maybe this isn''t exactly what you need, but maybe:
>
> for fs in `zfs list -H -o name` ; do echo $fs ; zfs get
reservation,refreservation,usedbyrefreservation $fs ; done
What is the sacramental purpose of such construct in comparison to:

zfs list -H -o reservation,refreservation,usedbyrefreservation,name \
   -t filesystem {-r pool/interesting/dataset}

Just asking - or suggesting a simpler way to do stuff ;)

//Jim

Hello,

Any comments/suggestions about this would be very nice.. 

Thanks!

-- Pasi

On Fri, Feb 08, 2013 at 05:09:56PM +0200, Pasi K?rkk?inen
wrote:
> 
> I''m seeing weird output aswell:
> 
> # zpool list foo
> NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
> foo      5.44T  4.44T  1023G    81%  14.49x  ONLINE  -
> 
> # zfs list | grep foo
> foo                          62.9T      0   250G  /volumes/foo
> foo/.nza-reserve               31K   100M    31K  none
> foo/foo                      62.6T      0  62.6T  /volumes/foo/foo
> 
> # zfs list -o space foo
> NAME     AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
> foo          0  62.9T         0    250G              0      62.7T
> 
> # zfs list -o space foo/foo
> NAME             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
> foo/foo              0  62.6T         0   62.6T              0          0
> 
> 
> What''s the correct way of finding out what actually uses/reserves
that 1023G of FREE in the zpool?
> 
> At this point the filesystems are full, and it''s not possible to
write to them anymore.
> Also creating new filesystems to the pool fail:
> 
> "Operation completed with error: cannot create
''foo/Test'': out of space"
> 
> So the zpool is full for real.
> 
> I''d like to better understand what actually uses that 1023G of
FREE space reported by zpool..
> 1023G out of 4.32T is around 22% overhead..
> zpool "foo" consists of 3x mirror vdevs, so there''s no
raidz involved.
> 
> 62.6T / 14.49x dedup-ratio = 4.32T 
> Which is pretty close to the ALLOC value reported by zpool.. 
> 
> Data on the filesystem is VM images written over NFS.
> 
> 
> Thanks,
> 
> -- Pasi
> 
>