zfs discuss - Feb 2010 - Problems with sudden zfs capacity loss on snv_79a

Yes snv_79a is old, yes we''re working separately on migrating to
snv_111b or later.  But i need to solve this problem ASAP to buy me
some more time for that implementation.

We pull data from a variety of sources onto our zpool called Backups,
then we snapshot them.  We keep around 20 or so and then delete them
automatically.  We''ve been doing this for around two years on this
system and it''s been absolutely fantastic.  Free-space hovers around
300G.  But suddenly something has changed:

root at darling(/)$:zfs list | head -1 && zfs list | tail -7
NAME
    USED  AVAIL  REFER  MOUNTPOINT
Backups/natoffice/onsite at 20091231_2347_TriggeredBy_20091231_2330
   30.1G      -   287G  -
Backups/natoffice/onsite at 20100131_2349_TriggeredBy_20100131_2330
   17.7G      -   287G  -
Backups/natoffice/onsite at 20100205_0001_TriggeredBy_20100204_2330
   15.9G      -   287G  -
Backups/natoffice/onsite at 20100212_0424_TriggeredBy_20100211_2330
   152G      -   285G  -
Backups/natoffice/onsite at 20100216_0430_TriggeredBy_20100215_2330
   154G      -   287G  -
Backups/natoffice/onsite at 20100217_0431_TriggeredBy_20100216_2330
   154G      -   287G  -
Backups/natoffice/onsite at 20100218_0423_TriggeredBy_20100217_2330
   0      -   287G  -

Normally a snapshot shows USED around 15G to 30G.  But suddenly,
snapshots of the same filesystem are showing USED ~150G.  There are no
corresponding increases in any of the machines we copy data from, nor
has any of that data changed significantly.  You can see that the
REFER hasn''t changed much at all, this is normal.  So we''re
backing up
the same amount of data, but it now occupies so much more on disk.
That of course means we can''t keep nearly as many snapshots, and that
makes us all very nervous.

Any ideas?

Other info:
Normally the snapshots are consecutive days but we''ve had to cull data
as we were running out of space.

root at darling(/)$:zpool status
  pool: Backups
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        Backups     ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            c2d0    ONLINE       0     0     0
            c5d0    ONLINE       0     0     0
            c6d0    ONLINE       0     0     0

errors: No known data errors

root at darling(/)$:zfs list Backups
NAME      USED  AVAIL  REFER  MOUNTPOINT
Backups  1.34T   456G  25.3K  /Backups

-- 
Kind regards, Jules

Zen left me, then I remembered, nothing to forget.

On Thu, Feb 18, 2010 at 1:19 AM, Julius Roberts <hooliowobbits at
gmail.com>wrote:
> Yes snv_79a is old, yes we''re working separately on migrating to
> snv_111b or later.  But i need to solve this problem ASAP to buy me
> some more time for that implementation.
>
> We pull data from a variety of sources onto our zpool called Backups,
> then we snapshot them.  We keep around 20 or so and then delete them
> automatically.  We''ve been doing this for around two years on this
> system and it''s been absolutely fantastic.  Free-space hovers
around
> 300G.  But suddenly something has changed:
>
> root at darling(/)$:zfs list | head -1 && zfs list | tail -7
> NAME
>    USED  AVAIL  REFER  MOUNTPOINT
> Backups/natoffice/onsite at 20091231_2347_TriggeredBy_20091231_2330
>   30.1G      -   287G  -
> Backups/natoffice/onsite at 20100131_2349_TriggeredBy_20100131_2330
>   17.7G      -   287G  -
> Backups/natoffice/onsite at 20100205_0001_TriggeredBy_20100204_2330
>   15.9G      -   287G  -
> Backups/natoffice/onsite at 20100212_0424_TriggeredBy_20100211_2330
>   152G      -   285G  -
> Backups/natoffice/onsite at 20100216_0430_TriggeredBy_20100215_2330
>   154G      -   287G  -
> Backups/natoffice/onsite at 20100217_0431_TriggeredBy_20100216_2330
>   154G      -   287G  -
> Backups/natoffice/onsite at 20100218_0423_TriggeredBy_20100217_2330
>   0      -   287G  -
>
> Normally a snapshot shows USED around 15G to 30G.  But suddenly,
> snapshots of the same filesystem are showing USED ~150G.  There are no
> corresponding increases in any of the machines we copy data from, nor
> has any of that data changed significantly.  You can see that the
> REFER hasn''t changed much at all, this is normal.  So
we''re backing up
> the same amount of data, but it now occupies so much more on disk.
> That of course means we can''t keep nearly as many snapshots, and
that
> makes us all very nervous.
>
> Any ideas?
>

Is it possible that your users are now deleting everything before starting
to write the backup data ?


-- 
Giovanni Tirloni
sysdroid.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100218/39011c90/attachment.html>