Tom Fanning
2011-Mar-09 21:50 UTC
[zfs-discuss] Free space on ZFS file system unexpectedly missing
Hi all
This is a production issue I have outstanding for several months now.
I asked here a while ago but didn''t reach a resolution. I have also
got the question up on serverfault.com, question 192927
I have a FreeNAS 0.7.2 box, based on FreeBSD 7.3-RELEASE-p1, running
ZFS with 4x1TB SATA drives in RAIDz1.
I appear to have lost 1TB of usable space after creating and deleting
a 1TB sparse file. This happened months ago.
This table lays out the situation as it stands.
command actual expected ok/not ok
du -c 1.47TB used 1.47TB used ok
zfs list used 2.48TB used 1.47TB not ok
avail 206GB avail 1.2TB not ok
zpool list size 3.64TB size 3.64TB ok
used 3.31TB used 1.95TB not ok
avail 334GB avail 1.69TB not ok
Windows right Disk size 2.67TB Disk size 2.67TB ok
click disk, Used 2.47TB Used 1.47 not ok
properties free 206GB free 1.2TB not ok
Windows select total file size total file size
all files, 1.48TB 1.48TB ok
right click,
properties
Some information
- No snapshots anywhere in the pool
- Compression is off
- De-dupe is off
- ZFS pool verion is 13
- ZFS FS version is 3
- Using the "embedded" version of FreeNAS
- File was created with dd using /dev/zero as input, deleted using
rm, all as root
- File has definitely been deleted
- Windows 7 can see the folder via SMB
- Exported the pool from FreeBSD, imported it on OpenIndiana 148 -
but not upgraded - same problem, much newer ZFS implementation. Can''t
upgrade the pool to see if the issue goes away since for now I need a
route back to FreeBSD and I don''t have spare storage.
Any help whatsoever would be much appreciated - something''s not right
here.
Many thanks
Tom
Peter Jeremy
2011-Mar-09 22:37 UTC
[zfs-discuss] Free space on ZFS file system unexpectedly missing
On 2011-Mar-10 05:50:53 +0800, Tom Fanning <ml at tomfanning.eu> wrote:>I have a FreeNAS 0.7.2 box, based on FreeBSD 7.3-RELEASE-p1, running >ZFS with 4x1TB SATA drives in RAIDz1. > >I appear to have lost 1TB of usable space after creating and deleting >a 1TB sparse file. This happened months ago.AFAIR, ZFS on FreeBSD 7.x was always described as experimental. This is a known problem (OpenSolaris bug id 6792701) that was fixed in OpenSolaris onnv revision 9950:78fc41aa9bc5 which was committed to FreeBSD as r208775 in head and r208869 in 8-stable. The fix was never back-ported to 7.x and I am unable to locate any workaround.> - Exported the pool from FreeBSD, imported it on OpenIndiana 148 - >but not upgraded - same problem, much newer ZFS implementation. Can''t >upgrade the pool to see if the issue goes away since for now I need a >route back to FreeBSD and I don''t have spare storage.I thought that just importing a pool on a system with the bugfix would free the space. If that doesn''t work, your only options are to either upgrade to FreeBSD 8.1-RELEASE or later (preferably 8.2 since there are a number of other fairly important ZFS fixes since 8.1) and upgrade your pool to v15 or rebuild your pool (via send/recv or similar). -- Peter Jeremy -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110310/5cbaca25/attachment.bin>
Bob Friesenhahn
2011-Mar-09 22:48 UTC
[zfs-discuss] Free space on ZFS file system unexpectedly missing
On Wed, 9 Mar 2011, Tom Fanning wrote:> > I appear to have lost 1TB of usable space after creating and deleting > a 1TB sparse file. This happened months ago.This is an old zfs bug. Recent enough zfs will know how to recover the space.> - Exported the pool from FreeBSD, imported it on OpenIndiana 148 - > but not upgraded - same problem, much newer ZFS implementation. Can''t > upgrade the pool to see if the issue goes away since for now I need a > route back to FreeBSD and I don''t have spare storage.The good news is that FreeBSD will soon have zfs version 28 since it is already checked in for version 9, and patches are available for 8.2 (if it is not already in to the respository). Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Tom Fanning
2011-Mar-10 00:05 UTC
[zfs-discuss] Free space on ZFS file system unexpectedly missing
On Wed, Mar 9, 2011 at 10:37 PM, Peter Jeremy <peter.jeremy at alcatel-lucent.com> wrote:> On 2011-Mar-10 05:50:53 +0800, Tom Fanning <ml at tomfanning.eu> wrote: >>I have a FreeNAS 0.7.2 box, based on FreeBSD 7.3-RELEASE-p1, running >>ZFS with 4x1TB SATA drives in RAIDz1. >> >>I appear to have lost 1TB of usable space after creating and deleting >>a 1TB sparse file. This happened months ago. > > AFAIR, ZFS on FreeBSD 7.x was always described as experimental. > > This is a known problem (OpenSolaris bug id 6792701) that was fixed in > OpenSolaris onnv revision 9950:78fc41aa9bc5 which was committed to > FreeBSD as r208775 in head and r208869 in 8-stable. ?The fix was never > back-ported to 7.x and I am unable to locate any workaround. > >> - Exported the pool from FreeBSD, imported it on OpenIndiana 148 - >>but not upgraded - same problem, much newer ZFS implementation. Can''t >>upgrade the pool to see if the issue goes away since for now I need a >>route back to FreeBSD and I don''t have spare storage. > > I thought that just importing a pool on a system with the bugfix would > free the space. ?If that doesn''t work, your only options are to either > upgrade to FreeBSD 8.1-RELEASE or later (preferably 8.2 since there > are a number of other fairly important ZFS fixes since 8.1) and > upgrade your pool to v15 or rebuild your pool (via send/recv or similar). > > -- > Peter JeremyWell I never. Just by chance I did zfs list -t snapshot, and now it shows a 1TB snapshot which it wasn''t showing before. bunker:~# zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT tank0 at snap 1.00T - 2.47T - I know it wasn''t showing before - here''s my first post with the output of the same command: http://www.mail-archive.com/zfs-discuss at opensolaris.org/msg43138.html Have destroyed the snapshot, all now as it should be. bunker:~# zfs list NAME USED AVAIL REFER MOUNTPOINT tank0 1.48T 1.20T 1.48T /mnt/tank0 Very odd. Pool history is pasted here: http://pastebin.com/TRWcebSs Thanks for the input. Tom
Richard Elling
2011-Mar-10 03:38 UTC
[zfs-discuss] Free space on ZFS file system unexpectedly missing
On Mar 9, 2011, at 4:05 PM, Tom Fanning wrote:> On Wed, Mar 9, 2011 at 10:37 PM, Peter Jeremy > <peter.jeremy at alcatel-lucent.com> wrote: >> On 2011-Mar-10 05:50:53 +0800, Tom Fanning <ml at tomfanning.eu> wrote: >>> I have a FreeNAS 0.7.2 box, based on FreeBSD 7.3-RELEASE-p1, running >>> ZFS with 4x1TB SATA drives in RAIDz1. >>> >>> I appear to have lost 1TB of usable space after creating and deleting >>> a 1TB sparse file. This happened months ago. >> >> AFAIR, ZFS on FreeBSD 7.x was always described as experimental. >> >> This is a known problem (OpenSolaris bug id 6792701) that was fixed in >> OpenSolaris onnv revision 9950:78fc41aa9bc5 which was committed to >> FreeBSD as r208775 in head and r208869 in 8-stable. The fix was never >> back-ported to 7.x and I am unable to locate any workaround. >> >>> - Exported the pool from FreeBSD, imported it on OpenIndiana 148 - >>> but not upgraded - same problem, much newer ZFS implementation. Can''t >>> upgrade the pool to see if the issue goes away since for now I need a >>> route back to FreeBSD and I don''t have spare storage. >> >> I thought that just importing a pool on a system with the bugfix would >> free the space. If that doesn''t work, your only options are to either >> upgrade to FreeBSD 8.1-RELEASE or later (preferably 8.2 since there >> are a number of other fairly important ZFS fixes since 8.1) and >> upgrade your pool to v15 or rebuild your pool (via send/recv or similar). >> >> -- >> Peter Jeremy > > Well I never. Just by chance I did zfs list -t snapshot, and now it > shows a 1TB snapshot which it wasn''t showing before.There was a change where snapshots are no longer shown by default. This can be configured back to the old behaviour setting the zpool "listsnapshots" property to "on" Otherwise, you need to use the "-t snapshot" list. But, a much better method of tracking this down is to use: zfs list -o space That will show the accounting for all dataset objects. -- richard
Craig Morgan
2011-Mar-10 09:15 UTC
[zfs-discuss] Free space on ZFS file system unexpectedly missing
But even the ''zfs list -o space'' is now limited by not displaying snapshots by default, so the catch all is now zfs list -o space -t all shouldn''t miss anything then ? ;-) Craig On 10 Mar 2011, at 03:38, Richard Elling wrote:> > On Mar 9, 2011, at 4:05 PM, Tom Fanning wrote: > >> On Wed, Mar 9, 2011 at 10:37 PM, Peter Jeremy >> <peter.jeremy at alcatel-lucent.com> wrote: >>> On 2011-Mar-10 05:50:53 +0800, Tom Fanning <ml at tomfanning.eu> wrote: >>>> I have a FreeNAS 0.7.2 box, based on FreeBSD 7.3-RELEASE-p1, running >>>> ZFS with 4x1TB SATA drives in RAIDz1. >>>> >>>> I appear to have lost 1TB of usable space after creating and deleting >>>> a 1TB sparse file. This happened months ago. >>> >>> AFAIR, ZFS on FreeBSD 7.x was always described as experimental. >>> >>> This is a known problem (OpenSolaris bug id 6792701) that was fixed in >>> OpenSolaris onnv revision 9950:78fc41aa9bc5 which was committed to >>> FreeBSD as r208775 in head and r208869 in 8-stable. The fix was never >>> back-ported to 7.x and I am unable to locate any workaround. >>> >>>> - Exported the pool from FreeBSD, imported it on OpenIndiana 148 - >>>> but not upgraded - same problem, much newer ZFS implementation. Can''t >>>> upgrade the pool to see if the issue goes away since for now I need a >>>> route back to FreeBSD and I don''t have spare storage. >>> >>> I thought that just importing a pool on a system with the bugfix would >>> free the space. If that doesn''t work, your only options are to either >>> upgrade to FreeBSD 8.1-RELEASE or later (preferably 8.2 since there >>> are a number of other fairly important ZFS fixes since 8.1) and >>> upgrade your pool to v15 or rebuild your pool (via send/recv or similar). >>> >>> -- >>> Peter Jeremy >> >> Well I never. Just by chance I did zfs list -t snapshot, and now it >> shows a 1TB snapshot which it wasn''t showing before. > > There was a change where snapshots are no longer shown by default. > This can be configured back to the old behaviour setting the zpool > "listsnapshots" property to "on" > > Otherwise, you need to use the "-t snapshot" list. > > But, a much better method of tracking this down is to use: > zfs list -o space > > That will show the accounting for all dataset objects. > -- richard > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Craig Morgan Cinnabar Solutions Ltd t: +44 (0)791 338 3190 f: +44 (0)870 705 1726 e: craig at cinnabar-solutions.com w: www.cinnabar-solutions.com