Accidentally posted the below earlier against ZFS Code, rather than ZFS Discuss. My ESXi box now uses ZFS filesystems which have been shared over NFS. Spotted something odd this afternoon - a filesystem which I thought didn''t have any files in it, weighs in at 14GB. Before I start deleting the empty folders to see what happens, any ideas what''s happened here? # zfs list | grep temp zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp # ls -la /zp/nfs/esx_temp total 20 drwxr-xr-x 5 root root 5 Aug 13 12:54 . drwxr-xr-x 7 root root 7 Aug 13 12:40 .. drwxr-xr-x 2 root root 2 Aug 13 12:53 iguana drwxr-xr-x 2 root root 2 Aug 13 12:54 meerkat drwxr-xr-x 2 root root 2 Aug 16 19:39 panda # ls -la /zp/nfs/esx_temp/iguana/ total 8 drwxr-xr-x 2 root root 2 Aug 13 12:53 . drwxr-xr-x 5 root root 5 Aug 13 12:54 .. # ls -la /zp/nfs/esx_temp/meerkat/ total 8 drwxr-xr-x 2 root root 2 Aug 13 12:54 . drwxr-xr-x 5 root root 5 Aug 13 12:54 .. # ls -la /zp/nfs/esx_temp/panda/ total 8 drwxr-xr-x 2 root root 2 Aug 16 19:39 . drwxr-xr-x 5 root root 5 Aug 13 12:54 .. # Could there be something super-hidden, which I can''t see here? There don''t appear to be any snapshots relating to zp/nfs/esx_temp. On a suggestion, I have ran the following: # zfs list -r zp/nfs/esx_temp NAME USED AVAIL REFER MOUNTPOINT zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp # du -sh /zp/nfs/esx_temp 8K /zp/nfs/esx_temp # Thanks, Chris -- This message posted from opensolaris.org
Chris Murray wrote:> Accidentally posted the below earlier against ZFS Code, rather than ZFS Discuss. > > My ESXi box now uses ZFS filesystems which have been shared over NFS. Spotted something odd this afternoon - a filesystem which I thought didn''t have any files in it, weighs in at 14GB. Before I start deleting the empty folders to see what happens, any ideas what''s happened here? > > # zfs list | grep temp > zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp > # ls -la /zp/nfs/esx_temp > total 20 > drwxr-xr-x 5 root root 5 Aug 13 12:54 . > drwxr-xr-x 7 root root 7 Aug 13 12:40 .. > drwxr-xr-x 2 root root 2 Aug 13 12:53 iguana > drwxr-xr-x 2 root root 2 Aug 13 12:54 meerkat > drwxr-xr-x 2 root root 2 Aug 16 19:39 panda > # ls -la /zp/nfs/esx_temp/iguana/ > total 8 > drwxr-xr-x 2 root root 2 Aug 13 12:53 . > drwxr-xr-x 5 root root 5 Aug 13 12:54 .. > # ls -la /zp/nfs/esx_temp/meerkat/ > total 8 > drwxr-xr-x 2 root root 2 Aug 13 12:54 . > drwxr-xr-x 5 root root 5 Aug 13 12:54 .. > # ls -la /zp/nfs/esx_temp/panda/ > total 8 > drwxr-xr-x 2 root root 2 Aug 16 19:39 . > drwxr-xr-x 5 root root 5 Aug 13 12:54 .. > # > > Could there be something super-hidden, which I can''t see here? > > There don''t appear to be any snapshots relating to zp/nfs/esx_temp. > > On a suggestion, I have ran the following: > > # zfs list -r zp/nfs/esx_temp > NAME USED AVAIL REFER MOUNTPOINT > zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp > # du -sh /zp/nfs/esx_temp > 8K /zp/nfs/esx_temp > # >Does zfs list -t snapshot -r zp/nfs/esx_temp show anything? What about zfs get refquota,refreservation,quota,reservation zp/fs/esx_tmp -tim> Thanks, > Chris
> Chris Murray wrote: >> Accidentally posted the below earlier against ZFS Code, rather than ZFS >> Discuss. >> >> My ESXi box now uses ZFS filesystems which have been shared over NFS. >> Spotted something odd this afternoon - a filesystem which I thought >> didn''t have any files in it, weighs in at 14GB. Before I start deleting >> the empty folders to see what happens, any ideas what''s happened here? >> >> # zfs list | grep temp >> zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp >> # ls -la /zp/nfs/esx_temp >> total 20 >> drwxr-xr-x 5 root root 5 Aug 13 12:54 . >> drwxr-xr-x 7 root root 7 Aug 13 12:40 .. >> drwxr-xr-x 2 root root 2 Aug 13 12:53 iguana >> drwxr-xr-x 2 root root 2 Aug 13 12:54 meerkat >> drwxr-xr-x 2 root root 2 Aug 16 19:39 panda >> # ls -la /zp/nfs/esx_temp/iguana/ >> total 8 >> drwxr-xr-x 2 root root 2 Aug 13 12:53 . >> drwxr-xr-x 5 root root 5 Aug 13 12:54 .. >> # ls -la /zp/nfs/esx_temp/meerkat/ >> total 8 >> drwxr-xr-x 2 root root 2 Aug 13 12:54 . >> drwxr-xr-x 5 root root 5 Aug 13 12:54 .. >> # ls -la /zp/nfs/esx_temp/panda/ >> total 8 >> drwxr-xr-x 2 root root 2 Aug 16 19:39 . >> drwxr-xr-x 5 root root 5 Aug 13 12:54 .. >> # >> >> Could there be something super-hidden, which I can''t see here? >> >> There don''t appear to be any snapshots relating to zp/nfs/esx_temp. >> >> On a suggestion, I have ran the following: >> >> # zfs list -r zp/nfs/esx_temp >> NAME USED AVAIL REFER MOUNTPOINT >> zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp >> # du -sh /zp/nfs/esx_temp >> 8K /zp/nfs/esx_temp >> # >> > > Does > > zfs list -t snapshot -r zp/nfs/esx_temp > > show anything? > > What about > > zfs get refquota,refreservation,quota,reservation zp/fs/esx_tmp >pardon me for butting in .. but I thought that was a spelling error. It wasn''t : # zfs get refquota,refreservation,quota,reservation fibre0 NAME PROPERTY VALUE SOURCE fibre0 refquota none default fibre0 refreservation none default fibre0 quota none default fibre0 reservation none default what the heck is refreservation ?? 8-) -- Dennis Clarke dclarke at opensolaris.ca <- Email related to the open source Solaris dclarke at blastwave.org <- Email related to open source for Solaris
Thanks Tim. Results are below: # zfs list -t snapshot -r zp/nfs/esx_temp no datasets available # zfs get refquota,refreservation,quota,reservation zp/nfs/esx_temp NAME PROPERTY VALUE SOURCE zp/nfs/esx_temp refquota none default zp/nfs/esx_temp refreservation none default zp/nfs/esx_temp quota none default zp/nfs/esx_temp reservation none default # I''m 99% sure I''ve done something ''obvious'', which is escaping me at the minute. The filesystem only has 3 folders in it, which are all empty, so I could very quickly destroy and recreate, but it has piqued my curiosity enough to wonder what''s happened here. -- This message posted from opensolaris.org
On Mon, Aug 17, 2009 at 12:31 AM, Dennis Clarke<dclarke at blastwave.org> wrote:> >> Chris Murray wrote: >>> Accidentally posted the below earlier against ZFS Code, rather than ZFS >>> Discuss. >>> >>> My ESXi box now uses ZFS filesystems which have been shared over NFS. >>> Spotted something odd this afternoon - a filesystem which I thought >>> didn''t have any files in it, weighs in at 14GB. Before I start deleting >>> the empty folders to see what happens, any ideas what''s happened here? >>> >>> # zfs list | grep temp >>> zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp >>> # ls -la /zp/nfs/esx_temp >>> total 20 >>> drwxr-xr-x 5 root root 5 Aug 13 12:54 . >>> drwxr-xr-x 7 root root 7 Aug 13 12:40 .. >>> drwxr-xr-x 2 root root 2 Aug 13 12:53 iguana >>> drwxr-xr-x 2 root root 2 Aug 13 12:54 meerkat >>> drwxr-xr-x 2 root root 2 Aug 16 19:39 panda >>> # ls -la /zp/nfs/esx_temp/iguana/ >>> total 8 >>> drwxr-xr-x 2 root root 2 Aug 13 12:53 . >>> drwxr-xr-x 5 root root 5 Aug 13 12:54 .. >>> # ls -la /zp/nfs/esx_temp/meerkat/ >>> total 8 >>> drwxr-xr-x 2 root root 2 Aug 13 12:54 . >>> drwxr-xr-x 5 root root 5 Aug 13 12:54 .. >>> # ls -la /zp/nfs/esx_temp/panda/ >>> total 8 >>> drwxr-xr-x 2 root root 2 Aug 16 19:39 . >>> drwxr-xr-x 5 root root 5 Aug 13 12:54 .. >>> # >>> >>> Could there be something super-hidden, which I can''t see here? >>> >>> There don''t appear to be any snapshots relating to zp/nfs/esx_temp. >>> >>> On a suggestion, I have ran the following: >>> >>> # zfs list -r zp/nfs/esx_temp >>> NAME USED AVAIL REFER MOUNTPOINT >>> zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp >>> # du -sh /zp/nfs/esx_temp >>> 8K /zp/nfs/esx_temp >>> # >>> >> >> Does >> >> zfs list -t snapshot -r zp/nfs/esx_temp >> >> show anything? >> >> What about >> >> zfs get refquota,refreservation,quota,reservation zp/fs/esx_tmp >> > > pardon me for butting in .. but I thought that was a spelling error. > > It wasn''t : > > # zfs get refquota,refreservation,quota,reservation fibre0 > NAME ? ?PROPERTY ? ? ? ?VALUE ? ? ?SOURCE > fibre0 ?refquota ? ? ? ?none ? ? ? default > fibre0 ?refreservation ?none ? ? ? default > fibre0 ?quota ? ? ? ? ? none ? ? ? default > fibre0 ?reservation ? ? none ? ? ? default > > what the heck is refreservation ?? ?8-) >PSARC/2009/204 ZFS user/group quotas & space accounting [1] Integrated in build 114 [1] http://arc.opensolaris.org/caselog/PSARC/2009/204/ [2] http://mountall.blogspot.com/2009/05/sxce-build-114-is-out.html -- Regards, Cyril
>> what the heck is refreservation ?? ?8-) >> > > PSARC/2009/204 ZFS user/group quotas & space accounting [1] > Integrated in build 114 > > [1] http://arc.opensolaris.org/caselog/PSARC/2009/204/ > [2] http://mountall.blogspot.com/2009/05/sxce-build-114-is-out.htmlthat was fast . Cyril, long time no hear. :-( Hows life the universe and risc processors for you these days ? -- Dennis Clarke dclarke at opensolaris.ca <- Email related to the open source Solaris dclarke at blastwave.org <- Email related to open source for Solaris ps: I have been busy porting as per usual. New 64-bit ready Tk/Tcl released today.
i''m not sure if it applies to your situation, but when i FIRST starting using ZFS on freebsd, i noticed a similar error, where a filesystem would be totally empty and still show 14 GB space...it''s weird but i remember this because i had quotas set (freebsd didn''t have refquotas at the time). We had a user who kept running out of space, but swore he shouldn'' be, so we deleted all the files and sure enough, it was empty with 14 gb still showing...the only way i was able to solve it was to destroy the filesystem and create it again...i remember this happening 2 times on freebsd 7.2 which used version 6 of ZFS (a much older version) I''m not sure if this is a common bug or not but i HAVE seen it. On Sun, Aug 16, 2009 at 4:49 PM, Chris Murray <no-reply at opensolaris.org>wrote:> Accidentally posted the below earlier against ZFS Code, rather than ZFS > Discuss. > > My ESXi box now uses ZFS filesystems which have been shared over NFS. > Spotted something odd this afternoon - a filesystem which I thought didn''t > have any files in it, weighs in at 14GB. Before I start deleting the empty > folders to see what happens, any ideas what''s happened here? > > # zfs list | grep temp > zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp > # ls -la /zp/nfs/esx_temp > total 20 > drwxr-xr-x 5 root root 5 Aug 13 12:54 . > drwxr-xr-x 7 root root 7 Aug 13 12:40 .. > drwxr-xr-x 2 root root 2 Aug 13 12:53 iguana > drwxr-xr-x 2 root root 2 Aug 13 12:54 meerkat > drwxr-xr-x 2 root root 2 Aug 16 19:39 panda > # ls -la /zp/nfs/esx_temp/iguana/ > total 8 > drwxr-xr-x 2 root root 2 Aug 13 12:53 . > drwxr-xr-x 5 root root 5 Aug 13 12:54 .. > # ls -la /zp/nfs/esx_temp/meerkat/ > total 8 > drwxr-xr-x 2 root root 2 Aug 13 12:54 . > drwxr-xr-x 5 root root 5 Aug 13 12:54 .. > # ls -la /zp/nfs/esx_temp/panda/ > total 8 > drwxr-xr-x 2 root root 2 Aug 16 19:39 . > drwxr-xr-x 5 root root 5 Aug 13 12:54 .. > # > > Could there be something super-hidden, which I can''t see here? > > There don''t appear to be any snapshots relating to zp/nfs/esx_temp. > > On a suggestion, I have ran the following: > > # zfs list -r zp/nfs/esx_temp > NAME USED AVAIL REFER MOUNTPOINT > zp/nfs/esx_temp 14.0G 225G 14.0G /zp/nfs/esx_temp > # du -sh /zp/nfs/esx_temp > 8K /zp/nfs/esx_temp > # > > Thanks, > Chris > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090816/a4e37f57/attachment.html>
I don''t have quotas set, so I think I''ll have to put this down to some sort of bug. I''m on SXCE 105 at the minute, ZFS version is 3, but zpool is version 13 (could be 14 if I upgrade). I don''t have everything backed-up so won''t do a "zpool upgrade" just at the minute. I think when SXCE 120 is released, I''ll install that, upgrade my pool and see if the filesystem still registers as 14GB. If it does, I''ll destroy and recreate - no biggie! :-) -- This message posted from opensolaris.org
Nicolas Williams
2009-Aug-18 19:13 UTC
[zfs-discuss] *Almost* empty ZFS filesystem - 14GB?
Perhaps an open 14GB, zero-link file?
Nico, what is a zero-link file, and how would I go about finding whether I have one? You''ll have to bear with me, I''m afraid, as I''m still building my Solaris knowledge at the minute - I was brought up on Windows. I use Solaris for my storage needs now though, and slowly improving on my knowledge so I can move away from Windows one day :) If it makes any difference, the problem persists after a full reboot, and I''ve deleted the three folders, so now there is literally nothing in that filesystem .. yet it reports 14GB. It''s not too much of an inconvenience, but it does make me wonder whether the ''used'' figures on my other filesystems and zvols are correct. -- This message posted from opensolaris.org
Mark Shellenbaum
2009-Aug-21 13:59 UTC
[zfs-discuss] *Almost* empty ZFS filesystem - 14GB?
Chris Murray wrote:> Nico, what is a zero-link file, and how would I go about finding whether I have one? You''ll have to bear with me, I''m afraid, as I''m still building my Solaris knowledge at the minute - I was brought up on Windows. I use Solaris for my storage needs now though, and slowly improving on my knowledge so I can move away from Windows one day :) > > If it makes any difference, the problem persists after a full reboot, and I''ve deleted the three folders, so now there is literally nothing in that filesystem .. yet it reports 14GB. > > It''s not too much of an inconvenience, but it does make me wonder whether the ''used'' figures on my other filesystems and zvols are correct.You could be running into an instance of 6792701 Removing large holey file does not free space A fix for this was integrated into build 118 -Mark
Francois Napoleoni
2009-Aug-21 14:36 UTC
[zfs-discuss] *Almost* empty ZFS filesystem - 14GB?
Yup try to see what the ouput of
# zdb -dddd <name of your pool>/<name of your fs>
if you find big file(s) without pathname you are in ...
it should look like this :
...
Object lvl iblk dblk lsize asize type
6 5 16K 128K 300G 70.0G ZFS plain file
264 bonus ZFS znode
path ???<object#6> <============================ this
uid 0
gid 0
atime Thu Mar 26 18:08:51 2009
mtime Thu Mar 26 18:12:42 2009
ctime Thu Mar 26 18:12:42 2009
crtime Thu Mar 26 18:08:51 2009
gen 6075
mode 100600
size 322122547200
parent 3
links 0
xattr 0
rdev 0x0000000000000000
...
but 14Gb ... we should rename
6792701 "Removing large holey file does not free space"
into
6792701 "Removing NOT THAT large holey file does not free space"
:)
F.
On 08/21/09 15:59, Mark Shellenbaum wrote:> Chris Murray wrote:
>> Nico, what is a zero-link file, and how would I go about finding
>> whether I have one? You''ll have to bear with me, I''m
afraid, as I''m
>> still building my Solaris knowledge at the minute - I was brought up
>> on Windows. I use Solaris for my storage needs now though, and slowly
>> improving on my knowledge so I can move away from Windows one day :)
>>
>> If it makes any difference, the problem persists after a full reboot,
>> and I''ve deleted the three folders, so now there is literally
nothing
>> in that filesystem .. yet it reports 14GB.
>>
>> It''s not too much of an inconvenience, but it does make me
wonder
>> whether the ''used'' figures on my other filesystems
and zvols are correct.
>
>
> You could be running into an instance of
>
> 6792701 Removing large holey file does not free space
>
> A fix for this was integrated into build 118
>
>
> -Mark
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
Francois Napoleoni / Sun Support Engineer
mail : francois.napoleoni at sun.com
phone : +33 (0)1 3403 1707
fax : +33 (0)1 3403 1114
Nicolas Williams
2009-Aug-21 17:50 UTC
[zfs-discuss] *Almost* empty ZFS filesystem - 14GB?
On Fri, Aug 21, 2009 at 06:46:32AM -0700, Chris Murray wrote:> Nico, what is a zero-link file, and how would I go about finding > whether I have one? You''ll have to bear with me, I''m afraid, as I''m > still building my Solaris knowledge at the minute - I was brought up > on Windows. I use Solaris for my storage needs now though, and slowly > improving on my knowledge so I can move away from Windows one day :)I see that Mark S. thinks this may be a specific ZFS bug, and there''s a followup with instructions on how to detect if that''s the case. However, it can also be a zero-link file. I''ve certainly run into that problem before myself, on UFS and other filesystems. A zero-link file is a file that has been removed (unlink(2)ed), but which remains open in some process(es). Such a file continues to consume space until the processes that have it open are killed. Typically you''d use pfiles(1) or lsof to find such files.> If it makes any difference, the problem persists after a full reboot,Yeah, if you rebooted and there''s no 14GB .nfs* files, then this is not a zero-link file. See the followups. Nico --
That looks like it indeed. Output of zdb -dddd
Object lvl iblk dblk lsize asize type
9 5 16K 8K 150G 14.0G ZFS plain file
264 bonus ZFS znode
path ???<object#9>
Thanks for the help in clearing this up - satisfies my curiosity. Nico,
I''ll add those commands to the little list in my mind and
they''ll push the Windows ones out in no time :)
I''ll go to b120 when it is available.
--
This message posted from opensolaris.org