Hi all, we have some problems with ZFS. Our configuration: X4100 + dual 3510 JBOD, 2 zpool, Solaris 10U7 # zfs create dr/netapp11bkpVOL34 # cd /dr/netapp11bkpVOL34 # rsync -av --numeric-ids --delete /netapp11/vol/vol34/* . # zfs list|egrep netapp11bkpVOL34 dr/netapp11bkpVOL34 1.34T 158G 1.34T /dr/netapp11bkpVOL34 # cd /dr/netapp11bkpVOL34 # rm -r * # ls -la # Now there are no files in /dr/netapp11bkpVOL34, but # zfs list|egrep netapp11bkpVOL34 dr/netapp11bkpVOL34 1.34T 158G 1.34T /dr/netapp11bkpVOL34 Space has not been freed up! gino -- This message posted from opensolaris.org
On Thu, Sep 10, 2009 at 3:27 PM, Gino<dandr.ch at gmail.com> wrote:> # cd /dr/netapp11bkpVOL34 > # rm -r * > # ls -la > # > > Now there are no files in /dr/netapp11bkpVOL34, but > > # zfs list|egrep netapp11bkpVOL34 > dr/netapp11bkpVOL34 ? ? ? ? ? ? ? ? ? ? ? 1.34T ? ? ? 158G ? ?1.34T ? ? ?/dr/netapp11bkpVOL34 > > Space has not been freed up!Are there hidden files in that directory? Try "ls -la" What happens when you export-import that pool? -- Fajar
Already done "ls -la". No hidden files here. Import/export doesn''t change anything. Done a "zfs destroy dr/netapp11bkpVOL34" and is running since 7 minutes with very high I/O gino Fajar A. Nugraha wrote:> On Thu, Sep 10, 2009 at 3:27 PM, Gino<dandr.ch at gmail.com> wrote: > >> # cd /dr/netapp11bkpVOL34 >> # rm -r * >> # ls -la >> # >> >> Now there are no files in /dr/netapp11bkpVOL34, but >> >> # zfs list|egrep netapp11bkpVOL34 >> dr/netapp11bkpVOL34 1.34T 158G 1.34T /dr/netapp11bkpVOL34 >> >> Space has not been freed up! >> > > Are there hidden files in that directory? Try "ls -la" > What happens when you export-import that pool? > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090910/d1b402e6/attachment.html>
On 10 Sep 2009, at 09:38, Fajar A. Nugraha wrote:> On Thu, Sep 10, 2009 at 3:27 PM, Gino<dandr.ch at gmail.com> wrote: >> # cd /dr/netapp11bkpVOL34 >> # rm -r * >> # ls -la >> # >> >> Now there are no files in /dr/netapp11bkpVOL34, but >> >> # zfs list|egrep netapp11bkpVOL34 >> dr/netapp11bkpVOL34 1.34T 158G >> 1.34T /dr/netapp11bkpVOL34 >> >> Space has not been freed up! > > Are there hidden files in that directory? Try "ls -la" > What happens when you export-import that pool?Also are there snapshots? "zfs list -t all" Cheers, Chris
> >> # cd /dr/netapp11bkpVOL34 > >> # rm -r * > >> # ls -la > >> # > >> > >> Now there are no files in /dr/netapp11bkpVOL34, > but > >> > >> # zfs list|egrep netapp11bkpVOL34 > >> dr/netapp11bkpVOL34 1.34T > 158G > 4T /dr/netapp11bkpVOL34 > >> > >> Space has not been freed up! > > > > Are there hidden files in that directory? Try "ls > -la" > > What happens when you export-import that pool? > > Also are there snapshots? "zfs list -t all" > > Cheers, > > Chrisno snapshot at all gino -- This message posted from opensolaris.org
On Thu, September 10, 2009 04:27, Gino wrote:> # cd /dr/netapp11bkpVOL34 > # rm -r * > # ls -la > > Now there are no files in /dr/netapp11bkpVOL34, but > > # zfs list|egrep netapp11bkpVOL34 > dr/netapp11bkpVOL34 1.34T 158G 1.34T > /dr/netapp11bkpVOL34 > > Space has not been freed up!The other common cause for this is if an application has an open file descriptor open. If you have ''lsof'' on the machine, can you try running it and seeing if any progress are still accessing the file system.
>On Thu, September 10, 2009 04:27, Gino wrote: > >> # cd /dr/netapp11bkpVOL34 >> # rm -r * >> # ls -la >> >> Now there are no files in /dr/netapp11bkpVOL34, but >> >> # zfs list|egrep netapp11bkpVOL34 >> dr/netapp11bkpVOL34 1.34T 158G 1.34T >> /dr/netapp11bkpVOL34 >> >> Space has not been freed up! > >The other common cause for this is if an application has an open file >descriptor open. If you have ''lsof'' on the machine, can you try running it >and seeing if any progress are still accessing the file system.or use the built-in tool: find /proc -type f -links 0 (Will list all files which have been unlinked but which are still being used by applications) Casper
> On Thu, September 10, 2009 04:27, Gino wrote: > > > # cd /dr/netapp11bkpVOL34 > > # rm -r * > > # ls -la > > > > Now there are no files in /dr/netapp11bkpVOL34, but > > > > # zfs list|egrep netapp11bkpVOL34 > > dr/netapp11bkpVOL34 1.34T > 158G 1.34T > netapp11bkpVOL34 > > > > Space has not been freed up! > > The other common cause for this is if an application > has an open file > descriptor open. If you have ''lsof'' on the machine, > can you try running it > and seeing if any progress are still accessing the > file system. >I know but there was no open file. We also rebooted the server .. gino -- This message posted from opensolaris.org
Actually there is great chance that you are hitting this bug :
"6792701 Removing large holey file does not free space"
To check run :
# zdb -dddd <name of your pool>/<name of your fs>
if you find object(s) without pathname you are in ...
it should look like this :
...
Object lvl iblk dblk lsize asize type
6 5 16K 128K 300G 70.0G ZFS plain file
264 bonus ZFS znode
path ???<object#6> <============================ this
uid 0
gid 0
atime Thu Mar 26 18:08:51 2009
mtime Thu Mar 26 18:12:42 2009
ctime Thu Mar 26 18:12:42 2009
crtime Thu Mar 26 18:08:51 2009
gen 6075
mode 100600
size 322122547200
parent 3
links 0
xattr 0
rdev 0x0000000000000000
...
F.
Casper.Dik at Sun.COM wrote:>> On Thu, September 10, 2009 04:27, Gino wrote:
>>
>>> # cd /dr/netapp11bkpVOL34
>>> # rm -r *
>>> # ls -la
>>>
>>> Now there are no files in /dr/netapp11bkpVOL34, but
>>>
>>> # zfs list|egrep netapp11bkpVOL34
>>> dr/netapp11bkpVOL34 1.34T 158G 1.34T
>>> /dr/netapp11bkpVOL34
>>>
>>> Space has not been freed up!
>> The other common cause for this is if an application has an open file
>> descriptor open. If you have ''lsof'' on the machine,
can you try running it
>> and seeing if any progress are still accessing the file system.
>
> or use the built-in tool:
>
>
> find /proc -type f -links 0
>
> (Will list all files which have been unlinked but which are still being
> used by applications)
>
> Casper
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
Francois Napoleoni / Sun Support Engineer
mail : francois.napoleoni at sun.com
phone : +33 (0)1 3403 1707
Francois you''re right! We just found that it''s happening only with files >100GB and S10U7. We have no problem with SNV_101a. gino> Actually there is great chance that you are hitting > this bug : > > "6792701 Removing large holey file does not free > space" > > > To check run : > > # zdb -dddd <name of your pool>/<name of your fs> > > if you find object(s) without pathname you are in ... > > it should look like this : > > ... > Object lvl iblk dblk lsize asize type > 6 5 16K 128K 300G 70.0G ZFS > plain file > 264 bonus ZFS znode > =========================== this > uid 0 > gid 0 > atime Thu Mar 26 18:08:51 2009 > mtime Thu Mar 26 18:12:42 2009 > ctime Thu Mar 26 18:12:42 2009 > crtime Thu Mar 26 18:08:51 2009 > gen 6075 > mode 100600 > size 322122547200 > parent 3 > links 0 > xattr 0 > rdev 0x0000000000000000 > > > > > > > > Casper.Dik at Sun.COM wrote: > >> On Thu, September 10, 2009 04:27, Gino wrote: > >> > >>> # cd /dr/netapp11bkpVOL34 > >>> # rm -r * > >>> # ls -la > >>> > >>> Now there are no files in /dr/netapp11bkpVOL34, > but > >>> > >>> # zfs list|egrep netapp11bkpVOL34 > >>> dr/netapp11bkpVOL34 1.34T > 158G 1.34T > r/netapp11bkpVOL34 > >>> > >>> Space has not been freed up! > >> The other common cause for this is if an > application has an open file > >> descriptor open. If you have ''lsof'' on the > machine, can you try running it > >> and seeing if any progress are still accessing the > file system. > > > > or use the built-in tool: > > > > > > find /proc -type f -links 0 > > > > (Will list all files which have been unlinked but > which are still being > > used by applications) > > > > Casper > > > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss at opensolaris.org > > > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss > > -- > Francois Napoleoni / Sun Support Engineer > mail : francois.napoleoni at sun.com > phone : +33 (0)1 3403 1707 >-- This message posted from opensolaris.org