During our tests we noticed very disturbing behavior, what would be causing
this?
System is running latest stable opensolaris.
Any other means to remove ghost files rather than destroying pool and restoring
from backups?
root@~# zpool status testpool
pool: testpool
state: ONLINE
scrub: scrub completed after 0h26m with 0 errors on Fri Jul 24 10:32:09 2009
config:
NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c0t5000C5000505C31Bd0 ONLINE 0 0 0
c0t5000C5000498A9D3d0 ONLINE 0 0 0
c0t5000C5000505B523d0 ONLINE 0 0 0
c0t5000C5000505BB83d0 ONLINE 0 0 0
c0t5000C5000505B727d0 ONLINE 0 0 0
c0t5000C50004987B6Bd0 ONLINE 0 0 0
errors: No known data errors
root@~# zpool list testpool
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
testpool 408G 402G 6.37G 98% ONLINE -
root@~# ls -lasht /testpool/
total 4.0K
1.5K drwxr-xr-x 29 root root 30 2009-07-24 09:56 ..
2.5K drwxr-xr-x 2 root root 2 2009-07-23 18:23 .
root@~# df /testpool
Filesystem 1K-blocks Used Available Use% Mounted on
testpool 280481377 280481377 0 100% /testpool
root@~# df -i /testpool
Filesystem Inodes IUsed IFree IUse% Mounted on
testpool 7 7 0 100% /testpool
root@~# zdb -dddd testpool
...
Object lvl iblk dblk lsize asize type
6 5 16K 128K 1000G 262G ZFS plain file
264 bonus ZFS znode
path ???<object#6>
uid 0
gid 0
atime Thu Jul 23 17:23:19 2009
mtime Thu Jul 23 17:50:17 2009
ctime Thu Jul 23 17:50:17 2009
crtime Thu Jul 23 17:23:19 2009
gen 19
mode 100600
size 1073741824000
parent 3
links 0
xattr 0
rdev 0x0000000000000000
Yours
Markus Kovero
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090724/a66b6d56/attachment.html>
On Fri, Jul 24, 2009 at 09:33, Markus Kovero<Markus.Kovero at nebula.fi> wrote:> During our tests we noticed very disturbing behavior, what would be causing > this? > > System is running latest stable opensolaris. > > Any other means to remove ghost files rather than destroying pool and > restoring from backups?You may have snapshots, try: zfs list -t snapshot
root@~# zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT rpool/ROOT/opensolaris at install 146M - 2.82G - root@~# -----Original Message----- From: pantzare at gmail.com [mailto:pantzare at gmail.com] On Behalf Of Mattias Pantzare Sent: 24. hein?kuuta 2009 10:56 To: Markus Kovero Cc: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] No files but pool is full? On Fri, Jul 24, 2009 at 09:33, Markus Kovero<Markus.Kovero at nebula.fi> wrote:> During our tests we noticed very disturbing behavior, what would be causing > this? > > System is running latest stable opensolaris. > > Any other means to remove ghost files rather than destroying pool and > restoring from backups?You may have snapshots, try: zfs list -t snapshot
On Fri, Jul 24, 2009 at 09:57, Markus Kovero<Markus.Kovero at nebula.fi> wrote:> root@~# zfs list -t snapshot > NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? USED ?AVAIL ?REFER ?MOUNTPOINT > rpool/ROOT/opensolaris at install ? 146M ? ? ?- ?2.82G ?- > root@~#Then it is probably some process that has a deleted file open. You can find those with: fuser -c /testpool But if you can''t find the space after a reboot something is not right...> > -----Original Message----- > From: pantzare at gmail.com [mailto:pantzare at gmail.com] On Behalf Of Mattias Pantzare > Sent: 24. hein?kuuta 2009 10:56 > To: Markus Kovero > Cc: zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] No files but pool is full? > > On Fri, Jul 24, 2009 at 09:33, Markus Kovero<Markus.Kovero at nebula.fi> wrote: >> During our tests we noticed very disturbing behavior, what would be causing >> this? >> >> System is running latest stable opensolaris. >> >> Any other means to remove ghost files rather than destroying pool and >> restoring from backups? > > You may have snapshots, try: > zfs list -t snapshot > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Yes, server has been rebooted several times and there is no available space, is it possible to delete ghosts that zdb sees somehow? how this can happen? Yours Markus Kovero -----Original Message----- From: pantzare at gmail.com [mailto:pantzare at gmail.com] On Behalf Of Mattias Pantzare Sent: 24. hein?kuuta 2009 11:22 To: Markus Kovero Cc: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] No files but pool is full? On Fri, Jul 24, 2009 at 09:57, Markus Kovero<Markus.Kovero at nebula.fi> wrote:> root@~# zfs list -t snapshot > NAME ? ? ? ? ? ? ? ? ? ? ? ? ? ? USED ?AVAIL ?REFER ?MOUNTPOINT > rpool/ROOT/opensolaris at install ? 146M ? ? ?- ?2.82G ?- > root@~#Then it is probably some process that has a deleted file open. You can find those with: fuser -c /testpool But if you can''t find the space after a reboot something is not right...> > -----Original Message----- > From: pantzare at gmail.com [mailto:pantzare at gmail.com] On Behalf Of Mattias Pantzare > Sent: 24. hein?kuuta 2009 10:56 > To: Markus Kovero > Cc: zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] No files but pool is full? > > On Fri, Jul 24, 2009 at 09:33, Markus Kovero<Markus.Kovero at nebula.fi> wrote: >> During our tests we noticed very disturbing behavior, what would be causing >> this? >> >> System is running latest stable opensolaris. >> >> Any other means to remove ghost files rather than destroying pool and >> restoring from backups? > > You may have snapshots, try: > zfs list -t snapshot > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
On 24 jul 2009, at 09.33, Markus Kovero <Markus.Kovero at nebula.fi> wrote:> During our tests we noticed very disturbing behavior, what would be > causing this? > > System is running latest stable opensolaris. > > Any other means to remove ghost files rather than destroying pool > and restoring from backups? > >This looks like bug i filed a while ago, CR 6792701 removing large holey files does bot free space. The only solution I found to clean the pool when isolating the bug was to recreate it. The fix was integrated inbuild post OSOL 2009.06. Mkfile of a certain size will trigger this. Henrik http://sparcv9.blogspot.com
Hi, thanks for pointing out issue, we haven''t run updates on server yet. Yours Markus Kovero -----Original Message----- From: Henrik Johansson [mailto:henrikj at henkis.net] Sent: 24. hein?kuuta 2009 12:26 To: Markus Kovero Cc: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] No files but pool is full? On 24 jul 2009, at 09.33, Markus Kovero <Markus.Kovero at nebula.fi> wrote:> During our tests we noticed very disturbing behavior, what would be > causing this? > > System is running latest stable opensolaris. > > Any other means to remove ghost files rather than destroying pool > and restoring from backups? > >This looks like bug i filed a while ago, CR 6792701 removing large holey files does bot free space. The only solution I found to clean the pool when isolating the bug was to recreate it. The fix was integrated inbuild post OSOL 2009.06. Mkfile of a certain size will trigger this. Henrik http://sparcv9.blogspot.com