Ocfs2 users - Aug 2009 - Ghost files in OCFS2 filesystem

Hi,

I have encountered an issue on an Oracle RAC cluster using ocfs2, OS is RH Linux
5.3.

One of the ocfs2 filesystems appears to be 97% full, yet when I look at the
files in there they only equal about 13gig (filesystems is 40gig in size).

I have seen this sort of thing in HP-UX but that involved a process who's
output file was deleted but the process hadn't been stopped properly, once
we killed the offending process the space was released, but I can't seem to
find any process on this Linux server that is using or writing files to that
filesystem.

File system:

Filesystem            Size  Used Avail Use% Mounted on
/dev/emcpoweri1        40G   39G  1.9G  96% /oraexport
File listing: Other than directories, these are the only files in that
filesystem, nothing in lost+found either..

[root at aumel21db01cn01]# ll
total 11926784
-rw-rw---- 1 oracle oinstall 12210978816 Aug 21 05:23   00000.Full.090821.dmp
-rw-rw-r-- 1 oracle oinstall     1920327 Aug 21 05:23      00000.Full.090821.log
Any assistance with what is going on would be greatly appreciated.


________________________________
This e-mail and any attachments may be confidential or legally privileged.
If you received this message in error or are not the intended recipient, you
should destroy the e-mail message and any attachments or copies, and you are
prohibited from retaining, distributing, disclosing or using any information
contained herein. Please inform us of the erroneous delivery by return
e-mail.

Thank you for your cooperation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090821/aa4d0d4e/attachment.html

Yes, this is a known issue in OCFS2 1.4.1 and 1.4.2. That is assuming
no process in the cluster has that file open. We have the fix. It will be
available with 1.4.3 which is in testing.

This was discussed in the email announcing the 1.4.2 release.
===================================================================http://oss.oracle.com/pipermail/ocfs2-announce/2009-June/000028.html

1. Oracle# 7643059 - Orphan files not getting deleted
When one unlinks a file, its inode is moved to the orphan directory and
deleted when it is no longer in-use across the cluster. As part of the
scheme, the node that unlinks the file, informs interested nodes of the
same, asking the last node to stop using that inode to recover the space
allocated to it. However, this scheme fails if memory pressure forces a
node to forget to delete the inode on close.
This issue was introduced in OCFS2 1.4.1. While we have fixed this issue,
the fix did not make it into this release. Users running into this issue
can call Oracle Support and ask for an interim release with this fix.
===================================================================
Workaround:
If on 1.4.2, mount the fs on another node. Chances are it will delete
the orphans. If not or if you are on 1.4.1, umount vol on all nodes and
run fsck.ocfs2 -f.

You could ping support to get an interim fix. But we are close to releasing
1.4.3. So maybe better if you wait for that.

Sunil


Shave, Chris wrote:
> Hi,
>  
> I have encountered an issue on an Oracle RAC cluster using ocfs2, OS 
> is RH Linux 5.3.
>  
> One of the ocfs2 filesystems appears to be 97% full, yet when I look 
> at the files in there they only equal about 13gig (filesystems is 
> 40gig in size).
>  
> I have seen this sort of thing in HP-UX but that involved a process 
> who's output file was deleted but the process hadn't been stopped 
> properly, once we killed the offending process the space was released, 
> but I can't seem to find any process on this Linux server that is 
> using or writing files to that filesystem.
>  
> File system:
>  
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/emcpoweri1        40G   39G  1.9G  96% /oraexport
> File listing: Other than directories, these are the only files in that 
> filesystem, nothing in lost+found either..
>  
> [root at aumel21db01cn01]# ll
> total 11926784
> -rw-rw---- 1 oracle oinstall 12210978816 Aug 21 05:23   
> 00000.Full.090821.dmp
> -rw-rw-r-- 1 oracle oinstall     1920327 Aug 21 05:23      
> 00000.Full.090821.log
> Any assistance with what is going on would be greatly appreciated.