Tim Hughes
2009-Feb-24 19:08 UTC
[Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ?
I am looking for a little help with some orphaned files that are taking up diskspace. I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from a three node ocfs2 cluster. The files appeared removed from the filesystem but the results of a `df -h /var/lib/mysql` showed that no disk space has been cleared. A `du -sh /var/lib/mysql` on the other hand says that ~ 30GB was removed. <snip> [root at host1 mysql]# du -sh /var/lib/mysql 50G /var/lib/mysql [root at host1 mysql]# df -h /var/lib/mysql/ Filesystem Size Used Avail Use% Mounted on /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql [root at host1 mysql]# <snip> First thoughts were that mysql was holding the file descriptors open but `lsof` showed nothing. After trying a few other things we discovered that the files were orphans of ocfs2. <snip> [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001" /dev/xvdb1 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 . 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 .. 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14 0000000000103eba 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14 0000000000103eb9 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33 0000000000103ebd 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37 0000000000103ebc 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18 00000000001236d9 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10 00000000001236da 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29 00000000001236db 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37 00000000001236dc 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59 00000000001236dd 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36 00000000001236de 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05 00000000001236df 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18 00000000001236e0 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46 00000000001236e1 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06 00000000001236e2 <snip> We have shutdown mysql and tried unmounting and mounting the ocfs2 file system from the node where we deleted the files. This made no difference so we decided to take down the mysql cluster and unmounting the ocfs2 filesystem from all nodes just incase something was for some reason holding these files open that we couldn't find. This didn't clear the space and the orphans were still there. Next we unmounted it all again and ran `fsck.ocfs2 /dev/xvdb1` on it with the following results which indicated there was nothing wrong. <snip> Checking OCFS2 filesystem in /dev/xvdb1: label: db-store01 uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb number of blocks: 26214047 bytes per block: 4096 number of clusters: 26214047 bytes per cluster: 4096 max slots: 16 /dev/xvdb1 is clean. It will be checked after 20 additional mounts. <snip> We are running : ocfs2-tools-1.4.1-1.el5 ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5 with kernel 2.6.18-92.1.10.el5xen on RHEL5 How do we reclaim this space ? Tim Hughes www.tradefair.com Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11 4AN The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company.
Tim Hughes
2009-Feb-27 18:32 UTC
[Ocfs2-users] Fwd: how do I remove ocfs2 orphaned files that are taking up diskspace ?
I dont think this made it the first time so I am sending it again I am looking for a little help with some orphaned files that are taking up diskspace. I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from a three node ocfs2 cluster. The files appeared removed from the filesystem but the results of a `df -h /var/lib/mysql` showed that no disk space has been cleared. A `du -sh /var/lib/mysql` on the other hand says that ~ 30GB was removed. <snip> [root at host1 mysql]# du -sh /var/lib/mysql 50G /var/lib/mysql [root at host1 mysql]# df -h /var/lib/mysql/ Filesystem Size Used Avail Use% Mounted on /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql [root at host1 mysql]# <snip> First thoughts were that mysql was holding the file descriptors open but `lsof` showed nothing. After trying a few other things we discovered that the files were orphans of ocfs2. <snip> [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001" /dev/xvdb1 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 . 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 .. 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14 0000000000103eba 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14 0000000000103eb9 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33 0000000000103ebd 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37 0000000000103ebc 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18 00000000001236d9 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10 00000000001236da 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29 00000000001236db 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37 00000000001236dc 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59 00000000001236dd 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36 00000000001236de 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05 00000000001236df 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18 00000000001236e0 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46 00000000001236e1 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06 00000000001236e2 <snip> We have shutdown mysql and tried unmounting and mounting the ocfs2 file system from the node where we deleted the files. This made no difference so we decided to take down the mysql cluster and unmounting the ocfs2 filesystem from all nodes just incase something was for some reason holding these files open that we couldn't find. This didn't clear the space and the orphans were still there. Next we unmounted it all again and ran `fsck.ocfs2 /dev/xvdb1` on it with the following results which indicated there was nothing wrong. <snip> Checking OCFS2 filesystem in /dev/xvdb1: label: db-store01 uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb number of blocks: 26214047 bytes per block: 4096 number of clusters: 26214047 bytes per cluster: 4096 max slots: 16 /dev/xvdb1 is clean. It will be checked after 20 additional mounts. <snip> We are running : ocfs2-tools-1.4.1-1.el5 ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5 with kernel 2.6.18-92.1.10.el5xen on RHEL5 How do we reclaim this space ? Tim Hughes www.tradefair.com Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11 4AN The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090227/5510d52d/attachment-0001.html
Daniel Keisling
2009-Feb-27 22:18 UTC
[Ocfs2-users] how do I remove ocfs2 orphaned files that are takingup diskspace ?
I had the exact same problem today. Use the '-f' option to fsck.> -----Original Message----- > From: ocfs2-users-bounces at oss.oracle.com > [mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of Tim Hughes > Sent: Tuesday, February 24, 2009 1:08 PM > To: ocfs2-users at oss.oracle.com > Subject: [Ocfs2-users] how do I remove ocfs2 orphaned files > that are takingup diskspace ? > > > > I am looking for a little help with some orphaned files that > are taking up diskspace. > > I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files > from a three node ocfs2 cluster. The files appeared removed > from the filesystem but the results of a `df -h > /var/lib/mysql` showed that no disk space has been cleared. A > `du -sh /var/lib/mysql` on the other hand says that ~ 30GB > was removed. > > <snip> > [root at host1 mysql]# du -sh /var/lib/mysql > 50G /var/lib/mysql > [root at host1 mysql]# df -h /var/lib/mysql/ > Filesystem Size Used Avail Use% Mounted on > /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql > [root at host1 mysql]# > <snip> > > First thoughts were that mysql was holding the file > descriptors open but `lsof` showed nothing. After trying a > few other things we discovered that the files were orphans of ocfs2. > > <snip> > [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001"/dev/xvdb1> 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 . > 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 .. > 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14 0000000000103eba > 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb > 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14 0000000000103eb9 > 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33 0000000000103ebd > 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37 0000000000103ebc > 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18 > 00000000001236d9 > 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10 > 00000000001236da > 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29 > 00000000001236db > 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37 > 00000000001236dc > 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59 > 00000000001236dd > 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36 > 00000000001236de > 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05 > 00000000001236df > 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18 > 00000000001236e0 > 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46 > 00000000001236e1 > 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06 > 00000000001236e2 > <snip> > > We have shutdown mysql and tried unmounting and mounting the > ocfs2 file system from the node where we deleted the files. > This made no difference so we decided to take down the mysql > cluster and unmounting the ocfs2 filesystem from all nodes > just incase something was for some reason holding these files > open that we couldn't find. This didn't clear the space and > the orphans were still there. > > Next we unmounted it all again and ran `fsck.ocfs2 > /dev/xvdb1` on it with the following results which indicated > there was nothing wrong. > > <snip> > Checking OCFS2 filesystem in /dev/xvdb1: > label: db-store01 > uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb > number of blocks: 26214047 > bytes per block: 4096 > number of clusters: 26214047 > bytes per cluster: 4096 > max slots: 16 > > /dev/xvdb1 is clean. It will be checked after 20 additional mounts. > <snip> > > We are running : > ocfs2-tools-1.4.1-1.el5 > ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5 > > with kernel 2.6.18-92.1.10.el5xen on RHEL5 > > > How do we reclaim this space ? > > > > Tim Hughes > > www.tradefair.com > > Tradefair | Level 2, Yellow Building | 1 Nicholas Road | > London | W11 4AN > > The information in this e-mail and any attachment is > confidential and is intended only for the named recipient(s). > The e-mail may not be disclosed or used by any person other > than the addressee, nor may it be copied in any way. If you > are not a named recipient please notify the sender > immediately and delete any copies of this message. Any > unauthorized copying, disclosure or distribution of the > material in this e-mail is strictly forbidden. Any view or > opinions presented are solely those of the author and do not > necessarily represent those of the company. > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users > >______________________________________________________________________ This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
Srinivas Eeda
2009-Feb-27 22:21 UTC
[Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ?
please run fsck.ocfs2 -f to clean the orphans. If you can reproduce it at will, can you please enable tracing debugfs.ocfs2 -l JOURNAL, SUPER, INODE, NAMEI, DLM_GLUE ENTRY EXIT allow and send us the messages files from all nodes. thanks, --Srini Tim Hughes wrote:> I am looking for a little help with some orphaned files that are taking up diskspace. > > I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from a three node ocfs2 cluster. The files appeared removed from the filesystem but the results of a `df -h /var/lib/mysql` showed that no disk space has been cleared. A `du -sh /var/lib/mysql` on the other hand says that ~ 30GB was removed. > > <snip> > [root at host1 mysql]# du -sh /var/lib/mysql > 50G /var/lib/mysql > [root at host1 mysql]# df -h /var/lib/mysql/ > Filesystem Size Used Avail Use% Mounted on > /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql > [root at host1 mysql]# > <snip> > > First thoughts were that mysql was holding the file descriptors open but `lsof` showed nothing. After trying a few other things we discovered that the files were orphans of ocfs2. > > <snip> > [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001" /dev/xvdb1 > 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 . > 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 .. > 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14 0000000000103eba > 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb > 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14 0000000000103eb9 > 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33 0000000000103ebd > 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37 0000000000103ebc > 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18 00000000001236d9 > 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10 00000000001236da > 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29 00000000001236db > 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37 00000000001236dc > 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59 00000000001236dd > 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36 00000000001236de > 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05 00000000001236df > 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18 00000000001236e0 > 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46 00000000001236e1 > 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06 00000000001236e2 > <snip> > > We have shutdown mysql and tried unmounting and mounting the ocfs2 file system from the node where we deleted the files. This made no difference so we decided to take down the mysql cluster and unmounting the ocfs2 filesystem from all nodes just incase something was for some reason holding these files open that we couldn't find. This didn't clear the space and the orphans were still there. > > Next we unmounted it all again and ran `fsck.ocfs2 /dev/xvdb1` on it with the following results which indicated there was nothing wrong. > > <snip> > Checking OCFS2 filesystem in /dev/xvdb1: > label: db-store01 > uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb > number of blocks: 26214047 > bytes per block: 4096 > number of clusters: 26214047 > bytes per cluster: 4096 > max slots: 16 > > /dev/xvdb1 is clean. It will be checked after 20 additional mounts. > <snip> > > We are running : > ocfs2-tools-1.4.1-1.el5 > ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5 > > with kernel 2.6.18-92.1.10.el5xen on RHEL5 > > > How do we reclaim this space ? > > > > Tim Hughes > > www.tradefair.com > > Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11 4AN > > The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company. > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users >
Daniel Keisling
2009-Feb-27 22:46 UTC
[Ocfs2-users] how do I remove ocfs2 orphaned files that are takingup diskspace ?
The same thing happened to me today. I had to run 'fsck -f' to clear the orphaned inodes. The only thing that recently changed was I resized the partition earlier in the week.> -----Original Message----- > From: ocfs2-users-bounces at oss.oracle.com > [mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of Tim Hughes > Sent: Tuesday, February 24, 2009 1:08 PM > To: ocfs2-users at oss.oracle.com > Subject: [Ocfs2-users] how do I remove ocfs2 orphaned files > that are takingup diskspace ? > > > > I am looking for a little help with some orphaned files that > are taking up diskspace. > > I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files > from a three node ocfs2 cluster. The files appeared removed > from the filesystem but the results of a `df -h > /var/lib/mysql` showed that no disk space has been cleared. A > `du -sh /var/lib/mysql` on the other hand says that ~ 30GB > was removed. > > <snip> > [root at host1 mysql]# du -sh /var/lib/mysql > 50G /var/lib/mysql > [root at host1 mysql]# df -h /var/lib/mysql/ > Filesystem Size Used Avail Use% Mounted on > /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql > [root at host1 mysql]# > <snip> > > First thoughts were that mysql was holding the file > descriptors open but `lsof` showed nothing. After trying a > few other things we discovered that the files were orphans of ocfs2. > > <snip> > [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001"/dev/xvdb1> 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 . > 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 .. > 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14 0000000000103eba > 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb > 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14 0000000000103eb9 > 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33 0000000000103ebd > 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37 0000000000103ebc > 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18 > 00000000001236d9 > 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10 > 00000000001236da > 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29 > 00000000001236db > 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37 > 00000000001236dc > 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59 > 00000000001236dd > 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36 > 00000000001236de > 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05 > 00000000001236df > 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18 > 00000000001236e0 > 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46 > 00000000001236e1 > 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06 > 00000000001236e2 > <snip> > > We have shutdown mysql and tried unmounting and mounting the > ocfs2 file system from the node where we deleted the files. > This made no difference so we decided to take down the mysql > cluster and unmounting the ocfs2 filesystem from all nodes > just incase something was for some reason holding these files > open that we couldn't find. This didn't clear the space and > the orphans were still there. > > Next we unmounted it all again and ran `fsck.ocfs2 > /dev/xvdb1` on it with the following results which indicated > there was nothing wrong. > > <snip> > Checking OCFS2 filesystem in /dev/xvdb1: > label: db-store01 > uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb > number of blocks: 26214047 > bytes per block: 4096 > number of clusters: 26214047 > bytes per cluster: 4096 > max slots: 16 > > /dev/xvdb1 is clean. It will be checked after 20 additional mounts. > <snip> > > We are running : > ocfs2-tools-1.4.1-1.el5 > ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5 > > with kernel 2.6.18-92.1.10.el5xen on RHEL5 > > > How do we reclaim this space ? > > > > Tim Hughes > > www.tradefair.com > > Tradefair | Level 2, Yellow Building | 1 Nicholas Road | > London | W11 4AN > > The information in this e-mail and any attachment is > confidential and is intended only for the named recipient(s). > The e-mail may not be disclosed or used by any person other > than the addressee, nor may it be copied in any way. If you > are not a named recipient please notify the sender > immediately and delete any copies of this message. Any > unauthorized copying, disclosure or distribution of the > material in this e-mail is strictly forbidden. Any view or > opinions presented are solely those of the author and do not > necessarily represent those of the company. > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users > >______________________________________________________________________ This email transmission and any documents, files or previous email messages attached to it may contain information that is confidential or legally privileged. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you are hereby notified that you must not read this transmission and that any disclosure, copying, printing, distribution or use of this transmission is strictly prohibited. If you have received this transmission in error, please immediately notify the sender by telephone or return email and delete the original transmission and its attachments without reading or saving in any manner.
Joel Becker
2009-Feb-28 00:21 UTC
[Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ?
On Tue, Feb 24, 2009 at 07:08:00PM +0000, Tim Hughes wrote:> I am looking for a little help with some orphaned files that are taking up diskspace. > > I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from a three node ocfs2 cluster. The files appeared removed from the filesystem but the results of a `df -h /var/lib/mysql` showed that no disk space has been cleared. A `du -sh /var/lib/mysql` on the other hand says that ~ 30GB was removed.Others have pointed you to 'fsck -f'. What I want to know is whether you had any nodes die or killed while this filesystem was running? Eg an 'xm destroy' of a vm that had the filesystem mounted. That sort of thing. Joel -- Life's Little Instruction Book #407 "Every once in a while, take the scenic route." Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
Jan Kara
2009-Mar-02 16:24 UTC
[Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ?
> On Tue, Feb 24, 2009 at 07:08:00PM +0000, Tim Hughes wrote: > > I am looking for a little help with some orphaned files that are taking up diskspace. > > > > I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from a three node ocfs2 cluster. The files appeared removed from the filesystem but the results of a `df -h /var/lib/mysql` showed that no disk space has been cleared. A `du -sh /var/lib/mysql` on the other hand says that ~ 30GB was removed. > > Others have pointed you to 'fsck -f'. What I want to know is > whether you had any nodes die or killed while this filesystem was > running? Eg an 'xm destroy' of a vm that had the filesystem mounted. > That sort of thing.I think this could be caused by a problem I've described in: http://www.mail-archive.com/ocfs2-devel at oss.oracle.com/msg02886.html The question is what is the right fix for this race. I describe one possibility in http://www.mail-archive.com/ocfs2-devel at oss.oracle.com/msg02938.html but I got no response so far and honestly it is a bit hacky solution. I can write the fix if people here agree that that's the way to go. Honza
Tim Hughes
2009-Mar-10 16:27 UTC
[Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ?
Sorry about the delay on this. I managed to get some downtime on the system and the "debugfs.ocfs2 -f" command did the trick. I have taken a snapshot of the lun while it was in the bad state so if I get some time I will try and investigate it further. Srinivas, Is there anything you would like me to run on this snapshot to get some diagnostics for you or will i need to have those traces running when the issue initially occures ? In its current state anything I delete from the snapshot causes orphans. Would that be enough to get some traces out of it? Tim Hughes www.tradefair.com Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11 4AN The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company. ----- Original Message ----- From: "Srinivas Eeda" <srinivas.eeda at oracle.com> To: "Tim Hughes" <tim.hughes at tradefair.com> Cc: ocfs2-users at oss.oracle.com Sent: Friday, 27 February, 2009 10:21:50 PM GMT +00:00 GMT Britain, Ireland, Portugal Subject: Re: [Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ? please run fsck.ocfs2 -f to clean the orphans. If you can reproduce it at will, can you please enable tracing debugfs.ocfs2 -l JOURNAL, SUPER, INODE, NAMEI, DLM_GLUE ENTRY EXIT allow and send us the messages files from all nodes. thanks, --Srini Tim Hughes wrote:> I am looking for a little help with some orphaned files that are taking up diskspace. > > I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from a three node ocfs2 cluster. The files appeared removed from the filesystem but the results of a `df -h /var/lib/mysql` showed that no disk space has been cleared. A `du -sh /var/lib/mysql` on the other hand says that ~ 30GB was removed. > > <snip> > [root at host1 mysql]# du -sh /var/lib/mysql > 50G /var/lib/mysql > [root at host1 mysql]# df -h /var/lib/mysql/ > Filesystem Size Used Avail Use% Mounted on > /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql > [root at host1 mysql]# > <snip> > > First thoughts were that mysql was holding the file descriptors open but `lsof` showed nothing. After trying a few other things we discovered that the files were orphans of ocfs2. > > <snip> > [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001" /dev/xvdb1 > 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 . > 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 .. > 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14 0000000000103eba > 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb > 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14 0000000000103eb9 > 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33 0000000000103ebd > 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37 0000000000103ebc > 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18 00000000001236d9 > 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10 00000000001236da > 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29 00000000001236db > 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37 00000000001236dc > 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59 00000000001236dd > 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36 00000000001236de > 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05 00000000001236df > 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18 00000000001236e0 > 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46 00000000001236e1 > 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06 00000000001236e2 > <snip> > > We have shutdown mysql and tried unmounting and mounting the ocfs2 file system from the node where we deleted the files. This made no difference so we decided to take down the mysql cluster and unmounting the ocfs2 filesystem from all nodes just incase something was for some reason holding these files open that we couldn't find. This didn't clear the space and the orphans were still there. > > Next we unmounted it all again and ran `fsck.ocfs2 /dev/xvdb1` on it with the following results which indicated there was nothing wrong. > > <snip> > Checking OCFS2 filesystem in /dev/xvdb1: > label: db-store01 > uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb > number of blocks: 26214047 > bytes per block: 4096 > number of clusters: 26214047 > bytes per cluster: 4096 > max slots: 16 > > /dev/xvdb1 is clean. It will be checked after 20 additional mounts. > <snip> > > We are running : > ocfs2-tools-1.4.1-1.el5 > ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5 > > with kernel 2.6.18-92.1.10.el5xen on RHEL5 > > > How do we reclaim this space ? > > > > Tim Hughes > > www.tradefair.com > > Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11 4AN > > The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company. > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090310/53bdfd78/attachment.html