Ocfs2 devel - Jul 2011 - [PATCH 4/4] ocfs2: limit the number of inodes in orphanscan

During orphanscan, limit the scan number in a single loop to avoid OOM.

Signed-off-by: Wengang Wang <wen.gang.wang at oracle.com>
---
 fs/ocfs2/journal.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/fs/ocfs2/journal.c b/fs/ocfs2/journal.c
index 7d4e1ca..1c0487b 100644
--- a/fs/ocfs2/journal.c
+++ b/fs/ocfs2/journal.c
@@ -56,6 +56,7 @@
 DEFINE_SPINLOCK(trans_inc_lock);
 
 #define ORPHAN_SCAN_SCHEDULE_TIMEOUT 300000
+#define ORPHAN_SCAN_ITEMS_PER_LOOP 1000
 
 static int ocfs2_force_read_journal(struct inode *inode);
 static int ocfs2_recover_node(struct ocfs2_super *osb,
@@ -1926,6 +1927,7 @@ struct ocfs2_orphan_filldir_priv {
 	struct inode		*head;
 	struct ocfs2_super	*osb;
 	struct 			inode *orphan_dir;
+	int			count;
 };
 
 static int ocfs2_orphan_filldir(void *priv, const char *name, int name_len,
@@ -1964,6 +1966,12 @@ static int ocfs2_orphan_filldir(void *priv, const char
*name, int name_len,
 	}
 	spin_unlock(&o_orphandir->ip_lock);
 
+	/*
+	 * reading limited number of inodes in a single loop to avoid OOM
+	 */
+	if ( ++p->count == ORPHAN_SCAN_ITEMS_PER_LOOP )
+		return -EAGAIN;
+
 	return 0;
 }
 
@@ -1978,6 +1986,7 @@ static int ocfs2_queue_orphans(struct ocfs2_super *osb,
 
 	priv.osb = osb;
 	priv.head = *head;
+	priv.count = 0;
 
 	orphan_dir_inode = ocfs2_get_system_file_inode(osb,
 						       ORPHAN_DIR_SYSTEM_INODE,
-- 
1.7.5.2

On Fri, Jul 29, 2011 at 06:06:23PM +0800, Wengang Wang
wrote:
> During orphanscan, limit the scan number in a single loop to avoid OOM.
How does this OOM?  Please explain this in the patch and in the patch
description.  Or, conversely, free the resources the orphan scan is done
using.

Joel
> 
> Signed-off-by: Wengang Wang <wen.gang.wang at oracle.com>
> ---
>  fs/ocfs2/journal.c |    9 +++++++++
>  1 files changed, 9 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/ocfs2/journal.c b/fs/ocfs2/journal.c
> index 7d4e1ca..1c0487b 100644
> --- a/fs/ocfs2/journal.c
> +++ b/fs/ocfs2/journal.c
> @@ -56,6 +56,7 @@
>  DEFINE_SPINLOCK(trans_inc_lock);
>  
>  #define ORPHAN_SCAN_SCHEDULE_TIMEOUT 300000
> +#define ORPHAN_SCAN_ITEMS_PER_LOOP 1000
>  
>  static int ocfs2_force_read_journal(struct inode *inode);
>  static int ocfs2_recover_node(struct ocfs2_super *osb,
> @@ -1926,6 +1927,7 @@ struct ocfs2_orphan_filldir_priv {
>  	struct inode		*head;
>  	struct ocfs2_super	*osb;
>  	struct 			inode *orphan_dir;
> +	int			count;
>  };
>  
>  static int ocfs2_orphan_filldir(void *priv, const char *name, int
name_len,
> @@ -1964,6 +1966,12 @@ static int ocfs2_orphan_filldir(void *priv, const
char *name, int name_len,
>  	}
>  	spin_unlock(&o_orphandir->ip_lock);
>  
> +	/*
> +	 * reading limited number of inodes in a single loop to avoid OOM
> +	 */
> +	if ( ++p->count == ORPHAN_SCAN_ITEMS_PER_LOOP )
> +		return -EAGAIN;
> +
>  	return 0;
>  }
>  
> @@ -1978,6 +1986,7 @@ static int ocfs2_queue_orphans(struct ocfs2_super
*osb,
>  
>  	priv.osb = osb;
>  	priv.head = *head;
> +	priv.count = 0;
>  
>  	orphan_dir_inode = ocfs2_get_system_file_inode(osb,
>  						       ORPHAN_DIR_SYSTEM_INODE,
> -- 
> 1.7.5.2
> 
> 
> _______________________________________________
> Ocfs2-devel mailing list
> Ocfs2-devel at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-devel
-- 

"Get right to the heart of matters.
 It's the heart that matters more."

			http://www.jlbec.org/
			jlbec at evilplan.org

Hi Joel,

On 11-08-21 21:47, Joel Becker wrote:
> On Fri, Jul 29, 2011 at 06:06:23PM +0800, Wengang Wang wrote:
> > During orphanscan, limit the scan number in a single loop to avoid
OOM.
> 
> How does this OOM?  Please explain this in the patch and in the patch
> description.  Or, conversely, free the resources the orphan scan is done
> using.
Explaination why it OOM:
In current code, when recoverying orphan dirs, we are doing it like this:

lock orphandir
read out all the inodes under the orphandir to a list
unlock the orphandir
#above see ocfs2_queue_orphans()
and the iput all the inodes in the list to delete the on disk inode
#above see ocfs2_recover_orphans()

For normal case, there are no many files under orphandir, there is no problem.
But for an extreme case, there are a huge number of files under orphandir,
huge size of memory is needed to hold the inodes. It is possible to exhaust
memory, then OOM occurs.

The fix in this patch depends on patch
  "ocfs2: ocfs2_dir_foreach returns the error turned by filldir"
Since NAK for that patch, please ignore this too.

thanks,
wengang.
 
> Joel
> 
> >