Linux Virtualization - May 2018 - [dm-devel] [PATCH v5] fault-injection: introduce kvmalloc fallback options

On 04/25/2018 01:02 PM, Mikulas Patocka wrote:
> 
> 
> From: Mikulas Patocka <mpatocka at redhat.com>
> Subject: [PATCH v4] fault-injection: introduce kvmalloc fallback options
> 
> This patch introduces a fault-injection option
"kvmalloc_fallback". This
> option makes kvmalloc randomly fall back to vmalloc.
> 
> Unfortunatelly, some kernel code has bugs - it uses kvmalloc and then
  Unfortunately,
> uses DMA-API on the returned memory or frees it with kfree. Such bugs were
> found in the virtio-net driver, dm-integrity or RHEL7 powerpc-specific
> code. This options helps to test for these bugs.
> 
> The patch introduces a config option FAIL_KVMALLOC_FALLBACK_PROBABILITY.
> It can be enabled in distribution debug kernels, so that kvmalloc abuse
> can be tested by the users. The default can be overriden with
                                                 overridden
> "kvmalloc_fallback" parameter or in
/sys/kernel/debug/kvmalloc_fallback/.
> 
> Signed-off-by: Mikulas Patocka <mpatocka at redhat.com>
> 
> ---
>  Documentation/fault-injection/fault-injection.txt |    7 +++++
>  include/linux/fault-inject.h                      |    9 +++---
>  kernel/futex.c                                    |    2 -
>  lib/Kconfig.debug                                 |   15 +++++++++++
>  mm/failslab.c                                     |    2 -
>  mm/page_alloc.c                                   |    2 -
>  mm/util.c                                         |   30
++++++++++++++++++++++
>  7 files changed, 60 insertions(+), 7 deletions(-)
> 
> Index: linux-2.6/Documentation/fault-injection/fault-injection.txt
> ==================================================================> ---
linux-2.6.orig/Documentation/fault-injection/fault-injection.txt	2018-04-16
21:08:34.000000000 +0200
> +++ linux-2.6/Documentation/fault-injection/fault-injection.txt	2018-04-25
21:36:36.000000000 +0200
> @@ -15,6 +15,12 @@ o fail_page_alloc
>  
>    injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
>  
> +o kvmalloc_faillback
     kvmalloc_fallback
> +
> +  makes the function kvmalloc randonly fall back to vmalloc. This could be
used
                                 randomly
> +  to detects bugs such as using DMA-API on the result of kvmalloc or
freeing
> +  the result of kvmalloc with free.
> +
>  o fail_futex
>  
>    injects futex deadlock and uaddr fault errors.
> @@ -167,6 +173,7 @@ use the boot option:
>  
>  	failslab>  	fail_page_alloc> +	kvmalloc_faillback
	kvmalloc_fallback
>  	fail_make_request>  	fail_futex>  
mmc_core.fail_request=<interval>,<probability>,<space>,<times>
> Index: linux-2.6/lib/Kconfig.debug
> ==================================================================> ---
linux-2.6.orig/lib/Kconfig.debug	2018-04-25 15:56:16.000000000 +0200
> +++ linux-2.6/lib/Kconfig.debug	2018-04-25 21:39:45.000000000 +0200
> @@ -1527,6 +1527,21 @@ config FAIL_PAGE_ALLOC
>  	help
>  	  Provide fault-injection capability for alloc_pages().
>  
> +config FAIL_KVMALLOC_FALLBACK_PROBABILITY
> +	int "Default kvmalloc fallback probability"
> +	depends on FAULT_INJECTION
> +	range 0 100
> +	default "0"
> +	help
> +	  This option will make kvmalloc randomly fall back to vmalloc.
> +	  Normally, kvmalloc falls back to vmalloc only rarely, if memory
> +	  is fragmented.
> +
> +	  This option helps to detect hard-to-reproduce driver bugs, for
> +	  example using DMA API on the result of kvmalloc.
> +
> +	  The default may be overriden with the kvmalloc_faillback parameter.
	                     overridden         kvmalloc_fallback
> +
>  config FAIL_MAKE_REQUEST
>  	bool "Fault-injection capability for disk IO"
>  	depends on FAULT_INJECTION && BLOCK
-- 
~Randy

On Wed, 25 Apr 2018, Randy Dunlap wrote:
> On 04/25/2018 01:02 PM, Mikulas Patocka wrote:
> > 
> > 
> > From: Mikulas Patocka <mpatocka at redhat.com>
> > Subject: [PATCH v4] fault-injection: introduce kvmalloc fallback
options
> > 
> > This patch introduces a fault-injection option
"kvmalloc_fallback". This
> > option makes kvmalloc randomly fall back to vmalloc.
> > 
> > Unfortunatelly, some kernel code has bugs - it uses kvmalloc and then
> 
>   Unfortunately,
OK - here I fixed the typos:


From: Mikulas Patocka <mpatocka at redhat.com>
Subject: [PATCH] fault-injection: introduce kvmalloc fallback options

This patch introduces a fault-injection option "kvmalloc_fallback".
This
option makes kvmalloc randomly fall back to vmalloc.

Unfortunately, some kernel code has bugs - it uses kvmalloc and then
uses DMA-API on the returned memory or frees it with kfree. Such bugs were
found in the virtio-net driver, dm-integrity or RHEL7 powerpc-specific
code. This options helps to test for these bugs.

The patch introduces a config option FAIL_KVMALLOC_FALLBACK_PROBABILITY.
It can be enabled in distribution debug kernels, so that kvmalloc abuse
can be tested by the users. The default can be overridden with
"kvmalloc_fallback" parameter or in
/sys/kernel/debug/kvmalloc_fallback/.

Signed-off-by: Mikulas Patocka <mpatocka at redhat.com>

---
 Documentation/fault-injection/fault-injection.txt |    7 +++++
 include/linux/fault-inject.h                      |    9 +++---
 kernel/futex.c                                    |    2 -
 lib/Kconfig.debug                                 |   15 +++++++++++
 mm/failslab.c                                     |    2 -
 mm/page_alloc.c                                   |    2 -
 mm/util.c                                         |   30 ++++++++++++++++++++++
 7 files changed, 60 insertions(+), 7 deletions(-)

Index: linux-2.6/Documentation/fault-injection/fault-injection.txt
==================================================================---
linux-2.6.orig/Documentation/fault-injection/fault-injection.txt	2018-04-16
21:08:34.000000000 +0200
+++ linux-2.6/Documentation/fault-injection/fault-injection.txt	2018-04-25
21:36:36.000000000 +0200
@@ -15,6 +15,12 @@ o fail_page_alloc
 
   injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
 
+o kvmalloc_fallback
+
+  makes the function kvmalloc randomly fall back to vmalloc. This could be used
+  to detects bugs such as using DMA-API on the result of kvmalloc or freeing
+  the result of kvmalloc with free.
+
 o fail_futex
 
   injects futex deadlock and uaddr fault errors.
@@ -167,6 +173,7 @@ use the boot option:
 
 	failslab 	fail_page_alloc+	kvmalloc_fallback 	fail_make_request 	fail_futex 
mmc_core.fail_request=<interval>,<probability>,<space>,<times>
Index: linux-2.6/include/linux/fault-inject.h
==================================================================---
linux-2.6.orig/include/linux/fault-inject.h	2018-04-16 21:08:36.000000000 +0200
+++ linux-2.6/include/linux/fault-inject.h	2018-04-25 21:38:22.000000000 +0200
@@ -31,17 +31,18 @@ struct fault_attr {
 	struct dentry *dname;
 };
 
-#define FAULT_ATTR_INITIALIZER {					\
+#define FAULT_ATTR_INITIALIZER(p) {					\
+		.probability = (p),					\
 		.interval = 1,						\
-		.times = ATOMIC_INIT(1),				\
+		.times = ATOMIC_INIT((p) ? -1 : 1),			\
+		.verbose = (p) ? 0 : 2,					\
 		.require_end = ULONG_MAX,				\
 		.stacktrace_depth = 32,					\
 		.ratelimit_state = RATELIMIT_STATE_INIT_DISABLED,	\
-		.verbose = 2,						\
 		.dname = NULL,						\
 	}
 
-#define DECLARE_FAULT_ATTR(name) struct fault_attr name =
FAULT_ATTR_INITIALIZER
+#define DECLARE_FAULT_ATTR(name) struct fault_attr name =
FAULT_ATTR_INITIALIZER(0)
 int setup_fault_attr(struct fault_attr *attr, char *str);
 bool should_fail(struct fault_attr *attr, ssize_t size);
 
Index: linux-2.6/lib/Kconfig.debug
==================================================================---
linux-2.6.orig/lib/Kconfig.debug	2018-04-25 15:56:16.000000000 +0200
+++ linux-2.6/lib/Kconfig.debug	2018-04-25 21:39:45.000000000 +0200
@@ -1527,6 +1527,21 @@ config FAIL_PAGE_ALLOC
 	help
 	  Provide fault-injection capability for alloc_pages().
 
+config FAIL_KVMALLOC_FALLBACK_PROBABILITY
+	int "Default kvmalloc fallback probability"
+	depends on FAULT_INJECTION
+	range 0 100
+	default "0"
+	help
+	  This option will make kvmalloc randomly fall back to vmalloc.
+	  Normally, kvmalloc falls back to vmalloc only rarely, if memory
+	  is fragmented.
+
+	  This option helps to detect hard-to-reproduce driver bugs, for
+	  example using DMA API on the result of kvmalloc.
+
+	  The default may be overridden with the kvmalloc_fallback parameter.
+
 config FAIL_MAKE_REQUEST
 	bool "Fault-injection capability for disk IO"
 	depends on FAULT_INJECTION && BLOCK
Index: linux-2.6/mm/util.c
==================================================================---
linux-2.6.orig/mm/util.c	2018-04-25 15:48:39.000000000 +0200
+++ linux-2.6/mm/util.c	2018-04-25 21:43:31.000000000 +0200
@@ -14,6 +14,7 @@
 #include <linux/hugetlb.h>
 #include <linux/vmalloc.h>
 #include <linux/userfaultfd_k.h>
+#include <linux/fault-inject.h>
 
 #include <asm/sections.h>
 #include <linux/uaccess.h>
@@ -377,6 +378,29 @@ unsigned long vm_mmap(struct file *file,
 }
 EXPORT_SYMBOL(vm_mmap);
 
+#ifdef CONFIG_FAULT_INJECTION
+
+static struct fault_attr kvmalloc_fallback +
FAULT_ATTR_INITIALIZER(CONFIG_FAIL_KVMALLOC_FALLBACK_PROBABILITY);
+
+static int __init setup_kvmalloc_fallback(char *str)
+{
+	return setup_fault_attr(&kvmalloc_fallback, str);
+}
+
+__setup("kvmalloc_fallback=", setup_kvmalloc_fallback);
+
+#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
+static int __init kvmalloc_fallback_debugfs_init(void)
+{
+	fault_create_debugfs_attr("kvmalloc_fallback", NULL,
&kvmalloc_fallback);
+	return 0;
+}
+late_initcall(kvmalloc_fallback_debugfs_init);
+#endif
+
+#endif
+
 /**
  * kvmalloc_node - attempt to allocate physically contiguous memory, but upon
  * failure, fall back to non-contiguous (vmalloc) allocation.
@@ -404,6 +428,11 @@ void *kvmalloc_node(size_t size, gfp_t f
 	 */
 	WARN_ON_ONCE((flags & GFP_KERNEL) != GFP_KERNEL);
 
+#ifdef CONFIG_FAULT_INJECTION
+	if (should_fail(&kvmalloc_fallback, size))
+		goto do_vmalloc;
+#endif
+
 	/*
 	 * We want to attempt a large physically contiguous block first because
 	 * it is less likely to fragment multiple larger blocks and therefore
@@ -427,6 +456,7 @@ void *kvmalloc_node(size_t size, gfp_t f
 	if (ret || size <= PAGE_SIZE)
 		return ret;
 
+do_vmalloc: __maybe_unused
 	return __vmalloc_node_flags_caller(size, node, flags,
 			__builtin_return_address(0));
 }
Index: linux-2.6/kernel/futex.c
==================================================================---
linux-2.6.orig/kernel/futex.c	2018-02-14 20:24:42.000000000 +0100
+++ linux-2.6/kernel/futex.c	2018-04-25 21:11:33.000000000 +0200
@@ -288,7 +288,7 @@ static struct {
 
 	bool ignore_private;
 } fail_futex = {
-	.attr = FAULT_ATTR_INITIALIZER,
+	.attr = FAULT_ATTR_INITIALIZER(0),
 	.ignore_private = false,
 };
 
Index: linux-2.6/mm/failslab.c
==================================================================---
linux-2.6.orig/mm/failslab.c	2018-04-16 21:08:36.000000000 +0200
+++ linux-2.6/mm/failslab.c	2018-04-25 21:11:40.000000000 +0200
@@ -9,7 +9,7 @@ static struct {
 	bool ignore_gfp_reclaim;
 	bool cache_filter;
 } failslab = {
-	.attr = FAULT_ATTR_INITIALIZER,
+	.attr = FAULT_ATTR_INITIALIZER(0),
 	.ignore_gfp_reclaim = true,
 	.cache_filter = false,
 };
Index: linux-2.6/mm/page_alloc.c
==================================================================---
linux-2.6.orig/mm/page_alloc.c	2018-04-16 21:08:36.000000000 +0200
+++ linux-2.6/mm/page_alloc.c	2018-04-25 21:11:47.000000000 +0200
@@ -3055,7 +3055,7 @@ static struct {
 	bool ignore_gfp_reclaim;
 	u32 min_order;
 } fail_page_alloc = {
-	.attr = FAULT_ATTR_INITIALIZER,
+	.attr = FAULT_ATTR_INITIALIZER(0),
 	.ignore_gfp_reclaim = true,
 	.ignore_gfp_highmem = true,
 	.min_order = 1,

On 04/25/2018 01:57 PM, Mikulas Patocka wrote:
> 
> 
> On Wed, 25 Apr 2018, Randy Dunlap wrote:
> 
>> On 04/25/2018 01:02 PM, Mikulas Patocka wrote:
>>>
>>>
>>> From: Mikulas Patocka <mpatocka at redhat.com>
>>> Subject: [PATCH v4] fault-injection: introduce kvmalloc fallback
options
>>>
>>> This patch introduces a fault-injection option
"kvmalloc_fallback". This
>>> option makes kvmalloc randomly fall back to vmalloc.
>>>
>>> Unfortunatelly, some kernel code has bugs - it uses kvmalloc and
then
>>
>>   Unfortunately,
> 
> OK - here I fixed the typos:
> 
> 
> From: Mikulas Patocka <mpatocka at redhat.com>
> Subject: [PATCH] fault-injection: introduce kvmalloc fallback options
> 
> This patch introduces a fault-injection option
"kvmalloc_fallback". This
> option makes kvmalloc randomly fall back to vmalloc.
> 
> Unfortunately, some kernel code has bugs - it uses kvmalloc and then
> uses DMA-API on the returned memory or frees it with kfree. Such bugs were
> found in the virtio-net driver, dm-integrity or RHEL7 powerpc-specific
> code. This options helps to test for these bugs.
> 
> The patch introduces a config option FAIL_KVMALLOC_FALLBACK_PROBABILITY.
> It can be enabled in distribution debug kernels, so that kvmalloc abuse
> can be tested by the users. The default can be overridden with
> "kvmalloc_fallback" parameter or in
/sys/kernel/debug/kvmalloc_fallback/.
> 
> Signed-off-by: Mikulas Patocka <mpatocka at redhat.com>
> 
> ---
>  Documentation/fault-injection/fault-injection.txt |    7 +++++
>  include/linux/fault-inject.h                      |    9 +++---
>  kernel/futex.c                                    |    2 -
>  lib/Kconfig.debug                                 |   15 +++++++++++
>  mm/failslab.c                                     |    2 -
>  mm/page_alloc.c                                   |    2 -
>  mm/util.c                                         |   30
++++++++++++++++++++++
>  7 files changed, 60 insertions(+), 7 deletions(-)
Acked-by: Randy Dunlap <rdunlap at infradead.org> # Documentation and
Kconfig only

thanks.
-- 
~Randy

On Wed, 25 Apr 2018, David Rientjes wrote:
> On Wed, 25 Apr 2018, Mikulas Patocka wrote:
> 
> > From: Mikulas Patocka <mpatocka at redhat.com>
> > Subject: [PATCH] fault-injection: introduce kvmalloc fallback options
> > 
> > This patch introduces a fault-injection option
"kvmalloc_fallback". This
> > option makes kvmalloc randomly fall back to vmalloc.
> > 
> > Unfortunately, some kernel code has bugs - it uses kvmalloc and then
> > uses DMA-API on the returned memory or frees it with kfree. Such bugs
were
> > found in the virtio-net driver, dm-integrity or RHEL7 powerpc-specific
> > code. This options helps to test for these bugs.
> > 
> > The patch introduces a config option
FAIL_KVMALLOC_FALLBACK_PROBABILITY.
> > It can be enabled in distribution debug kernels, so that kvmalloc
abuse
> > can be tested by the users. The default can be overridden with
> > "kvmalloc_fallback" parameter or in
/sys/kernel/debug/kvmalloc_fallback/.
> > 
> 
> Do we really need the new config option?  This could just be manually 
> tunable via fault injection IIUC.
We do, because we want to enable it in RHEL and Fedora debugging kernels, 
so that it will be tested by the users.

The users won't use some extra magic kernel options or debugfs files.

Mikulas

> > Signed-off-by: Mikulas Patocka <mpatocka at redhat.com>
> > 
> > ---
> >  Documentation/fault-injection/fault-injection.txt |    7 +++++
> >  include/linux/fault-inject.h                      |    9 +++---
> >  kernel/futex.c                                    |    2 -
> >  lib/Kconfig.debug                                 |   15 +++++++++++
> >  mm/failslab.c                                     |    2 -
> >  mm/page_alloc.c                                   |    2 -
> >  mm/util.c                                         |   30
++++++++++++++++++++++
> >  7 files changed, 60 insertions(+), 7 deletions(-)
> > 
> > Index: linux-2.6/Documentation/fault-injection/fault-injection.txt
> > ==================================================================>
> --- linux-2.6.orig/Documentation/fault-injection/fault-injection.txt
2018-04-16 21:08:34.000000000 +0200
> > +++ linux-2.6/Documentation/fault-injection/fault-injection.txt
2018-04-25 21:36:36.000000000 +0200
> > @@ -15,6 +15,12 @@ o fail_page_alloc
> >  
> >    injects page allocation failures. (alloc_pages(), get_free_pages(),
...)
> >  
> > +o kvmalloc_fallback
> > +
> > +  makes the function kvmalloc randomly fall back to vmalloc. This
could be used
> > +  to detects bugs such as using DMA-API on the result of kvmalloc or
freeing
> > +  the result of kvmalloc with free.
> > +
> >  o fail_futex
> >  
> >    injects futex deadlock and uaddr fault errors.
> > @@ -167,6 +173,7 @@ use the boot option:
> >  
> >  	failslab> >  	fail_page_alloc> > +	kvmalloc_fallback>
>  	fail_make_request> >  	fail_futex> >  
mmc_core.fail_request=<interval>,<probability>,<space>,<times>
> > Index: linux-2.6/include/linux/fault-inject.h
> > ==================================================================>
> --- linux-2.6.orig/include/linux/fault-inject.h	2018-04-16
21:08:36.000000000 +0200
> > +++ linux-2.6/include/linux/fault-inject.h	2018-04-25
21:38:22.000000000 +0200
> > @@ -31,17 +31,18 @@ struct fault_attr {
> >  	struct dentry *dname;
> >  };
> >  
> > -#define FAULT_ATTR_INITIALIZER {					\
> > +#define FAULT_ATTR_INITIALIZER(p) {					\
> > +		.probability = (p),					\
> >  		.interval = 1,						\
> > -		.times = ATOMIC_INIT(1),				\
> > +		.times = ATOMIC_INIT((p) ? -1 : 1),			\
> > +		.verbose = (p) ? 0 : 2,					\
> >  		.require_end = ULONG_MAX,				\
> >  		.stacktrace_depth = 32,					\
> >  		.ratelimit_state = RATELIMIT_STATE_INIT_DISABLED,	\
> > -		.verbose = 2,						\
> >  		.dname = NULL,						\
> >  	}
> >  
> > -#define DECLARE_FAULT_ATTR(name) struct fault_attr name =
FAULT_ATTR_INITIALIZER
> > +#define DECLARE_FAULT_ATTR(name) struct fault_attr name =
FAULT_ATTR_INITIALIZER(0)
> >  int setup_fault_attr(struct fault_attr *attr, char *str);
> >  bool should_fail(struct fault_attr *attr, ssize_t size);
> >  
> > Index: linux-2.6/lib/Kconfig.debug
> > ==================================================================>
> --- linux-2.6.orig/lib/Kconfig.debug	2018-04-25 15:56:16.000000000 +0200
> > +++ linux-2.6/lib/Kconfig.debug	2018-04-25 21:39:45.000000000 +0200
> > @@ -1527,6 +1527,21 @@ config FAIL_PAGE_ALLOC
> >  	help
> >  	  Provide fault-injection capability for alloc_pages().
> >  
> > +config FAIL_KVMALLOC_FALLBACK_PROBABILITY
> > +	int "Default kvmalloc fallback probability"
> > +	depends on FAULT_INJECTION
> > +	range 0 100
> > +	default "0"
> > +	help
> > +	  This option will make kvmalloc randomly fall back to vmalloc.
> > +	  Normally, kvmalloc falls back to vmalloc only rarely, if memory
> > +	  is fragmented.
> > +
> > +	  This option helps to detect hard-to-reproduce driver bugs, for
> > +	  example using DMA API on the result of kvmalloc.
> > +
> > +	  The default may be overridden with the kvmalloc_fallback
parameter.
> > +
> >  config FAIL_MAKE_REQUEST
> >  	bool "Fault-injection capability for disk IO"
> >  	depends on FAULT_INJECTION && BLOCK
> > Index: linux-2.6/mm/util.c
> > ==================================================================>
> --- linux-2.6.orig/mm/util.c	2018-04-25 15:48:39.000000000 +0200
> > +++ linux-2.6/mm/util.c	2018-04-25 21:43:31.000000000 +0200
> > @@ -14,6 +14,7 @@
> >  #include <linux/hugetlb.h>
> >  #include <linux/vmalloc.h>
> >  #include <linux/userfaultfd_k.h>
> > +#include <linux/fault-inject.h>
> >  
> >  #include <asm/sections.h>
> >  #include <linux/uaccess.h>
> > @@ -377,6 +378,29 @@ unsigned long vm_mmap(struct file *file,
> >  }
> >  EXPORT_SYMBOL(vm_mmap);
> >  
> > +#ifdef CONFIG_FAULT_INJECTION
> > +
> > +static struct fault_attr kvmalloc_fallback > > +
FAULT_ATTR_INITIALIZER(CONFIG_FAIL_KVMALLOC_FALLBACK_PROBABILITY);
> > +
> > +static int __init setup_kvmalloc_fallback(char *str)
> > +{
> > +	return setup_fault_attr(&kvmalloc_fallback, str);
> > +}
> > +
> > +__setup("kvmalloc_fallback=", setup_kvmalloc_fallback);
> > +
> > +#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
> > +static int __init kvmalloc_fallback_debugfs_init(void)
> > +{
> > +	fault_create_debugfs_attr("kvmalloc_fallback", NULL,
&kvmalloc_fallback);
> > +	return 0;
> > +}
> > +late_initcall(kvmalloc_fallback_debugfs_init);
> > +#endif
> > +
> > +#endif
> > +
> >  /**
> >   * kvmalloc_node - attempt to allocate physically contiguous memory,
but upon
> >   * failure, fall back to non-contiguous (vmalloc) allocation.
> > @@ -404,6 +428,11 @@ void *kvmalloc_node(size_t size, gfp_t f
> >  	 */
> >  	WARN_ON_ONCE((flags & GFP_KERNEL) != GFP_KERNEL);
> >  
> > +#ifdef CONFIG_FAULT_INJECTION
> > +	if (should_fail(&kvmalloc_fallback, size))
> > +		goto do_vmalloc;
> > +#endif
> > +
> >  	/*
> >  	 * We want to attempt a large physically contiguous block first
because
> >  	 * it is less likely to fragment multiple larger blocks and
therefore
> > @@ -427,6 +456,7 @@ void *kvmalloc_node(size_t size, gfp_t f
> >  	if (ret || size <= PAGE_SIZE)
> >  		return ret;
> >  
> > +do_vmalloc: __maybe_unused
> >  	return __vmalloc_node_flags_caller(size, node, flags,
> >  			__builtin_return_address(0));
> >  }
> > Index: linux-2.6/kernel/futex.c
> > ==================================================================>
> --- linux-2.6.orig/kernel/futex.c	2018-02-14 20:24:42.000000000 +0100
> > +++ linux-2.6/kernel/futex.c	2018-04-25 21:11:33.000000000 +0200
> > @@ -288,7 +288,7 @@ static struct {
> >  
> >  	bool ignore_private;
> >  } fail_futex = {
> > -	.attr = FAULT_ATTR_INITIALIZER,
> > +	.attr = FAULT_ATTR_INITIALIZER(0),
> >  	.ignore_private = false,
> >  };
> >  
> > Index: linux-2.6/mm/failslab.c
> > ==================================================================>
> --- linux-2.6.orig/mm/failslab.c	2018-04-16 21:08:36.000000000 +0200
> > +++ linux-2.6/mm/failslab.c	2018-04-25 21:11:40.000000000 +0200
> > @@ -9,7 +9,7 @@ static struct {
> >  	bool ignore_gfp_reclaim;
> >  	bool cache_filter;
> >  } failslab = {
> > -	.attr = FAULT_ATTR_INITIALIZER,
> > +	.attr = FAULT_ATTR_INITIALIZER(0),
> >  	.ignore_gfp_reclaim = true,
> >  	.cache_filter = false,
> >  };
> > Index: linux-2.6/mm/page_alloc.c
> > ==================================================================>
> --- linux-2.6.orig/mm/page_alloc.c	2018-04-16 21:08:36.000000000 +0200
> > +++ linux-2.6/mm/page_alloc.c	2018-04-25 21:11:47.000000000 +0200
> > @@ -3055,7 +3055,7 @@ static struct {
> >  	bool ignore_gfp_reclaim;
> >  	u32 min_order;
> >  } fail_page_alloc = {
> > -	.attr = FAULT_ATTR_INITIALIZER,
> > +	.attr = FAULT_ATTR_INITIALIZER(0),
> >  	.ignore_gfp_reclaim = true,
> >  	.ignore_gfp_highmem = true,
> >  	.min_order = 1,
> > 
> > 
>

On Wed, 25 Apr 2018, James Bottomley wrote:
> On Wed, 2018-04-25 at 17:22 -0400, Mikulas Patocka wrote:
> > 
> > On Wed, 25 Apr 2018, David Rientjes wrote:
> > > 
> > > Do we really need the new config option???This could just be
> > > manually? tunable via fault injection IIUC.
> > 
> > We do, because we want to enable it in RHEL and Fedora debugging
> > kernels,?so that it will be tested by the users.
> > 
> > The users won't use some extra magic kernel options or debugfs
files.
> 
> If it can be enabled via a tunable, then the distro can turn it on
> without the user having to do anything.
You need to enable it on boot. Enabling it when the kernel starts to 
execute userspace code is already too late (because you would miss 
kvmalloc calls in the kernel boot path).

These are files in the kernel-debug rpm package. Where would you put the 
extra kernel parameter to enable this feature? None of these files contain 
kernel parameters.

kernel-debug              /boot/.vmlinuz-3.10.0-693.21.1.el7.x86_64.debug.hmac
kernel-debug              /boot/System.map-3.10.0-693.21.1.el7.x86_64.debug
kernel-debug              /boot/config-3.10.0-693.21.1.el7.x86_64.debug
kernel-debug              /boot/initramfs-3.10.0-693.21.1.el7.x86_64.debug.img
kernel-debug              /boot/symvers-3.10.0-693.21.1.el7.x86_64.debug.gz
kernel-debug              /boot/vmlinuz-3.10.0-693.21.1.el7.x86_64.debug
kernel-debug             
/etc/ld.so.conf.d/kernel-3.10.0-693.21.1.el7.x86_64.debug.conf
kernel-debug              /lib/modules/3.10.0-693.21.1.el7.x86_64.debug
> If you want to present the user with a different boot option, you can 
> (just have the tunable set on the command line), but being tunable 
> driven means that you don't have to choose that option, you could 
> automatically enable it under a range of circumstances.  I think most 
> sane distributions would want that flexibility.
> 
> Kconfig proliferation, conversely, is a bit of a nightmare from both
> the user and the tester's point of view, so we're trying to avoid
it
> unless absolutely necessary.
> 
> James
I already offered that we don't need to introduce a new kernel option and 
we can bind this feature to any other kernel option, that is enabled in 
the debug kernel, for example CONFIG_DEBUG_SG. Michal said no and he said 
that he wants a new kernel option instead.

Mikulas

On Wed, 25 Apr 2018, David Rientjes wrote:
> On Wed, 25 Apr 2018, Mikulas Patocka wrote:
> 
> > You need to enable it on boot. Enabling it when the kernel starts to 
> > execute userspace code is already too late (because you would miss 
> > kvmalloc calls in the kernel boot path).
> 
> Is your motivation that since kvmalloc() never falls back to vmalloc() on 
> boot because fragmentation is not be an issue at boot that we should catch 
> bugs where it would matter if it had fallen back?  If we are worrying 
> about falling back to vmalloc before even initscripts have run I think we 
> have bigger problems.
The same driver can be compiled directly into the kernel or be loaded as a 
module. If the user (or the person preparing distro kernel) compiles the 
driver directly into the kernel, kvmalloc should be tested on that driver, 
because a different user or distribution can compile that driver as a 
module.

Mikulas

On Wed, 25 Apr 2018, James Bottomley wrote:
> > > Do we really need the new config option???This could just be
> > > manually? tunable via fault injection IIUC.
> > 
> > We do, because we want to enable it in RHEL and Fedora debugging
> > kernels,?so that it will be tested by the users.
> > 
> > The users won't use some extra magic kernel options or debugfs
files.
> 
> If it can be enabled via a tunable, then the distro can turn it on
> without the user having to do anything.  If you want to present the
> user with a different boot option, you can (just have the tunable set
> on the command line), but being tunable driven means that you don't
> have to choose that option, you could automatically enable it under a
> range of circumstances.  I think most sane distributions would want
> that flexibility.
> 
> Kconfig proliferation, conversely, is a bit of a nightmare from both
> the user and the tester's point of view, so we're trying to avoid
it
> unless absolutely necessary.
> 
> James
BTW. even developers who compile their own kernel should have this enabled 
by a CONFIG option - because if the developer sees the option when 
browsing through menuconfig, he may enable it. If he doesn't see the 
option, he won't even know that such an option exists.

Mikulas

On Wed, 2 May 2018, John Stoffel wrote:
> You miss my point, which is that there's no explanation of what the
> difference is between SLAB and SLUB and which I should choose.  The
> same goes here.  If the KConfig option doesn't give useful info,
it's
> useless.
So what, we could write explamantion of that option.
> >> Now I also think that Linus has the right idea to not just
sprinkle
> >> BUG_ONs into the code, just dump and oops and keep going if you
can.
> >> If it's a filesystem or a device, turn it read only so that
people
> >> notice right away.
> 
> Mikulas> This vmalloc fallback is similar to
> Mikulas> CONFIG_DEBUG_KOBJECT_RELEASE.  CONFIG_DEBUG_KOBJECT_RELEASE
> Mikulas> changes the behavior of kobject_put in order to cause
> Mikulas> deliberate crashes (that wouldn't happen otherwise) in
> Mikulas> drivers that misuse kobject_put. In the same sense, we want
> Mikulas> to cause deliberate crashes (that wouldn't happen
otherwise)
> Mikulas> in drivers that misuse kvmalloc.
> 
> Mikulas> The crashes will only happen in debugging kernels, not in
> Mikulas> production kernels.
> 
> Says you.  What about people or distros that enable it
> unconditionally?  They're going to get all kinds of reports and then
> turn it off again.  Crashing the system isn't the answer here.  
I've made that kvmalloc bug too (in the function 
dm_integrity_free_journal_scatterlist). I'd much rather like if the kernel 
crashed (because then - I would fix the bug). The kernel didn't crash and 
the bug sneaked into the official linux tree, where may be causing random 
crashes for other users.

Mikulas