thr3ads.net - Linux Virtualization - [PATCH] x86/paravirt: Do not trace _paravirt_ident

If this information is useful, please help other people find it:
Share via:

Steven Rostedt

2016-May-25 17:47 UTC

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

?ukasz Daniluk reported that on a RHEL kernel that his machine would lock up
after enabling function tracer. I asked him to bisect the functions within
available_filter_functions, which he did and it came down to three:

 _paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()

It was found that this is only an issue when noreplace-paravirt is added to
the kernel command line.

This means that those functions are most likely called within critical
sections of the funtion tracer, and must not be traced.

In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
longer an issue. But both _paravirt_ident_{32,64}() causes the following
splat when they are traced:

 mm/pgtable-generic.c:33: bad pmd ffff8800d2435150(0000000001d00054)
 mm/pgtable-generic.c:33: bad pmd ffff8800d3624190(0000000001d00070)
 mm/pgtable-generic.c:33: bad pmd ffff8800d36a5110(0000000001d00054)
 mm/pgtable-generic.c:33: bad pmd ffff880118eb1450(0000000001d00054)
 NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemd-journal:469]
 Modules linked in: e1000e
 CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
 Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05
05/07/2012
 task: ffff880118f740c0 ti: ffff8800d4aec000 task.ti: ffff8800d4aec000
 RIP: 0010:[<ffffffff81134148>]  [<ffffffff81134148>]
queued_spin_lock_slowpath+0x118/0x1a0
 RSP: 0018:ffff8800d4aefb90  EFLAGS: 00000246
 RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88011eb16d40
 RDX: ffffffff82485760 RSI: 000000001f288820 RDI: ffffea0000008030
 RBP: ffff8800d4aefb90 R08: 00000000000c0000 R09: 0000000000000000
 R10: ffffffff821c8e0e R11: 0000000000000000 R12: ffff880000200fb8
 R13: 00007f7a4e3f7000 R14: ffffea000303f600 R15: ffff8800d4b562e0
 FS:  00007f7a4e3d7840(0000) GS:ffff88011eb00000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 00007f7a4e3f7000 CR3: 00000000d3e71000 CR4: 00000000001406e0
 Stack:
  ffff8800d4aefba0 ffffffff81cc5f47 ffff8800d4aefc60 ffffffff8122c15b
  ffff8800d4aefcb0 ffff8800d4aefbd0 ffffffff811bf4cb 0000000000000002
  0000000000000015 ffff8800d2276050 80000000c0fd8867 ffffea0000008030
 Call Trace:
  [<ffffffff81cc5f47>] _raw_spin_lock+0x27/0x30
  [<ffffffff8122c15b>] handle_pte_fault+0x13db/0x16b0
  [<ffffffff811bf4cb>] ? function_trace_call+0x15b/0x180
  [<ffffffff8122ad85>] ? handle_pte_fault+0x5/0x16b0
  [<ffffffff8122e322>] handle_mm_fault+0x312/0x670
  [<ffffffff81231068>] ? find_vma+0x68/0x70
  [<ffffffff810ab741>] __do_page_fault+0x1b1/0x4e0
  [<ffffffff810aba92>] do_page_fault+0x22/0x30
  [<ffffffff81cc7f68>] page_fault+0x28/0x30
  [<ffffffff81574af5>] ? copy_user_enhanced_fast_string+0x5/0x10
  [<ffffffff8129dec5>] ? seq_read+0x305/0x370
  [<ffffffff81279668>] __vfs_read+0x28/0xe0
  [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
  [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
  [<ffffffff81279df6>] vfs_read+0x86/0x130
  [<ffffffff8127b216>] SyS_read+0x46/0xa0
  [<ffffffff81cc6176>] entry_SYSCALL_64_fastpath+0x1e/0xa8
 Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
 c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7
 4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b

Reported-by: ?ukasz Daniluk <lukasz.daniluk at intel.com>
Signed-off-by: Steven Rostedt <rostedt at goodmis.org>

diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
index f08ac28b8136..f975d226be6e 100644
--- a/arch/x86/kernel/paravirt.c
+++ b/arch/x86/kernel/paravirt.c
@@ -55,12 +55,12 @@ asm (".pushsection .entry.text,
\"ax\"\n"
      ".popsection");
 
 /* identity function, which can be inlined */
-u32 _paravirt_ident_32(u32 x)
+u32 notrace _paravirt_ident_32(u32 x)
 {
 	return x;
 }
 
-u64 _paravirt_ident_64(u64 x)
+u64 notrace _paravirt_ident_64(u64 x)
 {
 	return x;
 }

Steven Rostedt

2016-Aug-08 20:09 UTC

head link

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

Hmm, I'm guessing this patch got lost.

-- Steve


On Wed, 25 May 2016 13:47:26 -0400
Steven Rostedt <rostedt at goodmis.org> wrote:
> ?ukasz Daniluk reported that on a RHEL kernel that his machine would lock
up
> after enabling function tracer. I asked him to bisect the functions within
> available_filter_functions, which he did and it came down to three:
> 
>  _paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()
> 
> It was found that this is only an issue when noreplace-paravirt is added to
> the kernel command line.
> 
> This means that those functions are most likely called within critical
> sections of the funtion tracer, and must not be traced.
> 
> In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
> longer an issue. But both _paravirt_ident_{32,64}() causes the following
> splat when they are traced:
> 
>  mm/pgtable-generic.c:33: bad pmd ffff8800d2435150(0000000001d00054)
>  mm/pgtable-generic.c:33: bad pmd ffff8800d3624190(0000000001d00070)
>  mm/pgtable-generic.c:33: bad pmd ffff8800d36a5110(0000000001d00054)
>  mm/pgtable-generic.c:33: bad pmd ffff880118eb1450(0000000001d00054)
>  NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
[systemd-journal:469]
>  Modules linked in: e1000e
>  CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
>  Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01
v02.05 05/07/2012
>  task: ffff880118f740c0 ti: ffff8800d4aec000 task.ti: ffff8800d4aec000
>  RIP: 0010:[<ffffffff81134148>]  [<ffffffff81134148>]
queued_spin_lock_slowpath+0x118/0x1a0
>  RSP: 0018:ffff8800d4aefb90  EFLAGS: 00000246
>  RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88011eb16d40
>  RDX: ffffffff82485760 RSI: 000000001f288820 RDI: ffffea0000008030
>  RBP: ffff8800d4aefb90 R08: 00000000000c0000 R09: 0000000000000000
>  R10: ffffffff821c8e0e R11: 0000000000000000 R12: ffff880000200fb8
>  R13: 00007f7a4e3f7000 R14: ffffea000303f600 R15: ffff8800d4b562e0
>  FS:  00007f7a4e3d7840(0000) GS:ffff88011eb00000(0000)
knlGS:0000000000000000
>  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>  CR2: 00007f7a4e3f7000 CR3: 00000000d3e71000 CR4: 00000000001406e0
>  Stack:
>   ffff8800d4aefba0 ffffffff81cc5f47 ffff8800d4aefc60 ffffffff8122c15b
>   ffff8800d4aefcb0 ffff8800d4aefbd0 ffffffff811bf4cb 0000000000000002
>   0000000000000015 ffff8800d2276050 80000000c0fd8867 ffffea0000008030
>  Call Trace:
>   [<ffffffff81cc5f47>] _raw_spin_lock+0x27/0x30
>   [<ffffffff8122c15b>] handle_pte_fault+0x13db/0x16b0
>   [<ffffffff811bf4cb>] ? function_trace_call+0x15b/0x180
>   [<ffffffff8122ad85>] ? handle_pte_fault+0x5/0x16b0
>   [<ffffffff8122e322>] handle_mm_fault+0x312/0x670
>   [<ffffffff81231068>] ? find_vma+0x68/0x70
>   [<ffffffff810ab741>] __do_page_fault+0x1b1/0x4e0
>   [<ffffffff810aba92>] do_page_fault+0x22/0x30
>   [<ffffffff81cc7f68>] page_fault+0x28/0x30
>   [<ffffffff81574af5>] ? copy_user_enhanced_fast_string+0x5/0x10
>   [<ffffffff8129dec5>] ? seq_read+0x305/0x370
>   [<ffffffff81279668>] __vfs_read+0x28/0xe0
>   [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
>   [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
>   [<ffffffff81279df6>] vfs_read+0x86/0x130
>   [<ffffffff8127b216>] SyS_read+0x46/0xa0
>   [<ffffffff81cc6176>] entry_SYSCALL_64_fastpath+0x1e/0xa8
>  Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
>  c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0
74 f7
>  4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b
> 
> Reported-by: ?ukasz Daniluk <lukasz.daniluk at intel.com>
> Signed-off-by: Steven Rostedt <rostedt at goodmis.org>
> 
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index f08ac28b8136..f975d226be6e 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -55,12 +55,12 @@ asm (".pushsection .entry.text,
\"ax\"\n"
>       ".popsection");
>  
>  /* identity function, which can be inlined */
> -u32 _paravirt_ident_32(u32 x)
> +u32 notrace _paravirt_ident_32(u32 x)
>  {
>  	return x;
>  }
>  
> -u64 _paravirt_ident_64(u64 x)
> +u64 notrace _paravirt_ident_64(u64 x)
>  {
>  	return x;
>  }

Steven Rostedt

2016-Sep-02 14:46 UTC

head link

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

I just spent half a day bisecting function tracing because I tripped
over this again. I thought this was merged, but I guess it was missed
again.

Can someone please pull this in. And mark it for stable, it goes
probably as far back as 2.6.32.

Thanks!

-- Steve

On Wed, 25 May 2016 13:47:26 -0400
Steven Rostedt <rostedt at goodmis.org> wrote:
> ?ukasz Daniluk reported that on a RHEL kernel that his machine would lock
up
> after enabling function tracer. I asked him to bisect the functions within
> available_filter_functions, which he did and it came down to three:
> 
>  _paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()
> 
> It was found that this is only an issue when noreplace-paravirt is added to
> the kernel command line.
> 
> This means that those functions are most likely called within critical
> sections of the funtion tracer, and must not be traced.
> 
> In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
> longer an issue. But both _paravirt_ident_{32,64}() causes the following
> splat when they are traced:
> 
>  mm/pgtable-generic.c:33: bad pmd ffff8800d2435150(0000000001d00054)
>  mm/pgtable-generic.c:33: bad pmd ffff8800d3624190(0000000001d00070)
>  mm/pgtable-generic.c:33: bad pmd ffff8800d36a5110(0000000001d00054)
>  mm/pgtable-generic.c:33: bad pmd ffff880118eb1450(0000000001d00054)
>  NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
[systemd-journal:469]
>  Modules linked in: e1000e
>  CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
>  Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01
v02.05 05/07/2012
>  task: ffff880118f740c0 ti: ffff8800d4aec000 task.ti: ffff8800d4aec000
>  RIP: 0010:[<ffffffff81134148>]  [<ffffffff81134148>]
queued_spin_lock_slowpath+0x118/0x1a0
>  RSP: 0018:ffff8800d4aefb90  EFLAGS: 00000246
>  RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88011eb16d40
>  RDX: ffffffff82485760 RSI: 000000001f288820 RDI: ffffea0000008030
>  RBP: ffff8800d4aefb90 R08: 00000000000c0000 R09: 0000000000000000
>  R10: ffffffff821c8e0e R11: 0000000000000000 R12: ffff880000200fb8
>  R13: 00007f7a4e3f7000 R14: ffffea000303f600 R15: ffff8800d4b562e0
>  FS:  00007f7a4e3d7840(0000) GS:ffff88011eb00000(0000)
knlGS:0000000000000000
>  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>  CR2: 00007f7a4e3f7000 CR3: 00000000d3e71000 CR4: 00000000001406e0
>  Stack:
>   ffff8800d4aefba0 ffffffff81cc5f47 ffff8800d4aefc60 ffffffff8122c15b
>   ffff8800d4aefcb0 ffff8800d4aefbd0 ffffffff811bf4cb 0000000000000002
>   0000000000000015 ffff8800d2276050 80000000c0fd8867 ffffea0000008030
>  Call Trace:
>   [<ffffffff81cc5f47>] _raw_spin_lock+0x27/0x30
>   [<ffffffff8122c15b>] handle_pte_fault+0x13db/0x16b0
>   [<ffffffff811bf4cb>] ? function_trace_call+0x15b/0x180
>   [<ffffffff8122ad85>] ? handle_pte_fault+0x5/0x16b0
>   [<ffffffff8122e322>] handle_mm_fault+0x312/0x670
>   [<ffffffff81231068>] ? find_vma+0x68/0x70
>   [<ffffffff810ab741>] __do_page_fault+0x1b1/0x4e0
>   [<ffffffff810aba92>] do_page_fault+0x22/0x30
>   [<ffffffff81cc7f68>] page_fault+0x28/0x30
>   [<ffffffff81574af5>] ? copy_user_enhanced_fast_string+0x5/0x10
>   [<ffffffff8129dec5>] ? seq_read+0x305/0x370
>   [<ffffffff81279668>] __vfs_read+0x28/0xe0
>   [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
>   [<ffffffff81279645>] ? __vfs_read+0x5/0xe0
>   [<ffffffff81279df6>] vfs_read+0x86/0x130
>   [<ffffffff8127b216>] SyS_read+0x46/0xa0
>   [<ffffffff81cc6176>] entry_SYSCALL_64_fastpath+0x1e/0xa8
>  Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
>  c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0
74 f7
>  4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b
> 
> Reported-by: ?ukasz Daniluk <lukasz.daniluk at intel.com>
> Signed-off-by: Steven Rostedt <rostedt at goodmis.org>
> 
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index f08ac28b8136..f975d226be6e 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -55,12 +55,12 @@ asm (".pushsection .entry.text,
\"ax\"\n"
>       ".popsection");
>  
>  /* identity function, which can be inlined */
> -u32 _paravirt_ident_32(u32 x)
> +u32 notrace _paravirt_ident_32(u32 x)
>  {
>  	return x;
>  }
>  
> -u64 _paravirt_ident_64(u64 x)
> +u64 notrace _paravirt_ident_64(u64 x)
>  {
>  	return x;
>  }

Linus Torvalds

2016-Sep-02 16:41 UTC

head link

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

On Fri, Sep 2, 2016 at 7:46 AM, Steven Rostedt <rostedt at goodmis.org>
wrote:>
> Can someone please pull this in. And mark it for stable, it goes
> probably as far back as 2.6.32.
Applied.

             Linus

Possibly Parallel Threads

Search for more apparently analagous threads

Linux Virtualization - May 2016 - [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

Possibly Parallel Threads