Hi Sergey, On Wed, Jun 15, 2016 at 04:59:09PM +0900, Sergey Senozhatsky wrote:> Hello Minchan, > > -next 4.7.0-rc3-next-20160614 > > > [ 315.146533] kasan: CONFIG_KASAN_INLINE enabled > [ 315.146538] kasan: GPF could be caused by NULL-ptr deref or user memory access > [ 315.146546] general protection fault: 0000 [#1] PREEMPT SMP KASAN > [ 315.146576] Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel r8169 i2c_i801 mii snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core acpi_cpufreq snd_pcm snd_timer snd soundcore lpc_ich mfd_core processor sch_fq_codel sd_mod hid_generic usbhid hid ahci libahci libata ehci_pci ehci_hcd scsi_mod usbcore usb_common > [ 315.146785] CPU: 3 PID: 38 Comm: khugepaged Not tainted 4.7.0-rc3-next-20160614-dbg-00004-ga1c2cbc-dirty #488 > [ 315.146841] task: ffff8800bfaf2900 ti: ffff880112468000 task.ti: ffff880112468000 > [ 315.146859] RIP: 0010:[<ffffffffa02c413d>] [<ffffffffa02c413d>] zs_page_migrate+0x355/0xaa0 [zsmalloc]Thanks for the report! zs_page_migrate+0x355? Could you tell me what line is it? It seems to be related to obj_to_head. Could you test with [zsmalloc: keep first object offset in struct page] in mmotm?> [ 315.146892] RSP: 0000:ffff88011246f138 EFLAGS: 00010293 > [ 315.146906] RAX: 736761742d6f6e2c RBX: ffff880017ad9a80 RCX: 0000000000000000 > [ 315.146924] RDX: 1ffffffff064d704 RSI: ffff88000511469a RDI: ffffffff8326ba20 > [ 315.146942] RBP: ffff88011246f328 R08: 0000000000000001 R09: 0000000000000000 > [ 315.146959] R10: ffff88011246f0a8 R11: ffff8800bfc07fff R12: ffff88011246f300 > [ 315.146977] R13: ffffed0015523e6f R14: ffff8800aa91f378 R15: ffffea0000144500 > [ 315.146995] FS: 0000000000000000(0000) GS:ffff880113780000(0000) knlGS:0000000000000000 > [ 315.147015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 315.147030] CR2: 00007f3f97911000 CR3: 0000000002209000 CR4: 00000000000006e0 > [ 315.147046] Stack: > [ 315.147052] 1ffff10015523e0f ffff88011246f240 ffff880005116800 00017f80e0000000 > [ 315.147083] ffff880017ad9aa8 736761742d6f6e2c 1ffff1002248de34 ffff880017ad9a90 > [ 315.147113] 0000069a1246f660 000000000000069a ffff880005114000 ffffea0002ff0180 > [ 315.147143] Call Trace: > [ 315.147154] [<ffffffffa02c3de8>] ? obj_to_head+0x9d/0x9d [zsmalloc] > [ 315.147175] [<ffffffff81d31dbc>] ? _raw_spin_unlock_irqrestore+0x47/0x5c > [ 315.147195] [<ffffffff812275b1>] ? isolate_freepages_block+0x2f9/0x5a6 > [ 315.147213] [<ffffffff8127f15c>] ? kasan_poison_shadow+0x2f/0x31 > [ 315.147230] [<ffffffff8127f66a>] ? kasan_alloc_pages+0x39/0x3b > [ 315.147246] [<ffffffff812267e6>] ? map_pages+0x1f3/0x3ad > [ 315.147262] [<ffffffff812265f3>] ? update_pageblock_skip+0x18d/0x18d > [ 315.147280] [<ffffffff81115972>] ? up_read+0x1a/0x30 > [ 315.147296] [<ffffffff8111ec7e>] ? debug_check_no_locks_freed+0x150/0x22b > [ 315.147315] [<ffffffff812842d1>] move_to_new_page+0x4dd/0x615 > [ 315.147332] [<ffffffff81283df4>] ? migrate_page+0x75/0x75 > [ 315.147347] [<ffffffff8122785e>] ? isolate_freepages_block+0x5a6/0x5a6 > [ 315.147366] [<ffffffff812851c1>] migrate_pages+0xadd/0x131a > [ 315.147382] [<ffffffff8122785e>] ? isolate_freepages_block+0x5a6/0x5a6 > [ 315.147399] [<ffffffff81226375>] ? kzfree+0x2b/0x2b > [ 315.147414] [<ffffffff812846e4>] ? buffer_migrate_page+0x2db/0x2db > [ 315.147431] [<ffffffff8122a6cf>] compact_zone+0xcdb/0x1155 > [ 315.147448] [<ffffffff812299f4>] ? compaction_suitable+0x76/0x76 > [ 315.147465] [<ffffffff8122ac29>] compact_zone_order+0xe0/0x167 > [ 315.147481] [<ffffffff8111f0ac>] ? debug_show_all_locks+0x226/0x226 > [ 315.147499] [<ffffffff8122ab49>] ? compact_zone+0x1155/0x1155 > [ 315.147515] [<ffffffff810d58d1>] ? finish_task_switch+0x3de/0x484 > [ 315.147533] [<ffffffff8122bcff>] try_to_compact_pages+0x2f1/0x648 > [ 315.147550] [<ffffffff8122bcff>] ? try_to_compact_pages+0x2f1/0x648 > [ 315.147568] [<ffffffff8122ba0e>] ? compaction_zonelist_suitable+0x3a6/0x3a6 > [ 315.147589] [<ffffffff811ee129>] ? get_page_from_freelist+0x2c0/0x129a > [ 315.147608] [<ffffffff811ef1ed>] __alloc_pages_direct_compact+0xea/0x30d > [ 315.147626] [<ffffffff811ef103>] ? get_page_from_freelist+0x129a/0x129a > [ 315.147645] [<ffffffff811f0422>] __alloc_pages_nodemask+0x840/0x16b6 > [ 315.147663] [<ffffffff810dba27>] ? try_to_wake_up+0x696/0x6c8 > [ 315.149147] [<ffffffff811efbe2>] ? warn_alloc_failed+0x226/0x226 > [ 315.150615] [<ffffffff810dba69>] ? wake_up_process+0x10/0x12 > [ 315.152078] [<ffffffff810dbaf4>] ? wake_up_q+0x89/0xa7 > [ 315.153539] [<ffffffff81128b6f>] ? rwsem_wake+0x131/0x15c > [ 315.155007] [<ffffffff812922e7>] ? khugepaged+0x4072/0x484f > [ 315.156471] [<ffffffff8128e449>] khugepaged+0x1d4/0x484f > [ 315.157940] [<ffffffff8128e275>] ? hugepage_vma_revalidate+0xef/0xef > [ 315.159402] [<ffffffff810d58d1>] ? finish_task_switch+0x3de/0x484 > [ 315.160870] [<ffffffff81d31df8>] ? _raw_spin_unlock_irq+0x27/0x45 > [ 315.162341] [<ffffffff8111cde6>] ? trace_hardirqs_on_caller+0x3d2/0x492 > [ 315.163814] [<ffffffff8111112e>] ? prepare_to_wait_event+0x3f7/0x3f7 > [ 315.165295] [<ffffffff81d27ad5>] ? __schedule+0xa4d/0xd16 > [ 315.166763] [<ffffffff810ccde3>] kthread+0x252/0x261 > [ 315.168214] [<ffffffff8128e275>] ? hugepage_vma_revalidate+0xef/0xef > [ 315.169646] [<ffffffff810ccb91>] ? kthread_create_on_node+0x377/0x377 > [ 315.171056] [<ffffffff81d3277f>] ret_from_fork+0x1f/0x40 > [ 315.172462] [<ffffffff810ccb91>] ? kthread_create_on_node+0x377/0x377 > [ 315.173869] Code: 03 b5 60 fe ff ff e8 2e fc ff ff a8 01 74 4c 48 83 e0 fe bf 01 00 00 00 48 89 85 38 fe ff ff e8 41 18 e1 e0 48 8b 85 38 fe ff ff <f0> 0f ba 28 00 73 29 bf 01 00 00 00 41 bc f5 ff ff ff e8 ea 27 > [ 315.175573] RIP [<ffffffffa02c413d>] zs_page_migrate+0x355/0xaa0 [zsmalloc] > [ 315.177084] RSP <ffff88011246f138> > [ 315.186572] ---[ end trace 0962b8ee48c98bbc ]--- > > > > > [ 315.186577] BUG: sleeping function called from invalid context at include/linux/sched.h:2960 > [ 315.186580] in_atomic(): 1, irqs_disabled(): 0, pid: 38, name: khugepaged > [ 315.186581] INFO: lockdep is turned off. > [ 315.186583] Preemption disabled at:[<ffffffffa02c3f1d>] zs_page_migrate+0x135/0xaa0 [zsmalloc] > > [ 315.186594] CPU: 3 PID: 38 Comm: khugepaged Tainted: G D 4.7.0-rc3-next-20160614-dbg-00004-ga1c2cbc-dirty #488 > [ 315.186599] 0000000000000000 ffff88011246ed58 ffffffff814d56bf ffff8800bfaf2900 > [ 315.186604] 0000000000000004 ffff88011246ed98 ffffffff810d5e6a 0000000000000000 > [ 315.186609] ffff8800bfaf2900 ffffffff81e39820 0000000000000b90 0000000000000000 > [ 315.186614] Call Trace: > [ 315.186618] [<ffffffff814d56bf>] dump_stack+0x68/0x92 > [ 315.186622] [<ffffffff810d5e6a>] ___might_sleep+0x3bd/0x3c9 > [ 315.186625] [<ffffffff810d5fd1>] __might_sleep+0x15b/0x167 > [ 315.186630] [<ffffffff810ac4c1>] exit_signals+0x7a/0x34f > [ 315.186633] [<ffffffff810ac447>] ? get_signal+0xd9b/0xd9b > [ 315.186636] [<ffffffff811aee21>] ? irq_work_queue+0x101/0x11c > [ 315.186640] [<ffffffff8111f0ac>] ? debug_show_all_locks+0x226/0x226 > [ 315.186645] [<ffffffff81096357>] do_exit+0x34d/0x1b4e > [ 315.186648] [<ffffffff81130e16>] ? vprintk_emit+0x4b1/0x4d3 > [ 315.186652] [<ffffffff8109600a>] ? is_current_pgrp_orphaned+0x8c/0x8c > [ 315.186655] [<ffffffff81122c56>] ? lock_acquire+0xec/0x147 > [ 315.186658] [<ffffffff811321ef>] ? kmsg_dump+0x12/0x27a > [ 315.186662] [<ffffffff81132448>] ? kmsg_dump+0x26b/0x27a > [ 315.186666] [<ffffffff81036507>] oops_end+0x9d/0xa4 > [ 315.186669] [<ffffffff8103662c>] die+0x55/0x5e > [ 315.186672] [<ffffffff81032aa0>] do_general_protection+0x16c/0x337 > [ 315.186676] [<ffffffff81d33abf>] general_protection+0x1f/0x30 > [ 315.186681] [<ffffffffa02c413d>] ? zs_page_migrate+0x355/0xaa0 [zsmalloc] > [ 315.186686] [<ffffffffa02c4136>] ? zs_page_migrate+0x34e/0xaa0 [zsmalloc] > [ 315.186691] [<ffffffffa02c3de8>] ? obj_to_head+0x9d/0x9d [zsmalloc] > [ 315.186695] [<ffffffff81d31dbc>] ? _raw_spin_unlock_irqrestore+0x47/0x5c > [ 315.186699] [<ffffffff812275b1>] ? isolate_freepages_block+0x2f9/0x5a6 > [ 315.186702] [<ffffffff8127f15c>] ? kasan_poison_shadow+0x2f/0x31 > [ 315.186706] [<ffffffff8127f66a>] ? kasan_alloc_pages+0x39/0x3b > [ 315.186709] [<ffffffff812267e6>] ? map_pages+0x1f3/0x3ad > [ 315.186712] [<ffffffff812265f3>] ? update_pageblock_skip+0x18d/0x18d > [ 315.186716] [<ffffffff81115972>] ? up_read+0x1a/0x30 > [ 315.186719] [<ffffffff8111ec7e>] ? debug_check_no_locks_freed+0x150/0x22b > [ 315.186723] [<ffffffff812842d1>] move_to_new_page+0x4dd/0x615 > [ 315.186726] [<ffffffff81283df4>] ? migrate_page+0x75/0x75 > [ 315.186730] [<ffffffff8122785e>] ? isolate_freepages_block+0x5a6/0x5a6 > [ 315.186733] [<ffffffff812851c1>] migrate_pages+0xadd/0x131a > [ 315.186737] [<ffffffff8122785e>] ? isolate_freepages_block+0x5a6/0x5a6 > [ 315.186740] [<ffffffff81226375>] ? kzfree+0x2b/0x2b > [ 315.186743] [<ffffffff812846e4>] ? buffer_migrate_page+0x2db/0x2db > [ 315.186747] [<ffffffff8122a6cf>] compact_zone+0xcdb/0x1155 > [ 315.186751] [<ffffffff812299f4>] ? compaction_suitable+0x76/0x76 > [ 315.186754] [<ffffffff8122ac29>] compact_zone_order+0xe0/0x167 > [ 315.186757] [<ffffffff8111f0ac>] ? debug_show_all_locks+0x226/0x226 > [ 315.186761] [<ffffffff8122ab49>] ? compact_zone+0x1155/0x1155 > [ 315.186764] [<ffffffff810d58d1>] ? finish_task_switch+0x3de/0x484 > [ 315.186768] [<ffffffff8122bcff>] try_to_compact_pages+0x2f1/0x648 > [ 315.186771] [<ffffffff8122bcff>] ? try_to_compact_pages+0x2f1/0x648 > [ 315.186775] [<ffffffff8122ba0e>] ? compaction_zonelist_suitable+0x3a6/0x3a6 > [ 315.186780] [<ffffffff811ee129>] ? get_page_from_freelist+0x2c0/0x129a > [ 315.186783] [<ffffffff811ef1ed>] __alloc_pages_direct_compact+0xea/0x30d > [ 315.186787] [<ffffffff811ef103>] ? get_page_from_freelist+0x129a/0x129a > [ 315.186791] [<ffffffff811f0422>] __alloc_pages_nodemask+0x840/0x16b6 > [ 315.186794] [<ffffffff810dba27>] ? try_to_wake_up+0x696/0x6c8 > [ 315.186798] [<ffffffff811efbe2>] ? warn_alloc_failed+0x226/0x226 > [ 315.186801] [<ffffffff810dba69>] ? wake_up_process+0x10/0x12 > [ 315.186804] [<ffffffff810dbaf4>] ? wake_up_q+0x89/0xa7 > [ 315.186807] [<ffffffff81128b6f>] ? rwsem_wake+0x131/0x15c > [ 315.186811] [<ffffffff812922e7>] ? khugepaged+0x4072/0x484f > [ 315.186815] [<ffffffff8128e449>] khugepaged+0x1d4/0x484f > [ 315.186819] [<ffffffff8128e275>] ? hugepage_vma_revalidate+0xef/0xef > [ 315.186822] [<ffffffff810d58d1>] ? finish_task_switch+0x3de/0x484 > [ 315.186826] [<ffffffff81d31df8>] ? _raw_spin_unlock_irq+0x27/0x45 > [ 315.186829] [<ffffffff8111cde6>] ? trace_hardirqs_on_caller+0x3d2/0x492 > [ 315.186832] [<ffffffff8111112e>] ? prepare_to_wait_event+0x3f7/0x3f7 > [ 315.186836] [<ffffffff81d27ad5>] ? __schedule+0xa4d/0xd16 > [ 315.186840] [<ffffffff810ccde3>] kthread+0x252/0x261 > [ 315.186843] [<ffffffff8128e275>] ? hugepage_vma_revalidate+0xef/0xef > [ 315.186846] [<ffffffff810ccb91>] ? kthread_create_on_node+0x377/0x377 > [ 315.186851] [<ffffffff81d3277f>] ret_from_fork+0x1f/0x40 > [ 315.186854] [<ffffffff810ccb91>] ? kthread_create_on_node+0x377/0x377 > [ 315.186869] note: khugepaged[38] exited with preempt_count 4 > > > > [ 340.319852] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [jbd2/zram0-8:405] > [ 340.319856] Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel r8169 i2c_i801 mii snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core acpi_cpufreq snd_pcm snd_timer snd soundcore lpc_ich mfd_core processor sch_fq_codel sd_mod hid_generic usbhid hid ahci libahci libata ehci_pci ehci_hcd scsi_mod usbcore usb_common > [ 340.319900] irq event stamp: 834296 > [ 340.319902] hardirqs last enabled at (834295): [<ffffffff81280b07>] quarantine_put+0xa1/0xe6 > [ 340.319911] hardirqs last disabled at (834296): [<ffffffff81d31e68>] _raw_write_lock_irqsave+0x13/0x4c > [ 340.319917] softirqs last enabled at (833836): [<ffffffff81d3455e>] __do_softirq+0x406/0x48f > [ 340.319922] softirqs last disabled at (833831): [<ffffffff8109914a>] irq_exit+0x6a/0x113 > [ 340.319929] CPU: 2 PID: 405 Comm: jbd2/zram0-8 Tainted: G D 4.7.0-rc3-next-20160614-dbg-00004-ga1c2cbc-dirty #488 > [ 340.319935] task: ffff8800bb512900 ti: ffff8800a69c0000 task.ti: ffff8800a69c0000 > [ 340.319937] RIP: 0010:[<ffffffff814ed772>] [<ffffffff814ed772>] delay_tsc+0x0/0xa4 > [ 340.319943] RSP: 0018:ffff8800a69c70f8 EFLAGS: 00000206 > [ 340.319945] RAX: 0000000000000001 RBX: ffff8800aa91f300 RCX: 0000000000000000 > [ 340.319947] RDX: 0000000000000003 RSI: ffffffff81ed2840 RDI: 0000000000000001 > [ 340.319949] RBP: ffff8800a69c7100 R08: 0000000000000001 R09: 0000000000000000 > [ 340.319951] R10: ffff8800a69c70e8 R11: 000000007e7516b9 R12: ffff8800aa91f310 > [ 340.319954] R13: ffff8800aa91f308 R14: 000000001f3306fa R15: 0000000000000000 > [ 340.319956] FS: 0000000000000000(0000) GS:ffff880113700000(0000) knlGS:0000000000000000 > [ 340.319959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 340.319961] CR2: 00007fc99caba080 CR3: 00000000b9796000 CR4: 00000000000006e0 > [ 340.319963] Stack: > [ 340.319964] ffffffff814ed89c ffff8800a69c7148 ffffffff8112795d ffffed0015523e60 > [ 340.319970] 000000009e857390 ffff8800aa91f300 ffff8800bbe21cc0 ffff8800047d6f80 > [ 340.319975] ffff8800a69c72b0 ffff8800aa91f300 ffff8800a69c7168 ffffffff81d31bed > [ 340.319980] Call Trace: > [ 340.319983] [<ffffffff814ed89c>] ? __delay+0xa/0xc > [ 340.319988] [<ffffffff8112795d>] do_raw_spin_lock+0x197/0x257 > [ 340.319991] [<ffffffff81d31bed>] _raw_spin_lock+0x35/0x3c > [ 340.319998] [<ffffffffa02c6062>] ? zs_free+0x191/0x27a [zsmalloc] > [ 340.320003] [<ffffffffa02c6062>] zs_free+0x191/0x27a [zsmalloc] > [ 340.320008] [<ffffffffa02c5ed1>] ? free_zspage+0xe8/0xe8 [zsmalloc] > [ 340.320012] [<ffffffff810d58d1>] ? finish_task_switch+0x3de/0x484 > [ 340.320015] [<ffffffff810d58a6>] ? finish_task_switch+0x3b3/0x484 > [ 340.320021] [<ffffffff81d27ad5>] ? __schedule+0xa4d/0xd16 > [ 340.320024] [<ffffffff81d28086>] ? preempt_schedule+0x1f/0x21 > [ 340.320028] [<ffffffff81d27ff9>] ? preempt_schedule_common+0xb7/0xe8 > [ 340.320034] [<ffffffffa02d3f0e>] zram_free_page+0x112/0x1f6 [zram] > [ 340.320039] [<ffffffffa02d5e6c>] zram_make_request+0x45d/0x89f [zram] > [ 340.320045] [<ffffffffa02d5a0f>] ? zram_rw_page+0x21d/0x21d [zram] > [ 340.320048] [<ffffffff81493657>] ? blk_exit_rl+0x39/0x39 > [ 340.320053] [<ffffffff8148fe3f>] ? handle_bad_sector+0x192/0x192 > [ 340.320056] [<ffffffff8127f83e>] ? kasan_slab_alloc+0x12/0x14 > [ 340.320059] [<ffffffff8127ca68>] ? kmem_cache_alloc+0xf3/0x101 > [ 340.320062] [<ffffffff81494e37>] generic_make_request+0x2bc/0x496 > [ 340.320066] [<ffffffff81494b7b>] ? blk_plug_queued_count+0x103/0x103 > [ 340.320069] [<ffffffff8111ec7e>] ? debug_check_no_locks_freed+0x150/0x22b > [ 340.320072] [<ffffffff81495309>] submit_bio+0x2f8/0x324 > [ 340.320075] [<ffffffff81495011>] ? generic_make_request+0x496/0x496 > [ 340.320078] [<ffffffff811190fc>] ? lockdep_init_map+0x1ef/0x4b0 > [ 340.320082] [<ffffffff814880a4>] submit_bio_wait+0xff/0x138 > [ 340.320085] [<ffffffff81487fa5>] ? bio_add_page+0x292/0x292 > [ 340.320090] [<ffffffff814ab82c>] blkdev_issue_discard+0xee/0x148 > [ 340.320093] [<ffffffff814ab73e>] ? __blkdev_issue_discard+0x399/0x399 > [ 340.320097] [<ffffffff8111f0ac>] ? debug_show_all_locks+0x226/0x226 > [ 340.320101] [<ffffffff81404de8>] ext4_free_data_callback+0x2cc/0x8bc > [ 340.320104] [<ffffffff81404de8>] ? ext4_free_data_callback+0x2cc/0x8bc > [ 340.320107] [<ffffffff81404b1c>] ? ext4_mb_release_context+0x10aa/0x10aa > [ 340.320111] [<ffffffff81122c56>] ? lock_acquire+0xec/0x147 > [ 340.320115] [<ffffffff813c8a6a>] ? ext4_journal_commit_callback+0x203/0x220 > [ 340.320119] [<ffffffff813c8a61>] ext4_journal_commit_callback+0x1fa/0x220 > [ 340.320124] [<ffffffff81438bf5>] jbd2_journal_commit_transaction+0x3753/0x3c20 > [ 340.320128] [<ffffffff814354a2>] ? journal_submit_commit_record+0x777/0x777 > [ 340.320132] [<ffffffff8111f0ac>] ? debug_show_all_locks+0x226/0x226 > [ 340.320135] [<ffffffff811205a5>] ? __lock_acquire+0x14f9/0x33b8 > [ 340.320139] [<ffffffff81d31db0>] ? _raw_spin_unlock_irqrestore+0x3b/0x5c > [ 340.320143] [<ffffffff8111cde6>] ? trace_hardirqs_on_caller+0x3d2/0x492 > [ 340.320146] [<ffffffff81d31dbc>] ? _raw_spin_unlock_irqrestore+0x47/0x5c > [ 340.320151] [<ffffffff81156945>] ? try_to_del_timer_sync+0xa5/0xce > [ 340.320154] [<ffffffff8111cde6>] ? trace_hardirqs_on_caller+0x3d2/0x492 > [ 340.320157] [<ffffffff8143febd>] kjournald2+0x246/0x6e1 > [ 340.320160] [<ffffffff8143febd>] ? kjournald2+0x246/0x6e1 > [ 340.320163] [<ffffffff8143fc77>] ? commit_timeout+0xb/0xb > [ 340.320167] [<ffffffff8111112e>] ? prepare_to_wait_event+0x3f7/0x3f7 > [ 340.320171] [<ffffffff810ccde3>] kthread+0x252/0x261 > [ 340.320174] [<ffffffff8143fc77>] ? commit_timeout+0xb/0xb > [ 340.320177] [<ffffffff810ccb91>] ? kthread_create_on_node+0x377/0x377 > [ 340.320181] [<ffffffff81d3277f>] ret_from_fork+0x1f/0x40 > [ 340.320185] [<ffffffff810ccb91>] ? kthread_create_on_node+0x377/0x377 > [ 340.320186] Code: 5c 5d c3 55 48 8d 04 bd 00 00 00 00 65 48 8b 15 8d 59 b2 7e 48 69 d2 fa 00 00 00 48 89 e5 f7 e2 48 8d 7a 01 e8 22 01 00 00 5d c3 <55> 48 89 e5 41 56 41 55 41 54 53 49 89 fd bf 01 00 00 00 e8 ed > > -ss
Hi, On (06/16/16 08:12), Minchan Kim wrote:> > [ 315.146533] kasan: CONFIG_KASAN_INLINE enabled > > [ 315.146538] kasan: GPF could be caused by NULL-ptr deref or user memory access > > [ 315.146546] general protection fault: 0000 [#1] PREEMPT SMP KASAN > > [ 315.146576] Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel r8169 i2c_i801 mii snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core acpi_cpufreq snd_pcm snd_timer snd soundcore lpc_ich mfd_core processor sch_fq_codel sd_mod hid_generic usbhid hid ahci libahci libata ehci_pci ehci_hcd scsi_mod usbcore usb_common > > [ 315.146785] CPU: 3 PID: 38 Comm: khugepaged Not tainted 4.7.0-rc3-next-20160614-dbg-00004-ga1c2cbc-dirty #488 > > [ 315.146841] task: ffff8800bfaf2900 ti: ffff880112468000 task.ti: ffff880112468000 > > [ 315.146859] RIP: 0010:[<ffffffffa02c413d>] [<ffffffffa02c413d>] zs_page_migrate+0x355/0xaa0 [zsmalloc] > > Thanks for the report! > > zs_page_migrate+0x355? Could you tell me what line is it? > > It seems to be related to obj_to_head.reproduced. a bit different call stack this time. but the problem is still the same. zs_compact() ... 6371: e8 00 00 00 00 callq 6376 <zs_compact+0x22b> 6376: 0f 0b ud2 6378: 48 8b 95 a8 fe ff ff mov -0x158(%rbp),%rdx 637f: 4d 8d 74 24 78 lea 0x78(%r12),%r14 6384: 4c 89 ee mov %r13,%rsi 6387: 4c 89 e7 mov %r12,%rdi 638a: e8 86 c7 ff ff callq 2b15 <get_first_obj_offset> 638f: 41 89 c5 mov %eax,%r13d 6392: 4c 89 f0 mov %r14,%rax 6395: 48 c1 e8 03 shr $0x3,%rax 6399: 8a 04 18 mov (%rax,%rbx,1),%al 639c: 84 c0 test %al,%al 639e: 0f 85 f2 02 00 00 jne 6696 <zs_compact+0x54b> 63a4: 41 8b 44 24 78 mov 0x78(%r12),%eax 63a9: 41 0f af c7 imul %r15d,%eax 63ad: 41 01 c5 add %eax,%r13d 63b0: 4c 89 f0 mov %r14,%rax 63b3: 48 c1 e8 03 shr $0x3,%rax 63b7: 48 01 d8 add %rbx,%rax 63ba: 48 89 85 88 fe ff ff mov %rax,-0x178(%rbp) 63c1: 41 81 fd ff 0f 00 00 cmp $0xfff,%r13d 63c8: 0f 87 1a 03 00 00 ja 66e8 <zs_compact+0x59d> 63ce: 49 63 f5 movslq %r13d,%rsi 63d1: 48 03 b5 98 fe ff ff add -0x168(%rbp),%rsi 63d8: 48 8b bd a8 fe ff ff mov -0x158(%rbp),%rdi 63df: e8 67 d9 ff ff callq 3d4b <obj_to_head> 63e4: a8 01 test $0x1,%al 63e6: 0f 84 d9 02 00 00 je 66c5 <zs_compact+0x57a> 63ec: 48 83 e0 fe and $0xfffffffffffffffe,%rax 63f0: bf 01 00 00 00 mov $0x1,%edi 63f5: 48 89 85 b0 fe ff ff mov %rax,-0x150(%rbp) 63fc: e8 00 00 00 00 callq 6401 <zs_compact+0x2b6> 6401: 48 8b 85 b0 fe ff ff mov -0x150(%rbp),%rax ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 6408: f0 0f ba 28 00 lock btsl $0x0,(%rax) 640d: 0f 82 98 02 00 00 jb 66ab <zs_compact+0x560> 6413: 48 8b 85 10 fe ff ff mov -0x1f0(%rbp),%rax 641a: 48 8d b8 48 10 00 00 lea 0x1048(%rax),%rdi 6421: 48 89 f8 mov %rdi,%rax 6424: 48 c1 e8 03 shr $0x3,%rax 6428: 8a 04 18 mov (%rax,%rbx,1),%al 642b: 84 c0 test %al,%al 642d: 0f 85 c5 02 00 00 jne 66f8 <zs_compact+0x5ad> 6433: 48 8b 85 10 fe ff ff mov -0x1f0(%rbp),%rax 643a: 65 4c 8b 2c 25 00 00 mov %gs:0x0,%r13 6441: 00 00 6443: 49 8d bd 48 10 00 00 lea 0x1048(%r13),%rdi 644a: ff 88 48 10 00 00 decl 0x1048(%rax) 6450: 48 89 f8 mov %rdi,%rax 6453: 48 c1 e8 03 shr $0x3,%rax 6457: 8a 04 18 mov (%rax,%rbx,1),%al 645a: 84 c0 test %al,%al 645c: 0f 85 a8 02 00 00 jne 670a <zs_compact+0x5bf> 6462: 41 83 bd 48 10 00 00 cmpl $0x0,0x1048(%r13) which is _next/./arch/x86/include/asm/bitops.h:206 _next/./arch/x86/include/asm/bitops.h:219 _next/include/linux/bit_spinlock.h:44 _next/mm/zsmalloc.c:950 _next/mm/zsmalloc.c:1774 _next/mm/zsmalloc.c:1809 _next/mm/zsmalloc.c:2306 _next/mm/zsmalloc.c:2346 smells like race conditon. backtraces: [ 319.363646] kasan: CONFIG_KASAN_INLINE enabled [ 319.363650] kasan: GPF could be caused by NULL-ptr deref or user memory access [ 319.363658] general protection fault: 0000 [#1] PREEMPT SMP KASAN [ 319.363688] Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel snd_hda_codec_realtek snd_hda_codec_generic r8169 mii i2c_i801 snd_hda_intel snd_hda_codec snd_hda_core snd_pcm snd_timer acpi_cpufreq snd lpc_ich soundcore mfd_core processor sch_fq_codel sd_mod hid_generic usbhid hid ahci libahci ehci_pci libata ehci_hcd usbcore scsi_mod usb_common [ 319.363895] CPU: 0 PID: 45 Comm: kswapd0 Not tainted 4.7.0-rc3-next-20160615-dbg-00004-g550dc8a-dirty #490 [ 319.363950] task: ffff8800bfb93d80 ti: ffff880112200000 task.ti: ffff880112200000 [ 319.363968] RIP: 0010:[<ffffffffa03ce408>] [<ffffffffa03ce408>] zs_compact+0x2bd/0xf22 [zsmalloc] [ 319.364000] RSP: 0018:ffff8801122077f8 EFLAGS: 00010293 [ 319.364014] RAX: 2065676162726166 RBX: dffffc0000000000 RCX: 0000000000000000 [ 319.364032] RDX: 1ffffffff064c504 RSI: ffff88003217c770 RDI: ffffffff83262ae0 [ 319.364049] RBP: ffff880112207a18 R08: 0000000000000001 R09: 0000000000000000 [ 319.364067] R10: ffff880112207768 R11: 00000000a19f2c26 R12: ffff8800a7caab00 [ 319.364085] R13: 0000000000000770 R14: ffff8800a7caab78 R15: 0000000000000000 [ 319.364103] FS: 0000000000000000(0000) GS:ffff880113600000(0000) knlGS:0000000000000000 [ 319.364123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 319.364138] CR2: 00007fa154633d70 CR3: 00000000b183d000 CR4: 00000000000006f0 [ 319.364154] Stack: [ 319.364160] ffffed00163d6a81 1ffff10017f729b9 ffff8800bfb944a0 ffffed0017f729b9 [ 319.364191] ffff8800bfb93d80 ffff8800b1eb5408 ffff8800bfb93d80 ffff8800bfb94dc8 [ 319.364222] ffff8800bfb944f8 ffff880000000001 1ffff10022440f1a 0000000041b58ab3 [ 319.364252] Call Trace: [ 319.364264] [<ffffffff8111f405>] ? debug_show_all_locks+0x226/0x226 [ 319.364284] [<ffffffffa03ce14b>] ? zs_free+0x27a/0x27a [zsmalloc] [ 319.364303] [<ffffffff812303e3>] ? list_lru_count_one+0x65/0x6d [ 319.364320] [<ffffffff81122faf>] ? lock_acquire+0xec/0x147 [ 319.364336] [<ffffffff812303b7>] ? list_lru_count_one+0x39/0x6d [ 319.364353] [<ffffffff81d32e4f>] ? _raw_spin_unlock+0x2c/0x3f [ 319.364371] [<ffffffffa03cf0a8>] zs_shrinker_scan+0x3b/0x4e [zsmalloc] [ 319.364391] [<ffffffff81204eef>] shrink_slab.part.5.constprop.17+0x2e4/0x432 [ 319.364411] [<ffffffff81204c0b>] ? cpu_callback+0xb0/0xb0 [ 319.364426] [<ffffffff8120bfbc>] shrink_zone+0x19b/0x416 [ 319.364442] [<ffffffff8120be21>] ? shrink_zone_memcg.isra.14+0xd08/0xd08 [ 319.364461] [<ffffffff811f0b10>] ? zone_watermark_ok_safe+0x1e9/0x1f8 [ 319.364478] [<ffffffff81205fd7>] ? zone_reclaimable+0x14b/0x170 [ 319.364495] [<ffffffff8120d2fb>] kswapd+0xaad/0xcee [ 319.364510] [<ffffffff8120c84e>] ? try_to_free_pages+0x617/0x617 [ 319.364527] [<ffffffff8111d13f>] ? trace_hardirqs_on_caller+0x3d2/0x492 [ 319.364545] [<ffffffff81111487>] ? prepare_to_wait_event+0x3f7/0x3f7 [ 319.364564] [<ffffffff810cd0de>] kthread+0x252/0x261 [ 319.364578] [<ffffffff8120c84e>] ? try_to_free_pages+0x617/0x617 [ 319.364595] [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 [ 319.364614] [<ffffffff81d3387f>] ret_from_fork+0x1f/0x40 [ 319.364629] [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 [ 319.364645] Code: ff ff e8 67 d9 ff ff a8 01 0f 84 d9 02 00 00 48 83 e0 fe bf 01 00 00 00 48 89 85 b0 fe ff ff e8 71 78 d0 e0 48 8b 85 b0 fe ff ff <f0> 0f ba 28 00 0f 82 98 02 00 00 48 8b 85 10 fe ff ff 48 8d b8 [ 319.364913] RIP [<ffffffffa03ce408>] zs_compact+0x2bd/0xf22 [zsmalloc] [ 319.364937] RSP <ffff8801122077f8> [ 319.372870] ---[ end trace bcefd5a456f6b462 ]--- [ 319.372875] BUG: sleeping function called from invalid context at include/linux/sched.h:2960 [ 319.372877] in_atomic(): 1, irqs_disabled(): 0, pid: 45, name: kswapd0 [ 319.372879] INFO: lockdep is turned off. [ 319.372880] Preemption disabled at:[<ffffffffa03ce2c3>] zs_compact+0x178/0xf22 [zsmalloc] [ 319.372891] CPU: 0 PID: 45 Comm: kswapd0 Tainted: G D 4.7.0-rc3-next-20160615-dbg-00004-g550dc8a-dirty #490 [ 319.372895] 0000000000000000 ffff880112207418 ffffffff814d69b0 ffff8800bfb93d80 [ 319.372901] 0000000000000003 ffff880112207458 ffffffff810d6165 0000000000000000 [ 319.372906] ffff8800bfb93d80 ffffffff81e39860 0000000000000b90 0000000000000000 [ 319.372911] Call Trace: [ 319.372915] [<ffffffff814d69b0>] dump_stack+0x68/0x92 [ 319.372919] [<ffffffff810d6165>] ___might_sleep+0x3bd/0x3c9 [ 319.372922] [<ffffffff810d62cc>] __might_sleep+0x15b/0x167 [ 319.372927] [<ffffffff810ac7bf>] exit_signals+0x7a/0x34f [ 319.372931] [<ffffffff810ac745>] ? get_signal+0xd9b/0xd9b [ 319.372934] [<ffffffff811af758>] ? irq_work_queue+0x101/0x11c [ 319.372938] [<ffffffff8111f405>] ? debug_show_all_locks+0x226/0x226 [ 319.372943] [<ffffffff81096655>] do_exit+0x34d/0x1b4e [ 319.372947] [<ffffffff8113119f>] ? vprintk_emit+0x4b1/0x4d3 [ 319.372951] [<ffffffff81096308>] ? is_current_pgrp_orphaned+0x8c/0x8c [ 319.372954] [<ffffffff81122faf>] ? lock_acquire+0xec/0x147 [ 319.372957] [<ffffffff81132578>] ? kmsg_dump+0x12/0x27a [ 319.372961] [<ffffffff811327d1>] ? kmsg_dump+0x26b/0x27a [ 319.372965] [<ffffffff81036507>] oops_end+0x9d/0xa4 [ 319.372968] [<ffffffff81036641>] die+0x55/0x5e [ 319.372971] [<ffffffff81032aa0>] do_general_protection+0x16c/0x337 [ 319.372975] [<ffffffff81d34bbf>] general_protection+0x1f/0x30 [ 319.372981] [<ffffffffa03ce408>] ? zs_compact+0x2bd/0xf22 [zsmalloc] [ 319.372986] [<ffffffffa03ce401>] ? zs_compact+0x2b6/0xf22 [zsmalloc] [ 319.372989] [<ffffffff8111f405>] ? debug_show_all_locks+0x226/0x226 [ 319.372995] [<ffffffffa03ce14b>] ? zs_free+0x27a/0x27a [zsmalloc] [ 319.372999] [<ffffffff812303e3>] ? list_lru_count_one+0x65/0x6d [ 319.373002] [<ffffffff81122faf>] ? lock_acquire+0xec/0x147 [ 319.373005] [<ffffffff812303b7>] ? list_lru_count_one+0x39/0x6d [ 319.373009] [<ffffffff81d32e4f>] ? _raw_spin_unlock+0x2c/0x3f [ 319.373014] [<ffffffffa03cf0a8>] zs_shrinker_scan+0x3b/0x4e [zsmalloc] [ 319.373018] [<ffffffff81204eef>] shrink_slab.part.5.constprop.17+0x2e4/0x432 [ 319.373022] [<ffffffff81204c0b>] ? cpu_callback+0xb0/0xb0 [ 319.373025] [<ffffffff8120bfbc>] shrink_zone+0x19b/0x416 [ 319.373029] [<ffffffff8120be21>] ? shrink_zone_memcg.isra.14+0xd08/0xd08 [ 319.373032] [<ffffffff811f0b10>] ? zone_watermark_ok_safe+0x1e9/0x1f8 [ 319.373036] [<ffffffff81205fd7>] ? zone_reclaimable+0x14b/0x170 [ 319.373039] [<ffffffff8120d2fb>] kswapd+0xaad/0xcee [ 319.373043] [<ffffffff8120c84e>] ? try_to_free_pages+0x617/0x617 [ 319.373046] [<ffffffff8111d13f>] ? trace_hardirqs_on_caller+0x3d2/0x492 [ 319.373050] [<ffffffff81111487>] ? prepare_to_wait_event+0x3f7/0x3f7 [ 319.373054] [<ffffffff810cd0de>] kthread+0x252/0x261 [ 319.373057] [<ffffffff8120c84e>] ? try_to_free_pages+0x617/0x617 [ 319.373060] [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 [ 319.373064] [<ffffffff81d3387f>] ret_from_fork+0x1f/0x40 [ 319.373068] [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 [ 319.373071] note: kswapd0[45] exited with preempt_count 3 [ 322.891083] kmemleak: Cannot allocate a kmemleak_object structure [ 322.891091] kmemleak: Kernel memory leak detector disabled [ 322.891194] kmemleak: Automatic memory scanning thread ended [ 344.264076] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:3:108] [ 344.264080] Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel snd_hda_codec_realtek snd_hda_codec_generic r8169 mii i2c_i801 snd_hda_intel snd_hda_codec snd_hda_core snd_pcm snd_timer acpi_cpufreq snd lpc_ich soundcore mfd_core processor sch_fq_codel sd_mod hid_generic usbhid hid ahci libahci ehci_pci libata ehci_hcd usbcore scsi_mod usb_common [ 344.264118] irq event stamp: 13848655 [ 344.264119] hardirqs last enabled at (13848655): [<ffffffff8127dbd8>] __slab_alloc.isra.18.constprop.23+0x53/0x61 [ 344.264127] hardirqs last disabled at (13848654): [<ffffffff8127db9e>] __slab_alloc.isra.18.constprop.23+0x19/0x61 [ 344.264131] softirqs last enabled at (13848614): [<ffffffff81d3565e>] __do_softirq+0x406/0x48f [ 344.264136] softirqs last disabled at (13848593): [<ffffffff81099448>] irq_exit+0x6a/0x113 [ 344.264143] CPU: 1 PID: 108 Comm: kworker/u8:3 Tainted: G D 4.7.0-rc3-next-20160615-dbg-00004-g550dc8a-dirty #490 [ 344.264151] Workqueue: writeback wb_workfn (flush-254:0) [ 344.264155] task: ffff8800ba1c2900 ti: ffff8801122a0000 task.ti: ffff8801122a0000 [ 344.264157] RIP: 0010:[<ffffffff814eeae3>] [<ffffffff814eeae3>] delay_tsc+0x81/0xa4 [ 344.264162] RSP: 0018:ffff8801122a70d0 EFLAGS: 00000206 [ 344.264164] RAX: 000000000000001c RBX: 000000dc3a548e47 RCX: 0000000000000000 [ 344.264166] RDX: 000000dc3a548e63 RSI: ffffffff81ed2e80 RDI: ffffffff81ed2ec0 [ 344.264168] RBP: ffff8801122a70f0 R08: 0000000000000001 R09: 0000000000000000 [ 344.264170] R10: ffff8801122a70e8 R11: 0000000045cb5d4f R12: 000000dc3a548e63 [ 344.264172] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000 [ 344.264175] FS: 0000000000000000(0000) GS:ffff880113680000(0000) knlGS:0000000000000000 [ 344.264177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 344.264179] CR2: 00007fa26a978978 CR3: 0000000002209000 CR4: 00000000000006e0 [ 344.264180] Stack: [ 344.264181] ffff8800a7caab00 ffff8800a7caab10 ffff8800a7caab08 0000000022af534e [ 344.264186] ffff8801122a7100 ffffffff814eeb8c ffff8801122a7148 ffffffff81127ce6 [ 344.264191] ffffed0014f95560 000000009e85cd68 ffff8800a7caab00 ffff8800a7caab58 [ 344.264196] Call Trace: [ 344.264199] [<ffffffff814eeb8c>] __delay+0xa/0xc [ 344.264203] [<ffffffff81127ce6>] do_raw_spin_lock+0x197/0x257 [ 344.264206] [<ffffffff81d32d0d>] _raw_spin_lock+0x35/0x3c [ 344.264212] [<ffffffffa03ccd78>] ? zs_malloc+0x17e/0xb71 [zsmalloc] [ 344.264217] [<ffffffffa03ccd78>] zs_malloc+0x17e/0xb71 [zsmalloc] [ 344.264220] [<ffffffffa0190204>] ? lzo_decompress+0x11d/0x11d [lzo] [ 344.264223] [<ffffffff81122faf>] ? lock_acquire+0xec/0x147 [ 344.264228] [<ffffffffa03ccbfa>] ? obj_malloc+0x372/0x372 [zsmalloc] [ 344.264233] [<ffffffff81472ff9>] ? crypto_compress+0x87/0x93 [ 344.264238] [<ffffffffa041522d>] zram_bvec_rw+0x1073/0x1638 [zram] [ 344.264243] [<ffffffffa04141ba>] ? zram_slot_free_notify+0x1c8/0x1c8 [zram] [ 344.264247] [<ffffffff812fc37b>] ? wb_writeback+0x316/0x44c [ 344.264251] [<ffffffffa0416104>] zram_make_request+0x6f5/0x89f [zram] [ 344.264255] [<ffffffff81111ef0>] ? woken_wake_function+0x51/0x51 [ 344.264260] [<ffffffffa0415a0f>] ? zram_rw_page+0x21d/0x21d [zram] [ 344.264263] [<ffffffff81494948>] ? blk_exit_rl+0x39/0x39 [ 344.264267] [<ffffffff81491130>] ? handle_bad_sector+0x192/0x192 [ 344.264271] [<ffffffff811506a1>] ? call_rcu+0x12/0x14 [ 344.264274] [<ffffffff8129a684>] ? put_object+0x58/0x5b [ 344.264277] [<ffffffff81496128>] generic_make_request+0x2bc/0x496 [ 344.264280] [<ffffffff81495e6c>] ? blk_plug_queued_count+0x103/0x103 [ 344.264283] [<ffffffff814965fa>] submit_bio+0x2f8/0x324 [ 344.264286] [<ffffffff81496302>] ? generic_make_request+0x496/0x496 [ 344.264289] [<ffffffff813aa993>] ? ext4_reserve_inode_write+0x101/0x101 [ 344.264292] [<ffffffff813b44e8>] ext4_io_submit+0x12d/0x15d [ 344.264295] [<ffffffff813ac54d>] ext4_writepages+0x15f9/0x1660 [ 344.264298] [<ffffffff813aaf54>] ? ext4_mark_inode_dirty+0x5c1/0x5c1 [ 344.264301] [<ffffffff8111f405>] ? debug_show_all_locks+0x226/0x226 [ 344.264304] [<ffffffff8111f405>] ? debug_show_all_locks+0x226/0x226 [ 344.264307] [<ffffffff8111f9a4>] ? __lock_acquire+0x59f/0x33b8 [ 344.264311] [<ffffffff811fa6ea>] do_writepages+0x93/0xa1 [ 344.264315] [<ffffffff812fb7a0>] ? writeback_sb_inodes+0x270/0x85e [ 344.264317] [<ffffffff811fa6ea>] ? do_writepages+0x93/0xa1 [ 344.264321] [<ffffffff812fb287>] __writeback_single_inode+0x8b/0x334 [ 344.264324] [<ffffffff812fb9c9>] writeback_sb_inodes+0x499/0x85e [ 344.264327] [<ffffffff812fb530>] ? __writeback_single_inode+0x334/0x334 [ 344.264331] [<ffffffff81115e1c>] ? down_read_trylock+0x53/0xaf [ 344.264335] [<ffffffff812a7398>] ? trylock_super+0x16/0xaf [ 344.264338] [<ffffffff812fbe95>] __writeback_inodes_wb+0x107/0x17d [ 344.264341] [<ffffffff812fc37b>] wb_writeback+0x316/0x44c [ 344.264345] [<ffffffff812fc065>] ? writeback_inodes_wb.constprop.10+0x15a/0x15a [ 344.264348] [<ffffffff811f837f>] ? wb_over_bg_thresh+0x110/0x194 [ 344.264351] [<ffffffff811f826f>] ? balance_dirty_pages_ratelimited+0x14f5/0x14f5 [ 344.264354] [<ffffffff812fce5d>] ? wb_workfn+0x296/0x6d6 [ 344.264357] [<ffffffff812fced4>] wb_workfn+0x30d/0x6d6 [ 344.264360] [<ffffffff812fced4>] ? wb_workfn+0x30d/0x6d6 [ 344.264364] [<ffffffff812fcbc7>] ? inode_wait_for_writeback+0x2e/0x2e [ 344.264368] [<ffffffff810be6d0>] process_one_work+0x6f4/0xb2c [ 344.264371] [<ffffffff810bdfdc>] ? pwq_dec_nr_in_flight+0x22b/0x22b [ 344.264375] [<ffffffff810c0de0>] worker_thread+0x5bb/0x88e [ 344.264378] [<ffffffff810cd0de>] kthread+0x252/0x261 [ 344.264381] [<ffffffff810c0825>] ? rescuer_thread+0x879/0x879 [ 344.264383] [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 [ 344.264387] [<ffffffff81d3387f>] ret_from_fork+0x1f/0x40 [ 344.264390] [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 [ 344.264392] Code: 14 6a b2 7e 85 c0 75 05 e8 8b 35 b1 ff f3 90 bf 01 00 00 00 e8 a1 71 be ff e8 e6 f3 01 00 44 39 f0 74 b6 4c 29 e3 49 01 dd eb 97 <bf> 01 00 00 00 e8 4c 81 be ff 65 8b 05 dc 69 b2 7e 85 c0 75 05> Could you test with [zsmalloc: keep first object offset in struct page] > in mmotm?sure, I can. will it help, tho? we have a race condition here I think. -ss
On Thu, Jun 16, 2016 at 11:48:27AM +0900, Sergey Senozhatsky wrote:> Hi, > > On (06/16/16 08:12), Minchan Kim wrote: > > > [ 315.146533] kasan: CONFIG_KASAN_INLINE enabled > > > [ 315.146538] kasan: GPF could be caused by NULL-ptr deref or user memory access > > > [ 315.146546] general protection fault: 0000 [#1] PREEMPT SMP KASAN > > > [ 315.146576] Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel r8169 i2c_i801 mii snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core acpi_cpufreq snd_pcm snd_timer snd soundcore lpc_ich mfd_core processor sch_fq_codel sd_mod hid_generic usbhid hid ahci libahci libata ehci_pci ehci_hcd scsi_mod usbcore usb_common > > > [ 315.146785] CPU: 3 PID: 38 Comm: khugepaged Not tainted 4.7.0-rc3-next-20160614-dbg-00004-ga1c2cbc-dirty #488 > > > [ 315.146841] task: ffff8800bfaf2900 ti: ffff880112468000 task.ti: ffff880112468000 > > > [ 315.146859] RIP: 0010:[<ffffffffa02c413d>] [<ffffffffa02c413d>] zs_page_migrate+0x355/0xaa0 [zsmalloc] > > > > Thanks for the report! > > > > zs_page_migrate+0x355? Could you tell me what line is it? > > > > It seems to be related to obj_to_head. > > reproduced. a bit different call stack this time. but the problem is > still the same. > > zs_compact() > ... > 6371: e8 00 00 00 00 callq 6376 <zs_compact+0x22b> > 6376: 0f 0b ud2 > 6378: 48 8b 95 a8 fe ff ff mov -0x158(%rbp),%rdx > 637f: 4d 8d 74 24 78 lea 0x78(%r12),%r14 > 6384: 4c 89 ee mov %r13,%rsi > 6387: 4c 89 e7 mov %r12,%rdi > 638a: e8 86 c7 ff ff callq 2b15 <get_first_obj_offset> > 638f: 41 89 c5 mov %eax,%r13d > 6392: 4c 89 f0 mov %r14,%rax > 6395: 48 c1 e8 03 shr $0x3,%rax > 6399: 8a 04 18 mov (%rax,%rbx,1),%al > 639c: 84 c0 test %al,%al > 639e: 0f 85 f2 02 00 00 jne 6696 <zs_compact+0x54b> > 63a4: 41 8b 44 24 78 mov 0x78(%r12),%eax > 63a9: 41 0f af c7 imul %r15d,%eax > 63ad: 41 01 c5 add %eax,%r13d > 63b0: 4c 89 f0 mov %r14,%rax > 63b3: 48 c1 e8 03 shr $0x3,%rax > 63b7: 48 01 d8 add %rbx,%rax > 63ba: 48 89 85 88 fe ff ff mov %rax,-0x178(%rbp) > 63c1: 41 81 fd ff 0f 00 00 cmp $0xfff,%r13d > 63c8: 0f 87 1a 03 00 00 ja 66e8 <zs_compact+0x59d> > 63ce: 49 63 f5 movslq %r13d,%rsi > 63d1: 48 03 b5 98 fe ff ff add -0x168(%rbp),%rsi > 63d8: 48 8b bd a8 fe ff ff mov -0x158(%rbp),%rdi > 63df: e8 67 d9 ff ff callq 3d4b <obj_to_head> > 63e4: a8 01 test $0x1,%al > 63e6: 0f 84 d9 02 00 00 je 66c5 <zs_compact+0x57a> > 63ec: 48 83 e0 fe and $0xfffffffffffffffe,%rax > 63f0: bf 01 00 00 00 mov $0x1,%edi > 63f5: 48 89 85 b0 fe ff ff mov %rax,-0x150(%rbp) > 63fc: e8 00 00 00 00 callq 6401 <zs_compact+0x2b6> > 6401: 48 8b 85 b0 fe ff ff mov -0x150(%rbp),%raxRAX: 2065676162726166 so rax is totally garbage, I think. It means obj_to_head returns garbage because get_first_obj_offset is utter crab because (page_idx / class->pages_per_zspage) was totally wrong.> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > 6408: f0 0f ba 28 00 lock btsl $0x0,(%rax)<snip>> > Could you test with [zsmalloc: keep first object offset in struct page] > > in mmotm? > > sure, I can. will it help, tho? we have a race condition here I think.I guess root cause is caused by get_first_obj_offset. Please test with it. Thanks!