On Thu, Jun 16, 2016 at 01:23:43PM +0900, Sergey Senozhatsky wrote:> On (06/16/16 11:58), Minchan Kim wrote: > [..] > > RAX: 2065676162726166 so rax is totally garbage, I think. > > It means obj_to_head returns garbage because get_first_obj_offset is > > utter crab because (page_idx / class->pages_per_zspage) was totally > > wrong. > > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > 6408: f0 0f ba 28 00 lock btsl $0x0,(%rax) > > > > <snip> > > > > > > Could you test with [zsmalloc: keep first object offset in struct page] > > > > in mmotm? > > > > > > sure, I can. will it help, tho? we have a race condition here I think. > > > > I guess root cause is caused by get_first_obj_offset. > > sounds reasonable. > > > Please test with it. > > > this is what I'm getting with the [zsmalloc: keep first object offset in struct page] > applied: "count:0 mapcount:-127". which may be not related to zsmalloc at this point. > > kernel: BUG: Bad page state in process khugepaged pfn:101db8 > kernel: page:ffffea0004076e00 count:0 mapcount:-127 mapping: (null) index:0x1Hm, it seems double free. It doen't happen if you disable zram? IOW, it seems to be related zsmalloc migration? How easy can you reprodcue it? Could you bisect it?> kernel: flags: 0x8000000000000000() > kernel: page dumped because: nonzero mapcount > kernel: Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel snd_hda_codec_realtek i2c_i801 snd_hda_codec_generic r8169 mii snd_hda_intel snd_hda_codec snd_hda_core acpi_cpufreq snd_pcm snd_timer snd soundcore lpc_ich processor mfd_core sch_fq_codel sd_mod hid_generic usb > kernel: CPU: 3 PID: 38 Comm: khugepaged Not tainted 4.7.0-rc3-next-20160615-dbg-00005-gfd11984-dirty #491 > kernel: 0000000000000000 ffff8801124c73f8 ffffffff814d69b0 ffffea0004076e00 > kernel: ffffffff81e658a0 ffff8801124c7420 ffffffff811e9b63 0000000000000000 > kernel: ffffea0004076e00 ffffffff81e658a0 ffff8801124c7440 ffffffff811e9ca9 > kernel: Call Trace: > kernel: [<ffffffff814d69b0>] dump_stack+0x68/0x92 > kernel: [<ffffffff811e9b63>] bad_page+0x158/0x1a2 > kernel: [<ffffffff811e9ca9>] free_pages_check_bad+0xfc/0x101 > kernel: [<ffffffff811ee516>] free_hot_cold_page+0x135/0x5de > kernel: [<ffffffff811eea26>] __free_pages+0x67/0x72 > kernel: [<ffffffff81227c63>] release_freepages+0x13a/0x191 > kernel: [<ffffffff8122b3c2>] compact_zone+0x845/0x1155 > kernel: [<ffffffff8122ab7d>] ? compaction_suitable+0x76/0x76 > kernel: [<ffffffff8122bdb2>] compact_zone_order+0xe0/0x167 > kernel: [<ffffffff8122bcd2>] ? compact_zone+0x1155/0x1155 > kernel: [<ffffffff8122ce88>] try_to_compact_pages+0x2f1/0x648 > kernel: [<ffffffff8122ce88>] ? try_to_compact_pages+0x2f1/0x648 > kernel: [<ffffffff8122cb97>] ? compaction_zonelist_suitable+0x3a6/0x3a6 > kernel: [<ffffffff811ef1ea>] ? get_page_from_freelist+0x2c0/0x133c > kernel: [<ffffffff811f0350>] __alloc_pages_direct_compact+0xea/0x30d > kernel: [<ffffffff811f0266>] ? get_page_from_freelist+0x133c/0x133c > kernel: [<ffffffff811ee3b2>] ? drain_all_pages+0x1d6/0x205 > kernel: [<ffffffff811f21a8>] __alloc_pages_nodemask+0x143d/0x16b6 > kernel: [<ffffffff8111f405>] ? debug_show_all_locks+0x226/0x226 > kernel: [<ffffffff811f0d6b>] ? warn_alloc_failed+0x24c/0x24c > kernel: [<ffffffff81110ffc>] ? finish_wait+0x1a4/0x1b0 > kernel: [<ffffffff81122faf>] ? lock_acquire+0xec/0x147 > kernel: [<ffffffff81d32ed0>] ? _raw_spin_unlock_irqrestore+0x3b/0x5c > kernel: [<ffffffff81d32edc>] ? _raw_spin_unlock_irqrestore+0x47/0x5c > kernel: [<ffffffff81110ffc>] ? finish_wait+0x1a4/0x1b0 > kernel: [<ffffffff8128f73a>] khugepaged+0x1d4/0x484f > kernel: [<ffffffff8128f566>] ? hugepage_vma_revalidate+0xef/0xef > kernel: [<ffffffff810d5bcc>] ? finish_task_switch+0x3de/0x484 > kernel: [<ffffffff81d32f18>] ? _raw_spin_unlock_irq+0x27/0x45 > kernel: [<ffffffff8111d13f>] ? trace_hardirqs_on_caller+0x3d2/0x492 > kernel: [<ffffffff81111487>] ? prepare_to_wait_event+0x3f7/0x3f7 > kernel: [<ffffffff81d28bf5>] ? __schedule+0xa4d/0xd16 > kernel: [<ffffffff810cd0de>] kthread+0x252/0x261 > kernel: [<ffffffff8128f566>] ? hugepage_vma_revalidate+0xef/0xef > kernel: [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 > kernel: [<ffffffff81d3387f>] ret_from_fork+0x1f/0x40 > kernel: [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 > -- Reboot -- > > -ss
On (06/16/16 13:47), Minchan Kim wrote: [..]> > this is what I'm getting with the [zsmalloc: keep first object offset in struct page] > > applied: "count:0 mapcount:-127". which may be not related to zsmalloc at this point. > > > > kernel: BUG: Bad page state in process khugepaged pfn:101db8 > > kernel: page:ffffea0004076e00 count:0 mapcount:-127 mapping: (null) index:0x1 > > Hm, it seems double free. > > It doen't happen if you disable zram? IOW, it seems to be related > zsmalloc migration?need to test more, can't confidently answer now.> How easy can you reprodcue it? Could you bisect it?it takes some (um.. random) time to trigger the bug. I'll try to come up with more details. -ss> > kernel: flags: 0x8000000000000000() > > kernel: page dumped because: nonzero mapcount > > kernel: Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel snd_hda_codec_realtek i2c_i801 snd_hda_codec_generic r8169 mii snd_hda_intel snd_hda_codec snd_hda_core acpi_cpufreq snd_pcm snd_timer snd soundcore lpc_ich processor mfd_core sch_fq_codel sd_mod hid_generic usb > > kernel: CPU: 3 PID: 38 Comm: khugepaged Not tainted 4.7.0-rc3-next-20160615-dbg-00005-gfd11984-dirty #491 > > kernel: 0000000000000000 ffff8801124c73f8 ffffffff814d69b0 ffffea0004076e00 > > kernel: ffffffff81e658a0 ffff8801124c7420 ffffffff811e9b63 0000000000000000 > > kernel: ffffea0004076e00 ffffffff81e658a0 ffff8801124c7440 ffffffff811e9ca9 > > kernel: Call Trace: > > kernel: [<ffffffff814d69b0>] dump_stack+0x68/0x92 > > kernel: [<ffffffff811e9b63>] bad_page+0x158/0x1a2 > > kernel: [<ffffffff811e9ca9>] free_pages_check_bad+0xfc/0x101 > > kernel: [<ffffffff811ee516>] free_hot_cold_page+0x135/0x5de > > kernel: [<ffffffff811eea26>] __free_pages+0x67/0x72 > > kernel: [<ffffffff81227c63>] release_freepages+0x13a/0x191 > > kernel: [<ffffffff8122b3c2>] compact_zone+0x845/0x1155 > > kernel: [<ffffffff8122ab7d>] ? compaction_suitable+0x76/0x76 > > kernel: [<ffffffff8122bdb2>] compact_zone_order+0xe0/0x167 > > kernel: [<ffffffff8122bcd2>] ? compact_zone+0x1155/0x1155 > > kernel: [<ffffffff8122ce88>] try_to_compact_pages+0x2f1/0x648 > > kernel: [<ffffffff8122ce88>] ? try_to_compact_pages+0x2f1/0x648 > > kernel: [<ffffffff8122cb97>] ? compaction_zonelist_suitable+0x3a6/0x3a6 > > kernel: [<ffffffff811ef1ea>] ? get_page_from_freelist+0x2c0/0x133c > > kernel: [<ffffffff811f0350>] __alloc_pages_direct_compact+0xea/0x30d > > kernel: [<ffffffff811f0266>] ? get_page_from_freelist+0x133c/0x133c > > kernel: [<ffffffff811ee3b2>] ? drain_all_pages+0x1d6/0x205 > > kernel: [<ffffffff811f21a8>] __alloc_pages_nodemask+0x143d/0x16b6 > > kernel: [<ffffffff8111f405>] ? debug_show_all_locks+0x226/0x226 > > kernel: [<ffffffff811f0d6b>] ? warn_alloc_failed+0x24c/0x24c > > kernel: [<ffffffff81110ffc>] ? finish_wait+0x1a4/0x1b0 > > kernel: [<ffffffff81122faf>] ? lock_acquire+0xec/0x147 > > kernel: [<ffffffff81d32ed0>] ? _raw_spin_unlock_irqrestore+0x3b/0x5c > > kernel: [<ffffffff81d32edc>] ? _raw_spin_unlock_irqrestore+0x47/0x5c > > kernel: [<ffffffff81110ffc>] ? finish_wait+0x1a4/0x1b0 > > kernel: [<ffffffff8128f73a>] khugepaged+0x1d4/0x484f > > kernel: [<ffffffff8128f566>] ? hugepage_vma_revalidate+0xef/0xef > > kernel: [<ffffffff810d5bcc>] ? finish_task_switch+0x3de/0x484 > > kernel: [<ffffffff81d32f18>] ? _raw_spin_unlock_irq+0x27/0x45 > > kernel: [<ffffffff8111d13f>] ? trace_hardirqs_on_caller+0x3d2/0x492 > > kernel: [<ffffffff81111487>] ? prepare_to_wait_event+0x3f7/0x3f7 > > kernel: [<ffffffff81d28bf5>] ? __schedule+0xa4d/0xd16 > > kernel: [<ffffffff810cd0de>] kthread+0x252/0x261 > > kernel: [<ffffffff8128f566>] ? hugepage_vma_revalidate+0xef/0xef > > kernel: [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 > > kernel: [<ffffffff81d3387f>] ret_from_fork+0x1f/0x40 > > kernel: [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 > > -- Reboot --
On Thu, Jun 16, 2016 at 02:22:09PM +0900, Sergey Senozhatsky wrote:> On (06/16/16 13:47), Minchan Kim wrote: > [..] > > > this is what I'm getting with the [zsmalloc: keep first object offset in struct page] > > > applied: "count:0 mapcount:-127". which may be not related to zsmalloc at this point. > > > > > > kernel: BUG: Bad page state in process khugepaged pfn:101db8 > > > kernel: page:ffffea0004076e00 count:0 mapcount:-127 mapping: (null) index:0x1 > > > > Hm, it seems double free. > > > > It doen't happen if you disable zram? IOW, it seems to be related > > zsmalloc migration? > > need to test more, can't confidently answer now. > > > How easy can you reprodcue it? Could you bisect it? > > it takes some (um.. random) time to trigger the bug. > I'll try to come up with more details.Could you revert [1] and retest? [1] mm/compaction: split freepages without holding the zone lock> > -ss > > > > kernel: flags: 0x8000000000000000() > > > kernel: page dumped because: nonzero mapcount > > > kernel: Modules linked in: lzo zram zsmalloc mousedev coretemp hwmon crc32c_intel snd_hda_codec_realtek i2c_i801 snd_hda_codec_generic r8169 mii snd_hda_intel snd_hda_codec snd_hda_core acpi_cpufreq snd_pcm snd_timer snd soundcore lpc_ich processor mfd_core sch_fq_codel sd_mod hid_generic usb > > > kernel: CPU: 3 PID: 38 Comm: khugepaged Not tainted 4.7.0-rc3-next-20160615-dbg-00005-gfd11984-dirty #491 > > > kernel: 0000000000000000 ffff8801124c73f8 ffffffff814d69b0 ffffea0004076e00 > > > kernel: ffffffff81e658a0 ffff8801124c7420 ffffffff811e9b63 0000000000000000 > > > kernel: ffffea0004076e00 ffffffff81e658a0 ffff8801124c7440 ffffffff811e9ca9 > > > kernel: Call Trace: > > > kernel: [<ffffffff814d69b0>] dump_stack+0x68/0x92 > > > kernel: [<ffffffff811e9b63>] bad_page+0x158/0x1a2 > > > kernel: [<ffffffff811e9ca9>] free_pages_check_bad+0xfc/0x101 > > > kernel: [<ffffffff811ee516>] free_hot_cold_page+0x135/0x5de > > > kernel: [<ffffffff811eea26>] __free_pages+0x67/0x72 > > > kernel: [<ffffffff81227c63>] release_freepages+0x13a/0x191 > > > kernel: [<ffffffff8122b3c2>] compact_zone+0x845/0x1155 > > > kernel: [<ffffffff8122ab7d>] ? compaction_suitable+0x76/0x76 > > > kernel: [<ffffffff8122bdb2>] compact_zone_order+0xe0/0x167 > > > kernel: [<ffffffff8122bcd2>] ? compact_zone+0x1155/0x1155 > > > kernel: [<ffffffff8122ce88>] try_to_compact_pages+0x2f1/0x648 > > > kernel: [<ffffffff8122ce88>] ? try_to_compact_pages+0x2f1/0x648 > > > kernel: [<ffffffff8122cb97>] ? compaction_zonelist_suitable+0x3a6/0x3a6 > > > kernel: [<ffffffff811ef1ea>] ? get_page_from_freelist+0x2c0/0x133c > > > kernel: [<ffffffff811f0350>] __alloc_pages_direct_compact+0xea/0x30d > > > kernel: [<ffffffff811f0266>] ? get_page_from_freelist+0x133c/0x133c > > > kernel: [<ffffffff811ee3b2>] ? drain_all_pages+0x1d6/0x205 > > > kernel: [<ffffffff811f21a8>] __alloc_pages_nodemask+0x143d/0x16b6 > > > kernel: [<ffffffff8111f405>] ? debug_show_all_locks+0x226/0x226 > > > kernel: [<ffffffff811f0d6b>] ? warn_alloc_failed+0x24c/0x24c > > > kernel: [<ffffffff81110ffc>] ? finish_wait+0x1a4/0x1b0 > > > kernel: [<ffffffff81122faf>] ? lock_acquire+0xec/0x147 > > > kernel: [<ffffffff81d32ed0>] ? _raw_spin_unlock_irqrestore+0x3b/0x5c > > > kernel: [<ffffffff81d32edc>] ? _raw_spin_unlock_irqrestore+0x47/0x5c > > > kernel: [<ffffffff81110ffc>] ? finish_wait+0x1a4/0x1b0 > > > kernel: [<ffffffff8128f73a>] khugepaged+0x1d4/0x484f > > > kernel: [<ffffffff8128f566>] ? hugepage_vma_revalidate+0xef/0xef > > > kernel: [<ffffffff810d5bcc>] ? finish_task_switch+0x3de/0x484 > > > kernel: [<ffffffff81d32f18>] ? _raw_spin_unlock_irq+0x27/0x45 > > > kernel: [<ffffffff8111d13f>] ? trace_hardirqs_on_caller+0x3d2/0x492 > > > kernel: [<ffffffff81111487>] ? prepare_to_wait_event+0x3f7/0x3f7 > > > kernel: [<ffffffff81d28bf5>] ? __schedule+0xa4d/0xd16 > > > kernel: [<ffffffff810cd0de>] kthread+0x252/0x261 > > > kernel: [<ffffffff8128f566>] ? hugepage_vma_revalidate+0xef/0xef > > > kernel: [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 > > > kernel: [<ffffffff81d3387f>] ret_from_fork+0x1f/0x40 > > > kernel: [<ffffffff810cce8c>] ? kthread_create_on_node+0x377/0x377 > > > -- Reboot --