search for: khungtaskd

...are now taking hours and hours. Coinciding with this slow down I see these messages in /var/log/messages (this is hand typed, as I can't copy/paste or send email from this system, although for a change I do have remote access via 2 VPN hops): kernel: BUG: soft lockup - CPU#5 stuck for 67s! [khungtaskd] kernel: BUG: soft lockup - CPU#7 stuck for 67s! [khugepaged] These messages started appearing today around 7 AM, which is when users started reporting the slowdown. They are still occurring periodically and the system is still slow. Are these messages caused by the slowdown or are they the reason...

...ld by kworker/0:2/61: > [ 861.542406] #0: 000000002dbf8af5 ((wq_completion)"events"){+.+.}, at: process_one_work+0x1b3/0x620 > [ 861.543071] #1: 0000000076868126 ((work_completion)(&drm->hpd_work)){+.+.}, at: process_one_work+0x1b3/0x620 > [ 861.543814] 1 lock held by khungtaskd/64: > [ 861.544535] #0: 0000000059db4b53 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 > [ 861.545160] 3 locks held by kworker/6:2/320: > [ 861.545896] #0: 00000000d9e1bc59 ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 > [ 861.546702] #1: 00...

...ss_one_work+0x1b3/0x620 [ 246.748338] #2: 000000002a39c817 (&mgr->lock){+.+.}, at: drm_dp_mst_topology_mgr_suspend+0x33/0x90 [drm_kms_helper] [ 246.749120] #3: 00000000b7d2f3c0 (&aux->hw_mutex){+.+.}, at: drm_dp_dpcd_access+0x64/0x110 [drm_kms_helper] [ 246.749928] 1 lock held by khungtaskd/65: [ 246.750715] #0: 00000000407da5ec (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 [ 246.751535] 1 lock held by dmesg/1122: [ 246.752328] 2 locks held by zsh/1149: [ 246.753100] #0: 000000000a27c37b (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x37/0x40 [ 246.753901] #...

...696] 2 locks held by kworker/0:2/61: [ 861.542406] #0: 000000002dbf8af5 ((wq_completion)"events"){+.+.}, at: process_one_work+0x1b3/0x620 [ 861.543071] #1: 0000000076868126 ((work_completion)(&drm->hpd_work)){+.+.}, at: process_one_work+0x1b3/0x620 [ 861.543814] 1 lock held by khungtaskd/64: [ 861.544535] #0: 0000000059db4b53 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 [ 861.545160] 3 locks held by kworker/6:2/320: [ 861.545896] #0: 00000000d9e1bc59 ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 [ 861.546702] #1: 00000000c9f92d84 ((wor...

> > kernel: BUG: soft lockup - CPU#5 stuck for 67s! [khungtaskd] > kernel: BUG: soft lockup - CPU#7 stuck for 67s! [khugepaged] > > These messages started appearing today around 7 AM, which is when > users started reporting the slowdown. They are still occurring > periodically and the system is still slow. Are these messages caused > by the s...

...o_sys_open+0x74/0x106 [ 2161.204058] [<ffffffff811215fa>] sys_open+0x20/0x22 [ 2161.209488] [<ffffffff8146fec2>] system_call_fastpath+0x16/0x1b [ 2161.215279] INFO: lockdep is turned off. [ 2161.219841] Kernel panic - not syncing: hung_task: blocked tasks [ 2161.225647] Pid: 42, comm: khungtaskd Not tainted 2.6.39+ #3 [ 2161.231535] Call Trace: [ 2161.235146] [<ffffffff8146653b>] panic+0x91/0x1a9 [ 2161.240266] [<ffffffff810a8c81>] watchdog+0x1ae/0x219 [ 2161.244863] [<ffffffff810a8ad3>] ? rcu_read_unlock+0x23/0x23 [ 2161.250816] [<ffffffff8106c0e4>] kthread+0xa...

Next version of https://patchwork.freedesktop.org/series/46815/ Same as previous version, but some small changes made to commit messages and acks/rbs have been added Lyude Paul (5): drm/nouveau: Fix bogus drm_kms_helper_poll_enable() placement drm/nouveau: Remove duplicate poll_enable() in pmops_runtime_suspend() drm/nouveau: Fix deadlock with fb_helper with async RPM requests

Latest version of https://patchwork.freedesktop.org/series/46815/ , with one small change re: ilia Lyude Paul (5): drm/nouveau: Fix bogus drm_kms_helper_poll_enable() placement drm/nouveau: Remove duplicate poll_enable() in pmops_runtime_suspend() drm/nouveau: Fix deadlock with fb_helper with async RPM requests drm/nouveau: Use pm_runtime_get_noresume() in connector_detect()

Latest version of https://patchwork.freedesktop.org/series/46815/ with some significant improvements: - I finally figured out a clean way to do this entirely with runtime PM helpers, no avoiding grabbing refs required! - Since this new method removes the need for a lot of the other changes I made (although we probably still want those changes, but not for fixing these deadlocks)

This is the latest version of https://patchwork.freedesktop.org/series/46815/ With a bunch of fixes to the new fb_helper to prevent it from breaking module loading/unloading with nouveau. Also; lots of documentation fixes and one fix in response to a kbuild bot. Lyude Paul (8): drm/nouveau: Fix bogus drm_kms_helper_poll_enable() placement drm/nouveau: Enable polling even if we have runtime

[cc += linux-pm] Hi Lyude, First of all, thanks a lot for looking into this. On Mon, Jul 16, 2018 at 07:59:25PM -0400, Lyude Paul wrote: > In order to fix all of the spots that need to have runtime PM get/puts() > added, we need to ensure that it's possible for us to call > pm_runtime_get/put() in any context, regardless of how deep, since > almost all of the spots that are

...61516] 2 locks held by kworker/0:1/60: [ 246.762274] #0: 00000000fff6be0f ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.762982] #1: 000000005ab44fb4 ((work_completion)(&dev->power.work)){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.763890] 1 lock held by khungtaskd/64: [ 246.764664] #0: 000000008cb8b5c3 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 [ 246.765588] 5 locks held by kworker/4:2/422: [ 246.766440] #0: 00000000232f0959 ((wq_completion)"events_long"){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.767390] #1: 00000000bb59b...

...0 0.0 0 0 ? S< 19:33 0:00 [cqueue/6] root 298 0.0 0.0 0 0 ? S< 19:33 0:00 [cqueue/7] root 301 0.0 0.0 0 0 ? S< 19:33 0:00 [khubd] root 303 0.0 0.0 0 0 ? S< 19:33 0:00 [kseriod] root 387 0.0 0.0 0 0 ? S 19:33 0:00 [khungtaskd] root 388 0.0 0.0 0 0 ? S 19:33 0:00 [pdflush] root 389 0.0 0.0 0 0 ? S 19:33 0:00 [pdflush] root 390 0.0 0.0 0 0 ? S< 19:33 0:00 [kswapd0] root 391 0.0 0.0 0 0 ? S< 19:33 0:00 [aio/0] root 392 0.0 0.0 0 0 ? S< 19:3...

...held by kworker/0:1/60: > [ 246.762274] #0: 00000000fff6be0f ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.762982] #1: 000000005ab44fb4 ((work_completion)(&dev->power.work)){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.763890] 1 lock held by khungtaskd/64: > [ 246.764664] #0: 000000008cb8b5c3 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 > [ 246.765588] 5 locks held by kworker/4:2/422: > [ 246.766440] #0: 00000000232f0959 ((wq_completion)"events_long"){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.767390...

...61516] 2 locks held by kworker/0:1/60: [ 246.762274] #0: 00000000fff6be0f ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.762982] #1: 000000005ab44fb4 ((work_completion)(&dev->power.work)){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.763890] 1 lock held by khungtaskd/64: [ 246.764664] #0: 000000008cb8b5c3 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 [ 246.765588] 5 locks held by kworker/4:2/422: [ 246.766440] #0: 00000000232f0959 ((wq_completion)"events_long"){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.767390] #1: 00000000bb59b...

...held by kworker/0:1/60: > [ 246.762274] #0: 00000000fff6be0f ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.762982] #1: 000000005ab44fb4 ((work_completion)(&dev->power.work)){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.763890] 1 lock held by khungtaskd/64: > [ 246.764664] #0: 000000008cb8b5c3 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 > [ 246.765588] 5 locks held by kworker/4:2/422: > [ 246.766440] #0: 00000000232f0959 ((wq_completion)"events_long"){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.767390...

...held by kworker/0:1/60: > [ 246.762274] #0: 00000000fff6be0f ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.762982] #1: 000000005ab44fb4 ((work_completion)(&dev->power.work)){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.763890] 1 lock held by khungtaskd/64: > [ 246.764664] #0: 000000008cb8b5c3 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 > [ 246.765588] 5 locks held by kworker/4:2/422: > [ 246.766440] #0: 00000000232f0959 ((wq_completion)"events_long"){+.+.}, at: process_one_work+0x1b3/0x620 > [ 246.767390...

...61516] 2 locks held by kworker/0:1/60: [ 246.762274] #0: 00000000fff6be0f ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.762982] #1: 000000005ab44fb4 ((work_completion)(&dev->power.work)){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.763890] 1 lock held by khungtaskd/64: [ 246.764664] #0: 000000008cb8b5c3 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 [ 246.765588] 5 locks held by kworker/4:2/422: [ 246.766440] #0: 00000000232f0959 ((wq_completion)"events_long"){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.767390] #1: 00000000bb59b...

This is the next version of https://patchwork.freedesktop.org/series/46815/ With a lot more thought put into it so as to avoid the potential deadlock scenarios I missed. This also required fixing some bogus DRM helper usage. Try and deadlock me now, nouveau. I dare you!!! Lyude Paul (8): drm/nouveau: Fix bogus drm_kms_helper_poll_enable() placement drm/nouveau: Enable polling even if we

...61516] 2 locks held by kworker/0:1/60: [ 246.762274] #0: 00000000fff6be0f ((wq_completion)"pm"){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.762982] #1: 000000005ab44fb4 ((work_completion)(&dev->power.work)){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.763890] 1 lock held by khungtaskd/64: [ 246.764664] #0: 000000008cb8b5c3 (rcu_read_lock){....}, at: debug_show_all_locks+0x23/0x185 [ 246.765588] 5 locks held by kworker/4:2/422: [ 246.766440] #0: 00000000232f0959 ((wq_completion)"events_long"){+.+.}, at: process_one_work+0x1b3/0x620 [ 246.767390] #1: 00000000bb59b...