Virtualization - Nov 2017 - 4.14: WARNING: CPU: 4 PID: 2895 at block/blk-mq.c:1144 with virtio-blk (also 4.12 stable)

On 11/21/2017 10:27 AM, Jens Axboe wrote:
> On 11/21/2017 03:14 AM, Christian Borntraeger wrote:
>> Bisect points to
>>
>> 1b5a7455d345b223d3a4658a9e5fce985b7998c1 is the first bad commit
>> commit 1b5a7455d345b223d3a4658a9e5fce985b7998c1
>> Author: Christoph Hellwig <hch at lst.de>
>> Date:   Mon Jun 26 12:20:57 2017 +0200
>>
>>     blk-mq: Create hctx for each present CPU
>>     
>>     commit 4b855ad37194f7bdbb200ce7a1c7051fecb56a08 upstream.
>>     
>>     Currently we only create hctx for online CPUs, which can lead to a
lot
>>     of churn due to frequent soft offline / online operations.  Instead
>>     allocate one for each present CPU to avoid this and dramatically
simplify
>>     the code.
>>     
>>     Signed-off-by: Christoph Hellwig <hch at lst.de>
>>     Reviewed-by: Jens Axboe <axboe at kernel.dk>
>>     Cc: Keith Busch <keith.busch at intel.com>
>>     Cc: linux-block at vger.kernel.org
>>     Cc: linux-nvme at lists.infradead.org
>>     Link: http://lkml.kernel.org/r/20170626102058.10200-3-hch at lst.de
>>     Signed-off-by: Thomas Gleixner <tglx at linutronix.de>
>>     Cc: Oleksandr Natalenko <oleksandr at natalenko.name>
>>     Cc: Mike Galbraith <efault at gmx.de>
>>     Signed-off-by: Greg Kroah-Hartman <gregkh at
linuxfoundation.org>
> 
> I wonder if we're simply not getting the masks updated correctly.
I'll
> take a look.
Can't make it trigger here. We do init for each present CPU, which means
that if I offline a few CPUs here and register a queue, those still show
up as present (just offline) and get mapped accordingly.
>From the looks of it, your setup is different. If the CPU doesn't show
up as present and it gets hotplugged, then I can see how this condition
would trigger. What environment are you running this in? We might have
to re-introduce the cpu hotplug notifier, right now we just monitor
for a dead cpu and handle that.

-- 
Jens Axboe

On 11/21/2017 07:09 PM, Jens Axboe wrote:
> On 11/21/2017 10:27 AM, Jens Axboe wrote:
>> On 11/21/2017 03:14 AM, Christian Borntraeger wrote:
>>> Bisect points to
>>>
>>> 1b5a7455d345b223d3a4658a9e5fce985b7998c1 is the first bad commit
>>> commit 1b5a7455d345b223d3a4658a9e5fce985b7998c1
>>> Author: Christoph Hellwig <hch at lst.de>
>>> Date:   Mon Jun 26 12:20:57 2017 +0200
>>>
>>>     blk-mq: Create hctx for each present CPU
>>>     
>>>     commit 4b855ad37194f7bdbb200ce7a1c7051fecb56a08 upstream.
>>>     
>>>     Currently we only create hctx for online CPUs, which can lead
to a lot
>>>     of churn due to frequent soft offline / online operations. 
Instead
>>>     allocate one for each present CPU to avoid this and
dramatically simplify
>>>     the code.
>>>     
>>>     Signed-off-by: Christoph Hellwig <hch at lst.de>
>>>     Reviewed-by: Jens Axboe <axboe at kernel.dk>
>>>     Cc: Keith Busch <keith.busch at intel.com>
>>>     Cc: linux-block at vger.kernel.org
>>>     Cc: linux-nvme at lists.infradead.org
>>>     Link: http://lkml.kernel.org/r/20170626102058.10200-3-hch at
lst.de
>>>     Signed-off-by: Thomas Gleixner <tglx at linutronix.de>
>>>     Cc: Oleksandr Natalenko <oleksandr at natalenko.name>
>>>     Cc: Mike Galbraith <efault at gmx.de>
>>>     Signed-off-by: Greg Kroah-Hartman <gregkh at
linuxfoundation.org>
>>
>> I wonder if we're simply not getting the masks updated correctly.
I'll
>> take a look.
> 
> Can't make it trigger here. We do init for each present CPU, which
means
> that if I offline a few CPUs here and register a queue, those still show
> up as present (just offline) and get mapped accordingly.
> 
> From the looks of it, your setup is different. If the CPU doesn't show
> up as present and it gets hotplugged, then I can see how this condition
> would trigger. What environment are you running this in? We might have
> to re-introduce the cpu hotplug notifier, right now we just monitor
> for a dead cpu and handle that.
I am not doing a hot unplug and the replug, I use KVM and add a previously
not available CPU.

in libvirt/virsh speak:
  <vcpu placement='static' current='1'>4</vcpu>

On 11/21/2017 11:12 AM, Christian Borntraeger wrote:
> 
> 
> On 11/21/2017 07:09 PM, Jens Axboe wrote:
>> On 11/21/2017 10:27 AM, Jens Axboe wrote:
>>> On 11/21/2017 03:14 AM, Christian Borntraeger wrote:
>>>> Bisect points to
>>>>
>>>> 1b5a7455d345b223d3a4658a9e5fce985b7998c1 is the first bad
commit
>>>> commit 1b5a7455d345b223d3a4658a9e5fce985b7998c1
>>>> Author: Christoph Hellwig <hch at lst.de>
>>>> Date:   Mon Jun 26 12:20:57 2017 +0200
>>>>
>>>>     blk-mq: Create hctx for each present CPU
>>>>     
>>>>     commit 4b855ad37194f7bdbb200ce7a1c7051fecb56a08 upstream.
>>>>     
>>>>     Currently we only create hctx for online CPUs, which can
lead to a lot
>>>>     of churn due to frequent soft offline / online operations. 
Instead
>>>>     allocate one for each present CPU to avoid this and
dramatically simplify
>>>>     the code.
>>>>     
>>>>     Signed-off-by: Christoph Hellwig <hch at lst.de>
>>>>     Reviewed-by: Jens Axboe <axboe at kernel.dk>
>>>>     Cc: Keith Busch <keith.busch at intel.com>
>>>>     Cc: linux-block at vger.kernel.org
>>>>     Cc: linux-nvme at lists.infradead.org
>>>>     Link: http://lkml.kernel.org/r/20170626102058.10200-3-hch
at lst.de
>>>>     Signed-off-by: Thomas Gleixner <tglx at
linutronix.de>
>>>>     Cc: Oleksandr Natalenko <oleksandr at natalenko.name>
>>>>     Cc: Mike Galbraith <efault at gmx.de>
>>>>     Signed-off-by: Greg Kroah-Hartman <gregkh at
linuxfoundation.org>
>>>
>>> I wonder if we're simply not getting the masks updated
correctly. I'll
>>> take a look.
>>
>> Can't make it trigger here. We do init for each present CPU, which
means
>> that if I offline a few CPUs here and register a queue, those still
show
>> up as present (just offline) and get mapped accordingly.
>>
>> From the looks of it, your setup is different. If the CPU doesn't
show
>> up as present and it gets hotplugged, then I can see how this condition
>> would trigger. What environment are you running this in? We might have
>> to re-introduce the cpu hotplug notifier, right now we just monitor
>> for a dead cpu and handle that.
> 
> I am not doing a hot unplug and the replug, I use KVM and add a previously
> not available CPU.
> 
> in libvirt/virsh speak:
>   <vcpu placement='static' current='1'>4</vcpu>
So that's why we run into problems. It's not present when we load the
device,
but becomes present and online afterwards.

Christoph, we used to handle this just fine, your patch broke it.

I'll see if I can come up with an appropriate fix.

-- 
Jens Axboe

I can't reproduce it in my VM with adding a new CPU.  Do you have
any interesting blk-mq like actually using multiple queues?  I'll
give that a spin next.