Virtualization - Oct 2022 - [Bug] double ->queue_rq() because of timeout in ->queue_rq()

On Thu, Oct 20, 2022 at 04:01:11PM -0400, Stefan Hajnoczi
wrote:
> On Thu, Oct 20, 2022 at 05:10:13PM +0800, Ming Lei wrote:
> > Hi,
> > 
> > David Jeffery found one double ->queue_rq() issue, so far it can
> > be triggered in the following two cases:
> > 
> > 1) scsi driver in guest kernel
> > 
> > - the story could be long vmexit latency or long preempt latency of
> > vCPU pthread, then IO req is timed out before queuing the request
> > to hardware but after calling blk_mq_start_request() during
->queue_rq(),
> > then timeout handler handles it by requeue, then double
->queue_rq() is
> > caused, and kernel panic
> > 
> > 2) burst of kernel messages from irq handler 
> > 
> > For 1), I think it is one reasonable case, given latency from host
side
> > can come anytime in theory because vCPU is emulated by one normal host
> > pthread which can be preempted anywhere. For 2), I guess kernel
message is
> > supposed to be rate limited.
> > 
> > Firstly, is this kind of so long(30sec) random latency when running
kernel
> > code something normal? Or do we need to take care of it? IMO, it looks
> > reasonable in case of VM, but our VM experts may have better idea
about this
> > situation. Also the default 30sec timeout could be reduced via sysfs
or
> > drivers.
> 
> 30 seconds is a long latency that does not occur during normal
> operation, but unfortunately does happen on occasion.
Thanks for the confirmation!
> 
> I think there's an interest in understanding the root cause and solving
> long latencies (if possible) in the QEMU/KVM communities. We can
> investigate specific cases on kvm at vger.kernel.org and/or
> qemu-devel at nongnu.org.
The issue was original reported on VMware VM, but maybe David can figure
out how to trigger it on QEMU/KVM.



Thanks, 
Ming

On Fri, Oct 21, 2022 at 10:23:57AM +0800, Ming Lei
wrote:
> On Thu, Oct 20, 2022 at 04:01:11PM -0400, Stefan Hajnoczi wrote:
> > On Thu, Oct 20, 2022 at 05:10:13PM +0800, Ming Lei wrote:
> > > Hi,
> > > 
> > > David Jeffery found one double ->queue_rq() issue, so far it
can
> > > be triggered in the following two cases:
> > > 
> > > 1) scsi driver in guest kernel
> > > 
> > > - the story could be long vmexit latency or long preempt latency
of
> > > vCPU pthread, then IO req is timed out before queuing the request
> > > to hardware but after calling blk_mq_start_request() during
->queue_rq(),
> > > then timeout handler handles it by requeue, then double
->queue_rq() is
> > > caused, and kernel panic
> > > 
> > > 2) burst of kernel messages from irq handler 
> > > 
> > > For 1), I think it is one reasonable case, given latency from
host side
> > > can come anytime in theory because vCPU is emulated by one normal
host
> > > pthread which can be preempted anywhere. For 2), I guess kernel
message is
> > > supposed to be rate limited.
> > > 
> > > Firstly, is this kind of so long(30sec) random latency when
running kernel
> > > code something normal? Or do we need to take care of it? IMO, it
looks
> > > reasonable in case of VM, but our VM experts may have better idea
about this
> > > situation. Also the default 30sec timeout could be reduced via
sysfs or
> > > drivers.
> > 
> > 30 seconds is a long latency that does not occur during normal
> > operation, but unfortunately does happen on occasion.
> 
> Thanks for the confirmation!
> 
> > 
> > I think there's an interest in understanding the root cause and
solving
> > long latencies (if possible) in the QEMU/KVM communities. We can
> > investigate specific cases on kvm at vger.kernel.org and/or
> > qemu-devel at nongnu.org.
> 
> The issue was original reported on VMware VM, but maybe David can figure
> out how to trigger it on QEMU/KVM.
A very basic question:

The virtio_blk driver has no q->mq_ops->timeout() callback. Why does the
block layer still enable the timeout mechanism when the driver doesn't
implement ->timeout()?

I saw there was some "idle" hctx logic and I guess the requests are
resubmitted (although it wasn't obvious to me how that happens in the
code)? Maybe that's why the timer is still used if the driver doesn't
care about timeouts...

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL:
<http://lists.linuxfoundation.org/pipermail/virtualization/attachments/20221024/2898beb8/attachment-0001.sig>