Pkg xen devel - Nov 2019 - Bug#944247: xen domU crashes under high i/o load if you use qcow2 images

Source: xen
Severity: important

Dear Maintainer,

we have updated our server from debian oldstable (which unfortunately wasn't
running stable after the last update, bug reported) to debian buster.

unfortunately xen doesn't work reliably there either:

the virtual server crashes every 1-2 week with i/o problems and sometimes also
takes other domU instances with it.
we use qcow2 images.

the harddisk of the domU is simply no longer accessible for the linux kernel, no
logfiles are available. in the xl console the following last lines can be read,
login not possible:

[ 1450.976415] INFO: task nginx:376 blocked for more than 120 seconds.
[ 1450.976423] Not tainted 4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u5
[ 1450.976428] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1450.976469] INFO: task nginx:377 blocked for more than 120 seconds.
[ 1450.976474] Not tainted 4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u5
[ 1450.976479] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1450.976624] INFO: task nginx:378 blocked for more than 120 seconds.

the process varies:
[1523692.508073] INFO: task jbd2/xvda2-8:159 blocked for more than 120 seconds
[1523692.508084] Not tainted [...]

all hard disk accesses fail as if the i/o system is completely dead.
only "xl destroy <domid>" and recreate will help

you can easily reproduce this with the tool stress "stress -c 8 -i 8 -d
8".
it takes a maximum of 10 minutes until the vm crashes.

in our experience, as a workaround you can convert all images to raw. after our
tests, the error will no longer occur.
but since we need the snapshot functions of qcow2 images, this is not a
permanent solution.

does anyone else have problems with qcow2 images and xen under buster?
maybe this also concerns qemu?

xen vm.cfg:

we using pygrub, 4 vcpus, 1024 memory

#
# Disk device(s).
#
root = '/dev/xvda2 ro'
disk = [
                  'tap:qcow2:/[...]/disk.qcow2,xvda2,w',
                  file:/[...]/swap.img,xvda1,w',
              ]

-- System Information:
Debian Release: 10.1
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-6-amd64 (SMP w/16 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Your message dated Sat, 5 Dec 2020 19:49:50 +0100
with message-id <501a3fdb-35b9-6d5a-80f8-298d310d2f5c at knorrie.org>
and subject line Re: [Pkg-xen-devel] Bug#944247: xen domU crashes under high i/o
load if you use qcow2 images
has caused the Debian Bug report #944247,
regarding xen domU crashes under high i/o load if you use qcow2 images
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
944247: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944247
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: mario <info at munex.de>
Subject: xen domU crashes under high i/o load if you use qcow2 images
Date: Wed, 06 Nov 2019 16:46:05 +0100
Size: 4506
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20201205/6dc9eb51/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Hans van Kranenburg <hans at knorrie.org>
Subject: Re: [Pkg-xen-devel] Bug#944247: xen domU crashes under high i/o load if
you use qcow2 images
Date: Sat, 5 Dec 2020 19:49:50 +0100
Size: 2558
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20201205/6dc9eb51/attachment-0001.mht>