Libguestfs - Jul 2012 - NOTICE: Data corruption bug when writing to qcow2 files

As you might have seen for the past 3 days I've been tackling a nasty
data corruption bug[1][2].

The bug occurs when ALL of the following conditions are true:

(a) You are using a qcow2 image file.

(b) You are writing out data to the image file using libguestfs or a
    libguestfs-using tool like guestfish or virt-resize.

(c) The data is not being written to a filesystem (to files or
    directories) but is being written directly to a block device
    within libguestfs, eg. updates to the partition table or writes
    directly to /dev/sdaX.

(d) You are using qemu < 1.1.0.

When the guestfs handle is closed, the data might not be written to
the qcow2 image file.  This data loss, if it happens, is silent.

This peculiar combination of factors happened to occur in the
virt-resize test program[3], and this was where I first spotted it[4]
although at first it didn't look like a data corruption bug at all.

After analysis I found that there are four separate bugs involved:

  (i) qemu had a bug where it would segfault when you sent it a
      SIGTERM signal.  It turned out that where qemu was writing to a
      qcow2 file, and the qcow2 writeback cache is enabled [NB:
      cache=none enables this cache], and write requests were in
      flight at the point when the SIGERM is received, it would crash.

   ** This bug has been fixed in qemu/qemu-kvm >= 1.1.0.  It is highly
   ** recommended that you immediately upgrade to this version, not
   ** just for libguestfs but for all usage.

 (ii) The Linux kernel sync(2) system call doesn't issue a write
      barrier for dirty blocks that are written to a block device
      directly, only for mounted filesystems.

      This bug will probably be fixed if the following patch goes
      upstream: https://lkml.org/lkml/2012/7/3/277

(iii) libguestfs was issuing sync(2) in the expectation that it
      flushed everything.

      The implication for libguestfs is that the qemu cache still
      contains data at the point when we kill qemu.  Bugs (i) and (ii)
      unexpectedly interact.

 (iv) libguestfs didn't check the return value for waitpid(2) so it
      didn't know that qemu was segfaulting, so this loss of data was
      silently ignored.

Bug (i) can be fixed by updating to qemu 1.1.0.  Unfortunately we do
not know which precise commit between 1.0 and 1.1.0 fixed the bug, and
doing a git bisect is difficult because the data corruption bug is
very hard to reproduce reliably.

Bugs (iii) and (iv) will be fixed by forthcoming patches to
libguestfs
>= 1.19.16 which will be backported to 1.16 and 1.18 branches.  Note
that this requires a new API, guestfs_shutdown[5].  If your program
wants to handle write errors correctly it will need to use this new
API, otherwise an error will be printed and ignored.  All libguestfs
tools that modify disk images have been updated to use the new API.

Hans de Goede is currently updating Fedora to qemu-kvm 1.1.0.

Versions of libguestfs which contain fixes will be announced
separately.  It is likely that these versions will *require* qemu >= 1.1.0,
so effectively our baseline version of qemu has just increased from
1.0 to 1.1.0, and this change is noted in the README file.

(Thanks to Kevin Wolf, Paolo Bonzini, Avi Kivity, Padraig Brady for
invaluable help.)

Rich.

[1] https://www.redhat.com/archives/libguestfs/2012-July/msg00005.html
[2] https://www.redhat.com/archives/libguestfs/2012-July/msg00008.html
[3]
https://github.com/libguestfs/libguestfs/blob/cb24ceedd8a8ef7da71cfcce6db10669de47685c/resize/test-virt-resize.sh
[4] https://bugzilla.redhat.com/show_bug.cgi?id=836710
[5] https://www.redhat.com/archives/libguestfs/2012-July/msg00014.html

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

On Wed, Jul 04, 2012 at 05:28:05PM +0100, Richard W.M. Jones
wrote:
> Versions of libguestfs which contain fixes will be announced
> separately.  It is likely that these versions will *require* qemu >=
1.1.0,
> so effectively our baseline version of qemu has just increased from
> 1.0 to 1.1.0, and this change is noted in the README file.
I've just pushed libguestfs stable releases 1.18.4 and 1.16.27 to the
website:

  http://libguestfs.org/download/1.16-stable/
  http://libguestfs.org/download/1.18-stable/

Upgrading is recommended, but read the rest of this email first.

Usually a stable point-release is supposed to be a simple, safe update
that will only fix bugs, because of our policy[1] of only cherry-
picking simple bug fixes.

  [1] http://libguestfs.org/guestfs.3.html#libguestfs_version_numbers

However in this case I have to urge some caution.  These releases
contain a fix for the qemu data corruption bug[2], but if your qemu
still has this bug then the new libguestfs will loudly report when
qemu crashes (instead of ignoring it).  This means that virt programs
that worked before, and the test suite, may fail where before they ran
successfully.

  [2] https://bugzilla.redhat.com/show_bug.cgi?id=836710

I thought this was better than ignoring potential data corruption.
The solution to this is to use qemu >= 1.1.0.

Furthermore, and again highly unusual for a point release, these
versions contain backports of three new APIs:

The first is guestfs_shutdown[3] which can be used from your program
to detect write/finalization failures linked to the qemu bug.  It is
not necessary, but it's recommended, to use this API from your
programs, as described here[4] and in the updated examples.

  [3] http://libguestfs.org/guestfs.3.html#guestfs_shutdown
  [4] http://libguestfs.org/guestfs.3.html#guestfs_close

The other two[5][6] are there because of a bugfix[7] to the inspection
code which I felt was important to include in the stable branches.

  [5] http://libguestfs.org/guestfs.3.html#guestfs_device_index
  [6] http://libguestfs.org/guestfs.3.html#guestfs_nr_devices
  [7] https://bugzilla.redhat.com/show_bug.cgi?id=627675

If there are any problems, please file bugs in the usual place.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org