Libguestfs - Mar 2016 - [virt-builder] XFS metadata corruption in Fedora 23 template images (on resize)

Running this:

    $ virt-builder fedora-23 -o vm1.qcow2 --format qcow2 \
        --root-password password:123456  --selinux-relabel --size 100G

And, importing vm1.qcow2 into libvirt results in these "corrupted
metadata buffer" errors, in a continuous loop, on the serial console:

[...]
[   51.687988] XFS (vda3): First 64 bytes of corrupted metadata buffer:
[   51.688938] ffff8800bad08a00: 58 41 47 46 00 00 00 01 00 00 00 4f 00 04 e9 00
XAGF.......O....
[   51.690212] ffff8800bad08a10: 00 00 00 01 00 00 00 02 00 00 00 00 00 00 00 01
................
[   51.691486] ffff8800bad08a20: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 76
...............v
[   51.692767] ffff8800bad08a30: 00 00 00 00 00 04 e8 fb 00 04 e8 fb 00 00 00 00
................
[   51.694054] XFS (vda3): metadata I/O error: block 0xc1f3801
("xfs_trans_read_buf_map") error 117 numblks 1
[   51.695814] XFS (vda3): Metadata corruption detected at
xfs_agf_read_verify+0x70/0x120 [xfs], block 0xc468001
[   51.697312] XFS (vda3): Unmount and run xfs_repair
[   51.698057] XFS (vda3): First 64 bytes of corrupted metadata buffer:
[   51.699012] ffff8800bad08a00: 58 41 47 46 00 00 00 01 00 00 00 50 00 02 d2 c0
XAGF.......P....
[   51.700296] ffff8800bad08a10: 00 00 00 01 00 00 00 02 00 00 00 00 00 00 00 01
................
[   51.701581] ffff8800bad08a20: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 76
...............v
[   51.702872] ffff8800bad08a30: 00 00 00 00 00 02 d2 bb 00 02 d2 bb 00 00 00 00
................
[   51.704166] XFS (vda3): metadata I/O error: block 0xc468001
("xfs_trans_read_buf_map") error 117 numblks 1
[   51.705779] systemd-journald[327]: /dev/kmsg buffer overrun, some messages
lost.
[   51.717069] XFS (vda3): Metadata corruption detected at
xfs_agf_read_verify+0x70/0x120 [xfs], block 0x29b
[...]

I could reproduce it twice so far with QCOW2 image _and_ raw image
formats.

Notably, these errors occur only when the image is resized (as is the
case above).

I confirmed this by testing both the default raw & QCOW2 Fedora-23 image
(_without_ the "--size 100G"), there are no such corrupted metadata
errors in either case, and I get the login prompt on the serial console
just fine.

I'm using these on the host:

    libguestfs-tools-c-1.32.3-2.fc23.x86_64
    libguestfs-xfs-1.32.3-2.fc23.x86_64
    qemu-system-x86-2.4.1-7.fc23.x86_64
    kernel-4.4.4-301.fc23.x86_64

-- 
/kashyap

On Wed, Mar 16, 2016 at 09:06:50AM +0100, Kashyap Chamarthy
wrote:
> Running this:
> 
>     $ virt-builder fedora-23 -o vm1.qcow2 --format qcow2 \
>         --root-password password:123456  --selinux-relabel --size 100G
> 
> And, importing vm1.qcow2 into libvirt results in these "corrupted
> metadata buffer" errors, in a continuous loop, on the serial console:
> [   51.717069] XFS (vda3): Metadata corruption detected at
xfs_agf_read_verify+0x70/0x120 [xfs], block 0x29b
I can also reproduce this.

It's interesting that it happens on x86_64.  I have seem the same
problem, but it was on aarch64:

  https://bugzilla.redhat.com/show_bug.cgi?id=1315895

That bug is rather inconclusive.  Although I found a way to reproduce
the problem (involving creating disk images on aarch64 and copying
them over to x86_64), I don't know if the problem could be caused by
the way we run xfs_growfs or by XFS itself, nor if the problem is
pagesize / architecture specific or not.

Anyway, it needs deeper investigation.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW