Kashyap Chamarthy
2016-Mar-16 08:06 UTC
[Libguestfs] [virt-builder] XFS metadata corruption in Fedora 23 template images (on resize)
Running this: $ virt-builder fedora-23 -o vm1.qcow2 --format qcow2 \ --root-password password:123456 --selinux-relabel --size 100G And, importing vm1.qcow2 into libvirt results in these "corrupted metadata buffer" errors, in a continuous loop, on the serial console: [...] [ 51.687988] XFS (vda3): First 64 bytes of corrupted metadata buffer: [ 51.688938] ffff8800bad08a00: 58 41 47 46 00 00 00 01 00 00 00 4f 00 04 e9 00 XAGF.......O.... [ 51.690212] ffff8800bad08a10: 00 00 00 01 00 00 00 02 00 00 00 00 00 00 00 01 ................ [ 51.691486] ffff8800bad08a20: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 76 ...............v [ 51.692767] ffff8800bad08a30: 00 00 00 00 00 04 e8 fb 00 04 e8 fb 00 00 00 00 ................ [ 51.694054] XFS (vda3): metadata I/O error: block 0xc1f3801 ("xfs_trans_read_buf_map") error 117 numblks 1 [ 51.695814] XFS (vda3): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120 [xfs], block 0xc468001 [ 51.697312] XFS (vda3): Unmount and run xfs_repair [ 51.698057] XFS (vda3): First 64 bytes of corrupted metadata buffer: [ 51.699012] ffff8800bad08a00: 58 41 47 46 00 00 00 01 00 00 00 50 00 02 d2 c0 XAGF.......P.... [ 51.700296] ffff8800bad08a10: 00 00 00 01 00 00 00 02 00 00 00 00 00 00 00 01 ................ [ 51.701581] ffff8800bad08a20: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 76 ...............v [ 51.702872] ffff8800bad08a30: 00 00 00 00 00 02 d2 bb 00 02 d2 bb 00 00 00 00 ................ [ 51.704166] XFS (vda3): metadata I/O error: block 0xc468001 ("xfs_trans_read_buf_map") error 117 numblks 1 [ 51.705779] systemd-journald[327]: /dev/kmsg buffer overrun, some messages lost. [ 51.717069] XFS (vda3): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120 [xfs], block 0x29b [...] I could reproduce it twice so far with QCOW2 image _and_ raw image formats. Notably, these errors occur only when the image is resized (as is the case above). I confirmed this by testing both the default raw & QCOW2 Fedora-23 image (_without_ the "--size 100G"), there are no such corrupted metadata errors in either case, and I get the login prompt on the serial console just fine. I'm using these on the host: libguestfs-tools-c-1.32.3-2.fc23.x86_64 libguestfs-xfs-1.32.3-2.fc23.x86_64 qemu-system-x86-2.4.1-7.fc23.x86_64 kernel-4.4.4-301.fc23.x86_64 -- /kashyap
Richard W.M. Jones
2016-Mar-17 10:39 UTC
Re: [Libguestfs] [virt-builder] XFS metadata corruption in Fedora 23 template images (on resize)
On Wed, Mar 16, 2016 at 09:06:50AM +0100, Kashyap Chamarthy wrote:> Running this: > > $ virt-builder fedora-23 -o vm1.qcow2 --format qcow2 \ > --root-password password:123456 --selinux-relabel --size 100G > > And, importing vm1.qcow2 into libvirt results in these "corrupted > metadata buffer" errors, in a continuous loop, on the serial console:> [ 51.717069] XFS (vda3): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120 [xfs], block 0x29bI can also reproduce this. It's interesting that it happens on x86_64. I have seem the same problem, but it was on aarch64: https://bugzilla.redhat.com/show_bug.cgi?id=1315895 That bug is rather inconclusive. Although I found a way to reproduce the problem (involving creating disk images on aarch64 and copying them over to x86_64), I don't know if the problem could be caused by the way we run xfs_growfs or by XFS itself, nor if the problem is pagesize / architecture specific or not. Anyway, it needs deeper investigation. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW