Hi all,
It seems guestmount always fails to flush changes to the mounted qcow2 snapshot.
To reproduce this problem:
1. take an external snapshot for a qcow2
2. guestmount the snapshot
3. do some changes, for example: `echo 123 > /test; cat /test`, right now
/test can be seen created and containing the right content
4. fusermount -u {mount-point}
5. virt-cat changed file, for the example above, sometimes /test doesn't
exist, sometimes /test does exist but has no content
My libguestfs version is 1.20.11, and it's running on CentOS 6.6.
I tried several ways to work this around, but none of them worked.
I suspect it's a cache problem, so I tried to wait a while before umounting
the image, sometimes this way works, sometimes it doesn't.
I also looked into the documentation for syncing and caching options, I tried to
mount the image with fuse options like:
guestmount -o direct_io -o sync -o sync_read -o big_writes -w
no luck either.
What could be the cause? Is mounting a snapshot a problem?
Thanks.
Richard W.M. Jones
2017-Mar-20 09:12 UTC
Re: [Libguestfs] Guestmount seems not to sync changes
On Mon, Mar 20, 2017 at 07:40:18AM +0000, 张强 wrote:> Hi all, > > It seems guestmount always fails to flush changes to the mounted qcow2 snapshot. > > To reproduce this problem: > 1. take an external snapshot for a qcow2 > 2. guestmount the snapshot > 3. do some changes, for example: `echo 123 > /test; cat /test`, right now /test can be seen created and containing the right content > 4. fusermount -u {mount-point} > 5. virt-cat changed file, for the example above, sometimes /test doesn't exist, sometimes /test does exist but has no content > > My libguestfs version is 1.20.11, and it's running on CentOS 6.6.This is a very old version of libguestfs and CentOS.> I tried several ways to work this around, but none of them worked. > > I suspect it's a cache problem, so I tried to wait a while before umounting the image, sometimes this way works, sometimes it doesn't. > > I also looked into the documentation for syncing and caching options, I tried to mount the image with fuse options like: > guestmount -o direct_io -o sync -o sync_read -o big_writes -w > no luck either. > > What could be the cause? Is mounting a snapshot a problem?It should definitely work. Does it work if you add the 'sync' command after fusermount? There was a bug in handling the Linux (host) kernel page cache which was fixed a long time ago, but might still affect that very old version of libguestfs. You could also see if qemu is segfaulting (try enabling core dumps). In all modern versions of libguestfs, this should be caught and flagged as an error, but maybe you are missing the error because guestmount runs in the background. You can also try debugging guestmount (with --trace --verbose), which will show the libguestfs API calls being made. Check in particular that the "/test" file gets created through the API, and that the libguestfs shutdown API is called and there is no error. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Rich, Thanks for your feedback. Adding sync didn't fix it either. Since I'm working on an urgent project right now, i'll just try the easiest possible fix which is to migrate the service to a higher version of OS and use a newer libguestfs for now. Once I have time, i'll debug the older version and let you know if there's a bug. From: Richard W.M. Jones<mailto:rjones@redhat.com> Date: 2017-03-20 17:12 To: 张强<mailto:zhangqiang@meizu.com> CC: libguestfs<mailto:libguestfs@redhat.com>; dotslash.lu<mailto:dotslash.lu@gmail.com> Subject: Re: [Libguestfs] Guestmount seems not to sync changes On Mon, Mar 20, 2017 at 07:40:18AM +0000, 张强 wrote:> Hi all, > > It seems guestmount always fails to flush changes to the mounted qcow2 snapshot. > > To reproduce this problem: > 1. take an external snapshot for a qcow2 > 2. guestmount the snapshot > 3. do some changes, for example: `echo 123 > /test; cat /test`, right now /test can be seen created and containing the right content > 4. fusermount -u {mount-point} > 5. virt-cat changed file, for the example above, sometimes /test doesn't exist, sometimes /test does exist but has no content > > My libguestfs version is 1.20.11, and it's running on CentOS 6.6.This is a very old version of libguestfs and CentOS.> I tried several ways to work this around, but none of them worked. > > I suspect it's a cache problem, so I tried to wait a while before umounting the image, sometimes this way works, sometimes it doesn't. > > I also looked into the documentation for syncing and caching options, I tried to mount the image with fuse options like: > guestmount -o direct_io -o sync -o sync_read -o big_writes -w > no luck either. > > What could be the cause? Is mounting a snapshot a problem?It should definitely work. Does it work if you add the 'sync' command after fusermount? There was a bug in handling the Linux (host) kernel page cache which was fixed a long time ago, but might still affect that very old version of libguestfs. You could also see if qemu is segfaulting (try enabling core dumps). In all modern versions of libguestfs, this should be caught and flagged as an error, but maybe you are missing the error because guestmount runs in the background. You can also try debugging guestmount (with --trace --verbose), which will show the libguestfs API calls being made. Check in particular that the "/test" file gets created through the API, and that the libguestfs shutdown API is called and there is no error. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top