Libguestfs - Mar 2017 - Guestmount seems not to sync changes

Hi all,

It seems guestmount always fails to flush changes to the mounted qcow2 snapshot.

To reproduce this problem:
1. take an external snapshot for a qcow2
2. guestmount the snapshot
3. do some changes, for example: `echo 123 > /test; cat /test`, right now
/test can be seen created and containing the right content
4. fusermount -u {mount-point}
5. virt-cat changed file, for the example above, sometimes /test doesn't
exist, sometimes /test does exist but has no content

My libguestfs version is 1.20.11, and it's running on CentOS 6.6.

I tried several ways to work this around, but none of them worked.

I suspect it's a cache problem, so I tried to wait a while before umounting
the image, sometimes this way works, sometimes it doesn't.

I also looked into the documentation for syncing and caching options, I tried to
mount the image with fuse options like:
guestmount -o direct_io -o sync -o sync_read -o big_writes -w
no luck either.

What could be the cause? Is mounting a snapshot a problem?

Thanks.

On Mon, Mar 20, 2017 at 07:40:18AM +0000, 张强 wrote:
> Hi all,
> 
> It seems guestmount always fails to flush changes to the mounted qcow2
snapshot.
> 
> To reproduce this problem:
> 1. take an external snapshot for a qcow2
> 2. guestmount the snapshot
> 3. do some changes, for example: `echo 123 > /test; cat /test`, right
now /test can be seen created and containing the right content
> 4. fusermount -u {mount-point}
> 5. virt-cat changed file, for the example above, sometimes /test
doesn't exist, sometimes /test does exist but has no content
> 
> My libguestfs version is 1.20.11, and it's running on CentOS 6.6.
This is a very old version of libguestfs and CentOS.
> I tried several ways to work this around, but none of them worked.
> 
> I suspect it's a cache problem, so I tried to wait a while before
umounting the image, sometimes this way works, sometimes it doesn't.
> 
> I also looked into the documentation for syncing and caching options, I
tried to mount the image with fuse options like:
> guestmount -o direct_io -o sync -o sync_read -o big_writes -w
> no luck either.
> 
> What could be the cause? Is mounting a snapshot a problem?
It should definitely work.

Does it work if you add the 'sync' command after fusermount?  There
was a bug in handling the Linux (host) kernel page cache which was
fixed a long time ago, but might still affect that very old version of
libguestfs.

You could also see if qemu is segfaulting (try enabling core dumps).
In all modern versions of libguestfs, this should be caught and
flagged as an error, but maybe you are missing the error because
guestmount runs in the background.

You can also try debugging guestmount (with --trace --verbose), which
will show the libguestfs API calls being made.  Check in particular
that the "/test" file gets created through the API, and that the
libguestfs shutdown API is called and there is no error.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Rich,
Thanks for your feedback.
Adding sync didn't fix it either. Since I'm working on an urgent project
right now, i'll just try the easiest possible fix which is to migrate the
service to a higher version of OS and use a newer libguestfs for now.
Once I have time, i'll debug the older version and let you know if
there's a bug.


From: Richard W.M. Jones<mailto:rjones@redhat.com>
Date: 2017-03-20 17:12
To: 张强<mailto:zhangqiang@meizu.com>
CC: libguestfs<mailto:libguestfs@redhat.com>;
dotslash.lu<mailto:dotslash.lu@gmail.com>
Subject: Re: [Libguestfs] Guestmount seems not to sync changes

On Mon, Mar 20, 2017 at 07:40:18AM +0000, 张强 wrote:
> Hi all,
>
> It seems guestmount always fails to flush changes to the mounted qcow2
snapshot.
>
> To reproduce this problem:
> 1. take an external snapshot for a qcow2
> 2. guestmount the snapshot
> 3. do some changes, for example: `echo 123 > /test; cat /test`, right
now /test can be seen created and containing the right content
> 4. fusermount -u {mount-point}
> 5. virt-cat changed file, for the example above, sometimes /test
doesn't exist, sometimes /test does exist but has no content
>
> My libguestfs version is 1.20.11, and it's running on CentOS 6.6.
This is a very old version of libguestfs and CentOS.
> I tried several ways to work this around, but none of them worked.
>
> I suspect it's a cache problem, so I tried to wait a while before
umounting the image, sometimes this way works, sometimes it doesn't.
>
> I also looked into the documentation for syncing and caching options, I
tried to mount the image with fuse options like:
> guestmount -o direct_io -o sync -o sync_read -o big_writes -w
> no luck either.
>
> What could be the cause? Is mounting a snapshot a problem?
It should definitely work.

Does it work if you add the 'sync' command after fusermount?  There
was a bug in handling the Linux (host) kernel page cache which was
fixed a long time ago, but might still affect that very old version of
libguestfs.

You could also see if qemu is segfaulting (try enabling core dumps).
In all modern versions of libguestfs, this should be caught and
flagged as an error, but maybe you are missing the error because
guestmount runs in the background.

You can also try debugging guestmount (with --trace --verbose), which
will show the libguestfs API calls being made.  Check in particular
that the "/test" file gets created through the API, and that the
libguestfs shutdown API is called and there is no error.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top