Libguestfs - Feb 2015 - Re: getting guestfs_rsync_out to work

Richard-
> On Mon, Feb 09, 2015 at 08:01:31AM -0600, Jeff Brower wrote:
>> Richard-
>>
>> > On Sun, Feb 08, 2015 at 12:11:37PM -0600, Jeff Brower wrote:
>> >> With continuous loop testing, what we found is that we have to
shut
>> >> down and re-launch the image handle to see changes on the Win7
live
>> >> guest.  Unfortunately image re-launch takes time, 3-5 sec (the
image
>> >> size is 50 GByte).  I'm assuming this is because
libguestfs makes an
>> >> internal copy of filesystem, and works from that, and
doesn't
>> >> "refresh" this internal copy until re-launching the
image handle.
>> >
>> > This isn't surprising.  Libguestfs doesn't make an
internal copy of
>> > the whole filesystem, but the libguestfs appliance will have a
copy
>> > (in its kernel memory) of any parts of the disk that rsync read.
>> >
>> > See also:
>> > http://libguestfs.org/guestfs.3.html#architecture
>> >
>> >> Could we create a second partition on the guest that is much
>> >> smaller, say 50 MByte, attach the image to that, and thus
reduce the
>> >> shutdown and re-launch time into the msec range?  Is there
another
>> >> real-time method, not using rsync?
>> >
>> > The time to relaunch the appliance doesn't depend on the size
of the
>> > disk.
>> >
>> > You could try hotplugging but it may not be much faster:
>> >
>> > http://libguestfs.org/guestfs.3.html#hotplugging
>> >
>> > I think what you really need to do is to install a backup agent in
the
>> > Windows guest and use a regular network backup.
>>
>> We don't need to use rsync.  We know which files will change.  For
example even if we do this (error checking
>> removed):
>>
>>    while (1) {
>>
>>       guestfs_mount_options(g, "sync", "/dev/sda2",
"/");
>>
>>       printf("%s\n", guestfs_cat(g,
"/HostShared/temp.txt"));
>>
>>       guestfs_umount(g, "/");
>>
>>       guestfs_fsync(g);
>>
>>       usleep(1000*1000);
>>    }
>>
>> temp.txt from the Win7 guest is copied correctly the first time, but
>> afterwards any change we make to it (e.g. using Win7 Notepad) is
>> never seen.  That's the issue.
>
> This is completely expected.  Have a look at the architecture diagram
> here:
>
> http://libguestfs.org/guestfs.3.html#architecture
>
> Now imagine there is another line drawn from the "Device or disk
> image" box to your running Windows guest.  How does the appliance know
> that there's something else making changes to the disk?
Yes we know that it doesn't know.  We'd be ok to re-launch the image to
pick up changes except for the time it takes
(if we could get into 1 to 10 msec range it would be ok).
> BTW you really *must* be using the readonly flag when adding disks
> (guestfs_add_drive_opts) else you will get disk corruption.
Yes we have been doing that.
>> Is there any way at all to accomplish this using libguestfs, without
>> re-launching the image, which is time-consuming?
>
> You could try calling `guestfs_drop_caches (g, 3);' (which may or may
> not work depending on a bunch of details in how Linux and qemu works),
> or using hotplugging.
Ok we'll try that.  Thank you for the suggestion.

-Jeff

On Mon, Feb 09, 2015 at 08:27:22AM -0600, Jeff Brower
wrote:
> Yes we know that it doesn't know.  We'd be ok to re-launch the
image
> to pick up changes except for the time it takes (if we could get
> into 1 to 10 msec range it would be ok).
There's no feasible way to start a Linux kernel in under about 1-2
seconds.  It just takes that long for the kernel to initialize itself.
Even if you use the User-mode Linux backend (which just runs vmlinux
as a userspace process) you'll experience approximately the same
overhead.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Richard-
> On Mon, Feb 09, 2015 at 08:27:22AM -0600, Jeff Brower wrote:
>> Yes we know that it doesn't know.  We'd be ok to re-launch the
image
>> to pick up changes except for the time it takes (if we could get
>> into 1 to 10 msec range it would be ok).
> 
> There's no feasible way to start a Linux kernel in under about 1-2
> seconds.  It just takes that long for the kernel to initialize itself.
> Even if you use the User-mode Linux backend (which just runs vmlinux
> as a userspace process) you'll experience approximately the same
> overhead.
Ok got it.  Thanks again for your reply.  We tried guestfs_drop_caches() as
follows:

  while(1) {

    guestfs_mount(g, "/dev/sda2", "/");

    printf("%s\n", guestfs_cat(g, "/HostShared/temp.txt"));

    guestfs_umount(g, "/");

    guestfs_drop_caches(g, 3);

    usleep(1000*1000);   
 }

and still we cannot see changes made to temp.txt (from inside the Win7 guest VM)
while this loop continues to run on the host Linux.  We still have to re-launch
the
image to see changes.

In general, it would be great if the "kernel load" and "disk
refresh / sync"
functions in guestfs_launch() could be separated.  If there was a way to do
read-only
file sharing, a very limited subset of what virtFs/9p can do, that would be
great for
Win guests.

-Jeff