Libguestfs - Mar 2015 - Point-in-time snapshots (was: Re: Inspection of disk snapshots)

On Mon, Mar 23, 2015 at 10:41:01PM +0000, Richard W.M. Jones
wrote:
> On Mon, Mar 23, 2015 at 04:34:21PM +0200, NoxDaFox wrote:
> > Greetings,
> > 
> > I have the following typical scenario: given one or more qcow2 base
images
> > I clone them with COW and start the VMs.
> > 
> > At a certain point I'd like to inspect them in order to see their
evolution
> > compared to the known base images. To do so I was thinking about
taking a
> > disk snapshot of each VM and inspect its content through libguestfs
(using
> > it's Python bindings).
> >
> > Obviously I need the base image in order for libguestfs to correctly
guess
> > the OS, the FS structure etc.. Problem is that that point when I
inspect
> > the disk I get the whole disk state including the base image content
(30K+
> > files and directories).
> > 
> > This is not an issue but it's a very heavy operation considering
that some
> > of the snapshots are few megabytes while the base images are several
> > gigabytes.
> > 
> > Is there a way to programmatically instruct libguestfs to limit the
> > inspection to the sole snapshot?
> > Would it work as well with other disk format (vmdk linked clones for
> > example)?
> > 
> > For better comprehension I'll show the sequence I'm doing (I
might do it
> > wrong of course):
> > 
> > virsh -c "qemu:///system" snapshot-create --disk-only
<domain-ID>
> > 
> > I get the snapshot location from its XML description and then:
> > 
> > qemu-img convert -f qcow2 -O qcow2 base_image.qcow2 snapshot.qcow2
> 
> This makes a copy of the whole disk image.  It's also not a consistent
> (point in time) copy.
Oh I see that you're copying the _snapshot_ that you created with
libvirt; it's not a whole disk copy.  There's still not any point in
doing this, and what I said below stands.
> > At that point I mount it through libguestfs and inspect its content.
> 
> As long as you use the 'readonly=1' flag (which is really
*essential*,
> else you'll get disk corruption), you can just point libguestfs at the
> base image:
> 
>   g = guestfs.GuestFS (python_return_dict=True)
>   g.add_drive_opts ("base_image.qcow2", format="qcow2",
readonly=1)
> 
> That also doesn't get you a consistent snapshot, but it'll work
most
> of the time, and give you a clear error in libguestfs when it doesn't
> (and won't corrupt your base disk or anything like that, provided
> you're using readonly=1).
> 
> The effect of the readonly=1 flag is to create an external snapshot.
> It is roughly the equivalent of doing:
> 
>   qemu-img create -f qcow2 -b base_image snapshot.qcow2
>   < point libguestfs at snapshot.qcow2 >
> 
> If you want lightweight, consistent, point-in-time snapshots (which it
> sounds like you do), qemu has recently been adding this capability.
> See the 'drive-backup' monitor command.  I've not tried using
that and
> I don't know if it is wired up through libvirt, but libguestfs should
> be able to consume it since it's just an NBD source.
Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones wrote:

[. . .]
> > This makes a copy of the whole disk image.  It's also not a
consistent
> > (point in time) copy.
> 
> Oh I see that you're copying the _snapshot_ that you created with
> libvirt; it's not a whole disk copy.  There's still not any point
in
> doing this, and what I said below stands.
> 
> > > At that point I mount it through libguestfs and inspect its
content.
> > 
> > As long as you use the 'readonly=1' flag (which is really
*essential*,
> > else you'll get disk corruption), you can just point libguestfs at
the
> > base image:
> > 
> >   g = guestfs.GuestFS (python_return_dict=True)
> >   g.add_drive_opts ("base_image.qcow2",
format="qcow2", readonly=1)
> > 
> > That also doesn't get you a consistent snapshot, but it'll
work most
> > of the time, and give you a clear error in libguestfs when it
doesn't
> > (and won't corrupt your base disk or anything like that, provided
> > you're using readonly=1).
> > 
> > The effect of the readonly=1 flag is to create an external snapshot.
> > It is roughly the equivalent of doing:
> > 
> >   qemu-img create -f qcow2 -b base_image snapshot.qcow2
> >   < point libguestfs at snapshot.qcow2 >
> > 
> > If you want lightweight, consistent, point-in-time snapshots (which it
> > sounds like you do), qemu has recently been adding this capability.
> > See the 'drive-backup' monitor command.  I've not tried
using that
A small QMP test for 'drive-backup': 

    https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
> > and I don't know if it is wired up through libvirt,
I don't see it wired up yet in libvirt current `git` master.
> > but libguestfs should be able to consume it since it's just an NBD
> > source.
-- 
/kashyap

On Wed, Mar 25, 2015 at 07:38:03PM +0100, Kashyap Chamarthy
wrote:
> On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones wrote:
> 
> [. . .]
> 
> > > This makes a copy of the whole disk image.  It's also not a
consistent
> > > (point in time) copy.
> > 
> > Oh I see that you're copying the _snapshot_ that you created with
> > libvirt; it's not a whole disk copy.  There's still not any
point in
> > doing this, and what I said below stands.
> > 
> > > > At that point I mount it through libguestfs and inspect its
content.
> > > 
> > > As long as you use the 'readonly=1' flag (which is really
*essential*,
> > > else you'll get disk corruption), you can just point
libguestfs at the
> > > base image:
> > > 
> > >   g = guestfs.GuestFS (python_return_dict=True)
> > >   g.add_drive_opts ("base_image.qcow2",
format="qcow2", readonly=1)
> > > 
> > > That also doesn't get you a consistent snapshot, but
it'll work most
> > > of the time, and give you a clear error in libguestfs when it
doesn't
> > > (and won't corrupt your base disk or anything like that,
provided
> > > you're using readonly=1).
> > > 
> > > The effect of the readonly=1 flag is to create an external
snapshot.
> > > It is roughly the equivalent of doing:
> > > 
> > >   qemu-img create -f qcow2 -b base_image snapshot.qcow2
> > >   < point libguestfs at snapshot.qcow2 >
> > > 
> > > If you want lightweight, consistent, point-in-time snapshots
(which it
> > > sounds like you do), qemu has recently been adding this
capability.
> > > See the 'drive-backup' monitor command.  I've not
tried using that
> 
> A small QMP test for 'drive-backup': 
> 
>     https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
That's interesting, but even more interesting would be to see
drive-backup writing to NBD.
>From what I gather, drive-backup would take a nbd:... target, meaning
that it is the *client* (not the server).  What's interesting is what
you're supposed to connect to the server.  (Maybe I'm wrong about this
however)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW