Libguestfs - Mar 2015 - Re: Point-in-time snapshots

On Fri, Mar 27, 2015 at 12:31:41PM +0000, Richard W.M. Jones
wrote:
> On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote:
> > On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote:
> > > So, something like?
> > > 
> > >   . . .
> > >   { 'execute': 'drive-backup',
'arguments':
> > >         { 'device': 'drive-virtio-disk1',
'sync': 'full', 'target':
> > >         'nbd://localhost', 'mode':
'absolute-paths', 'format': 'qcow2' }
> > >   . . .
> > > 
> > > Same question as yours, what is the NBD server going to run?
> > > 
> > > 
> > > My only NBD testing so far has been with w/ NBD over Unix socket
or
> > > over TCP[**].
> > 
> > For 'sync': 'full' mode qemu-nbd or nbd-server can be
used as the
> > server.  You probably don't want 'format':
'qcow2', just raw data over
> > NBD.  That way the NBD server can implement whatever storage backend
it
> > likes (raw, qcow2, something else).
> > 
> > For incremental backup the NBD server would either be a qemu-nbd
> > instance with a qcow2 image and backing file:
> > 
> >   full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2
> > 
> > Or the NBD server would be a custom backup application that does
> > something smart with the incoming NBD WRITE requests.
> 
> What I care about is connecting libguestfs to qemu and reading a
> snapshot at some point in time, even though the guest is still writing
> away to its disks.  Is this possible with drive-backup (or otherwise)?
Yes, that is what drive-backup does.

New writes coming from the guest are held up until the old data has been
written to the NBD target.

That way you get a point-in-time snapshot while the guest continues
running.

Stefan

On Fri, Mar 27, 2015 at 03:21:25PM +0000, Stefan Hajnoczi
wrote:
> On Fri, Mar 27, 2015 at 12:31:41PM +0000, Richard W.M. Jones wrote:
> > On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote:
> > > On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy
wrote:
> > > > So, something like?
> > > > 
> > > >   . . .
> > > >   { 'execute': 'drive-backup',
'arguments':
> > > >         { 'device': 'drive-virtio-disk1',
'sync': 'full', 'target':
> > > >         'nbd://localhost', 'mode':
'absolute-paths', 'format': 'qcow2' }
> > > >   . . .
> > > > 
> > > > Same question as yours, what is the NBD server going to run?
> > > > 
> > > > 
> > > > My only NBD testing so far has been with w/ NBD over Unix
socket or
> > > > over TCP[**].
> > > 
> > > For 'sync': 'full' mode qemu-nbd or nbd-server
can be used as the
> > > server.  You probably don't want 'format':
'qcow2', just raw data over
> > > NBD.  That way the NBD server can implement whatever storage
backend it
> > > likes (raw, qcow2, something else).
> > > 
> > > For incremental backup the NBD server would either be a qemu-nbd
> > > instance with a qcow2 image and backing file:
> > > 
> > >   full-backup.img <- incremental-1.qcow2 <-
incremental-2.qcow2
> > > 
> > > Or the NBD server would be a custom backup application that does
> > > something smart with the incoming NBD WRITE requests.
> > 
> > What I care about is connecting libguestfs to qemu and reading a
> > snapshot at some point in time, even though the guest is still writing
> > away to its disks.  Is this possible with drive-backup (or otherwise)?
> 
> Yes, that is what drive-backup does.
> 
> New writes coming from the guest are held up until the old data has been
> written to the NBD target.
> 
> That way you get a point-in-time snapshot while the guest continues
> running.
I understand how that can work for backups, where you want to copy
a whole disk consistently.

But libguestfs doesn't want to do a backup, nor get a copy of the
whole disk, it just wants to access a scattering of blocks (maybe a
few hundred) but at a single point in time, in as lightweight a manner
as possible.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

On 03/27/2015 09:35 AM, Richard W.M. Jones wrote:
>>> What I care about is connecting libguestfs to qemu and reading a
>>> snapshot at some point in time, even though the guest is still
writing
>>> away to its disks.  Is this possible with drive-backup (or
otherwise)?
>>
>> Yes, that is what drive-backup does.
>>
>> New writes coming from the guest are held up until the old data has
been
>> written to the NBD target.
>>
>> That way you get a point-in-time snapshot while the guest continues
>> running.
> 
> I understand how that can work for backups, where you want to copy
> a whole disk consistently.
> 
> But libguestfs doesn't want to do a backup, nor get a copy of the
> whole disk, it just wants to access a scattering of blocks (maybe a
> few hundred) but at a single point in time, in as lightweight a manner
> as possible.
If you KNOW what sectors you want to read, then your NBD target can
ignore writes to the sectors you don't care about (the guest is changing
data on a sector you don't care about; yeah, bandwidth was spent in
telling you that, but you don't have to spend storage on it), while
focusing on reading the sectors you do care about as fast as possible
(to minimize the time spent dealing with uninteresting writes).  If you
DON'T know what sectors you want to read (because you are chasing file
system pointers and don't know a priori where those pointers will
resolve), then tracking ALL data flushed by guest writes IS the most
efficient manner for keeping your point in time accurate for the
duration of your fleecing operation.  Either way, if you really are
going to read only a few hundred sectors and then close the connection,
it shouldn't matter if the drive-backup failed to send all guest sectors
modified after the point in time, so long as every sector you read is
accurate (either because the guest hasn't touched it in the meantime; or
because even though the guest touched it after the point in time, you
were given the original contents through your NBD target).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

On Fri, Mar 27, 2015 at 03:35:02PM +0000, Richard W.M. Jones
wrote:
> On Fri, Mar 27, 2015 at 03:21:25PM +0000, Stefan Hajnoczi wrote:
> > On Fri, Mar 27, 2015 at 12:31:41PM +0000, Richard W.M. Jones wrote:
> > > On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote:
> > > > On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy
wrote:
> > > > > So, something like?
> > > > > 
> > > > >   . . .
> > > > >   { 'execute': 'drive-backup',
'arguments':
> > > > >         { 'device':
'drive-virtio-disk1', 'sync': 'full', 'target':
> > > > >         'nbd://localhost', 'mode':
'absolute-paths', 'format': 'qcow2' }
> > > > >   . . .
> > > > > 
> > > > > Same question as yours, what is the NBD server going to
run?
> > > > > 
> > > > > 
> > > > > My only NBD testing so far has been with w/ NBD over
Unix socket or
> > > > > over TCP[**].
> > > > 
> > > > For 'sync': 'full' mode qemu-nbd or
nbd-server can be used as the
> > > > server.  You probably don't want 'format':
'qcow2', just raw data over
> > > > NBD.  That way the NBD server can implement whatever storage
backend it
> > > > likes (raw, qcow2, something else).
> > > > 
> > > > For incremental backup the NBD server would either be a
qemu-nbd
> > > > instance with a qcow2 image and backing file:
> > > > 
> > > >   full-backup.img <- incremental-1.qcow2 <-
incremental-2.qcow2
> > > > 
> > > > Or the NBD server would be a custom backup application that
does
> > > > something smart with the incoming NBD WRITE requests.
> > > 
> > > What I care about is connecting libguestfs to qemu and reading a
> > > snapshot at some point in time, even though the guest is still
writing
> > > away to its disks.  Is this possible with drive-backup (or
otherwise)?
> > 
> > Yes, that is what drive-backup does.
> > 
> > New writes coming from the guest are held up until the old data has
been
> > written to the NBD target.
> > 
> > That way you get a point-in-time snapshot while the guest continues
> > running.
> 
> I understand how that can work for backups, where you want to copy
> a whole disk consistently.
> 
> But libguestfs doesn't want to do a backup, nor get a copy of the
> whole disk, it just wants to access a scattering of blocks (maybe a
> few hundred) but at a single point in time, in as lightweight a manner
> as possible.
There is a way to do that.

Create a temporary qcow2 backed by the disk:

guest.img <- tmp.qcow2

Now use drive-backup with sync mode 'none' so only new guest writes
cause old data to be stashed in tmp.qcow2 before it is overwritten.

Finally, use the run-time NBD server for random access to tmp.qcow2.

When you are done, kill the drive-backup job and delete tmp.qcow2.

Stefan