Libguestfs - Mar 2015 - Re: Point-in-time snapshots (was: Re: Inspection of disk snapshots)

On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy
wrote:
> [Added Stefan Hajnoczi, who wrote the 'drive-backup' QMP command
and
> Eric Blake.]
> 
> On Wed, Mar 25, 2015 at 07:41:23PM +0000, Richard W.M. Jones wrote:
> > On Wed, Mar 25, 2015 at 07:38:03PM +0100, Kashyap Chamarthy wrote:
> > > On Mon, Mar 23, 2015 at 10:43:30PM +0000, Richard W.M. Jones
wrote:
> 
> [. . .]
> 
> > > > > If you want lightweight, consistent, point-in-time
snapshots (which it
> > > > > sounds like you do), qemu has recently been adding this
capability.
> 
> Yeah, I have the two-part incremental backup series from John Snow's
> (who is the author of it) git repo. But, he said one more revision is
> about to hit the list, so I held off my testing until I can cleanly test
> that series on top of current QEMU git master.
> 
> > > > > See the 'drive-backup' monitor command. 
I've not tried using that
> > > 
> > > A small QMP test for 'drive-backup': 
> > > 
> > >    
https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
> > 
> > That's interesting, but even more interesting would be to see
> > drive-backup writing to NBD.
> >
> > From what I gather, drive-backup would take a nbd:... target, meaning
> > that it is the *client* (not the server). 
> 
> You're right -- the original cover letter[*] of the
'drive-backup' QEMU
> patch series that was merged says it:
> 
>     "The simplest use-case is to copy a point-in-time snapshot to a
>     local file.
> 
>     More advanced users may wish to make the target an NBD URL. The NBD
>     server listening on the other side can process the backup writes any
>     way it wishes."
> 
> > What's interesting is what you're supposed to connect to the
server.
> > (Maybe I'm wrong about this however)
> 
> Yeah, I too wonder what to connect on the server side when using NBD as
> target with 'drive-backup' (taking the simple example test I
pointed
> above).
> 
> So, something like?
> 
>   . . .
>   { 'execute': 'drive-backup', 'arguments':
>         { 'device': 'drive-virtio-disk1', 'sync':
'full', 'target':
>         'nbd://localhost', 'mode':
'absolute-paths', 'format': 'qcow2' }
>   . . .
> 
> Same question as yours, what is the NBD server going to run?
> 
> 
> My only NBD testing so far has been with w/ NBD over Unix socket or
> over TCP[**].
For 'sync': 'full' mode qemu-nbd or nbd-server can be used as
the
server.  You probably don't want 'format': 'qcow2', just raw
data over
NBD.  That way the NBD server can implement whatever storage backend it
likes (raw, qcow2, something else).

For incremental backup the NBD server would either be a qemu-nbd
instance with a qcow2 image and backing file:

  full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2

Or the NBD server would be a custom backup application that does
something smart with the incoming NBD WRITE requests.

Stefan
> 
> 
> [*] http://lists.nongnu.org/archive/html/qemu-devel/2013-06/msg04229.html
> 
> [**] NBD over TCP:
> 
> (1) Run an NBD server locally:
> 
>     $ qemu-nbd -f qcow2 -p10809 \
>         /var/lib/libvirt/images/cirros-0.3.3-x86_64-disk.img -t
> 
> (2) Create an overlay using the NBD server as backing file:
> 
>     $ qemu-img create -f qcow2 -F raw \
>         -o backing_file=nbd://localhost nbd-overlay1.qcow2
> 
> (3) Optionally, boot the overlay w/ a minimal CLI.
> 
>     $ qemu-system-x86_64               \
>        -nographic                      \
>        -nodefconfig                    \
>        -nodefaults                     \
>        -m 2048                         \
>        -device virtio-scsi-pci,id=scsi \
>        -device virtio-serial-pci       \
>        -serial stdio                   \
>        -drive
file=./nbd-overlay1.qcow2,format=qcow2,if=virtio,cache=writeback
> 
> 
> -- 
> /kashyap

On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi
wrote:
> On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote:
> > So, something like?
> > 
> >   . . .
> >   { 'execute': 'drive-backup', 'arguments':
> >         { 'device': 'drive-virtio-disk1',
'sync': 'full', 'target':
> >         'nbd://localhost', 'mode':
'absolute-paths', 'format': 'qcow2' }
> >   . . .
> > 
> > Same question as yours, what is the NBD server going to run?
> > 
> > 
> > My only NBD testing so far has been with w/ NBD over Unix socket or
> > over TCP[**].
> 
> For 'sync': 'full' mode qemu-nbd or nbd-server can be used
as the
> server.  You probably don't want 'format': 'qcow2',
just raw data over
> NBD.  That way the NBD server can implement whatever storage backend it
> likes (raw, qcow2, something else).
> 
> For incremental backup the NBD server would either be a qemu-nbd
> instance with a qcow2 image and backing file:
> 
>   full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2
> 
> Or the NBD server would be a custom backup application that does
> something smart with the incoming NBD WRITE requests.
What I care about is connecting libguestfs to qemu and reading a
snapshot at some point in time, even though the guest is still writing
away to its disks.  Is this possible with drive-backup (or otherwise)?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On Fri, Mar 27, 2015 at 12:31:41PM +0000, Richard W.M. Jones
wrote:
> On Fri, Mar 27, 2015 at 12:15:34PM +0000, Stefan Hajnoczi wrote:
> > On Thu, Mar 26, 2015 at 12:02:27AM +0100, Kashyap Chamarthy wrote:
> > > So, something like?
> > > 
> > >   . . .
> > >   { 'execute': 'drive-backup',
'arguments':
> > >         { 'device': 'drive-virtio-disk1',
'sync': 'full', 'target':
> > >         'nbd://localhost', 'mode':
'absolute-paths', 'format': 'qcow2' }
> > >   . . .
> > > 
> > > Same question as yours, what is the NBD server going to run?
> > > 
> > > 
> > > My only NBD testing so far has been with w/ NBD over Unix socket
or
> > > over TCP[**].
> > 
> > For 'sync': 'full' mode qemu-nbd or nbd-server can be
used as the
> > server.  You probably don't want 'format':
'qcow2', just raw data over
> > NBD.  That way the NBD server can implement whatever storage backend
it
> > likes (raw, qcow2, something else).
> > 
> > For incremental backup the NBD server would either be a qemu-nbd
> > instance with a qcow2 image and backing file:
> > 
> >   full-backup.img <- incremental-1.qcow2 <- incremental-2.qcow2
> > 
> > Or the NBD server would be a custom backup application that does
> > something smart with the incoming NBD WRITE requests.
> 
> What I care about is connecting libguestfs to qemu and reading a
> snapshot at some point in time, even though the guest is still writing
> away to its disks.  Is this possible with drive-backup (or otherwise)?
Yes, that is what drive-backup does.

New writes coming from the guest are held up until the old data has been
written to the NBD target.

That way you get a point-in-time snapshot while the guest continues
running.

Stefan