libvirt users - Dec 2014 - Re: Using virsh blockcopy -- what's it supposed to accomplish?

I am experimenting with the blockcopy command, and after figuring out 
how to integrate qemu-nbd, nbd-client and 
dumpxml/undefine/blockcopy/define/et. al. I have one remaining question:

What's the point?

The "replication" disk file is not, from what I can ascertain,
bootable.
I expect this operation to create a pristine copy of my source qcow2 
file (at a given point in time) which implies that I can swap that copy 
in and use it just like the original.

Neither using --finish nor --pivot (both appear successful) give me a 
mirror that seems to serve any purpose. It seems especially pointless if 
I use --pivot because anything that happens after the pivot ends up lost 
if I don't actually have a usable qcow2 file.

I find lots of discussion online about getting the steps to work, but as 
yet find nothing about using the resulting file.

What am I missing here?

libvirt (1.2.2) and qemu (2.2.0) as distributed with Ubuntu Trusty.

-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc

On 12/22/2014 03:27 PM, Gary R Hook wrote:
> I am experimenting with the blockcopy command, and after figuring out
> how to integrate qemu-nbd, nbd-client and
> dumpxml/undefine/blockcopy/define/et. al. I have one remaining question:
> 
> What's the point?
Among other uses, live storage migration.

Let's say you are running on a cluster, where your VM is running locally
but was booted from network-accessed storage.  You don't want any guest
downtime, but you want to have the faster performance made possible by
accessing local storage instead of the network-accessed storage.  virsh
blockcopy can be used to change qemu's notion of where the active layer
of the disk lives without any guest time, by copying then pivoting to a
local file.
> 
> The "replication" disk file is not, from what I can ascertain,
bootable.
Correct in the current implementation, if you don't manually freeze
guest I/O prior to the point where you abort the copy (whether you do a
straight abort, leaving the copy as the point in time, or whether you do
a pivot, leaving the original as the point in time).  But I would like
to add a --quiesce option to blockcopy, similar to what is already
available for snapshot-create --quiesce.  The idea is that just before
breaking sync, you tell the guest to freeze all I/O, so that when you do
break sync, the disk you are no longer using _is_ a consistent image
(and depending on how well your guest is able to freeze I/O, it may well
be bootable).  But until that is implemented, you can use 'virsh
domfsfreeze' as the manual access to freezing guest I/O, if you have new
enough qemu and also have qemu-guest-agent wired up in your guest.
> I expect this operation to create a pristine copy of my source qcow2
> file (at a given point in time) which implies that I can swap that copy
> in and use it just like the original.
> 
> Neither using --finish nor --pivot (both appear successful) give me a
> mirror that seems to serve any purpose. It seems especially pointless if
> I use --pivot because anything that happens after the pivot ends up lost
> if I don't actually have a usable qcow2 file.
How is it not usable?  When you break sync at the conclusion of a
blockcopy, the image that you no longer use is an accurate snapshot of
the state of the disk at the time you broke sync; but whether or not
that is useful to a guest depends on how much influence unflushed I/O
that was still in guest memory at the time you broke sync will have on
your data.
> 
> I find lots of discussion online about getting the steps to work, but as
> yet find nothing about using the resulting file.
> 
> What am I missing here?
> 
> libvirt (1.2.2) and qemu (2.2.0) as distributed with Ubuntu Trusty.
> 
-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

On Mon, Dec 22, 2014 at 03:50:58PM -0700, Eric Blake
wrote:
> On 12/22/2014 03:27 PM, Gary R Hook wrote:
> > I am experimenting with the blockcopy command, and after figuring out
> > how to integrate qemu-nbd, nbd-client and
> > dumpxml/undefine/blockcopy/define/et. al. I have one remaining
question:
> > 
> > What's the point?
> 
> Among other uses, live storage migration.
> 
> Let's say you are running on a cluster, where your VM is running
locally
> but was booted from network-accessed storage.  You don't want any guest
> downtime, but you want to have the faster performance made possible by
> accessing local storage instead of the network-accessed storage.  virsh
> blockcopy can be used to change qemu's notion of where the active layer
> of the disk lives without any guest time, by copying then pivoting to a
> local file.
To add to Eric's explanation, I recently wrote a small example about it
here (this was tested with libvirt 1.2.6 & QEMU 2.1):

    http://kashyapc.com/2014/07/06/live-disk-migration-with-libvirt-blockcopy/
> > The "replication" disk file is not, from what I can
ascertain, bootable.
> 
> Correct in the current implementation, if you don't manually freeze
> guest I/O prior to the point where you abort the copy (whether you do a
> straight abort, leaving the copy as the point in time, or whether you do
> a pivot, leaving the original as the point in time).  But I would like
> to add a --quiesce option to blockcopy, similar to what is already
> available for snapshot-create --quiesce.
I remember a RHEL7 bug you filed for that, Eric,

    https://bugzilla.redhat.com/show_bug.cgi?id=1151629 -- blockcopy
    --keep-overlay ought to have --quiesce option 

Something similar needs to be cloned upstream?

-- 
/kashyap

On 12/23/14 2:08 PM, Eric Blake wrote:
> On 12/23/2014 11:35 AM, Gary R Hook wrote:
>> On 12/22/14 4:50 PM, Eric Blake wrote:
>>> On 12/22/2014 03:27 PM, Gary R Hook wrote:
>>>> I am experimenting with the blockcopy command, and after
figuring out
>>>> how to integrate qemu-nbd, nbd-client and
>>>> dumpxml/undefine/blockcopy/define/et. al. I have one remaining
question:
>>>>
>>>> What's the point?
>>>
>>> Among other uses, live storage migration.
>>
>> There is so very much to say here, but I will endeavor to be brief and
>> to the point.  Please note that I am working with libvirt 1.2.2, as
>> stated in my OP. And up front; I'm pretty sure I'm missing data
points
>> and working from a position of ignorance and unreasonable expectation.
>
> Did you mean for this to go to the list?  Usually, having discussions
> like this in public archives will benefit more people, and allow others
> to chime in.
>
> https://rwmj.wordpress.com/2010/11/08/want-help-dont-email-me-directly/

Dang, dang, dang, dang, dang.  Yes, of course I did. *sigh*.  Will 
resend to the list. Please ignore.

-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc

On 12/22/14 4:50 PM, Eric Blake wrote:
> On 12/22/2014 03:27 PM, Gary R Hook wrote:
>> I am experimenting with the blockcopy command, and after figuring out
>> how to integrate qemu-nbd, nbd-client and
>> dumpxml/undefine/blockcopy/define/et. al. I have one remaining
question:
>>
>> What's the point?
>
> Among other uses, live storage migration.
There is so very much to say here, but I will endeavor to be brief and 
to the point:

And then what?

Please note that I am working with libvirt 1.2.2, as stated in my OP. 
And up front: I'm pretty sure I'm missing data points and working from a
position of ignorance and unreasonable expectation. Your bearing with me 
is much, much appreciated.
> Let's say you are running on a cluster, where your VM is running
locally
> but was booted from network-accessed storage.  You don't want any guest
> downtime, but you want to have the faster performance made possible by
> accessing local storage instead of the network-accessed storage.  virsh
> blockcopy can be used to change qemu's notion of where the active layer
> of the disk lives without any guest time, by copying then pivoting to a
> local file.
I think I totally understand that. But I don't care about the old file, 
I care about the new one. And also, once you've moved to a block device 
(post-pivot) there's no going back, is there? The old qcow2 file can be 
an old snapshot, but that's about it. And the new file is not, in and of 
itself, usable as it stands. I tested a pivot and redefine of my domain 
using the new block device (/dev/nbd2, e.g.) and I was able to shut down 
and then start the domain successfully.

Which is no help whatsoever without forever including the NBD device. 
Or: what am I missing here?

I would expect that the domain, no matter where its disk files are 
located, may at some point need to be shutdown, then restarted. If I 
don't have an actual "mirror" (actually, a replication is what I
want)
then I'm still missing the point of this feature. Although I guess I 
could block copy and then migrate with non-shared storage to get 
something as usable and flexible as the original. Except that requires 
another host.

Let me summarize: I want to mirror a disk and at any point in time 
switch over and use the copy qcow2 file directly, in every way possible, 
as my new backing file. Including defining, shutting down and starting 
up domains arbitrarily. The old file is old news; forget about it. I 
want to move on. How does one accomplish that?
>> The "replication" disk file is not, from what I can
ascertain, bootable.
>
> Correct in the current implementation, if you don't manually freeze
> guest I/O prior to the point where you abort the copy (whether you do a
> straight abort, leaving the copy as the point in time, or whether you do
> a pivot, leaving the original as the point in time).  But I would like
> to add a --quiesce option to blockcopy, similar to what is already
> available for snapshot-create --quiesce.  The idea is that just before
> breaking sync, you tell the guest to freeze all I/O, so that when you do
> break sync, the disk you are no longer using _is_ a consistent image
> (and depending on how well your guest is able to freeze I/O, it may well
> be bootable).  But until that is implemented, you can use 'virsh
> domfsfreeze' as the manual access to freezing guest I/O, if you have
new
> enough qemu and also have qemu-guest-agent wired up in your guest.
Wow. Clearly something past 1.2.2. So my first step is to update to a 
current level? Dang it.

>> I expect this operation to create a pristine copy of my source qcow2
>> file (at a given point in time) which implies that I can swap that copy
>> in and use it just like the original.
>>
>> Neither using --finish nor --pivot (both appear successful) give me a
>> mirror that seems to serve any purpose. It seems especially pointless
if
>> I use --pivot because anything that happens after the pivot ends up
lost
>> if I don't actually have a usable qcow2 file.
>
> How is it not usable?  When you break sync at the conclusion of a
> blockcopy, the image that you no longer use is an accurate snapshot of
> the state of the disk at the time you broke sync; but whether or not
> that is useful to a guest depends on how much influence unflushed I/O
> that was still in guest memory at the time you broke sync will have on
> your data.
As stated above, I don't want the old file, I want the _new_ one. If I 
am running a block copy to a remote system (because, you know, that's 
one of the cool things about NBD) and my local system dies or otherwise 
becomes unrecoverable, I'd like to go to the remote system and bring up 
the guest there with minimal downtime in the event of a catastrophic 
failure. Right now that appears to be impossible. And I'm the only 
person to think this is reasonable.

For _me_, if I shut down my local guest, which causes the nbd-client 
process to exit, I expect the far end's server to flush blocks and 
create a disk file with integrity. What could it possibly be waiting for 
to prevent that from happening almost immediately? (He asks, without 
looking at the code...)
>> libvirt (1.2.2) and qemu (2.2.0) as distributed with Ubuntu Trusty.

-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc

On 12/24/14 4:42 AM, Kashyap Chamarthy wrote:
> On Tue, Dec 23, 2014 at 12:38:57PM -0600, Gary R Hook wrote:
>
> [. . .]
>
> In my case, the block device is a QCOW2 disk image file. If I boot
> without using the disk image file which has the operating system, the
> domain will fail to boot, no?
>
> I see you're playing with NBD disks. I'll admit, I haven't
played much
> with QEMU NBD, will have to experiment post holidays.
Back from the holidays, and back on this issue. I've learned a lot.

I've learned how to use the blockcopy command to create a local copy in 
a simple disk file:

virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait 
--verbose  --finish
virsh define my_domain.xml

and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It 
appears to be a perfect copy, as I expect it to be.

But while I see (per Kashyap's article, etc) that it can be useful in 
certain scenarios, it's not interesting to me. I would like to my copy 
to be off-system, and was hoping to use the NBD interface to accomplish 
that. So I tried this (a variant of the above), working on the same 
system because it's easier:

qemu-img create -f qcow2 /tmp/dsk.test.qcow2
qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
nbd-client localhost 11112 /dev/nbd2
virsh dumpxml my_domain > my_domain.xml
virsh undefine my_domain
virsh blockcopy --domain my_domain --wait --verbose --finish
virsh define my_domain.xml
nbd-client -d /dev/nbd2

and the qemu-nbd process exits, as I wish. I presume at this point that 
the new file has integrity.

I can take the qcow2 file that belongs to the domain and serve it up via 
NBD:

qemu-nbd --partition=1 -p11112 /path/to/my/qcow2/file.qcow2
nbd-client localhost 11112 /dev/nbd2
mount /dev/nbd2 -oloop /mnt/foo

and lo! in /mnt/foo I found my root filesytem. Seems perfectly reasonable.

If, however, I try to use my generated-via-NBD file, I get this:

# qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
[1] 7672
# qemu-nbd: Could not find partition 1: Invalid argument

[1]+  Exit 1                  qemu-nbd --partition=1 -p11112 
$PWD/dsk.test.qcow2
# qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 &
[1] 7686
# qemu-nbd: Invalid partition 0
^C
[1]+  Exit 1                  qemu-nbd --partition=0 -p11112 
$PWD/dsk.test.qcow2
# qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 &
[1] 7699
# qemu-nbd: Could not find partition 2: Invalid argument
^C
[1]+  Exit 1                  qemu-nbd --partition=2 -p11112 
$PWD/dsk.test.qcow2
# qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 &
[1] 7830
# qemu-nbd: Could not find partition 3: Invalid argument

[1]+  Exit 1                  qemu-nbd --partition=3 -p11112 
$PWD/dsk.test.qcow2

I don't know what has been created, but it's not a copy of the original 
guest's disk. There's no partition there, it seems.

So yes, blockcopy works fine under certain conditions. But the NBD layer 
seems to really muck things up.

Or, more likely, I'm doing things wrong. I'm hoping someone can point 
out something obvious.

There's a recent thread about "Block Replication for Continuous 
Checkpointing" that is heading towards using NBD. I fail to understand 
how this is ever going to work, based on my explorations.


-- 
Gary R Hook
Senior Kernel Engineer
NIMBOXX, Inc

On Thu, Jan 08, 2015 at 01:44:58PM -0600, Gary R Hook
wrote:
> On 12/24/14 4:42 AM, Kashyap Chamarthy wrote:
> >On Tue, Dec 23, 2014 at 12:38:57PM -0600, Gary R Hook wrote:
> >
> >[. . .]
> >
> >In my case, the block device is a QCOW2 disk image file. If I boot
> >without using the disk image file which has the operating system, the
> >domain will fail to boot, no?
> >
> >I see you're playing with NBD disks. I'll admit, I haven't
played much
> >with QEMU NBD, will have to experiment post holidays.
> 
> Back from the holidays, and back on this issue. I've learned a lot.
> 
> I've learned how to use the blockcopy command to create a local copy in
a
> simple disk file:
> 
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait --verbose
> --finish
> virsh define my_domain.xml
Yes, that's correct.
> and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It appears
to
> be a perfect copy, as I expect it to be.
Good, this time you find it bootable, compared to your previous test.
> But while I see (per Kashyap's article, etc) that it can be useful in
> certain scenarios, it's not interesting to me. I would like to my copy
to be
> off-system, and was hoping to use the NBD interface to accomplish that. So
I
> tried this (a variant of the above), working on the same system because
it's
> easier:
> 
> qemu-img create -f qcow2 /tmp/dsk.test.qcow2
A typo? You also need to provide a size here:

    $ qemu-img create -f qcow2 /tmp/dsk.test.qcow2 1G


For the rest, I'm afraid I still didn't manage time to test
the NBD scenario to give a meaningful response here. I'll let the others
who deal with NBD more often respond to it.
> qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
> nbd-client localhost 11112 /dev/nbd2
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain --wait --verbose --finish
> virsh define my_domain.xml
> nbd-client -d /dev/nbd2
> 
> and the qemu-nbd process exits, as I wish. I presume at this point that the
> new file has integrity.
> 
> I can take the qcow2 file that belongs to the domain and serve it up via
> NBD:
> 
> qemu-nbd --partition=1 -p11112 /path/to/my/qcow2/file.qcow2
> nbd-client localhost 11112 /dev/nbd2
> mount /dev/nbd2 -oloop /mnt/foo
> 
> and lo! in /mnt/foo I found my root filesytem. Seems perfectly reasonable.
> 
> If, however, I try to use my generated-via-NBD file, I get this:
> 
> # qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7672
> # qemu-nbd: Could not find partition 1: Invalid argument
> 
> [1]+  Exit 1                  qemu-nbd --partition=1 -p11112
> $PWD/dsk.test.qcow2
> # qemu-nbd --partition=0 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7686
> # qemu-nbd: Invalid partition 0
> ^C
> [1]+  Exit 1                  qemu-nbd --partition=0 -p11112
> $PWD/dsk.test.qcow2
> # qemu-nbd --partition=2 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7699
> # qemu-nbd: Could not find partition 2: Invalid argument
> ^C
> [1]+  Exit 1                  qemu-nbd --partition=2 -p11112
> $PWD/dsk.test.qcow2
> # qemu-nbd --partition=3 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7830
> # qemu-nbd: Could not find partition 3: Invalid argument
> 
> [1]+  Exit 1                  qemu-nbd --partition=3 -p11112
> $PWD/dsk.test.qcow2
> 
> I don't know what has been created, but it's not a copy of the
original
> guest's disk. There's no partition there, it seems.
> 
> So yes, blockcopy works fine under certain conditions. But the NBD layer
> seems to really muck things up.
> 
> Or, more likely, I'm doing things wrong. I'm hoping someone can
point out
> something obvious.
> 
> There's a recent thread about "Block Replication for Continuous
> Checkpointing" that is heading towards using NBD. I fail to understand
how
> this is ever going to work, based on my explorations.
-- 
/kashyap

On 01/08/2015 12:44 PM, Gary R Hook wrote:
> I've learned how to use the blockcopy command to create a local copy in
> a simple disk file:
> 
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain vda $PWD/dsk.copy.qcow2 --wait
> --verbose  --finish
> virsh define my_domain.xml
> 
> and the resulting copy in dsk.copy.qcow2 is, indeed, bootable. It
> appears to be a perfect copy, as I expect it to be.
> 
> But while I see (per Kashyap's article, etc) that it can be useful in
> certain scenarios, it's not interesting to me. I would like to my copy
> to be off-system,
Once you have the local dsk.copy.qcow2, you can then move it off-system
and serve it up however you'd like, including by NBD.
> and was hoping to use the NBD interface to accomplish
> that. So I tried this (a variant of the above), working on the same
> system because it's easier:
> 
> qemu-img create -f qcow2 /tmp/dsk.test.qcow2
> qemu-nbd -f qcow2 -p11112 /tmp/dsk.test.qcow2
> nbd-client localhost 11112 /dev/nbd2
> virsh dumpxml my_domain > my_domain.xml
> virsh undefine my_domain
> virsh blockcopy --domain my_domain --wait --verbose --finish
Missing a destination?  I'm guessing you meant /dev/nbd2 as the
pre-existing destination?
> virsh define my_domain.xml
> nbd-client -d /dev/nbd2
> 
> and the qemu-nbd process exits, as I wish. I presume at this point that
> the new file has integrity.
Where are you specifying the format?  I have not personally played with
NBD much.  But here's my guess: Even though /tmp/dsk.test.qcow2 is a
qcow2 file, the NBD server is serving up a RAW image through /dev/nbd2.
 Thus, if you are trying to treat /dev/nbd2 as the destination of your
copy, you MUST tell qemu that the file format of the copy is to be raw
(regardless of the file format of the original that is being copied
from).  If you omit the --raw (also spelled --format=raw in newer
libvirt) parameter to the virsh blockcopy command, then libvirt has to
guess at the destination format; if the source was qcow2, then libvirt
will guess that the destination should be qcow2 as well.  But writing
qcow2 data to a raw NBD disk means you have created a nested file in
/tmp/dsk.test.qcow2 - it is a qcow2 file whose contents are a qcow2 file
whose contents are the raw data (not typical usage, and a bit weird to
wrap your head around).
> 
> If, however, I try to use my generated-via-NBD file, I get this:
> 
> # qemu-nbd --partition=1 -p11112 $PWD/dsk.test.qcow2 &
> [1] 7672
> # qemu-nbd: Could not find partition 1: Invalid argument
It goes back to your format nesting mistake above - you are serving up a
nested qcow2 image rather than raw data, while the guest is trying to
treat the served up image as data.  Since the nested qcow2 data in
sector 0 of the served-up data is not a partition table, it is no wonder
that you can't find a partition.
> 
> I don't know what has been created, but it's not a copy of the
original
> guest's disk. There's no partition there, it seems.
Or there is, but only if you peel off yet another layer of qcow2 :)

Try doing 'virsh blockcopy --raw /dev/nbd2 ...', to see if I was right
and that makes the difference.  Remember, blockcopy CAN change things so
that the destination (as seen by qemu) is a different format than the
source, and in the case of doing a blockcopy from a qcow2 source to a
raw-image NBD server, this is a case where you need to be explicit about
the difference in format.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org