libvirt users - Apr 2014 - Re: Help with understanding and solving snapshot problem

Hello

Thanks heaps for your suggestions.  responses inline.

On 10 Apr 2014, at 10:34 pm, Eric Blake <eblake@rehost1hat.com>
wrote:
> 
> [ … ]
> [Can you convince your mailer to wrap long lines?  It makes it easier
> for other readers]
I’ll try and keep the lines short. I don’t see any obvious setting to change the
wrap.
>> 
>> Using RHEL 6.4 and libvirt version is 0.10.2 and qemu-img version is
0.12.1.2
> 
> Have you considered raising this as a support request with Red Hat?
My error. I am using RHEL 6.5. Sorry about the wrong info.

I could put this is to Red Hat but thought the mailing list might be a better
initial place.
> From the upstream perspective, 0.10.2 is quite old, and Red Hat may be
> better equipped to answer questions about what snapshot support they
> have backported to that version of RHEL.  In particular, the fact that
> you are not using RHEL 6.5 is a bit worrisome, and I also understand
> that RHEL 6.x tends to not support internal snapshots.
[ … ]
>> # virsh snapshot-list host1
>> Name                 Creation Time             State
>> ------------------------------------------------------------
>> snap1-host1           2014-01-19 16:59:10 +1100 shutoff
>> 
>> # 
>> # virsh snapshot-info host1 --current
>> Name:           snap1-host1
>> Domain:         host1
>> Current:        yes
>> State:          shutoff
>> Location:       internal
> 
> So the fact that you created an internal snapshot may have already put
> you in unsupported territory for the versions of software that you are
> using.  That said, I can still try to help, and I hope that upstream
> behaves nicer in this regards, although you have certainly given us
> enough steps to try and reproduce if this is still a bug in upstream.
> Or maybe the bug is here, and you really did create an external snapshot
> but the code is reporting it incorrectly.  Can you post the actual
> command that you used to create the snapshot?
> 
Unfortunately no as I cannot remember it.  I have a feeling that it was a menu
item
in the virt-manager gui, but that said I can’t find it.  It is most likely to
have been the simplest
form of the virsh snapshot-create command.

>> Parent:         -
>> Children:       0
>> Descendants:    0
>> Metadata:       yes
>> 
>> So I try and delete it:
>> 
>> # virsh snapshot-delete host1 --current
>> 
>> error: Failed to delete snapshot snap1-host1
>> error: unsupported configuration: deletion of 1 external disk snapshots
not supported yet
>> 
>> Why does it say “external” in the above error, when the snapshot-info
says its Location is internal? Are they not related ideas?
> 
> It's very likely that you have tripped up on a bug, and perhaps on a
bug
> that has been fixed in the meantime in newer libvirt, which either needs
> backporting to the RHEL version, or which Red Hat has deemed not worth
> backporting because of their level of limited snapshot support in RHEL
> 6.  But again, going through a Red Hat support ticket will get faster
> results than asking upstream.
Thanks.  I just did a yum update to libvirt and got
libvirt-0.10.2-29.el6_5.7.x86_64.rpm
Which is not a big change.

In any event the error persists unchanged when I try to delete the snapshot.
>> 
>> The img file /var/lib/libvirt/images/host1.img  is not being used by
KVM.  the snapshot files are in use instead:
>> 
>> # virsh domblklist host1
>> Target     Source
>> ------------------------------------------------
>> vda        /var/lib/libvirt/snap1-host1.qcow2
>> vdb        /var/lib/libvirt/snap1-host1-db.qcow2
>> hdc        -
>> 
>> Querying them:
>> 
>> # qemu-img info /var/lib/libvirt/snap1-host1.qcow2 
>> image: /var/lib/libvirt/snap1-host1.qcow2
>> file format: qcow2
>> virtual size: 24G (25769803776 bytes)
>> disk size: 3.5G
>> cluster_size: 65536
> 
> No external backing file listed; but I'm not sure how this relates to
> the original snapshot.  Maybe you also want to show 'virsh
> snapshot-dumpxml host1 snap1-host1' to make it more obvious what the
> domain was using at the time you took the snapshot?
ok. That output shows:

# virsh snapshot-dumpxml host1 snap1-host1
<domainsnapshot>
  <name>snap1-host1</name>
  <description>After install completed</description>
  <state>shutoff</state>
  <creationTime>1390111150</creationTime>
  <memory snapshot='no'/>
  <disks>
    <disk name='vda' snapshot='external'>
      <driver type='qcow2'/>
      <source file='/var/lib/libvirt/snap1-host1.qcow2'/>
    </disk>
    <disk name='vdb' snapshot='external'>
      <driver type='qcow2'/>
      <source file='/var/lib/libvirt/snap1-host1-db.qcow2'/>
    </disk>
    <disk name='hdc' snapshot='no'/>
  </disks>
  <domain type='kvm'>
    <name>host1</name>
    <uuid>e1a43a89-af8f-95e2-e242-a42a44afc127</uuid>
    <memory unit='KiB'>16777216</memory>
    <currentMemory unit='KiB'>16777216</currentMemory>
    <vcpu placement='static'>6</vcpu>
    <os>
      <type arch='x86_64'
machine='rhel6.5.0'>hvm</type>
      <boot dev='hd'/>
      <bootmenu enable='no'/>
    </os>
    <features>
      <acpi/>
      <apic/>
      <pae/>
    </features>
    <clock offset='utc'/>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>restart</on_crash>
    <devices>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw'
cache='none'/>
        <source file='/var/lib/libvirt/images/host1.img'/>
        <target dev='vda' bus='virtio'/>
        <address type='pci' domain='0x0000'
bus='0x00' slot='0x05' function='0x0'/>
      </disk>
      <disk type='block' device='disk'>
        <driver name='qemu' type='raw' cache='none'
io='native'/>
        <source dev='/dev/sdd1'/>
        <target dev='vdb' bus='virtio'/>
        <address type='pci' domain='0x0000'
bus='0x00' slot='0x07' function='0x0'/>
      </disk>
      <disk type='file' device='cdrom'>
        <driver name='qemu' type='raw'/>
        <target dev='hdc' bus='ide'/>
        <readonly/>
        <address type='drive' controller='0' bus='1'
target='0' unit='0'/>
      </disk>
      <controller type='ide' index='0'>
        <address type='pci' domain='0x0000'
bus='0x00' slot='0x01' function='0x1'/>
      </controller>
      <controller type='usb' index='0'/>
      <interface type='bridge'>
        <mac address='52:54:00:97:0e:67'/>
        <source bridge='br3'/>
        <model type='virtio'/>
        <address type='pci' domain='0x0000'
bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <serial type='pty'>
        <target port='0'/>
      </serial>
      <console type='pty'>
        <target type='serial' port='0'/>
      </console>
      <input type='tablet' bus='usb'/>
      <input type='mouse' bus='ps2'/>
      <graphics type='vnc' port='-1'
autoport='yes'/>
      <sound model='ich6'>
        <address type='pci' domain='0x0000'
bus='0x00' slot='0x04' function='0x0'/>
      </sound>
      <video>
        <model type='cirrus' vram='9216'
heads='1'/>
        <address type='pci' domain='0x0000'
bus='0x00' slot='0x02' function='0x0'/>
      </video>
      <memballoon model='virtio'>
        <address type='pci' domain='0x0000'
bus='0x00' slot='0x06' function='0x0'/>
      </memballoon>
    </devices>
  </domain>
</domainsnapshot>

Not completely sure what to derive from that output?
>> [root@cocoa libvirt]# qemu-img info
/var/lib/libvirt/snap1-host1-db.qcow2
>> image: /var/lib/libvirt/snap1-host1-db.qcow2
>> file format: qcow2
>> virtual size: 1.6T (1800279089664 bytes)
>> disk size: 325G
>> cluster_size: 65536
>> 
> 
> I don't quite have a full picture of how you got into the situation. 
If
> you are trying to just get rid of the snapshot, you could always try
> 'virsh snapshot-delete --metadata host1 snap1-host1', to make
libvirt
> forget about the snapshot without cleaning up any actual data (leaving
> any external backing chains intact, and not removing internal snapshots
> from qcow2 files).
ok. What implications does this have long term?
Is there then a related step to remove the data of the snapshot that is no
longer referenced?
Reclaiming the space would be handy.

And as before, given the structure of the VM and its two disks, how is a
snapshot created
excluding the second disk? vdb in the above xml output?

Many thanks for your help so far.

regards

rolf.

> 
> -- 
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org

On 04/10/2014 04:38 PM, rolf wrote:
>> [Can you convince your mailer to wrap long lines?  It makes it easier
>> for other readers]
> 
> I’ll try and keep the lines short. I don’t see any obvious setting to
change the wrap.
Thanks for being accommodating; this was indeed easier to read (alas,
many web mailers these days lack settings for automatic wrap, so I end
up whacking the Enter key for manual wrap when I'm forced to use a web
mail interface for sending to a technical list)

>>> # virsh snapshot-info host1 --current
>>> Name:           snap1-host1
>>> Domain:         host1
>>> Current:        yes
>>> State:          shutoff
>>> Location:       internal
>>
> # virsh snapshot-dumpxml host1 snap1-host1
> <domainsnapshot>
>   <name>snap1-host1</name>
>   <description>After install completed</description>
>   <state>shutoff</state>
>   <creationTime>1390111150</creationTime>
>   <memory snapshot='no'/>
>   <disks>
>     <disk name='vda' snapshot='external'>
Okay, we've definitely demonstrated a bug in at least 'virsh
snapshot-list' for that particular build of libvirt; this proves that
the snapshot is definitely external, even though the info output claimed
it was internal.  I didn't search whether a bugzilla entry was already
tracking this for RHEL 6.5; it's not a crasher, so it probably won't get
fixed until RHEL 6.6.  If you'd like, you can open a BZ, (it might get
closed as a dup if someone else finds where it was already reported,
even though I didn't do that search), to make sure it doesn't get lost.

Meanwhile, creating external snapshots is supported in that version of
RHEL, but not deleting (at least not via virsh directly), so you'll have
to get your hands a bit dirty with qemu-img and virsh edit.
>       <driver type='qcow2'/>
>       <source file='/var/lib/libvirt/snap1-host1.qcow2'/>
This says that snap1-host1.qcow2 is the wrapper file created at the time
of the snapshot, and that...
>     </disk>
>     <disk name='vdb' snapshot='external'>
>       <driver type='qcow2'/>
>       <source file='/var/lib/libvirt/snap1-host1-db.qcow2'/>
>     </disk>
>     <disk name='hdc' snapshot='no'/>
>   </disks>
>   <domain type='kvm'>
...
>     <devices>
>       <emulator>/usr/libexec/qemu-kvm</emulator>
>       <disk type='file' device='disk'>
>         <driver name='qemu' type='raw'
cache='none'/>
>         <source file='/var/lib/libvirt/images/host1.img'/>
it should have a backing file of /var/lib/libvirt/images/host1.img.
Wonder why your 'qemu-img info' output didn't show that fact?
>>
>> I don't quite have a full picture of how you got into the
situation.  If
>> you are trying to just get rid of the snapshot, you could always try
>> 'virsh snapshot-delete --metadata host1 snap1-host1', to make
libvirt
>> forget about the snapshot without cleaning up any actual data (leaving
>> any external backing chains intact, and not removing internal snapshots
>> from qcow2 files).
> 
> ok. What implications does this have long term?
> Is there then a related step to remove the data of the snapshot that is no
longer referenced?
> Reclaiming the space would be handy.
Are you trying to revert to that state, or just forget that you ever
took the snapshot?  Are you okay keeping the two files as a backing
chain, or do you want to collapse it into one?  And if you DO want to
collapse into one file (so you can delete the other), do you want the
kept file to be snap1-host1.qcow2 (do a blockpull operation) or
host1.img (do a commit operation)?

> 
> And as before, given the structure of the VM and its two disks, how is a
snapshot created
> excluding the second disk? vdb in the above xml output?
If you create the snapshot via command line, 'virsh snapshot-create-as
host1 --name ... --diskspec vda,file=/path/for/disk --diskspec
vdb,snapshot=no' should be sufficient to exclude vdb from the snapshot
(you can use the --print-xml option to see what those options would
actually pass to the virDomainSnapshotCreateXML command).
> 
> Many thanks for your help so far.
Glad to hear it, and hope we can continue to be helpful.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Hello

On 11 Apr 2014, at 8:52 am, Eric Blake <eblake@redhat.com> wrote:

[ … ]
> 
>>>> # virsh snapshot-info host1 --current
>>>> Name:           snap1-host1
>>>> Domain:         host1
>>>> Current:        yes
>>>> State:          shutoff
>>>> Location:       internal
>>> 
> 
>> # virsh snapshot-dumpxml host1 snap1-host1
>> <domainsnapshot>
>>  <name>snap1-host1</name>
>>  <description>After install completed</description>
>>  <state>shutoff</state>
>>  <creationTime>1390111150</creationTime>
>>  <memory snapshot='no'/>
>>  <disks>
>>    <disk name='vda' snapshot='external'>
> 
> Okay, we've definitely demonstrated a bug in at least 'virsh
> snapshot-list' for that particular build of libvirt; this proves that
> the snapshot is definitely external, even though the info output claimed
> it was internal.  I didn't search whether a bugzilla entry was already
> tracking this for RHEL 6.5; it's not a crasher, so it probably
won't get
> fixed until RHEL 6.6.  If you'd like, you can open a BZ, (it might get
> closed as a dup if someone else finds where it was already reported,
> even though I didn't do that search), to make sure it doesn't get
lost
Thanks I’ll sort out getting bugzilla report put it. But will search it first.
> Meanwhile, creating external snapshots is supported in that version of
> RHEL, but not deleting (at least not via virsh directly), so you'll
have
> to get your hands a bit dirty with qemu-img and virsh edit.
> 
>>      <driver type='qcow2'/>
>>      <source file='/var/lib/libvirt/snap1-host1.qcow2'/>
> 
> This says that snap1-host1.qcow2 is the wrapper file created at the time
> of the snapshot, and that...
> 
>>    </disk>
>>    <disk name='vdb' snapshot='external'>
>>      <driver type='qcow2'/>
>>      <source
file='/var/lib/libvirt/snap1-host1-db.qcow2'/>
>>    </disk>
>>    <disk name='hdc' snapshot='no'/>
>>  </disks>
>>  <domain type='kvm'>
> ...
>>    <devices>
>>      <emulator>/usr/libexec/qemu-kvm</emulator>
>>      <disk type='file' device='disk'>
>>        <driver name='qemu' type='raw'
cache='none'/>
>>        <source file='/var/lib/libvirt/images/host1.img'/>
> 
> it should have a backing file of /var/lib/libvirt/images/host1.img.
> Wonder why your 'qemu-img info' output didn't show that fact?
I’m not sure.

File is definitely there but hasn’t been used since creation it seems.
Something else I don’t quite understand.

# ls -l /var/lib/libvirt/images/host1.img 
-rw------- 1 qemu qemu 25769803776 Jan 19 16:58
/var/lib/libvirt/images/host1.img

I presume that had I never created a snapshot then its this file 
that would change as the VM is used?

>>> 
>>> I don't quite have a full picture of how you got into the
situation.  If
>>> you are trying to just get rid of the snapshot, you could always
try
>>> 'virsh snapshot-delete --metadata host1 snap1-host1', to
make libvirt
>>> forget about the snapshot without cleaning up any actual data
(leaving
>>> any external backing chains intact, and not removing internal
snapshots
>>> from qcow2 files).
>> 
>> ok. What implications does this have long term?
>> Is there then a related step to remove the data of the snapshot that is
no longer referenced?
>> Reclaiming the space would be handy.
> 
> Are you trying to revert to that state, or just forget that you ever
> took the snapshot?  Are you okay keeping the two files as a backing
> chain, or do you want to collapse it into one?  And if you DO want to
> collapse into one file (so you can delete the other), do you want the
> kept file to be snap1-host1.qcow2 (do a blockpull operation) or
> host1.img (do a commit operation)?
ok.  What I’d like is to have a state where the snapshot I took never existed. 
Which I assume means that the img file in /var/lib/libvirt/images becomes
the file used as the VM as its used?

Then I’d like to make a snapshot of the guest - memory and disk state. 
But the second disk be excluded 
(its a separate mount point for the guest for MySQL data)

But if I can get it working reliably a snapshot of both disks would
also be desirable for me.
>> And as before, given the structure of the VM and its two disks, how is
a snapshot created
>> excluding the second disk? vdb in the above xml output?
> 
> If you create the snapshot via command line, 'virsh snapshot-create-as
> host1 --name ... --diskspec vda,file=/path/for/disk --diskspec
> vdb,snapshot=no' should be sufficient to exclude vdb from the snapshot
> (you can use the --print-xml option to see what those options would
> actually pass to the virDomainSnapshotCreateXML command).
ok.  in “/path/for/disk” is that a path or a filename - called “disk” and can it
be
anything?  The default seem sot have been /var/lib/libvirt/  Is that not ideal??

Thanks again.  This is helping.

regards

rolf.

>> 
>> Many thanks for your help so far.
> 
> Glad to hear it, and hope we can continue to be helpful.
> 
> -- 
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>