libvirt users - Apr 2019 - Libvirt pool cannot see or create rbd clones

Hello everyone,

To increase my odds of finding an answer I also wanted to ask here.
This is my post from serverfault[1] in verbatim:

While trying to get a cloned disk running from my OS snapshot I run
into the problem that Libvirt cannot see existing images cloned from a
snapshot. Created via:

# rbd -p vmdisks clone vmdisks/coreos_2023@base vmdisks/coreos00.disk

The base image has the one snapshot 'base' and is protected. The cloned
disk is created just fine:

# rbd -p vmdisks info coreos00.disk
rbd image 'coreos00.disk':
        size 8.49GiB in 2174 objects
        order 22 (4MiB objects)
        block_name_prefix: rbd_data.48a99c6b8b4567
        format: 2
        features: layering
        flags: 
        create_timestamp: Thu Apr 25 14:46:52 2019
        parent: vmdisks/coreos_2023@base
        overlap: 8.49GiB

I temporarily have Libvirt configured with a rbd pool that uses the
ceph admin user. But I cannot see the cloned disk. Just the parent:

virsh # vol-list --pool rbd_image_root
 Name                 Path                                    
---------------------------------------------------------------------
---------
 coreos_2023 vmdisks/coreos_2023     

If I try to create the cloned image from within virsh I run into the
following issue:

virsh # vol-clone --pool rbd_image_root coreos_2023 coreos00.disk
error: Failed to clone vol from coreos_2023
error: failed to iterate RBD snapshot coreos_2023@base: Operation not
permitted

Note that this pool uses the Ceph admin user which makes the Operation
not permitted a tad odd.

Am I missing a configuration option here that would allow for the pool
to use clones? I can't find any information on this in the
documentation so far. And the source code of libvirt looks like it
should support both features.

Versions:

Libvirt Machine: Ubuntu 18.04
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1

Ceph Machine: openSUSE Leap 42.3
Ceph 12.2.5


[1] 
https://serverfault.com/questions/964586/libvirt-pool-cannot-see-or-create-rbd-clones

Some additional information that I have found since yesterday:

When I create a volume like this:

<volume>
   <name>coreos00.disk</name>
   <capacity unit="bytes">9116319744</capacity>
   <target>
       <format type="raw"></format>
       <permissions>
           <mode>644</mode>
       </permissions>
   </target>
   <backingStore>
       <path>vmdisks/coreos_2023</path>
       <format type="raw"></format>
   </backingStore>
</volume>

Which does not seem to work with the backing store either as it creates
an empty volume without the underlying parent. But when I now delete
the volume via rbd and create a new clone it shows up in libvirt and
can be used by the machine.

# rbd -p vmdisks rm coreos00.disk
# rbd clone vmdisks/coreos_2023@basis vmdisks/coreos00.disk

Sadly this is not a very efficient way to create the clones I would
say.

Anyone got any, like literally any, ideas what might be wrong in this
setup?

On 4/26/19 11:53 AM, Weller, Lennart wrote:
> Hello everyone,
> 
> To increase my odds of finding an answer I also wanted to ask here.
> This is my post from serverfault[1] in verbatim:
> 
> While trying to get a cloned disk running from my OS snapshot I run
> into the problem that Libvirt cannot see existing images cloned from a
> snapshot. Created via:
> 
> # rbd -p vmdisks clone vmdisks/coreos_2023@base vmdisks/coreos00.disk
> 
> The base image has the one snapshot 'base' and is protected. The
cloned
> disk is created just fine:
> 
> # rbd -p vmdisks info coreos00.disk
> rbd image 'coreos00.disk':
>          size 8.49GiB in 2174 objects
>          order 22 (4MiB objects)
>          block_name_prefix: rbd_data.48a99c6b8b4567
>          format: 2
>          features: layering
>          flags:
>          create_timestamp: Thu Apr 25 14:46:52 2019
>          parent: vmdisks/coreos_2023@base
>          overlap: 8.49GiB
> 
> I temporarily have Libvirt configured with a rbd pool that uses the
> ceph admin user. But I cannot see the cloned disk. Just the parent:
> 
> virsh # vol-list --pool rbd_image_root
>   Name                 Path
> ---------------------------------------------------------------------
> ---------
>   coreos_2023 vmdisks/coreos_2023
> 
> If I try to create the cloned image from within virsh I run into the
> following issue:
> 
> virsh # vol-clone --pool rbd_image_root coreos_2023 coreos00.disk
> error: Failed to clone vol from coreos_2023
> error: failed to iterate RBD snapshot coreos_2023@base: Operation not
> permitted
> 
> Note that this pool uses the Ceph admin user which makes the Operation
> not permitted a tad odd.
> 
> Am I missing a configuration option here that would allow for the pool
> to use clones? I can't find any information on this in the
> documentation so far. And the source code of libvirt looks like it
> should support both features.
> 
> Versions:
> 
> Libvirt Machine: Ubuntu 18.04
> Compiled against library: libvirt 4.0.0
> Using library: libvirt 4.0.0
> Using API: QEMU 4.0.0
> Running hypervisor: QEMU 2.11.1
> 
> Ceph Machine: openSUSE Leap 42.3
> Ceph 12.2.5
> 
I wonder if this is a libvirt bug. Looks like it. The error comes from 
rbd_diff_iterate() (or rbd_diff_iterate2 if compiled against newer 
librbd). Anyway, a callback is passed to the itreate() function and from 
librbd code it looks like if the callback returns a negative value then 
an error is signalized from the iterate() function. Well, we're passing 
a callback that does nothing but return an error. This might be a 
possible fix then:

diff --git i/src/storage/storage_backend_rbd.c
w/src/storage/storage_backend_rbd.c
index f8c968e682..08f8123678 100644
--- i/src/storage/storage_backend_rbd.c
+++ w/src/storage/storage_backend_rbd.c
@@ -1038,7 +1038,7 @@ virStorageBackendRBDIterateCb(uint64_t offset
ATTRIBUTE_UNUSED,
      * searching any further.
      */
     *(int*) arg = 1;
-    return -1;
+    return 0;
 }
 
 static int



Can you please give it a try?

Thanks,
Michal