libvirt users - Feb 2016 - Unable to create raw volume on netfs storage (Operation not permitted)

Hello,

we are trying to create a new setup with a centralized storage server. Currently
we are running multiple "All-in-One" hosts using raw volumes stored on
the local disk of the server.

I tried it first with samba but it did not work so currently I have configured
it as NFS.

This is my current setup:

For Storage:
OS: Ubuntu 16.04 beta (to match vHost version),
IP: 123.123.123.2
NFS: v1.2.8
Samba: v4.3.3

For vHost:
OS: Ubuntu 16.04 beta (as the Package in Ubuntu 14.04 has still a bug that
prevents mounting of samba shares)
IP: 123.123.123.3
Libvirt: v2.21
Qemu: 2.5

Both are simply connected via switched network for testing.

So on the Storage server I created a user and a group called vstorage
(uid:1000,gid:500)
Then created a folder /pool owned by this user, permission 777
And exported it via nfs in /etc/exports:

/pool  
123.123.123.3(rw,async,no_subtree_check,all_squash,anonuid=1000,anongid=500)


On the vHost I defined the pool via XML:

<pool type='netfs'>
  <name>pool</name>
 <source>
    <host name='123.123.123.2'/>
    <dir path='pool'/>
    <format type='nfs'/>
  </source>
  <target>
    <path>/var/lib/virt/images</path>
  </target>
</pool>


When I try to define a volume in the pool, it will always fail if raw format
(default) is used.

virsh vol-create-as --pool pool --name test --capacity 512M
error: Failed to create vol test
error: Failed to create file '/var/lib/virt/images/test': Operation not
permitted

In /var/log/syslog I can find the following
libvirtd[889]: cannot chown '/var/lib/virt/images/test' to (0, 0):
Operation not permitted
libvirtd[889]: Failed to create file '/var/lib/virt/images/test':
Operation not permitted

When I try it with a different format, like qcow2 it is working fine:

virsh vol-create-as --pool pool --name test --capacity 512M --format qcow2
Vol test created

I am unable to find the reason for this problem, it also took me quite a while
find out that this is a format issue as the error message is actually not really
helpful.

I found a small guide on setting up KVM from IBM and they are using NFS with raw
images, so I'm not sure where the error is
https://www.ibm.com/developerworks/library/l-using-kvm/

Maybe someone could shed some light on this? The same error message happens btw.
with Samba.

Best regards,

Benjamin Weik

On 10.02.2016 14:10, Benjamin Weik wrote:
> Hello,
> 
> we are trying to create a new setup with a centralized storage server.
Currently we are running multiple "All-in-One" hosts using raw volumes
stored on the local disk of the server.
> 
> I tried it first with samba but it did not work so currently I have
configured it as NFS.
> 
> This is my current setup:
> 
> For Storage:
> OS: Ubuntu 16.04 beta (to match vHost version),
> IP: 123.123.123.2
> NFS: v1.2.8
> Samba: v4.3.3
> 
> For vHost:
> OS: Ubuntu 16.04 beta (as the Package in Ubuntu 14.04 has still a bug that
prevents mounting of samba shares)
> IP: 123.123.123.3
> Libvirt: v2.21
> Qemu: 2.5
> 
> Both are simply connected via switched network for testing.
> 
> So on the Storage server I created a user and a group called vstorage
(uid:1000,gid:500)
> Then created a folder /pool owned by this user, permission 777
> And exported it via nfs in /etc/exports:
> 
> /pool  
123.123.123.3(rw,async,no_subtree_check,all_squash,anonuid=1000,anongid=500)
> 
> 
> On the vHost I defined the pool via XML:
> 
> <pool type='netfs'>
>   <name>pool</name>
>  <source>
>     <host name='123.123.123.2'/>
>     <dir path='pool'/>
>     <format type='nfs'/>
>   </source>
>   <target>
>     <path>/var/lib/virt/images</path>
>   </target>
> </pool>
> 
> 
> When I try to define a volume in the pool, it will always fail if raw
format (default) is used.
> 
> virsh vol-create-as --pool pool --name test --capacity 512M
> error: Failed to create vol test
> error: Failed to create file '/var/lib/virt/images/test': Operation
not permitted
> 
> In /var/log/syslog I can find the following
> libvirtd[889]: cannot chown '/var/lib/virt/images/test' to (0, 0):
Operation not permitted
> libvirtd[889]: Failed to create file '/var/lib/virt/images/test':
Operation not permitted
> 
> When I try it with a different format, like qcow2 it is working fine:
> 
> virsh vol-create-as --pool pool --name test --capacity 512M --format qcow2
> Vol test created
Unfortunately, this will create the volume with the default permissions
as there are none specified in vol-create XML. This vol-create-as
command creates a very minimalistic volume XML and passes it to
virStorageVolCreateXML() API which if no perms specified in the XML will
use current euid/egid of libvirtd (0:0). Hence the error message.

What you can do here is create your own volume XML with perms specified
and use vol-create instead.

What we can do is:
1) introduce --print-xml option to vol-crate-as like other commands
constructing an XML have.
2) Make libvirt to inherit pool perms for the volume if none specified.
3) Teach vol-create-as to accept permissions.

Michal