libvirt users - Nov 2017 - virtlock - a VM goes read-only

Dear colleagues,

I am facing a problem that has been troubling me for last week and a half.
Please if you are able to help or offer some guidance.

I have a non-prod POC environment with 2 CentOS7 fully updated hypervisors
and an NFS filer that serves as a VM image storage. The overall environment
works exceptionally well. However, starting a few weeks ago I have been
trying to implement virtlock in order to prevent a VM running on 2
hypervisors at the same time.

Here is the description how the environment looks like in terms of virtlock
configuration on both hypervisors:

-- Content of /etc/libvirt/qemu.conf --
lock_manager = "lockd"

Only the above line is uncommented for direct locking.

# libvirtd --version; python -c "import platform;
print(platform.platform())"; virtlockd -V
libvirtd (libvirt) 3.2.0
Linux-3.10.0-693.2.2.el7.x86_64-x86_64-with-centos-7.4.1708-Core
virtlockd (libvirt) 3.2.0

# getenforce
Permissive

Here is the issue:

h1 # virsh list
 Id    Name                           State
----------------------------------------------------
 1     test09                         running

h1 # virsh domblklist test09
Target     Source
------------------------------------------------
vda        /storage_nfs/images_001/test09.qcow2


h1 #

h2 # virsh list
 Id    Name                           State
----------------------------------------------------

h2 # virsh list --all | grep test09
 -     test09                         shut off

h2 # virsh start test09
error: Failed to start domain test09
error: resource busy: Lockspace resource
'/storage_nfs/images_001/test09.qcow2' is locked

h2 # virsh list
 Id    Name                           State
----------------------------------------------------

h2 #

Before I start test09 I open a console to the guest and observe what is
going on in it. Once I try to start test09 (and get a message about locked
resource) on h2 hypervisor, I can see the following messages in the console
and the vm goes to ro mode:

on test09's console:

[  567.394148] blk_update_request: I/O error, dev vda, sector 13296056
[  567.395883] blk_update_request: I/O error, dev vda, sector 13296056

[  572.871905] blk_update_request: I/O error, dev vda, sector 8654040
[  572.872627] Aborting journal on device vda1-8.
[  572.873978] blk_update_request: I/O error, dev vda, sector 8652800
[  572.874707] Buffer I/O error on dev vda1, logical block 1081344, lost
sync page write
[  572.875472] blk_update_request: I/O error, dev vda, sector 2048
[  572.876009] Buffer I/O error on dev vda1, logical block 0, lost sync
page write
[  572.876727] EXT4-fs error (device vda1): ext4_journal_check_start:56:
Detected aborted journal[  572.878061] JBD2: Error -5 detected when
updating journal superblock for vda1-8.

[  572.878807] EXT4-fs (vda1): Remounting filesystem read-only
[  572.879311] EXT4-fs (vda1): previous I/O error to superblock detected
[  572.880937] blk_update_request: I/O error, dev vda, sector 2048
[  572.881538] Buffer I/O error on dev vda1, logical block 0, lost sync
page write

I also observe the guests'log:

-- /var/log/libvirt/qemu/test09.log --

block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)

If it helps, here is the disk portion of an XML file:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'
cache='none'/>
      <source file='/storage_nfs/images_001/test09.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
function='0x0'/>
    </disk>

I usually do implement SELinux on a hypervisor to isolate guests even
further but this time I set it to permissive mode just to rule out SELinux
factor. The same thing happens when SELinux is in enforcing mode
(virt_use_nfs is set to on in that case) and audit2why doesn't report any
anomalies when parsing audit logs.

I have tried to use indirect locking via the same filer and with a
separated export for the hashes by removing the comment in
/etc/libvirt/qemu-lockd.conf for the following line:

file_lockspace_dir = "/var/lib/libvirt/lockd/files"

In this case the hashes are normally created on the NFS export mounted
under /var/lib/libvirt/lockd/files. I have also tried playing with both
QCOW2 and raw disk images for VMs (and even with XFS/ext4 based guests) but
the outcome is always the same. I have a couple of KVM books - consulted
them on this topic, consulted  Red Hat and SUSE docs but pretty much the
configuration instructions are, naturally, the same. I saw that  some
colleagues posted a few emails (ie
https://www.redhat.com/archives/libvirt-users/2015-September/msg00004.html)
to the list related to virtlock but it seems that it is not the same issue.
I have also, as a last resort, completely disabled SELinux, rebooted both
hypervisors, created a new vm, repeated all the steps listed above but with
the same results.

Now, I am pretty sure that I am missing something simple here since this is
a standard feature and should work out of the box if set correctly but so
far I cannot see what I am missing.

I would really appreciate any tip/help.

Thank you very much!!


Regards,

Branimir

On Wed, Nov 15, 2017 at 02:24:48PM +0000, Branimir Pejakovic
wrote:
> Dear colleagues,
> 
> I am facing a problem that has been troubling me for last week and a half.
> Please if you are able to help or offer some guidance.
> 
> I have a non-prod POC environment with 2 CentOS7 fully updated hypervisors
> and an NFS filer that serves as a VM image storage. The overall environment
> works exceptionally well. However, starting a few weeks ago I have been
> trying to implement virtlock in order to prevent a VM running on 2
> hypervisors at the same time.
[snip]
> h2 # virsh start test09
> error: Failed to start domain test09
> error: resource busy: Lockspace resource
> '/storage_nfs/images_001/test09.qcow2' is locked
[snip]
> Now, I am pretty sure that I am missing something simple here since this is
> a standard feature and should work out of the box if set correctly but so
> far I cannot see what I am missing.
So I think you are hitting the little surprise in the way our locking
works. Specifically, right now the locking only protects the image
file contents from concurrent writes. We don't have locking around
the file attributes (permissions, user/group ownership, selinux label,
etc)

Unfortunately with the current libvirt design, the security drivers run
before locking takes effect. So what happens is that you have your first
VM running normally. It has been granted ability to write to the image
in terms of SELinux label & permissions/owership. The lock manager is
holding locks protecting the image contents

Now you try to start the second guest, and libvirt will apply the SELinux
label & permissions/ownership needed for that second guest, despite it
being used by the first guest. Only then do we acquire the locks for the
disk image, and fail because the first guest holds the lock. We now
reset the permissions/ownership we just granted for the second guest,
and thus unfortunately blocks the first guest from using the images,
causing the I/O errors you mention

We *have* successfully prevented 2 guests from writing to the same
image at once, so your data is still safe. Unfortunately though the
first guest cannot write any further datas, so that previously
running guest is now fubar :-(

I appreciated this is rather surprising & unhelpful in general. Just
console yourself with the fact that at least your disk iamge is not
corrupted.

Note, this should only happen with SELinux enforcing though - if it is
permissive, then I'd expect the first guest to carry on working.

We would like to improve our locking so that we can apply locks before
we even try to change ownership/permissions/selinux, which would make
it far more useful. We've never succesfully completed that work though.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Thu, Nov 16, 2017 at 12:48 PM, Daniel P. Berrange <berrange@redhat.com>
wrote:
> On Wed, Nov 15, 2017 at 02:24:48PM +0000, Branimir Pejakovic wrote:
> > Dear colleagues,
> >
> > I am facing a problem that has been troubling me for last week and a
> half.
> > Please if you are able to help or offer some guidance.
> >
> > I have a non-prod POC environment with 2 CentOS7 fully updated
> hypervisors
> > and an NFS filer that serves as a VM image storage. The overall
> environment
> > works exceptionally well. However, starting a few weeks ago I have
been
> > trying to implement virtlock in order to prevent a VM running on 2
> > hypervisors at the same time.
>
> [snip]
>
> > h2 # virsh start test09
> > error: Failed to start domain test09
> > error: resource busy: Lockspace resource
> > '/storage_nfs/images_001/test09.qcow2' is locked
>
> [snip]
>
> > Now, I am pretty sure that I am missing something simple here since
this
> is
> > a standard feature and should work out of the box if set correctly but
so
> > far I cannot see what I am missing.
>
> So I think you are hitting the little surprise in the way our locking
> works. Specifically, right now the locking only protects the image
> file contents from concurrent writes. We don't have locking around
> the file attributes (permissions, user/group ownership, selinux label,
> etc)
>
> Unfortunately with the current libvirt design, the security drivers run
> before locking takes effect. So what happens is that you have your first
> VM running normally. It has been granted ability to write to the image
> in terms of SELinux label & permissions/owership. The lock manager is
> holding locks protecting the image contents
>
> Now you try to start the second guest, and libvirt will apply the SELinux
> label & permissions/ownership needed for that second guest, despite it
> being used by the first guest. Only then do we acquire the locks for the
> disk image, and fail because the first guest holds the lock. We now
> reset the permissions/ownership we just granted for the second guest,
> and thus unfortunately blocks the first guest from using the images,
> causing the I/O errors you mention
>
> We *have* successfully prevented 2 guests from writing to the same
> image at once, so your data is still safe. Unfortunately though the
> first guest cannot write any further datas, so that previously
> running guest is now fubar :-(
>
> I appreciated this is rather surprising & unhelpful in general. Just
> console yourself with the fact that at least your disk iamge is not
> corrupted.
>
> Note, this should only happen with SELinux enforcing though - if it is
> permissive, then I'd expect the first guest to carry on working.
>
> We would like to improve our locking so that we can apply locks before
> we even try to change ownership/permissions/selinux, which would make
> it far more useful. We've never succesfully completed that work though.
>
>
Hi Daniel

Thank you very much for your answer. Apologies for late reply.

I got it working but I had to do a few modifications.

Usually, qemu-kvm runs as a qemu user as configured in
/etc/libvirt/qemu.conf (user/group parameters). My QCOW files were owned by
root during this experiment (usually I set them to be owned by qemu user).
Once a VM starts, the ownership is changed to qemu and it keeps that way
until I try to start the same VM on another hypervisor and lock kicks in.
In that moment, the file ownership is changed to root again (observed via
watch in 2nd terminal) and the VM goes read only.

I can do a workaround and then lock works normally (no ro-VM) if I do the
following:
- set 0777 permissions on the QEMU file
- change user/group parameters in qemu.conf to root and restart libvirtd

I like to have a bit of a security so I searched through qemu.conf file and
found option dynamic_ownership. The option is set to 1 by default. I set it
to 0 and then changed ownership of the image files to qemu, set user/group
in qemu.conf
to qemu, normal permissions on the files and finally restarted libvirtd.
After that, lock works as expected.

The side effect is that if you want to do template based provisioning via
python-libvirt based app or from the shell by using virt-sysprep,
virt-clone or some other libvirt/libguestfs based app and you want to
modify a VM after cloning (hostname/network/vCPU/etc..), it will throw an
exception - permission denied on the image file. This one was solved by
looking at https://access.redhat.com/solutions/2110391 and it works great
once applied. During all this time, SELinux was disabled.

I would like to take this opportunity to personally thank you and the team
in Red Hat for all hard work on libvirt and libvirt-based tools. I heavily
use python-libvirt module and what I can say is that if you don't have
RHEV-M/oVirt as a single
pane of glass for your virtualization layer, it helps you a great deal in
managing and inspecting (and making statistics!) of a large pool of KVM
hypervisors. Please keep up a good work!

Regards,

Branimir