libvirt users - Jan 2015 - Libvirt guest can't boot up when use ceph as storage backend with Selinux enabled

Hi there, 

I met one problem that guest fail to boot up when Selinux is enabled with guest
storage
based on ceph. However, I can boot the guest with qemu directly. I also can boot
it up
with Selinux disabled. Not sure it is a libvirt bug or wrong use case. 

1. Enable Selinux 

# getenforce && iptables -L 
Enforcing 
Chain INPUT (policy ACCEPT) 
target prot opt source destination 

Chain FORWARD (policy ACCEPT) 
target prot opt source destination 

Chain OUTPUT (policy ACCEPT) 
target prot opt source destination 

2. Define a guest with source file based on ceph 
# virsh define /dev/stdin <<EOF 
<domain type='kvm' id='13'> 
<name>ceph</name> 
<memory unit='KiB'>4048896</memory> 
<currentMemory unit='KiB'>4048576</currentMemory> 
<vcpu placement='static'>4</vcpu> 
<resource> 
<partition>/machine</partition> 
</resource> 
<os> 
<type arch='x86_64'
machine='pc-i440fx-rhel7.1.0'>hvm</type>
<boot dev='hd'/> 
</os> 
<features> 
</features> 
<devices> 
<emulator>/usr/libexec/qemu-kvm</emulator> 
<disk type='network' device='disk'> 
<driver name='qemu' type='raw' cache='none'/> 
<auth username='libvirt'> 
<secret type='ceph' usage='client.libvirt secret'/> 
</auth> 
<source protocol='rbd' name='libvirt-pool/rhel7-rbd.img'> 
<host name='10.66.xxx.xx' port='6789'/> 
</source> 
<backingStore/> 
<target dev='vda' bus='virtio'/> 
<alias name='virtio-disk0'/> 
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05' function='0x0'/>
</disk> 
</devices> 
</domain> 
EOF 
Domain ceph defined from /dev/stdin 

3. Try to start the guest by virsh 
# virsh start ceph 
error: Failed to start domain ceph 
error: internal error: process exited while connecting to monitor: 

What I can see from libvirtd log is below: 

2015-01-08 08:07:32.376+0000: 22552: warning : qemuDomainObjTaint:1890 : Domain
id=19 name='ceph' uuid=e4412366-1f16-4c54-b121-dfb565672427 is tainted:
high-privileges
Detaching after fork from child process 23015. 
2015-01-08 08:07:32.684+0000: 22552: error : qemuMonitorOpenUnix:309 : failed to
connect to monitor socket: No such process
2015-01-08 08:07:32.684+0000: 22552: error : qemuProcessWaitForMonitor:2207 :
internal error: process exited while connecting to monitor:
2015-01-08 08:07:32.684+0000: 22552: error : virDBusCall:1542 : error from
service: TerminateMachine: No such file or directory

4. Start it by qemu cmd 

# cat /usr/local/var/log/libvirt/qemu/ceph.log 
2015-01-08 08:08:12.179+0000: starting up 
LC_ALL=C
PATH=/root/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm
-name ceph -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off -m 3954 -realtime
mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
e4412366-1f16-4c54-b121-dfb565672427 -nographic -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/usr/local/var/lib/libvirt/qemu/ceph.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-no-acpi -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive
file=rbd:libvirt-pool/rhel7-rbd.img:id=libvirt:key=AQAQLq5UwO8PMRAA5qftTrdfzXnFZdnunN1WeQ==:auth_supported=cephx\;none:mon_host=10.66.106.92\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x2 -msg timestamp=on
Domain id=20 is tainted: high-privileges 
2015-01-08 08:08:12.473+0000: shutting down 

# /usr/libexec/qemu-kvm -name ceph -S -machine
pc-i440fx-rhel7.1.0,accel=kvm,usb=off -m 3954 -realtime mlock=off -smp
4,sockets=4,cores=1,threads=1 -uuid 319cf458-8740-4d83-9317-e2d52025aa9e
-nographic -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/usr/local/var/lib/libvirt/qemu/ceph.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-no-acpi -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive
file=rbd:libvirt-pool/rhel7-rbd.img:id=libvirt:key=AQAQLq5UwO8PMRAA5qftTrdfzXnFZdnunN1WeQ==:auth_supported=cephx\;none:mon_host=10.66.106.92,if=none,id=drive-virtio-disk0,format=raw,cache=none


# ps aux|grep qemu 
root 23075 2.5 0.2 5114492 16352 pts/5 Sl+ 16:09 0:00 /usr/libexec/qemu-kvm
-name ceph -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off -m 3954 -realtime
mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
319cf458-8740-4d83-9317-e2d52025aa9e -nographic -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/usr/local/var/lib/libvirt/qemu/ceph.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-no-acpi -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive
file=rbd:libvirt-pool/rhel7-rbd.img:id=libvirt:key=AQAQLq5UwO8PMRAA5qftTrdfzXnFZdnunN1WeQ==:auth_supported=cephx;none:mon_host=10.66.106.92,if=none,id=drive-virtio-disk0,format=raw,cache=none

5. Disable Selinux, start guest by virsh will succeed 
# setenforce 0 && virsh start ceph 
Domain ceph started 

# virsh list 
Id Name State 
---------------------------------------------------- 
21 ceph running 

# virsh dumpxml ceph|grep disk -A 10 
.. 
<disk type='network' device='disk'> 
<driver name='qemu' type='raw' cache='none'/> 
<auth username='libvirt'> 
<secret type='ceph' usage='client.libvirt secret'/> 
</auth> 
<source protocol='rbd' name='libvirt-pool/rhel7-rbd.img'> 
<host name='10.66.xxx.xx' port='6789'/> 
</source> 
<backingStore/> 
<target dev='vda' bus='virtio'/> 
<alias name='virtio-disk0'/> 
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05' function='0x0'/>
</disk> 
.. 

I use latest libvirt build from git and 

# rpm -q librbd1 librados2 qemu-kvm-rhev 
librbd1-0.87-0.el7.x86_64 
librados2-0.87-0.el7.x86_64 
qemu-kvm-rhev-2.1.2-17.el7.x86_64 


-- 
Regards 
shyu