I'm having some trouble creating KVM domains with RBD block devices using virsh. I've managed to get virsh to define the domain, but it gives an error when trying to start the domain: error: Failed to start domain test0 error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/3 kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: error connecting kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:rbd/test0: No such file or directory>From libvirt's QEMU log:2012-08-06 18:36:57.788+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -name test0 -S -M pc-1.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid 3e727624-f336-d674-4e0c-234641fc9987 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/test0.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=19,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:8c:e4:92,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 Domain id=5 is tainted: high-privileges char device redirected to /dev/pts/2 kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: error connecting kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:rbd/test0: No such file or directory 2012-08-06 18:36:58.353+0000: shutting down I can, however, successfully run a KVM virtual machine with an RBD block device with the following command: kvm -drive file=rbd:rbd/test1,id=drive-virtio-disk0,format=raw -cdrom /var/lib/libvirt/images/ubuntu-12.04-server-amd64.iso kvm will boot with the "-drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw" options, but the disk is not detected unless "if=none" is removed. Has anyone else figured this out? Thank you, Jeff Here is my rbd pool definition: ######### BEGIN <pool type="rbd"> <name>rbd</name> <source> <name>rbd</name> <host name='thinkmate3' port='6789'/> <host name='thinkmate4' port='6789'/> <auth username='admin' type='ceph'> <secret uuid='f0836fcb-9c05-5ccf-653a-eb1d0308e247'/> </auth> </source> </pool> ######### END My rbd volumes: # virsh vol-list rbd Name Path ----------------------------------------- test0 rbd/test0 test1 rbd/test1 test2 rbd/test2 Here is my domain definition: ######### BEGIN <domain type='kvm'> <name>test0</name> <uuid>3e727624-f336-d674-4e0c-234641fc9987</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='pc-1.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='network' device='disk'> <driver name='qemu' type='raw'/> <source protocol='rbd' name='rbd/test0'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <interface type='network'> <mac address='52:54:00:8c:e4:92'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> </domain> ######## END
On 08/06/2012 12:12 PM, Jeff Strunk wrote:> I'm having some trouble creating KVM domains with RBD block devices using > virsh. I've managed to get virsh to define the domain, but it gives an error > when trying to start the domain: > error: Failed to start domain test0 > error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/3 > kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: error connectingThis is the relevant error. If you don't have an /etc/ceph/ceph.conf file, or it (or the keyring file possibly) is unreadable by qemu due to apparmor/selinux etc, you'll need to specify all the information needed to connect in the drive xml. This includes the monitor host and the libvirt secret uuid, i.e.: <disk type='network' device='disk'> <driver name='qemu' type='raw'/> <source protocol='rbd' name='rbd/test0'> <host name='thinkmate3' port='6789'/> <host name='thinkmate4' port='6789'/> </source> <auth username='admin'> <secret type='ceph' uuid='f0836fcb-9c05-5ccf-653a-eb1d0308e247'/> </auth> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> Josh> kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:rbd/test0: No such file or directory > >>From libvirt's QEMU log: > 2012-08-06 18:36:57.788+0000: starting up > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -name test0 -S -M pc-1.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid 3e727624-f336-d674-4e0c-234641fc9987 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/test0.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=19,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:8c:e4:92,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus!-!> device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 > Domain id=5 is tainted: high-privileges > char device redirected to /dev/pts/2 > kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: error connecting > kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:rbd/test0: No such file or directory > 2012-08-06 18:36:58.353+0000: shutting down > > I can, however, successfully run a KVM virtual machine with an RBD block > device with the following command: > kvm -drive file=rbd:rbd/test1,id=drive-virtio-disk0,format=raw -cdrom /var/lib/libvirt/images/ubuntu-12.04-server-amd64.iso > > kvm will boot with the "-drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw" > options, but the disk is not detected unless "if=none" is removed. > > Has anyone else figured this out? > > Thank you, > Jeff > > Here is my rbd pool definition: > ######### BEGIN > <pool type="rbd"> > <name>rbd</name> > <source> > <name>rbd</name> > <host name='thinkmate3' port='6789'/> > <host name='thinkmate4' port='6789'/> > <auth username='admin' type='ceph'> > <secret uuid='f0836fcb-9c05-5ccf-653a-eb1d0308e247'/> > </auth> > </source> > </pool> > ######### END > > My rbd volumes: > # virsh vol-list rbd > Name Path > ----------------------------------------- > test0 rbd/test0 > test1 rbd/test1 > test2 rbd/test2 > > Here is my domain definition: > ######### BEGIN > <domain type='kvm'> > <name>test0</name> > <uuid>3e727624-f336-d674-4e0c-234641fc9987</uuid> > <memory unit='KiB'>1048576</memory> > <currentMemory unit='KiB'>1048576</currentMemory> > <vcpu placement='static'>1</vcpu> > <os> > <type arch='x86_64' machine='pc-1.0'>hvm</type> > <boot dev='hd'/> > </os> > <features> > <acpi/> > <apic/> > <pae/> > </features> > <clock offset='utc'/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <devices> > <emulator>/usr/bin/kvm</emulator> > <disk type='network' device='disk'> > <driver name='qemu' type='raw'/> > <source protocol='rbd' name='rbd/test0'/> > <target dev='vda' bus='virtio'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> > </disk> > <disk type='block' device='cdrom'> > <driver name='qemu' type='raw'/> > <target dev='hdc' bus='ide'/> > <readonly/> > <address type='drive' controller='0' bus='1' target='0' unit='0'/> > </disk> > <controller type='ide' index='0'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> > </controller> > <controller type='usb' index='0'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> > </controller> > <interface type='network'> > <mac address='52:54:00:8c:e4:92'/> > <source network='default'/> > <model type='virtio'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> > </interface> > <serial type='pty'> > <target port='0'/> > </serial> > <console type='pty'> > <target type='serial' port='0'/> > </console> > <input type='mouse' bus='ps2'/> > <graphics type='vnc' port='-1' autoport='yes'/> > <video> > <model type='cirrus' vram='9216' heads='1'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> > </video> > <memballoon model='virtio'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> > </memballoon> > </devices> > </domain> > ######## END
Thank you it works now. The issue was the lack of the auth element. qemu has access to /etc/ceph/ceph.conf. Specifying the host elements in the source element caused a crash: error: Failed to start domain test0 error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/3 server name not found: thinkmate3:6789;thinkmate4:6789 (No such file or directory) unable to parse addrs in 'thinkmate3:6789;thinkmate4:6789' mon/MonClient.cc: In function 'void MonClient::_pick_new_mon()' thread 7fd1de5d67c0 time 2012-08-06 15:40:38.191920 mon/MonClient.cc: 424: FAILED assert(monmap.size() > 0) ceph version 0.48argonaut (commit:c2b20ca74249892c8e5e40c12aa14446a2bf2030) 1: (MonClient::_pick_new_mon()+0x40a) [0x7fd1dd6f4e8a] 2: (MonClient::_reopen_session()+0x187) [0x7fd1dd6f8c47] 3: (MonClient::authenticate(double)+0x1aa) [0x7fd1dd6f9fea] 4: (librados::RadosClient::connect()+0xb6c) [0x7fd1dd63d6ec] 5: (()+0x94393) [0x7fd1de698393] 6: (()+0x801e5) [0x7fd1de6841e5] 7: (()+0x80366) [0x7fd1de684366] 8: (()+0x802d9) [0x7fd1de6842d9] 9: (()+0x80ee7) [0x7fd1de684ee7] 10: (()+0xa3b03) [0x7fd1de6a7b03] 11: (()+0xfec4b) [0x7fd1de702c4b] 12: (()+0x11fbc2 On Monday, August 06, 2012 01:33:34 PM Josh Durgin wrote:> On 08/06/2012 12:12 PM, Jeff Strunk wrote: > > I'm having some trouble creating KVM domains with RBD block devices using > > virsh. I've managed to get virsh to define the domain, but it gives an > > error when trying to start the domain: > > error: Failed to start domain test0 > > error: internal error process exited while connecting to monitor: char > > device redirected to /dev/pts/3 kvm: -drive > > file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: error > > connecting > This is the relevant error. If you don't have an /etc/ceph/ceph.conf > file, or it (or the keyring file possibly) is unreadable by qemu due to > apparmor/selinux etc, you'll need to specify all the information needed > to connect in the drive xml. This includes the monitor host and the > libvirt secret uuid, i.e.: > > <disk type='network' device='disk'> > <driver name='qemu' type='raw'/> > <source protocol='rbd' name='rbd/test0'> > <host name='thinkmate3' port='6789'/> > <host name='thinkmate4' port='6789'/> > </source> > <auth username='admin'> > <secret type='ceph' uuid='f0836fcb-9c05-5ccf-653a-eb1d0308e247'/> > </auth> > <target dev='vda' bus='virtio'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x0'/> > </disk> > > Josh > > > kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: > > could not open disk image rbd:rbd/test0: No such file or directory> > >>From libvirt's QEMU log: > > 2012-08-06 18:36:57.788+0000: starting up > > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin > > QEMU_AUDIO_DRV=none /usr/bin/kvm -name test0 -S -M pc-1.0 -enable-kvm -m > > 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid > > 3e727624-f336-d674-4e0c-234641fc9987 -nodefconfig -nodefaults -chardev > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/test0.monitor,server,now > > ait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc > > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive > > file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw -device > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=vi > > rtio-disk0,bootindex=1 -drive > > if=none,id=drive-ide0-1-0,readonly=on,format=raw -device > > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > > tap,fd=19,id=hostnet0 -device > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:8c:e4:92,bus=pci.0,ad > > dr=0x3 -chardev pty,id=charserial0 -device > > isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -!> > > device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 > > > > Domain id=5 is tainted: high-privileges > > char device redirected to /dev/pts/2 > > kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: > > error connecting kvm: -drive > > file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: could not > > open disk image rbd:rbd/test0: No such file or directory 2012-08-06 > > 18:36:58.353+0000: shutting down > > > > I can, however, successfully run a KVM virtual machine with an RBD block > > device with the following command: > > kvm -drive file=rbd:rbd/test1,id=drive-virtio-disk0,format=raw -cdrom > > /var/lib/libvirt/images/ubuntu-12.04-server-amd64.iso > > > > kvm will boot with the "-drive > > file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw" options, but > > the disk is not detected unless "if=none" is removed. > > > > Has anyone else figured this out? > > > > Thank you, > > Jeff > > > > Here is my rbd pool definition: > > ######### BEGIN > > <pool type="rbd"> > > <name>rbd</name> > > <source> > > <name>rbd</name> > > <host name='thinkmate3' port='6789'/> > > <host name='thinkmate4' port='6789'/> > > <auth username='admin' type='ceph'> > > <secret uuid='f0836fcb-9c05-5ccf-653a-eb1d0308e247'/> > > </auth> > > </source> > > </pool> > > ######### END > > > > My rbd volumes: > > # virsh vol-list rbd > > Name Path > > ----------------------------------------- > > test0 rbd/test0 > > test1 rbd/test1 > > test2 rbd/test2 > > > > Here is my domain definition: > > ######### BEGIN > > <domain type='kvm'> > > > > <name>test0</name> > > <uuid>3e727624-f336-d674-4e0c-234641fc9987</uuid> > > <memory unit='KiB'>1048576</memory> > > <currentMemory unit='KiB'>1048576</currentMemory> > > <vcpu placement='static'>1</vcpu> > > <os> > > > > <type arch='x86_64' machine='pc-1.0'>hvm</type> > > <boot dev='hd'/> > > > > </os> > > <features> > > > > <acpi/> > > <apic/> > > <pae/> > > > > </features> > > <clock offset='utc'/> > > <on_poweroff>destroy</on_poweroff> > > <on_reboot>restart</on_reboot> > > <on_crash>restart</on_crash> > > <devices> > > > > <emulator>/usr/bin/kvm</emulator> > > <disk type='network' device='disk'> > > > > <driver name='qemu' type='raw'/> > > <source protocol='rbd' name='rbd/test0'/> > > <target dev='vda' bus='virtio'/> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > > function='0x0'/> > > > > </disk> > > <disk type='block' device='cdrom'> > > > > <driver name='qemu' type='raw'/> > > <target dev='hdc' bus='ide'/> > > <readonly/> > > <address type='drive' controller='0' bus='1' target='0' unit='0'/> > > > > </disk> > > <controller type='ide' index='0'> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > > function='0x1'/> > > > > </controller> > > <controller type='usb' index='0'> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > > function='0x2'/> > > > > </controller> > > <interface type='network'> > > > > <mac address='52:54:00:8c:e4:92'/> > > <source network='default'/> > > <model type='virtio'/> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > > function='0x0'/> > > > > </interface> > > <serial type='pty'> > > > > <target port='0'/> > > > > </serial> > > <console type='pty'> > > > > <target type='serial' port='0'/> > > > > </console> > > <input type='mouse' bus='ps2'/> > > <graphics type='vnc' port='-1' autoport='yes'/> > > <video> > > > > <model type='cirrus' vram='9216' heads='1'/> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > > function='0x0'/> > > > > </video> > > <memballoon model='virtio'> > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > > function='0x0'/> > > > > </memballoon> > > > > </devices> > > > > </domain> > > ######## END