Hi everybody,
my background: I'm doing Xen since 10++ years, many years with DRBD for
high availability, since some time I'm using preferable GlusterFS with
FUSE as replicated storage, where I place the image-files for the vms.
In my current project we started (successfully) with Xen/GlusterFS too,
but because the provider, where we placed the servers, uses widely CEPH,
we decided to switch too.
Unfortunately I'm new to CEPH, but with help of a technician, we have
running a 3 node CEPH-cluster now, that seems to work fine.
Hardware:
- Xeons, 24 Cores, 256 GB RAM,
2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID)
Software we are using:
- CentOS 7.5.1804
- Kernel: 4.9.86-30.el7 @centos-virt-xen-48
- Xen: 4.8.3-5.el7 @centos-virt-xen-48
- libvirt-xen: 4.1.0-2.xen48.el7 @centos-virt-xen-48
- Ceph: 2:12.2.5-0.el7 @Ceph
CEPH itself works, Xen too and even libvirt (I'm quite new in this too,
normally I use the XL-tools to manage my VMs) ;-) I can manually map an
RBD and start a PV-VM via libvirt, after modifying the XML-config and
defined the VM with libvirt.
That is what happens:
# virsh define xml/testvm3.xml
Domain testvm3 defined from xml/testvm3.xml
# virsh start --console testvm3
error: Failed to start domain testvm3
error: internal error: libxenlight failed to create new domain 'testvm3'
xml/testvm3.xml:
-> ...
<devices>
<disk type='network' device='disk'>
<source protocol='rbd'
name='devel-pool/testvm3.rbd'>
<host name="10.20.30.1" port="6789"/>
<host name="10.20.30.2" port="6789"/>
<host name="10.20.30.3" port="6789"/>
</source>
<auth username='libvirt'>
<secret type='ceph'
uuid='07f3a0fe-9178-xxxxxxxxx'/>
</auth>
<target dev='xvda' bus='xen'/>
</disk>
...
Logs:
/var/log/libvirt/libxl/libxl-driver.log:
-> ...
2018-05-27 11:24:36.666+0000: libxl:
libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
logfile /var/log/xen/bootloader.14.log
2018-05-27 11:24:36.666+0000: libxl:
libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216]
exited with error status 1
2018-05-27 11:24:36.667+0000: libxl:
libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3
/var/log/xen/bootloader.14.log:
->
Traceback (most recent call last):
File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
part_offs = get_partition_offsets(file)
File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
image_type = identify_disk_image(file)
File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:
'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'
So the Xen-Python-script, that searches for the kernel/bootloader of the
VM, does not find the imagefile. But if I grab the above output and use
it with qemu, it does find the blockdevice:
# qemu-img info
"rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789"
->
image:
rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\;none:mon_host=10.20.30.1\:6789\;10.20.30.2\:6789\;10.20.30.3\:6789
file format: raw
virtual size: 16G (17179869184 bytes)
disk size: unavailable
Thus it seems, that CEPH-auth works and the VM-XML-config is interpreted
correct by libvirt. But the next step fails, when libvirt should pass
the blockdevice to Xen?
Anybody out there who uses a similar configuration? Any hints how to
debug this? Unfortunately I do not have anymore logs to look at.
Thank you very much,
--
Kind regards,
thg
Jim Fehlig
2018-May-29 15:42 UTC
Re: [libvirt-users] Using libvirt to access Ceph RBDs with Xen
On 05/27/2018 05:42 AM, thg wrote:> Hi everybody, > > my background: I'm doing Xen since 10++ years, many years with DRBD for > high availability, since some time I'm using preferable GlusterFS with > FUSE as replicated storage, where I place the image-files for the vms. > > In my current project we started (successfully) with Xen/GlusterFS too, > but because the provider, where we placed the servers, uses widely CEPH, > we decided to switch too. > > Unfortunately I'm new to CEPH, but with help of a technician, we have > running a 3 node CEPH-cluster now, that seems to work fine. > > Hardware: > - Xeons, 24 Cores, 256 GB RAM, > 2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID) > > Software we are using: > - CentOS 7.5.1804 > - Kernel: 4.9.86-30.el7 @centos-virt-xen-48 > - Xen: 4.8.3-5.el7 @centos-virt-xen-48 > - libvirt-xen: 4.1.0-2.xen48.el7 @centos-virt-xen-48 > - Ceph: 2:12.2.5-0.el7 @Ceph > > > CEPH itself works, Xen too and even libvirt (I'm quite new in this too, > normally I use the XL-tools to manage my VMs) ;-) I can manually map an > RBD and start a PV-VM via libvirt, after modifying the XML-config and > defined the VM with libvirt. > > > That is what happens: > > # virsh define xml/testvm3.xml > Domain testvm3 defined from xml/testvm3.xml > > # virsh start --console testvm3 > error: Failed to start domain testvm3 > error: internal error: libxenlight failed to create new domain 'testvm3' > > xml/testvm3.xml: > -> ... > <devices> > <disk type='network' device='disk'> > <source protocol='rbd' name='devel-pool/testvm3.rbd'> > <host name="10.20.30.1" port="6789"/> > <host name="10.20.30.2" port="6789"/> > <host name="10.20.30.3" port="6789"/> > </source> > <auth username='libvirt'> > <secret type='ceph' uuid='07f3a0fe-9178-xxxxxxxxx'/> > </auth> > <target dev='xvda' bus='xen'/> > </disk> > ... > > Logs: > > /var/log/libvirt/libxl/libxl-driver.log: > -> ... > 2018-05-27 11:24:36.666+0000: libxl: > libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult > logfile /var/log/xen/bootloader.14.log > 2018-05-27 11:24:36.666+0000: libxl: > libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216] > exited with error status 1 > 2018-05-27 11:24:36.667+0000: libxl: > libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3 > > /var/log/xen/bootloader.14.log: > -> > Traceback (most recent call last): > File "/usr/lib64/xen/bin/pygrub", line 896, in <module> > part_offs = get_partition_offsets(file) > File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets > image_type = identify_disk_image(file) > File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image > fd = os.open(file, os.O_RDONLY) > OSError: [Errno 2] No such file or directory: > 'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'pygrub doesn't work with rbd. But why not use pvgrub instead, which will work with rbd, and doesn't require mounting foreign (possibly hostile) images in dom0? Regards, Jim