libvirt users - Jun 2013 - Re: [ceph-users] Setting RBD cache parameters for libvirt+qemu

I am running some qemu-kvm virtual machines via libvirt using Ceph RBD as the
back-end storage. Today I was testing an update to libvirt-1.0.6 on one of my
hosts and discovered that it includes this change:
	[libvirt] [PATCH] Forbid use of ':' in RBD pool names
		...People are known to be abusing the lack of escaping in current libvirt to
pass arbitrary args to QEMU.

I am one of those people--I have been storing colon-separated arguments in my
disk source 'name' attribute to control the RBD cache settings on a
per-volume basis. Under libvirt-1.0.6 such VM's refuse to start:
error: Failed to start domain foo
error: unsupported configuration: ':' not allowed in RBD source volume
name
'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5'

So my question is how do I control the cache settings now?

Thanks,

JN

On 06/07/2013 02:41 PM, John Nielsen wrote:
> I am running some qemu-kvm virtual machines via libvirt using Ceph RBD as
the back-end storage. Today I was testing an update to libvirt-1.0.6 on one of
my hosts and discovered that it includes this change:
> 	[libvirt] [PATCH] Forbid use of ':' in RBD pool names
> 		...People are known to be abusing the lack of escaping in current libvirt
to pass arbitrary args to QEMU.
>
> I am one of those people--I have been storing colon-separated arguments in
my disk source 'name' attribute to control the RBD cache settings on a
per-volume basis. Under libvirt-1.0.6 such VM's refuse to start:
> error: Failed to start domain foo
> error: unsupported configuration: ':' not allowed in RBD source
volume name
'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5'
>
> So my question is how do I control the cache settings now?
You can set the cache mode with the cache attribute on the <driver>
element inside a <disk> [1]. This works for rbd with QEMU 1.2 or later.

QEMU will try to read /etc/ceph/ceph.conf by default as well, so you
can put more specific settings in a [global] or [client.foo] section
there. Note that if you enable writeback caching here, you need to
include cache='writeback' in the xml as well, or qemu will not send
flushes.

Josh

[1] http://libvirt.org/formatdomain.html#elementsDisks

On Jun 7, 2013, at 5:01 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> On 06/07/2013 02:41 PM, John Nielsen wrote:
>> I am running some qemu-kvm virtual machines via libvirt using Ceph RBD
as the back-end storage. Today I was testing an update to libvirt-1.0.6 on one
of my hosts and discovered that it includes this change:
>> 	[libvirt] [PATCH] Forbid use of ':' in RBD pool names
>> 		...People are known to be abusing the lack of escaping in current
libvirt to pass arbitrary args to QEMU.
>> 
>> I am one of those people--I have been storing colon-separated arguments
in my disk source 'name' attribute to control the RBD cache settings on
a per-volume basis. Under libvirt-1.0.6 such VM's refuse to start:
>> error: Failed to start domain foo
>> error: unsupported configuration: ':' not allowed in RBD source
volume name
'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5'
>> 
>> So my question is how do I control the cache settings now?
> 
> You can set the cache mode with the cache attribute on the <driver>
> element inside a <disk> [1]. This works for rbd with QEMU 1.2 or
later.
> 
> QEMU will try to read /etc/ceph/ceph.conf by default as well, so you
> can put more specific settings in a [global] or [client.foo] section
> there. Note that if you enable writeback caching here, you need to
> include cache='writeback' in the xml as well, or qemu will not send
> flushes.
Thanks for the quick reply. Is it safe to assume that the "foo" in
"client.foo" should match the "auth username" attribute from
the XML?

If I want different cache settings for different VM's or volumes on the same
host is there a way to specify that any more? Can I specify multiple profiles in
ceph.conf or point different qemu instances to different ceph.conf files?

JN