libvirt users - Jun 2013 - Re: [ceph-users] Setting RBD cache parameters for libvirt+qemu

On Jun 7, 2013, at 5:01 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> On 06/07/2013 02:41 PM, John Nielsen wrote:
>> I am running some qemu-kvm virtual machines via libvirt using Ceph RBD
as the back-end storage. Today I was testing an update to libvirt-1.0.6 on one
of my hosts and discovered that it includes this change:
>> 	[libvirt] [PATCH] Forbid use of ':' in RBD pool names
>> 		...People are known to be abusing the lack of escaping in current
libvirt to pass arbitrary args to QEMU.
>> 
>> I am one of those people--I have been storing colon-separated arguments
in my disk source 'name' attribute to control the RBD cache settings on
a per-volume basis. Under libvirt-1.0.6 such VM's refuse to start:
>> error: Failed to start domain foo
>> error: unsupported configuration: ':' not allowed in RBD source
volume name
'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5'
>> 
>> So my question is how do I control the cache settings now?
> 
> You can set the cache mode with the cache attribute on the <driver>
> element inside a <disk> [1]. This works for rbd with QEMU 1.2 or
later.
> 
> QEMU will try to read /etc/ceph/ceph.conf by default as well, so you
> can put more specific settings in a [global] or [client.foo] section
> there. Note that if you enable writeback caching here, you need to
> include cache='writeback' in the xml as well, or qemu will not send
> flushes.
Thanks for the quick reply. Is it safe to assume that the "foo" in
"client.foo" should match the "auth username" attribute from
the XML?

If I want different cache settings for different VM's or volumes on the same
host is there a way to specify that any more? Can I specify multiple profiles in
ceph.conf or point different qemu instances to different ceph.conf files?

JN

On 06/07/2013 04:18 PM, John Nielsen wrote:
> On Jun 7, 2013, at 5:01 PM, Josh Durgin <josh.durgin@inktank.com>
wrote:
>
>> On 06/07/2013 02:41 PM, John Nielsen wrote:
>>> I am running some qemu-kvm virtual machines via libvirt using Ceph
RBD as the back-end storage. Today I was testing an update to libvirt-1.0.6 on
one of my hosts and discovered that it includes this change:
>>> 	[libvirt] [PATCH] Forbid use of ':' in RBD pool names
>>> 		...People are known to be abusing the lack of escaping in current
libvirt to pass arbitrary args to QEMU.
>>>
>>> I am one of those people--I have been storing colon-separated
arguments in my disk source 'name' attribute to control the RBD cache
settings on a per-volume basis. Under libvirt-1.0.6 such VM's refuse to
start:
>>> error: Failed to start domain foo
>>> error: unsupported configuration: ':' not allowed in RBD
source volume name
'mypool/foo:rbd_cache_size=268435456:rbd_cache_max_dirty=134217728:rbd_cache_target_dirty=33554432:rbd_cache_max_dirty_age=5'
>>>
>>> So my question is how do I control the cache settings now?
>>
>> You can set the cache mode with the cache attribute on the
<driver>
>> element inside a <disk> [1]. This works for rbd with QEMU 1.2 or
later.
>>
>> QEMU will try to read /etc/ceph/ceph.conf by default as well, so you
>> can put more specific settings in a [global] or [client.foo] section
>> there. Note that if you enable writeback caching here, you need to
>> include cache='writeback' in the xml as well, or qemu will not
send
>> flushes.
>
> Thanks for the quick reply. Is it safe to assume that the "foo"
in "client.foo" should match the "auth username" attribute
from the XML?
Yes, it should match the "auth usersname" attribute.
> If I want different cache settings for different VM's or volumes on the
same host is there a way to specify that any more? Can I specify multiple
profiles in ceph.conf or point different qemu instances to different ceph.conf
files?
Unfortunately there's no direct way anymore. You could use different
rados clients and put custom settings in e.g. [client.x] and [client.y]
sections of ceph.conf. That's really just a workaround though,
since you probably don't want to create as many clients as you have
sets of cache settings.

There's no way to tell QEMU to read a non-default configuration file
though, and libvirt runs QEMU in a clean environment so using CEPH_ARGS
won't work either.

Daniel and Eric, what do you think of exposing custom per-disk
properties? QEMU did add support for driver-specific options
recently. The rbd driver in QEMU could be modified to use those to
receive ceph-specific options like cache size, and libvirt could
provide a way to set them as opaque key/value pairs, i.e.:

<disk>
...
   <source ...>
     <option name="foo" value="bar"/>
   </source>
</disk>

If you don't want this kind of general pass-through as part of the
main api, maybe it could go in the qemu namespace?

Josh