CentOS virt - Jun 2015 - /var/lib/xenstored & Xen4CentOS

By default, /var/lib/xenstored is mounted tmpfs in centos 5 but this doesn't
appear to be true for Xen4CentOS. This can cause performance issues as
mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=446089

I'm not sure if this should be part of xencommons, but if not, adding a
tmpfs mount should probably be documented on the quick start wiki page.

Thanks, Sarah

On Thu, Jun 4, 2015 at 6:03 AM, Sarah Newman <srn at prgmr.com>
wrote:
> By default, /var/lib/xenstored is mounted tmpfs in centos 5 but this
doesn't appear to be true for Xen4CentOS. This can cause performance issues
as
> mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=446089
>
> I'm not sure if this should be part of xencommons, but if not, adding a
tmpfs mount should probably be documented on the quick start wiki page.
Sarah,

Thanks for looking into this.

Adding a tmpfs mount to xencommons, if using the C xenstored, might be
a good idea -- I'll float that with upstream and see what they say.

As an alternate, can you try using the ocaml-based oxenstored (which
doesn't use the filesystem as its atomicity mechanism)?

To use oxenstored:
* Install the xen-ocaml package
* Reboot dom0
* Verify that oxenstored is running:
# ps ax | grep xenstore
  654 ?        S      0:00 /usr/sbin/oxenstored --pid-file
/var/run/xenstored.pid

(On my to-do list is to get oxenstored into the core xen-runtime
package, which would also resolve this situation, I believe.)

Thanks,
 -George

On Thu, Jun 4, 2015 at 11:57 AM, George Dunlap <dunlapg at umich.edu>
wrote:
> On Thu, Jun 4, 2015 at 6:03 AM, Sarah Newman <srn at prgmr.com>
wrote:
>> By default, /var/lib/xenstored is mounted tmpfs in centos 5 but this
doesn't appear to be true for Xen4CentOS. This can cause performance issues
as
>> mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=446089
>>
>> I'm not sure if this should be part of xencommons, but if not,
adding a tmpfs mount should probably be documented on the quick start wiki page.
>
> Sarah,
>
> Thanks for looking into this.
>
> Adding a tmpfs mount to xencommons, if using the C xenstored, might be
> a good idea -- I'll float that with upstream and see what they say.
>
> As an alternate, can you try using the ocaml-based oxenstored (which
> doesn't use the filesystem as its atomicity mechanism)?
>
> To use oxenstored:
> * Install the xen-ocaml package
> * Reboot dom0
> * Verify that oxenstored is running:
> # ps ax | grep xenstore
>   654 ?        S      0:00 /usr/sbin/oxenstored --pid-file
> /var/run/xenstored.pid
>
> (On my to-do list is to get oxenstored into the core xen-runtime
> package, which would also resolve this situation, I believe.)
FWIW, I used the following shell snippet to create 80 VMs (which crash
immediately without an initrd but still take up space in xenstore):

for i in {1..80} ; do xl create null.cfg name=\"t$i\"
kernel=\"/boot/vmlinuz-3.18.12-11.el7.x86_64\" memory=\"40\"
on_crash=\"preserve\" ; done

And then used the shell snippet from the bug report to generate load:

while true; do xl list &> /dev/null; usleep 5000;done

With xenstored normally the CPU load was about 30%.  Mounted under
tmpfs, it was slightly smaller, around 20%, but swung around wildly.
With oxenstored it was about 15%.

This is with xfs as the filesystem.

That doesn't seem big enough to warrant a lot of effort: if you can
reproduce the 75% numbers from the bug report I'll pursue it further,
otherwise I'll probably just leave it.

 -George