libvirt users - Oct 2011 - Performance tuning questions for mail server

Hi,

I have a fedora15 x86_64 host with one fedora15 guest running
amavis+spamassassin+postfix and performance is horrible. The host is a
quad-core E13240 with 16GB and 3 1TB Seagate ST31000524NS and all
partitions are ext4. I've allocated 4 processors and 8GB of RAM to
this guest. I really hoped someone could help me identify areas in
which performance can be improved at both the guest and host.

Load on the server is regularly above 20, yet the processors generally
are idle and the host is still responsive.

Should the concentration of the performance tuning be done on the
guest or the host? I've read through the information at
http://libvirt.org/formatdomain.html#elementsMemoryTuning but I don't
know how the settings apply to my configuration and which ones apply
to my hardware.

I've included my libvirt xml config below. It was built using
virt-manager on fedora15. There appears to be quite a few other
options that are available and not provided by virt-manager that I
would like to be able to use.

I've done a little kernel tuning on the host, although it doesn't
appear to have made much difference. Are there a set of values for
kernel parameters that would be advisable to make for a mail server?
Here is what I currently have:

# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
vm.vfs_cache_pressure = 35
vm.nr_hugepages = 512
net.ipv4.tcp_max_syn_backlog = 2048
fs.aio-max-nr = 1048576
vm.dirty_background_ratio = 3
vm.dirty_ratio = 40

After making changes, do you have any recommendations on which tools
to use to monitor those changes and see how they perform?

I have noatime set in fstab in the guest for the /var partition, where
much of the spamassassin  occurs.

I've included below my libvirt xml config for the guest and hoped
someone could make some recommendations on how to apply the cputune
and memtune parameters to my system.

<domain type='kvm'>
  <name>mail02</name>
  <uuid>ec4f3cf5-2f27-fb3e-72f6-3fa3176b13b6</uuid>
  <memory>8388608</memory>
  <currentMemory>8388608</currentMemory>
  <vcpu>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.14'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/mail02.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1'
unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:67:2c:4c'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1'
autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
function='0x0'/>
    </memballoon>
  </devices>
</domain>

Thanks,
Alex

On Wed, Oct 05, 2011 at 02:28:30PM -0400, Alex wrote:
> 
> Load on the server is regularly above 20, yet the processors
> generally are idle and the host is still responsive.
That's completely normal for an email server running spamassassin,
in my experience, and has nothing to do with libvirt.  IME, the
issue is DNS lookups, which spamassassin and the RBLs it uses make
*heavy* use of.  Run a caching DNS on localhost with fairly
agressive caching (i.e. go ahead and have it ignore TTLs and keep
everything for at least 5 minutes) and I expect that you'll find
that problem being rapidly resolved.

-Robin