Xen users - Aug 2009 - High iowait in dom0 after creating a new guest, with file-based domU disk on GFS

Hopefully I summed up the jist of the problem in the subject line. ;)

I have a GFS cluster with ten Xen 3.0 dom0s sharing an iSCSI LUN.  There are
on average 8 domUs running on each Xen server.  The dom0 on each server is
hard-coded to 2 CPUs and 2GB of RAM with no ballooning, and has 2GB of
partition-based swap.

When creating a new domU on any of the Xen servers, just after the
completion of the operating system kickstart install as the new domU
completes its shutdown process, the dom0 will begin experiencing high load
and an iowait that consumes 100% of one of the CPUs.  This will last for
several minutes (10-15 generally), and the shutdown of the new domU will
usually hang during this time (xm console will not exit, and the domain will
continue running until the iowait drops).  No swap space will be consumed by
the dom0 OS, and no major disk activity will be apparent in iostat.
Watching the threads in top, the xenstored processes will consistently hang
out at the top of the process list, but without apparently consuming CPU
time (%cpu shows 0).  Nothing of much interest seems to show up in the
syslog.


dom0 syslog during the domU install time (minus audit entries):
Aug  6 17:40:25 xen03 kernel: device vif11.0 entered promiscuous mode
Aug  6 17:40:25 xen03 kernel: ADDRCONF(NETDEV_UP): vif11.0: link is not
ready
Aug  6 17:40:34 xen03 kernel: ADDRCONF(NETDEV_CHANGE): vif11.0: link becomes
ready
Aug  6 17:40:34 xen03 kernel: xenbr11: topology change detected, propagating
Aug  6 17:40:34 xen03 kernel: xenbr11: port 9(vif11.0) entering forwarding
state
Aug  6 17:40:34 xen03 kernel: blkback: ring-ref 522, event-channel 10,
protocol 2 (x86_32-abi)
Aug  6 17:43:26 xen03 kernel: xenbr11: port 9(vif11.0) entering disabled
state
Aug  6 17:43:26 xen03 kernel: device vif11.0 left promiscuous mode
Aug  6 17:43:26 xen03 kernel: xenbr11: port 9(vif11.0) entering disabled
state
(-- hang happens about here, and no further syslog messages are recorded
after iowait trails off--)


Any thoughts?


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users