Hi, we have a software accessing a lot of small files. We installed Xen 3.2 on a server with two quad core Xeons. We installed two domains with our software on Linux and put it into production. The problem is that I suspect that domains are not running at full (hardware) speed and they are getting stalled from time to time. I can see a lot of wait in top and "top -d .5" gets stalled also from time to time for some seconds (cca 5s) I found discussion that using the current implementation of IO handling can lead to huge delays - up to 30 seconds. http://xen.org/files/xensummit_fall07/13_ScottRixner.pdf xen-de:~# xm info host : xen-de release : 2.6.18.8-xen version : #2 SMP Fri Jun 27 21:27:02 UTC 2008 machine : x86_64 nr_cpus : 8 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 3000 hw_caps : bfebfbff:20100800:00000000:00000140:000ce3bd:00000000:00000001 total_memory : 16382 free_memory : 4 node_to_cpu : node0:0-7 xen_major : 3 xen_minor : 2 xen_extra : .1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21) cc_compile_by : root cc_compile_domain : xlab.si cc_compile_date : Fri Jun 27 23:35:01 CEST 2008 xend_config_format : 4 Could you point me how to improve IO performance? Damjan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Can somebody explain me if there are any issues with high IO ( a lot of DMA requests) usage and Xen? Is it possible that two domains starve each other and then one gets stalled? Is the pdf below relevant? Damjan Pipan -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Damjan Pipan Sent: Tuesday, October 14, 2008 1:49 PM To: xen-users@lists.xensource.com Subject: [Xen-users] domains gets stalled for N seconds Hi, we have a software accessing a lot of small files. We installed Xen 3.2 on a server with two quad core Xeons. We installed two domains with our software on Linux and put it into production. The problem is that I suspect that domains are not running at full (hardware) speed and they are getting stalled from time to time. I can see a lot of wait in top and "top -d .5" gets stalled also from time to time for some seconds (cca 5s) I found discussion that using the current implementation of IO handling can lead to huge delays - up to 30 seconds. http://xen.org/files/xensummit_fall07/13_ScottRixner.pdf xen-de:~# xm info host : xen-de release : 2.6.18.8-xen version : #2 SMP Fri Jun 27 21:27:02 UTC 2008 machine : x86_64 nr_cpus : 8 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 3000 hw_caps : bfebfbff:20100800:00000000:00000140:000ce3bd:00000000:00000001 total_memory : 16382 free_memory : 4 node_to_cpu : node0:0-7 xen_major : 3 xen_minor : 2 xen_extra : .1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21) cc_compile_by : root cc_compile_domain : xlab.si cc_compile_date : Fri Jun 27 23:35:01 CEST 2008 xend_config_format : 4 Could you point me how to improve IO performance? Damjan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
"Damjan Pipan" <damjan.pipan@xlab.si> writes:> The problem is that I suspect that domains are not running at full (hardware) speed and they are getting stalled from time to time. > I can see a lot of wait in top and "top -d .5" gets stalled also from time to time for some seconds (cca 5s)I''ve had similar problems that immediately went away after weighting the Dom0: xm sched-credit -d 0 -w 60000 I now put that in /etc/rc.local on all my Dom0s, and I haven''t seen the problem since. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thank you for your suggestion. I have set the weight on scheduler. I will try today again to run both domains which compete for IO. Damjan -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Luke S Crawford Sent: Wednesday, October 15, 2008 12:32 AM To: Damjan Pipan Cc: xen-users@lists.xensource.com Subject: Re: [Xen-users] domains gets stalled for N seconds "Damjan Pipan" <damjan.pipan@xlab.si> writes:> The problem is that I suspect that domains are not running at full (hardware) speed and they are getting stalled from time totime.> I can see a lot of wait in top and "top -d .5" gets stalled also from > time to time for some seconds (cca 5s)I''ve had similar problems that immediately went away after weighting the Dom0: xm sched-credit -d 0 -w 60000 I now put that in /etc/rc.local on all my Dom0s, and I haven''t seen the problem since. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users