Hi I have noticed that VMs are typically spending 3 microsecs or less before they are being prempted. I thought that the credit schduler time slice was 30 ms. I have 4 VMs running and they are all cpu intensive except for 1 (which is IO intensive) but having a VM spend max 3 micro secs before being kicked out seems strange. Is there something else going on that i am not aware of. Is the time slice really 30 millisecs? I am using default parameters of the credit scheduler. Thanks -Prabha _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
How many physical CPUs do you have, and what is the IO workload that one of the VMs is running? -- Keir On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:> Hi > > I have noticed that VMs are typically spending 3 microsecs or less before they > are being prempted. I thought that the credit schduler time slice was 30 ms. I > have 4 VMs running and they are all cpu intensive except for 1 (which is IO > intensive) but having a VM spend max 3 micro secs before being kicked out > seems strange. > > Is there something else going on that i am not aware of. Is the time slice > really 30 millisecs? I am using default parameters of the credit scheduler. > > Thanks > -Prabha > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim with 500 users Thanks -Prabha -------------- Original message ---------------------- From: Keir Fraser <keir@xensource.com>> How many physical CPUs do you have, and what is the IO workload that one of > the VMs is running? > > -- Keir > > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote: > > > Hi > > > > I have noticed that VMs are typically spending 3 microsecs or less before they > > are being prempted. I thought that the credit schduler time slice was 30 ms. I > > have 4 VMs running and they are all cpu intensive except for 1 (which is IO > > intensive) but having a VM spend max 3 micro secs before being kicked out > > seems strange. > > > > Is there something else going on that i am not aware of. Is the time slice > > really 30 millisecs? I am using default parameters of the credit scheduler. > > > > Thanks > > -Prabha > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Are these all Windows HVM guests? And do you exclude domain0 when you count the number of VMs in your system (so it¹s actually 4 user VMs + domain0 VM)? How did you come up with the 3us figure some kind of in-guest tracing, or using xentrace, or by some other means? Does the problem still happen if you run 2 cpu-intensive VMs instead of 3? -- Keir On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote:> > I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim > with 500 users > > Thanks > -Prabha > > > -------------- Original message ---------------------- > From: Keir Fraser <keir@xensource.com> >> > How many physical CPUs do you have, and what is the IO workload that one of >> > the VMs is running? >> > >> > -- Keir >> > >> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote: >> > >>> > > Hi >>> > > >>> > > I have noticed that VMs are typically spending 3 microsecs or less >>> before they >>> > > are being prempted. I thought that the credit schduler time slice was 30 >>> ms. I >>> > > have 4 VMs running and they are all cpu intensive except for 1 (which isIO>>> > > intensive) but having a VM spend max 3 micro secs before being kickedout>>> > > seems strange. >>> > > >>> > > Is there something else going on that i am not aware of. Is the time >>>slice>>> > > really 30 millisecs? I am using default parameters of the credit >>> scheduler. >>> > > >>> > > Thanks >>> > > -Prabha >>> > > >>> > > >>> > > >>> > > _______________________________________________ >>> > > Xen-devel mailing list >>> > > Xen-devel@lists.xensource.com >>> > > http://lists.xensource.com/xen-devel >> > >> > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yes all these are HVM guests. Yes there is dom0 and 4 other VMs. I added some trace statements in the schedule.c function and then postprocessed that data to come with the 3us example I have 16400 lines printed with xentrace during a span of 52 ms. Which means that i have entered the scheduler 16400 times during the 52ms time interval, which is roughly 3.2 us. i have not yet tried any other combination. my understanding of the schduler is that the scheduler kicks in on a timer interrupt or a block of any VM. so in a all cpu intensive VMs the VM would run interrupted for 30ms(credit schduler cap). since we have IO, whenever a VM needs IO service, it blocks and domain 0 kicks in and services the IO. is this correct? What I also see is that dom0 cpu utilization is only 28% across all 4 cpus. Assuming that the scheduler was entered because of an IO request, dom0 kicks in but spends very little time servicing the request. Dom0 kicks in 6800 times of the 16400 times that the scheduler function was called. Does this make sense? Thanks -Prabha -------------- Original message ---------------------- From: Keir Fraser <keir@xensource.com>> Are these all Windows HVM guests? And do you exclude domain0 when you count > the number of VMs in your system (so it¹s actually 4 user VMs + domain0 VM)? > How did you come up with the 3us figure ‹ some kind of in-guest tracing, or > using xentrace, or by some other means? > > Does the problem still happen if you run 2 cpu-intensive VMs instead of 3? > > -- Keir > > On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote: > > > > > I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim > > with 500 users > > > > Thanks > > -Prabha > > > > > > -------------- Original message ---------------------- > > From: Keir Fraser <keir@xensource.com> > >> > How many physical CPUs do you have, and what is the IO workload that one of > >> > the VMs is running? > >> > > >> > -- Keir > >> > > >> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote: > >> > > >>> > > Hi > >>> > > > >>> > > I have noticed that VMs are typically spending 3 microsecs or less > >>> before they > >>> > > are being prempted. I thought that the credit schduler time slice was 30 > >>> ms. I > >>> > > have 4 VMs running and they are all cpu intensive except for 1 (which is > IO > >>> > > intensive) but having a VM spend max 3 micro secs before being kicked > out > >>> > > seems strange. > >>> > > > >>> > > Is there something else going on that i am not aware of. Is the time >>> > slice > >>> > > really 30 millisecs? I am using default parameters of the credit > >>> scheduler. > >>> > > > >>> > > Thanks > >>> > > -Prabha > >>> > > > >>> > > > >>> > > > >>> > > _______________________________________________ > >>> > > Xen-devel mailing list > >>> > > Xen-devel@lists.xensource.com > >>> > > http://lists.xensource.com/xen-devel > >> > > >> > > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > >--NextPart_Webmail_9m3u9jl4l_485_1182909331_0 Content-Type: message/rfc822 From: Keir Fraser <keir@xensource.com> To: <pak333@comcast.net>, <xen-devel@lists.xensource.com> Subject: Re: [Xen-devel] Credit scheduler Date: Wed, 27 Jun 2007 01:17:50 +0000 Content-Type: Multipart/mixed; boundary="NextPart_Webmail_9m3u9jl4l_485_1182909331_1" --NextPart_Webmail_9m3u9jl4l_485_1182909331_1 Content-type: multipart/alternative; boundary="B_3265755453_7089961"> This message is in MIME format. Since your mail reader does not understandthis format, some or all of this message may not be legible. --B_3265755453_7089961 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Are these all Windows HVM guests? And do you exclude domain0 when you count the number of VMs in your system (so it=B9s actually 4 user VMs + domain0 VM)? How did you come up with the 3us figure =8B some kind of in-guest tracing, or using xentrace, or by some other means? Does the problem still happen if you run 2 cpu-intensive VMs instead of 3? -- Keir On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote:>=20 > I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim > with 500 users >=20 > Thanks > -Prabha >=20 >=20 > -------------- Original message ---------------------- > From: Keir Fraser <keir@xensource.com> >> > How many physical CPUs do you have, and what is the IO workload that one of >> > the VMs is running? >> >=20 >> > -- Keir >> >=20 >> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote: >> >=20 >>> > > Hi >>> > > =20 >>> > > I have noticed that VMs are typically spending 3 microsecs or less >>> before they >>> > > are being prempted. I thought that the credit schduler time slice was 30 >>> ms. I >>> > > have 4 VMs running and they are all cpu intensive except for 1 (which isIO>>> > > intensive) but having a VM spend max 3 micro secs before being kickedout>>> > > seems strange. >>> > > =20 >>> > > Is there something else going on that i am not aware of. Is the time >>>slice>>> > > really 30 millisecs? I am using default parameters of the credit >>> scheduler. >>> > > =20 >>> > > Thanks >>> > > -Prabha >>> > > =20 >>> > >=20 >>> > >=20 >>> > > _______________________________________________ >>> > > Xen-devel mailing list >>> > > Xen-devel@lists.xensource.com >>> > > http://lists.xensource.com/xen-devel >> >=20 >> >=20 >=20 >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel--B_3265755453_7089961 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable <HTML> <HEAD> <TITLE>Re: [Xen-devel] Credit scheduler</TITLE> </HEAD> <BODY> <FONT FACE=3D"Verdana, Helvetica, Arial"><SPAN STYLE=3D''font-size:12.0px''>Are these all Windows HVM guests? And do you exclude domain0 when you count the number of VMs in your system (so it’s actually 4 user VMs + domain0 VM)? How did you come up with the 3us figure — some kind of in-guest tracing, or using xentrace, or by some other means?<BR> <BR> Does the problem still happen if you run 2 cpu-intensive VMs instead of 3?<BR> <BR> -- Keir<BR> <BR> On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT SIZE=3D"2"><FONT FACE=3D"Monaco, Courier New"><SPAN STYLE=3D''font-size:10.0px''><BR> I have 4 physical cpus (2 dual core sockets) and the IO workload is loadsim with 500 users<BR> <BR> Thanks<BR> -Prabha<BR> <BR> <BR> -------------- Original message ----------------------<BR> From: Keir Fraser <keir@xensource.com><BR> > How many physical CPUs do you have, and what is the IO workload that one of<BR> > the VMs is running?<BR> > <BR> > -- Keir<BR> > <BR> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote:<BR> > <BR> > > Hi<BR> > > <BR> > > I have noticed that VMs are typically spending 3 microsecs or less before they<BR> > > are being prempted. I thought that the credit schduler time slice was 30 ms. I<BR> > > have 4 VMs running and they are all cpu intensive except for 1 (which is IO<BR> > > intensive) but having a VM spend max 3 micro secs before being kicked out<BR> > > seems strange.<BR> > > <BR> > > Is there something else going on that i am not aware of. Is the time slice<BR> > > really 30 millisecs? I am using default parameters of the credit scheduler.<BR> > > <BR> > > Thanks<BR> > > -Prabha<BR> > > <BR> > > <BR> > > <BR> > > _______________________________________________<BR> > > Xen-devel mailing list<BR> > > Xen-devel@lists.xensource.com<BR> > > <a href=3D"http://lists.xensource.com/xen-devel">http://lists.xensource.com/xen-devel</a><BR> > <BR> > <BR> <BR> <BR> <HR ALIGN=3DCENTER SIZE=3D"3" WIDTH=3D"95%">_______________________________________________<BR> Xen-devel mailing list<BR> Xen-devel@lists.xensource.com<BR> <a href=3D"http://lists.xensource.com/xen-devel">http://lists.xensource.com/xen-devel</a><BR> </SPAN></FONT></FONT></BLOCKQUOTE><FONT SIZE=3D"2"><FONT FACE=3D"Monaco, Courier New"><SPAN STYLE=3D''font-size:10.0px''><BR> </SPAN></FONT></FONT> </BODY> </HTML> --B_3265755453_7089961-- --NextPart_Webmail_9m3u9jl4l_485_1182909331_1 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --NextPart_Webmail_9m3u9jl4l_485_1182909331_1-- --NextPart_Webmail_9m3u9jl4l_485_1182909331_0 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --NextPart_Webmail_9m3u9jl4l_485_1182909331_0--
Hi, Prabha 30msec is the maximum time slice. And I guess your 3 microsecond is the response after SCHEDULE_SOFTIRQ is set. As you know, If SCHEDULE_SOFTIRQ is set, it waits softirq calls schedule(). (this time interval is 3microsec) And I/O intensive has higher priority than CPU intensive. So I/O intensive job is first dispatched domain in runq. This is because latency improvement for I/O intensive guest. So your behavior is not strange. Thanks Atsushi SAKAI pak333@comcast.net wrote:> Hi > > I have noticed that VMs are typically spending 3 microsecs or less before they are being prempted. I thought that the credit schduler time slice was 30 ms. I have 4 VMs running and they are all cpu intensive except for 1 (which is IO intensive) but having a VM spend max 3 micro secs before being kicked out seems strange. > > Is there something else going on that i am not aware of. Is the time slice really 30 millisecs? I am using default parameters of the credit scheduler. > > Thanks > -Prabha_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks for the explanation, i am not sure i understood it correctly. when does softirq get set. So when softirq is set a processor enters the scheduler every 3 micro secs? where in the source code is this handled. please could you send me some pointers thanks -Prabha -------------- Original message -------------- From: Atsushi SAKAI <sakaia@jp.fujitsu.com>> Hi,$B!!(BPrabha > > $B!!(B30msec is the maximum time slice. > And I guess your 3 microsecond is the response after > SCHEDULE_SOFTIRQ is set. > As you know, > If SCHEDULE_SOFTIRQ is set, > it waits softirq calls schedule(). (this time interval is 3microsec) > > And I/O intensive has higher priority than CPU intensive. > So I/O intensive job is first dispatched domain in runq. > This is because latency improvement for I/O intensive guest. > > So your behavior is not strange. > > Thanks > Atsushi SAKAI > > > pak333@comcast.net wrote: > > > Hi > > > > I have noticed that VMs are typically spending 3 microsecs or less before they > are being prempted. I thought that the credit schduler time slice was 30 ms. I > have 4 VMs running and they are all cpu intensive except for 1 (which is IO > intensive) but having a VM spend max 3 micro secs before being kicked out seems > strange. > > > > Is there something else going on that i am not aware of. Is the time slice > really 30 millisecs? I am using default parameters of the credit scheduler. > > > > Thanks > > -Prabha > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Prabha Please check do_softirq in assembler. for x86 call do_softirq is in. And usually do_softirq is executed at the end of system call. Thanks Atsushi SAKAI pak333@comcast.net wrote:> Thanks for the explanation, i am not sure i understood it correctly. > > > when does softirq get set. So when softirq is set a processor enters the scheduler every 3 micro secs? where in the source code is this handled. please could you send me some pointers > > thanks > -Prabha > > > > > -------------- Original message -------------- > From: Atsushi SAKAI <sakaia@jp.fujitsu.com> > > > Hi, Prabha > > > > 30msec is the maximum time slice. > > And I guess your 3 microsecond is the response after > > SCHEDULE_SOFTIRQ is set. > > As you know, > > If SCHEDULE_SOFTIRQ is set, > > it waits softirq calls schedule(). (this time interval is 3microsec) > > > > And I/O intensive has higher priority than CPU intensive. > > So I/O intensive job is first dispatched domain in runq. > > This is because latency improvement for I/O intensive guest. > > > > So your behavior is not strange. > > > > Thanks > > Atsushi SAKAI > > > > > > pak333@comcast.net wrote: > > > > > Hi > > > > > > I have noticed that VMs are typically spending 3 microsecs or less before they > > are being prempted. I thought that the credit schduler time slice was 30 ms. I > > have 4 VMs running and they are all cpu intensive except for 1 (which is IO > > intensive) but having a VM spend max 3 micro secs before being kicked out seems > > strange. > > > > > > Is there something else going on that i am not aware of. Is the time slice > > really 30 millisecs? I am using default parameters of the credit scheduler. > > > > > > Thanks > > > -Prabha > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
You¹ll probably do better by pinning dom0 to cpu0 and all other domains to all other cpus. -- Keir On 27/6/07 02:55, "pak333@comcast.net" <pak333@comcast.net> wrote:> > > Yes all these are HVM guests. > Yes there is dom0 and 4 other VMs. > > I added some trace statements in the schedule.c function and then > postprocessed that data to come with the 3us > > example I have 16400 lines printed with xentrace during a span of 52 ms. Which > means that i have entered the scheduler 16400 times during the 52ms time > interval, which is roughly 3.2 us. > > i have not yet tried any other combination. > > my understanding of the schduler is that the scheduler kicks in on a timer > interrupt or a block of any VM. > so in a all cpu intensive VMs the VM would run interrupted for 30ms(credit > schduler cap). > since we have IO, whenever a VM needs IO service, it blocks and domain 0 kicks > in and services the IO. > is this correct? > > What I also see is that dom0 cpu utilization is only 28% across all 4 cpus. > Assuming that the scheduler was entered because of an IO request, dom0 kicks > in but spends very little time servicing the request. Dom0 kicks in 6800 times > of the 16400 times that the scheduler function was called. Does this make > sense? > > Thanks > -Prabha > > > > > > > > > > -------------- Original message ---------------------- > From: Keir Fraser <keir@xensource.com> >> > Are these all Windows HVM guests? And do you exclude domain0 when you count >> > the number of VMs in your system (so it¹s actually 4 user VMs + domain0 >> VM)? >> > How did you come up with the 3us figure some kind of in-guest tracing, or >> > using xentrace, or by some other means? >> > >> > Does the problem still happen if you run 2 cpu-intensive VMs instead of 3? >> > >> > -- Keir >> > >> > On 27/6/07 02:09, "pak333@comcast.net" <pak333@comcast.net> wrote: >> > >>> > > >>> > > I have 4 physical cpus (2 dual core sockets) and the IO workload is >>> loadsim >>> > > with 500 users >>> > > >>> > > Thanks >>> > > -Prabha >>> > > >>> > > >>> > > -------------- Original message ---------------------- >>> > > From: Keir Fraser <keir@xensource.com> >>>>> > >> > How many physical CPUs do you have, and what is the IO workload >>>>> that one of >>>>> > >> > the VMs is running? >>>>> > >> > >>>>> > >> > -- Keir >>>>> > >> > >>>>> > >> > On 26/6/07 22:33, "pak333@comcast.net" <pak333@comcast.net> wrote: >>>>> > >> > >>>>>>> > >>> > > Hi >>>>>>> > >>> > > >>>>>>> > >>> > > I have noticed that VMs are typically spending 3 microsecs orless>>>>> > >>> before they >>>>>>> > >>> > > are being prempted. I thought that the credit schduler time >>>>>>> slice was 30 >>>>> > >>> ms. I >>>>>>> > >>> > > have 4 VMs running and they are all cpu intensive except for 1(which is>> > IO >>>>>>> > >>> > > intensive) but having a VM spend max 3 micro secs before beingkicked>> > out >>>>>>> > >>> > > seems strange. >>>>>>> > >>> > > >>>>>>> > >>> > > Is there something else going on that i am not aware of. Is >>>>>>> the time >>> >> > slice >>>>>>> > >>> > > really 30 millisecs? I am using default parameters of thecredit>>>>> > >>> scheduler. >>>>>>> > >>> > > >>>>>>> > >>> > > Thanks >>>>>>> > >>> > > -Prabha >>>>>>> > >>> > > >>>>>>> > >>> > > >>>>>>> > >>> > > >>>>>>> > >>> > > _______________________________________________ >>>>>>> > >>> > > Xen-devel mailing list >>>>>>> > >>> > > Xen-devel@lists.xensource.com >>>>>>> > >>> > > http://lists.xensource.com/xen-devel >>>>> > >> > >>>>> > >> > >>> > > >>> > > >>> > > >>> > > _______________________________________________ >>> > > Xen-devel mailing list >>> > > Xen-devel@lists.xensource.com >>> > > http://lists.xensource.com/xen-devel >> > >> > >> > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Atsushi I am still confused, so let me explain what i think should happen and understand from you why it may not happen. I have 2 cpu intensive VMs and 1 IO intensive VM. My system has 4 physical cpus and 8 virtual cpus. Now using default parameters for the credit scheduler, the cpu intersive vcpu should run for 30 millsec. It does not, it runs for less (few microsecs). You said this could be because of softirq which are raised after a system call. Can i disable the softirqs. What happens if i do? If i cannot disable is there a way to see what VM is raising the softirq. If the cpu-VM raises the softirq it gets preempted or does it continue to run. How can I montior the softirqs raised by this VM. Can I tell if the premption is from a softirq or from something else. Also, if the IO VM requests an IO, it will block and dom0 wakes up and gets scheduled to run to service the IO. Will it preempt the cpu intensive VM. If so why? Shouldn''t the cpuintensiveV M get its quantum of time. Or does the IOVM get higher priority to preempt the cpu intensive VM. How does the scheduler pick which cpu to run dom0 on ( if all the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus and Io-vcpus, which will be the victim As you see i have a lot of questions, so please bear with me and help me understand the scheduling behavior of Xen Thanks Prabha -------------- Original message -------------- From: Atsushi SAKAI <sakaia@jp.fujitsu.com>> Hi Prabha > > Please check do_softirq in assembler. > for x86 > call do_softirq > is in. > > And usually do_softirq is executed at the end of system call. > > Thanks > Atsushi SAKAI > > > > pak333@comcast.net wrote: > > > Thanks for the explanation, i am not sure i understood it correctly. > > > > > > when does softirq get set. So when softirq is set a processor enters the > scheduler every 3 micro secs? where in the source code is this handled. please > could you send me some pointers > > > > thanks > > -Prabha > > > > > > > > > > -------------- Original message -------------- > > From: Atsushi SAKAI > > > > > Hi,$B!!(BPrabha > > > > > > $B!!(B30msec is the maximum time slice. > > > And I guess your 3 microsecond is the response after > > > SCHEDULE_SOFTIRQ is set. > > > As you know, > > > If SCHEDULE_SOFTIRQ is set, > > > it waits softirq calls schedule(). (this time interval is 3microsec) > > > > > > And I/O intensive has higher priority than CPU intensive. > > > So I/O intensive job is first dispatched domain in runq. > > > This is because latency improvement for I/O intensive guest. > > > > > > So your behavior is not strange. > > > > > > Thanks > > > Atsushi SAKAI > > > > > > > > > pak333@comcast.net wrote: > > > > > > > Hi > > > > > > > > I have noticed that VMs are typically spending 3 microsecs or less before > they > > > are being prempted. I thought that the credit schduler time slice was 30 ms. > I > > > have 4 VMs running and they are all cpu intensive except for 1 (which is IO > > > intensive) but having a VM spend max 3 micro secs before being kicked out > seems > > > strange. > > > > > > > > Is there something else going on that i am not aware of. Is the time slice > > > really 30 millisecs? I am using default parameters of the credit scheduler. > > > > > > > > Thanks > > > > -Prabha > > > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 21:46 12/07/2007, pak333@comcast.net wrote:>Hi Atsushi > >I am still confused, so let me explain what i think should happen >and understand from you why it may not happen. > >I have 2 cpu intensive VMs and 1 IO intensive VM. My system has 4 >physical cpus and 8 virtual cpus. > >Now using default parameters for the credit scheduler, the cpu >intersive vcpu should run for 30 millsec. It does not, it runs for >less (few microsecs). > >You said this could be because of softirq which are raised after a >system call. Can i disable the softirqs. What happens if i do? If i >cannot disable is there a way to see what VM is raising the softirq. >If the cpu-VM raises the softirq it gets preempted or does it >continue to run. How can I montior the softirqs raised by this VM. >Can I tell if the premption is from a softirq or from something else.I don''t believe you can "disable" a softirq - you could of course avoid doing the call to "do_softirq", but that is most likely just going to completely remove any scheduling capability at all in the system, making the system "single-tasking". Not a recommended way, in my opinion.> >Also, if the IO VM requests an IO, it will block and dom0 wakes up >and gets scheduled to run to service the IO. Will it preempt the cpu >intensive VM. If so why? Shouldn''t the cpuintensiveV M get its >quantum of time. >Or does the IOVM get higher priority to preempt the cpu intensive >VM. How does the scheduler pick which cpu to run dom0 on ( if all >the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus >and Io-vcpus, which will be the victimThe whole idea of having separate queues for IO and CPU intensive VMs (or processes in a normal kernel scenario) is to allow the IO-intensive tasks to avoid waiting for the next time-slot when a CPU-hogging VM/process is running [1]. The normal behaviour for the scheduler is to determine dynamically if the current VM/process is IO or CPU intensive. Unless Dom0 is used for doing some silly CPU-intensive task (calculating PI with a million decimal points or compiling Xen + Linux kernel, for example), it will most likely look to the scheduler like a IO-intensive VM. So it will have the same priority as any other IO-intensive VM (assuming Dom0 has equal scheduler parameters as other guests, which I''m pretty sure is the default behaviour). If, like in your example, there are a number of processors busy with CPU-intensive tasks, and others with IO-intensive tasks, the most likely scenario is that any new IO-request will go be run on a "previously CPU-intensive" CPU - but on the other hand, if you have a lot of IO-intensive processing, there should be some processor(s) that are "asleep" - unless all CPU''s are busy running CPU-intensive tasks... Note that scheduling is a complex subject, and there have been dozens of PhD dissertations written on the subject. [1] The idea being that if we process the IO-request that just completed, we can set up another IO-request, and this will get better IO-throughput than waiting a long time (relatively speaking 30ms is an eternity to the processor). -- Mats> >As you see i have a lot of questions, so please bear with me and >help me understand the scheduling behavior of Xen > >Thanks >Prabha > > > > >-------------- Original message -------------- >From: Atsushi SAKAI <sakaia@jp.fujitsu.com> > > > Hi Prabha > > > > Please check do_softirq in assembler. > > for x86 > > call do_softirq > > is in. > > > > And usually do_softirq is executed at the end of system call. > > > > Thanks > > Atsushi SAKAI > > > > > > > > pak333@comcast.net wrote: > > > > > Thanks for the explanation, i am not sure i understood it correctly. > > > > > > > > > when does softirq get set. So when softirq is set a processor enters the > > scheduler every 3 micro secs? where in the source code is this > handled. please > > could you send me some pointers > > > > > > thanks > > > -Prabha > > > > > > > > > > > > > > > -------------- Or iginal message -------------- > > > From: Atsushi SAKAI > > > > > > > Hi,$B!!(BPrabha > > > > > > > > $B!!(B30msec is the maximum time slice. > > > > And I guess your 3 microsecond is the response after > > > > SCHEDULE_SOFTIRQ is set. > > > > As you know, > > > > If SCHEDULE_SOFTIRQ is set, > > > > it waits softirq calls schedule(). (this time interval is 3microsec) > > > > > > > > And I/O intensive has higher priority than CPU intensive. > > > > So I/O intensive job is first dispatched domain in runq. > > > > This is because latency improvement for I/O intensive guest. > > > > > > > > So your behavior is not strange. > > > > > > > > Thanks > > > > Atsushi SAKAI > > > > > > > > > > > > pak333@comcast.net wrote: > > &g >t; > > > > > > Hi > > > > > > > > > > I have noticed that VMs are typically spending 3 microsecs > or less before > > they > > > > are being prempted. I thought that the credit schduler time > slice was 30 ms. > > I > > > > have 4 VMs running and they are all cpu intensive except for > 1 (which is IO > > > > intensive) but having a VM spend max 3 micro secs before > being kicked out > > seems > > > > strange. > > > > > > > > > > Is there something else going on that i am not aware of. Is > the time slice > > > > really 30 millisecs? I am using default parameters of the > credit scheduler. > > > > > > > > > > Thanks > > > > > -Prabha > > > > > > > > > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen source.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks for the explanations. Some more questions Please see my comments inline: Thanks Prabha -------------- Original message -------------- From: Mats Petersson <mats@planetcatfish.com>> At 21:46 12/07/2007, pak333@comcast.net wrote: > >Hi Atsushi > > > >I am still confused, so let me explain what i think should happen > >and understand from you why it may not happen. > > > >I have 2 cpu intensive VMs and 1 IO intensive VM. My system has 4 > >physical cpus and 8 virtual cpus. > > > >Now using default parameters for the credit scheduler, the cpu > >intersive vcpu should run for 30 millsec. It does not, it runs for > >less (few microsecs). > > > >You said this could be because of softirq which are raised after a > >system call. Can i disable the softirqs. What happens if i do? If i > >cannot disable is there a way to see what VM is raising the softirq. > >If the cpu-VM raises the softirq it gets preempted or does it > >continue to run. How can I montior the softirqs raised by this VM. > >Can I tell if the premption is from a softirq or from something else. > > I don''t believe you can "disable" a softirq - you could of course > avoid doing the call to "do_softirq", but that is most likely just > going to completely remove any scheduling capability at all in the > system, making the system "single-tasking". Not a recommended way, in > my opinion. >Agreed if softirq is indeed a problem, and if a cpu-intensive VM is being preempted because of a softirq, then maybe affintizing the interrupts of dom0 to a pcpu will reduce this preemption. Comments?> > > >Also, if the IO VM requests an IO, it will block and dom0 wakes up > >and gets scheduled to run to service the IO. Will it preempt the cpu > >intensive VM. If so why? Shouldn''t the cpuintensiveV M get its > >quantum of time. > >Or does the IOVM get higher priority to preempt the cpu intensive > >VM. How does the scheduler pick which cpu to run dom0 on ( if all > >the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus > >and Io-vcpus, which will be the victim > > > The whole idea of having separate queues for IO and CPU intensive VMs > (or processes in a normal kernel scenario) is to allow the > IO-intensive tasks to avoid waiting for the next time-slot when a > CPU-hogging VM/process is running [1]. The normal behaviour for the > scheduler is to determine dynamically if the current VM/process is IO > or CPU intensive. > > Unless Dom0 is used for doing some silly CPU-intensive task > (calculating PI with a million decimal points or compiling Xen + > Linux kernel, for example), it will most likely look to the scheduler > like a IO-intensive VM. So it will have the same priority as any > other IO-intensive VM (assuming Dom0 has equal scheduler parameters > as other guests, which I''m pretty sure is the default behaviour). >Does dom0 have a higher pritority than any CPU intensive VM> If, like in your example, there are a number of processors busy with > CPU-intensive tasks, and others with IO-intensive tasks, the most > likely scenario is that any new IO-request will go be run on a > "previously CPU-intensive" CPU - but on the other hand, if you have a > lot of IO-intensive processing, there should be some processor(s) > that are "asleep" - unless all CPU''s are busy running CPU-intensive tasks...But will it preempt a CPU intensive vcpu if no cpu is available> > Note that scheduling is a complex subject, and there have been dozens > of PhD dissertations written on the subject.Yes, but what does the Xen credit scheduler actually do> > [1] The idea being that if we process the IO-request that just > completed, we can set up another IO-request, and this will get better > IO-throughput than waiting a long time (relatively speaking 30ms is > an eternity to the processor).Don''t understand what you mean here.> > -- > Mats > > > >As you see i have a lot of questions, so please bear with me and > >help me understand the scheduling behavior of Xen > > > >Thanks > >Prabha > > > > > > > > > >-------------- Original message -------------- > >From: Atsushi SAKAI > > > > > Hi Prabha > > > > > > Please check do_softirq in assembler. > > > for x86 > > > call do_softirq > > > is in. > > > > > > And usually do_softirq is executed at the end of system call. > > > > > > Thanks > > > Atsushi SAKAI > > > > > > > > > > > > pak333@comcast.net wrote: > > > > > > > Thanks for the explanation, i am not sure i understood it correctly. > > > > > > > > > > > > when does softirq get set. So when softirq is set a processor enters the > > > scheduler every 3 micro secs? where in the source code is this > > handled. please > > > could you send me some pointers > > > > > > > > thanks > > > > -Prabha > > > > > > > > > > > > > > > > > > > > -------------- Or iginal message -------------- > > > > From: Atsushi SAKAI > > > > > > > > > Hi,$B!!(BPrabha > > > > > > > > > > $B!!(B30msec is the maximum time slice. > > > > > And I guess your 3 microsecond is the response after > > > > > SCHEDULE_SOFTIRQ is set. > > > > > As you know, > > > > > If SCHEDULE_SOFTIRQ is set, > > > > > it waits softirq calls schedule(). (this time interval is 3microsec) > > > > > > > > > > And I/O intensive has higher priority than CPU intensive. > > > > > So I/O intensive job is first dispatched domain in runq. > > > > > This is because latency improvement for I/O intensive guest. > > > > > > > > > > So your behavior is not strange. > > > > > > > > > > Thanks > > > > > Atsushi SAKAI > > > > > > > > > > > > > > > pak333@comcast.net wrote: > > > &g > >t; > > > > > > > Hi > > > > > > > > > > > > I have noticed that VMs are typically spending 3 microsecs > > or less before > > > they > > > > > are being prempted. I thought that the credit schduler time > > slice was 30 ms. > > > I > > > > > have 4 VMs running and they are all cpu intensive except for > > 1 (which is IO > > > > > intensive) but having a VM spend max 3 micro secs before > > being kicked out > > > seems > > > > > strange. > > > > > > > > > > > > Is there something else going on that i am not aware of. Is > > the time slice > > > > > really 30 millisecs? I am using default parameters of the > > credit scheduler. > > > > > > > > > > > > Thanks > > > > > > -Prabha > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xen source.com > > > http://lists.xensource.com/xen-devel > > > >_______________________________________________ > >Xen-devel mailing list > >Xen-devel@lists.xensource.com > >http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, Prabha pak333@comcast.net wrote: [snip]> > >Also, if the IO VM requests an IO, it will block and dom0 wakes up > > >and gets scheduled to run to service the IO. Will it preempt the cpu > > >intensive VM. If so why? Shouldn''t the cpuintensiveV M get its > > >quantum of time. > > >Or does the IOVM get higher priority to preempt the cpu intensive > > >VM. How does the scheduler pick which cpu to run dom0 on ( if all > > >the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus > > >and Io-vcpus, which will be the victim > > > > > > The whole idea of having separate queues for IO and CPU intensive VMs > > (or processes in a normal kernel scenario) is to allow the > > IO-intensive tasks to avoid waiting for the next time-slot when a > > CPU-hogging VM/process is running [1]. The normal behaviour for the > > scheduler is to determine dynamically if the current VM/process is IO > > or CPU intensive. > > > > Unless Dom0 is used for doing some silly CPU-intensive task > > (calculating PI with a million decimal points or compiling Xen + > > Linux kernel, for example), it will most likely look to the scheduler > > like a IO-intensive VM. So it will have the same priority as any > > other IO-intensive VM (assuming Dom0 has equal scheduler parameters > > as other guests, which I''m pretty sure is the default behaviour). > > > Does dom0 have a higher pritority than any CPU intensive VMEasy way is to set higher weight to dom0. (This makes get higher priority like I/O intensive domain for domain dispatch) If you want to set higher priority of Dom0 see follow patch. (This is sample patch for boost Dom0 only not boost I/O intensive.) http://lists.xensource.com/archives/html/xen-devel/2007-05/msg00529.html Or just set CSCHED_PRI_TS_BOOST for domain0 in case dom0 priority is CSCHED_PRI_TS_UNDER.> > If, like in your example, there are a number of processors busy with > > CPU-intensive tasks, and others with IO-intensive tasks, the most > > likely scenario is that any new IO-request will go be run on a > > "previously CPU-intensive" CPU - but on the other hand, if you have a > > lot of IO-intensive processing, there should be some processor(s) > > that are "asleep" - unless all CPU''s are busy running CPU-intensive tasks... > But will it preempt a CPU intensive vcpu if no cpu is available[snip]> > > > [1] The idea being that if we process the IO-request that just > > completed, we can set up another IO-request, and this will get better > > IO-throughput than waiting a long time (relatively speaking 30ms is > > an eternity to the processor). > Don''t understand what you mean here.I guess Mats suggests to use AIO. Thanks Atsushi SAKAI _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 08:30 13/07/2007, Atsushi SAKAI wrote:>Hi, Prabha > >pak333@comcast.net wrote: > >[snip] > > > > >Also, if the IO VM requests an IO, it will block and dom0 wakes up > > > >and gets scheduled to run to service the IO. Will it preempt the cpu > > > >intensive VM. If so why? Shouldn''t the cpuintensiveV M get its > > > >quantum of time. > > > >Or does the IOVM get higher priority to preempt the cpu intensive > > > >VM. How does the scheduler pick which cpu to run dom0 on ( if all > > > >the vcpus running are cpu intensive)? If there is a mix of cpu-vcpus > > > >and Io-vcpus, which will be the victim > > > > > > > > > The whole idea of having separate queues for IO and CPU intensive VMs > > > (or processes in a normal kernel scenario) is to allow the > > > IO-intensive tasks to avoid waiting for the next time-slot when a > > > CPU-hogging VM/process is running [1]. The normal behaviour for the > > > scheduler is to determine dynamically if the current VM/process is IO > > > or CPU intensive. > > > > > > Unless Dom0 is used for doing some silly CPU-intensive task > > > (calculating PI with a million decimal points or compiling Xen + > > > Linux kernel, for example), it will most likely look to the scheduler > > > like a IO-intensive VM. So it will have the same priority as any > > > other IO-intensive VM (assuming Dom0 has equal scheduler parameters > > > as other guests, which I''m pretty sure is the default behaviour). > > > > > Does dom0 have a higher pritority than any CPU intensive VM > >Easy way is to set higher weight to dom0. >(This makes get higher priority like I/O intensive domain for domain >dispatch) > >If you want to set higher priority of Dom0 see follow patch. >(This is sample patch for boost Dom0 only not boost I/O intensive.) >http://lists.xensource.com/archives/html/xen-devel/2007-05/msg00529.html > >Or just set CSCHED_PRI_TS_BOOST for domain0 >in case dom0 priority is CSCHED_PRI_TS_UNDER. > > > > > If, like in your example, there are a number of processors busy with > > > CPU-intensive tasks, and others with IO-intensive tasks, the most > > > likely scenario is that any new IO-request will go be run on a > > > "previously CPU-intensive" CPU - but on the other hand, if you have a > > > lot of IO-intensive processing, there should be some processor(s) > > > that are "asleep" - unless all CPU''s are busy running > CPU-intensive tasks... > > But will it preempt a CPU intensive vcpu if no cpu is available > >[snip] > > > > > > > [1] The idea being that if we process the IO-request that just > > > completed, we can set up another IO-request, and this will get better > > > IO-throughput than waiting a long time (relatively speaking 30ms is > > > an eternity to the processor). > > Don''t understand what you mean here. > >I guess Mats suggests to use AIO.Well, sort of. I meant the idea behind waking an IO-intensive process by pre-empting a CPU intensive process is that it improves the overall IO throughput, because very often, once one IO request is completed, another one will be requested. Hardware will take some time to proces each request, at which time it''s fine to run the CPU-intensive process. If you prevent the IO intensive task from running until the CPU-intensive task has exhausted it''s time, it''s obviously going to take longer before the next IO-request is started, and thus longer until it gets completed. It makes little difference here if the APPLICATION is asynchronously or synchronously issuing the IO-request (assuming that under asynchronous circumstances there is a LIMITED amount of AIO queue - else it becomes a CPU-intensive task building until it''s issued all of the IO-requests and fall asleep). -- Mats>Thanks >Atsushi SAKAI_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel