Hi all, I need a periodic timer running at ideally at 125 microseconds and at least 500 microseconds. I''ve just found the VCPUOP_set_periodic_timer, however there is a comment saying "periods less than one millisecond may not be supported". I will be running on an x64 machine. Is this supported? If not, is there any alternate means of generating a fast interrupt? Regards. -- _ _ Debian GNU User Simon Martin | | (_)_ __ _ ___ __ Project Manager | | | | ''_ \| | | \ \/ / Milliways | |___| | | | | |_| |> < mailto: smartin@milliways.cl |_____|_|_| |_|\__,_/_/\_\ Si Hoc Legere Scis Nimium Eruditionis Habes _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 14/11/2013 21:18, Simon Martin wrote:> Hi all, > > I need a periodic timer running at ideally at 125 microseconds and at > least 500 microseconds. I''ve just found the VCPUOP_set_periodic_timer, > however there is a comment saying "periods less than one millisecond > may not be supported". > > I will be running on an x64 machine. Is this supported? If not, is > there any alternate means of generating a fast interrupt? > > Regards.What is the usecase here? 125us is very short indeed. Xen certainly cant guarantee anything more accurate than 50us. Unless the affected vcpu is running uncontested on the hardware, there is very little chance that the vcpu will indeed be woken up again in 125us. It sounds as if you are looking for some pseudo realtime system, at which point you might want to consider a different scheduler. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi Andrew, The operating system I am trying to port to Xen is an industrial servo controller. We are currently running at 125 microsecond servo update rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the 125 microsecond ideal interrupt period). We will be driving EtherCAT servo drives that need to be updated at 500 microseconds (hence at least 500 microsecond interrupt period). If we can achieve this on a single core ARM11 clocked at 532 MHz, it must be achievable on PC hardware. As I said before the idea is to dedicate a CPU core to this with all other functions running on the other cores. Luckily we can accept a reasonable amount of jitter on the EtherCAT network as we can hand over clock synchronisation to a slave. This gives us a little leeway. Regards. ------ Original Message ------ From: "Andrew Cooper" <andrew.cooper3@citrix.com> To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org" <xen-devel@lists.xen.org> Sent: 14/11/2013 18:39:21 Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer>On 14/11/2013 21:18, Simon Martin wrote: >>Hi all, >> >>I need a periodic timer running at ideally at 125 microseconds and at >>least 500 microseconds. I''ve just found the VCPUOP_set_periodic_timer, >>however there is a comment saying "periods less than one millisecond >>may not be supported". >> >>I will be running on an x64 machine. Is this supported? If not, is >>there any alternate means of generating a fast interrupt? >> >>Regards. > >What is the usecase here? 125us is very short indeed. Xen certainly >cant guarantee anything more accurate than 50us. Unless the affected >vcpu is running uncontested on the hardware, there is very little >chance that the vcpu will indeed be woken up again in 125us. > >It sounds as if you are looking for some pseudo realtime system, at >which point you might want to consider a different scheduler. > >~Andrew >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
The periodic_timer is really a coarse-grained timer for driving a legacy ticker. It is considered low accuracy and low priority for example, it is stopped entirely while a vcpu is descheduled! You should use VCPUOP_set_singleshot_timer, and call it again at the end of your timer handler to get the desired periodic behaviour. If you have multiple timers in your OS you would manage them in your OS and pick the nearest deadline to program down to Xen. -- Keir On 15/11/2013 11:24, "Simon Martin" <smartin@milliways.cl> wrote:> Hi Andrew, > > The operating system I am trying to port to Xen is an industrial servo > controller. We are currently running at 125 microsecond servo update rate on > our bespoke hardware (at the moment MIPS64 and ARM) (hence the 125 microsecond > ideal interrupt period). We will be driving EtherCAT servo drives that need to > be updated at 500 microseconds (hence at least 500 microsecond interrupt > period). If we can achieve this on a single core ARM11 clocked at 532 MHz, it > must be achievable on PC hardware. As I said before the idea is to dedicate a > CPU core to this with all other functions running on the other cores. > > Luckily we can accept a reasonable amount of jitter on the EtherCAT network as > we can hand over clock synchronisation to a slave. This gives us a little > leeway. > > Regards. > > > ------ Original Message ------ > From: "Andrew Cooper" <andrew.cooper3@citrix.com> > To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org" > <xen-devel@lists.xen.org> > Sent: 14/11/2013 18:39:21 > Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer > >> On 14/11/2013 21:18, Simon Martin wrote: >>> Hi all, >>> >>> I need a periodic timer running at ideally at 125 microseconds and at least >>> 500 microseconds. I''ve just found the VCPUOP_set_periodic_timer, however >>> there is a comment saying "periods less than one millisecond may not be >>> supported". >>> >>> I will be running on an x64 machine. Is this supported? If not, is there any >>> alternate means of generating a fast interrupt? >>> >>> Regards. >> >> What is the usecase here? 125us is very short indeed. Xen certainly cant >> guarantee anything more accurate than 50us. Unless the affected vcpu is >> running uncontested on the hardware, there is very little chance that the >> vcpu will indeed be woken up again in 125us. >> >> It sounds as if you are looking for some pseudo realtime system, at which >> point you might want to consider a different scheduler. >> >> ~Andrew >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 15/11/13 11:24, Simon Martin wrote:> Hi Andrew, > > The operating system I am trying to port to Xen is an industrial servo > controller. We are currently running at 125 microsecond servo update > rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the > 125 microsecond ideal interrupt period). We will be driving EtherCAT > servo drives that need to be updated at 500 microseconds (hence at > least 500 microsecond interrupt period). If we can achieve this on a > single core ARM11 clocked at 532 MHz, it must be achievable on PC > hardware. As I said before the idea is to dedicate a CPU core to this > with all other functions running on the other cores. > > Luckily we can accept a reasonable amount of jitter on the EtherCAT > network as we can hand over clock synchronisation to a slave. This > gives us a little leeway. > > Regards. > >It certainly is achievable on PC hardware, but not under the standard assumptions in virtualisation. You want to use VCPUOP_set_singleshot_timer, but be aware than there are still no normal guarantees that your domain vcpu will be run when the timer expires; the credit scheduler will pick the highest priority vcpu to run, which might not be the one which is expecting a timer interrupt. You should investigate using the arinc653 scheduler in Xen, which is a realtime scheduler, and might be more appropriate for your usecase. ~Andrew> ------ Original Message ------ > From: "Andrew Cooper" <andrew.cooper3@citrix.com > <mailto:andrew.cooper3@citrix.com>> > To: "Simon Martin" <smartin@milliways.cl > <mailto:smartin@milliways.cl>>; "xen-devel@lists.xen.org" > <xen-devel@lists.xen.org <mailto:xen-devel@lists.xen.org>> > Sent: 14/11/2013 18:39:21 > Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer > >> On 14/11/2013 21:18, Simon Martin wrote: >>> Hi all, >>> >>> I need a periodic timer running at ideally at 125 microseconds and >>> at least 500 microseconds. I''ve just found the >>> VCPUOP_set_periodic_timer, however there is a comment saying >>> "periods less than one millisecond may not be supported". >>> >>> I will be running on an x64 machine. Is this supported? If not, is >>> there any alternate means of generating a fast interrupt? >>> >>> Regards. >> >> What is the usecase here? 125us is very short indeed. Xen certainly >> cant guarantee anything more accurate than 50us. Unless the affected >> vcpu is running uncontested on the hardware, there is very little >> chance that the vcpu will indeed be woken up again in 125us. >> >> It sounds as if you are looking for some pseudo realtime system, at >> which point you might want to consider a different scheduler. >> >> ~Andrew >>--------------010806060005010103040205 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=UTF-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 15/11/13 11:24, Simon Martin wrote:<br> </div> <blockquote cite="mid:em0e63048f-7920-48f5-8392-cf0af10a1b08@smartin-alien" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <style id="eMClientCss">blockquote.cite { margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc } blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding-top: 0px; } .plain pre, .plain tt { font-family: monospace; font-size: 100%; font-weight: normal; font-style: normal; } body {font-family: Courier New;font-size: 10pt;} .plain pre, .plain tt {font-family: Courier New;font-size: 10pt;} </style> <style>#61e401be226d447880f34a151a1f4708 BLOCKQUOTE.cite2 {PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; MARGIN-TOP: 3px; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px} #61e401be226d447880f34a151a1f4708 .plain PRE, #61e401be226d447880f34a151a1f4708 .plain TT {FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal} #61e401be226d447880f34a151a1f4708, #61e401be226d447880f34a151a1f4708 .plain PRE, #61e401be226d447880f34a151a1f4708 .plain TT {FONT-SIZE: 10pt; FONT-FAMILY: Courier New} </style> <div>Hi Andrew,</div> <div> </div> <div>The operating system I am trying to port to Xen is an industrial servo controller. We are currently running at 125 microsecond servo update rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the 125 microsecond ideal interrupt period). We will be driving EtherCAT servo drives that need to be updated at 500 microseconds (hence at least 500 microsecond interrupt period). If we can achieve this on a single core ARM11 clocked at 532 MHz, it must be achievable on PC hardware. As I said before the idea is to dedicate a CPU core to this with all other functions running on the other cores.</div> <div> </div> <div>Luckily we can accept a reasonable amount of jitter on the EtherCAT network as we can hand over clock synchronisation to a slave. This gives us a little leeway.</div> <div> </div> <div>Regards.</div> <div> </div> <div> </div> </blockquote> <br> It certainly is achievable on PC hardware, but not under the standard assumptions in virtualisation. You want to use VCPUOP_set_singleshot_timer, but be aware than there are still no normal guarantees that your domain vcpu will be run when the timer expires; the credit scheduler will pick the highest priority vcpu to run, which might not be the one which is expecting a timer interrupt.<br> <br> You should investigate using the arinc653 scheduler in Xen, which is a realtime scheduler, and might be more appropriate for your usecase.<br> <br> ~Andrew<br> <br> <br> <blockquote cite="mid:em0e63048f-7920-48f5-8392-cf0af10a1b08@smartin-alien" type="cite"> <div>------ Original Message ------</div> <div>From: "Andrew Cooper" <<a moz-do-not-send="true" href="mailto:andrew.cooper3@citrix.com">andrew.cooper3@citrix.com</a>></div> <div>To: "Simon Martin" <<a moz-do-not-send="true" href="mailto:smartin@milliways.cl">smartin@milliways.cl</a>>; <a class="moz-txt-link-rfc2396E" href="mailto:xen-devel@lists.xen.org">"xen-devel@lists.xen.org"</a> <<a moz-do-not-send="true" href="mailto:xen-devel@lists.xen.org">xen-devel@lists.xen.org</a>></div> <div>Sent: 14/11/2013 18:39:21</div> <div>Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer</div> <div> </div> <div id="61e401be226d447880f34a151a1f4708" style="COLOR: #000000"> <blockquote class="cite2" cite="52854309.7000504@citrix.com" type="cite"> <div class="moz-cite-prefix">On 14/11/2013 21:18, Simon Martin wrote:<br> </div> <blockquote class="cite" cite="mid:em4158cadd-9eab-4a26-80ba-b3a3b311bee2@smartin-alien" type="cite"> <div>Hi all,</div> <div> </div> <div>I need a periodic timer running at ideally at 125 microseconds and at least 500 microseconds. I''ve just found the VCPUOP_set_periodic_timer, however there is a comment saying "periods less than one millisecond may not be supported".</div> <div> </div> <div>I will be running on an x64 machine. Is this supported? If not, is there any alternate means of generating a fast interrupt?</div> <div> </div> <div>Regards.</div> </blockquote> <br> What is the usecase here? 125us is very short indeed. Xen certainly cant guarantee anything more accurate than 50us. Unless the affected vcpu is running uncontested on the hardware, there is very little chance that the vcpu will indeed be woken up again in 125us.<br> <br> It sounds as if you are looking for some pseudo realtime system, at which point you might want to consider a different scheduler.<br> <br> ~Andrew<br> <br> </blockquote> </div> </blockquote> <br> </body> </html> --------------010806060005010103040205-- --===============8388215433939724451=Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============8388215433939724451==--
Thanks Keir, That sounds quite reasonable. What granularity can I achieve with this? Any idea on the jitter? Regards. ------ Original Message ------ From: "Keir Fraser" <keir.xen@gmail.com> To: "Simon Martin" <smartin@milliways.cl>; "Andrew Cooper" <andrew.cooper3@citrix.com> Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org> Sent: 15/11/2013 08:36:45 Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer>The periodic_timer is really a coarse-grained timer for driving a >legacy ticker. It is considered low accuracy and low priority – for >example, it is stopped entirely while a vcpu is descheduled! > >You should use VCPUOP_set_singleshot_timer, and call it again at the >end of your timer handler to get the desired periodic behaviour. If you >have multiple timers in your OS you would manage them in your OS and >pick the nearest deadline to program down to Xen. > > -- Keir > >On 15/11/2013 11:24, "Simon Martin" <smartin@milliways.cl> wrote: > >>Hi Andrew, >> >>The operating system I am trying to port to Xen is an industrial servo >>controller. We are currently running at 125 microsecond servo update >>rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the >>125 microsecond ideal interrupt period). We will be driving EtherCAT >>servo drives that need to be updated at 500 microseconds (hence at >>least 500 microsecond interrupt period). If we can achieve this on a >>single core ARM11 clocked at 532 MHz, it must be achievable on PC >>hardware. As I said before the idea is to dedicate a CPU core to this >>with all other functions running on the other cores. >> >>Luckily we can accept a reasonable amount of jitter on the EtherCAT >>network as we can hand over clock synchronisation to a slave. This >>gives us a little leeway. >> >>Regards. >> >> >>------ Original Message ------ >>From: "Andrew Cooper" <andrew.cooper3@citrix.com> >>To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org" >><xen-devel@lists.xen.org> >>Sent: 14/11/2013 18:39:21 >>Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer >> >>>On 14/11/2013 21:18, Simon Martin wrote: >>>>Hi all, >>>> >>>>I need a periodic timer running at ideally at 125 microseconds and >>>>at least 500 microseconds. I''ve just found the >>>>VCPUOP_set_periodic_timer, however there is a comment saying >>>>"periods less than one millisecond may not be supported". >>>> >>>>I will be running on an x64 machine. Is this supported? If not, is >>>>there any alternate means of generating a fast interrupt? >>>> >>>>Regards. >>> >>>What is the usecase here? 125us is very short indeed. Xen certainly >>>cant guarantee anything more accurate than 50us. Unless the affected >>>vcpu is running uncontested on the hardware, there is very little >>>chance that the vcpu will indeed be woken up again in 125us. >>> >>>It sounds as if you are looking for some pseudo realtime system, at >>>which point you might want to consider a different scheduler. >>> >>>~Andrew >>> >> >>-------------------------------------------------------------------------------- >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xen.org >>http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
If you have dedicated a core to the vcpu, so you can discount scheduling crosstalk, you shouldn¹t miss deadlines by more than a microsecond or two. Just enough time to wake up the CPU, handle the timer interrupt, and wake up the vcpu through the scheduler. There are infrequent operations that can take out all CPUs for a period: taking a CPU offline for example. You could avoid doing such things in your setup perhaps. :) Ultimately you will need to measure and confirm deadline-to-notification latency distribution if it is critical to you. -- Keir On 15/11/2013 11:45, "Simon Martin" <smartin@milliways.cl> wrote:> Thanks Keir, > > That sounds quite reasonable. What granularity can I achieve with this? Any > idea on the jitter? > > Regards. > > > ------ Original Message ------ > From: "Keir Fraser" <keir.xen@gmail.com> > To: "Simon Martin" <smartin@milliways.cl>; "Andrew Cooper" > <andrew.cooper3@citrix.com> > Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org> > Sent: 15/11/2013 08:36:45 > Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer > >> The periodic_timer is really a coarse-grained timer for driving a legacy >> ticker. It is considered low accuracy and low priority for example, it is >> stopped entirely while a vcpu is descheduled! >> >> You should use VCPUOP_set_singleshot_timer, and call it again at the end of >> your timer handler to get the desired periodic behaviour. If you have >> multiple timers in your OS you would manage them in your OS and pick the >> nearest deadline to program down to Xen. >> >> -- Keir >> >> On 15/11/2013 11:24, "Simon Martin" <smartin@milliways.cl> wrote: >> >>> Hi Andrew, >>> >>> The operating system I am trying to port to Xen is an industrial servo >>> controller. We are currently running at 125 microsecond servo update rate on >>> our bespoke hardware (at the moment MIPS64 and ARM) (hence the 125 >>> microsecond ideal interrupt period). We will be driving EtherCAT servo >>> drives that need to be updated at 500 microseconds (hence at least 500 >>> microsecond interrupt period). If we can achieve this on a single core ARM11 >>> clocked at 532 MHz, it must be achievable on PC hardware. As I said before >>> the idea is to dedicate a CPU core to this with all other functions running >>> on the other cores. >>> >>> Luckily we can accept a reasonable amount of jitter on the EtherCAT network >>> as we can hand over clock synchronisation to a slave. This gives us a little >>> leeway. >>> >>> Regards. >>> >>> >>> ------ Original Message ------ >>> From: "Andrew Cooper" <andrew.cooper3@citrix.com> >>> To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org" >>> <xen-devel@lists.xen.org> >>> Sent: 14/11/2013 18:39:21 >>> Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer >>> >>>> On 14/11/2013 21:18, Simon Martin wrote: >>>>> Hi all, >>>>> >>>>> I need a periodic timer running at ideally at 125 microseconds and at >>>>> least 500 microseconds. I''ve just found the VCPUOP_set_periodic_timer, >>>>> however there is a comment saying "periods less than one millisecond may >>>>> not be supported". >>>>> >>>>> I will be running on an x64 machine. Is this supported? If not, is there >>>>> any alternate means of generating a fast interrupt? >>>>> >>>>> Regards. >>>> >>>> What is the usecase here? 125us is very short indeed. Xen certainly cant >>>> guarantee anything more accurate than 50us. Unless the affected vcpu is >>>> running uncontested on the hardware, there is very little chance that the >>>> vcpu will indeed be woken up again in 125us. >>>> >>>> It sounds as if you are looking for some pseudo realtime system, at which >>>> point you might want to consider a different scheduler. >>>> >>>> ~Andrew >>>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Thanks Andrew, I''ve tried to load the arinc653 scheduler, however Xen crashes whenever I create a CPU pool with this scheduler. I reported this and it was confirmed by Seeing as I''ve been fighting to get my little port up, running and communicating via shared pages with a Windows domU I''ve kind of ignored this. However before I go and invest more time and effort with this I just wanted to confirm that I''m not barking up the wrong tree. Regards. ------ Original Message ------ From: "Andrew Cooper" <andrew.cooper3@citrix.com> To: "Simon Martin" <smartin@milliways.cl> Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org> Sent: 15/11/2013 08:41:44 Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer>On 15/11/13 11:24, Simon Martin wrote: >>Hi Andrew, >> >>The operating system I am trying to port to Xen is an industrial servo >>controller. We are currently running at 125 microsecond servo update >>rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the >>125 microsecond ideal interrupt period). We will be driving EtherCAT >>servo drives that need to be updated at 500 microseconds (hence at >>least 500 microsecond interrupt period). If we can achieve this on a >>single core ARM11 clocked at 532 MHz, it must be achievable on PC >>hardware. As I said before the idea is to dedicate a CPU core to this >>with all other functions running on the other cores. >> >>Luckily we can accept a reasonable amount of jitter on the EtherCAT >>network as we can hand over clock synchronisation to a slave. This >>gives us a little leeway. >> >>Regards. >> >> > >It certainly is achievable on PC hardware, but not under the standard >assumptions in virtualisation. You want to use >VCPUOP_set_singleshot_timer, but be aware than there are still no >normal guarantees that your domain vcpu will be run when the timer >expires; the credit scheduler will pick the highest priority vcpu to >run, which might not be the one which is expecting a timer interrupt. > >You should investigate using the arinc653 scheduler in Xen, which is a >realtime scheduler, and might be more appropriate for your usecase. > >~Andrew > > >>------ Original Message ------ >>From: "Andrew Cooper" <andrew.cooper3@citrix.com> >>To: "Simon Martin" <smartin@milliways.cl>; >>mailto:xen-devel@lists.xen.org <xen-devel@lists.xen.org> >>Sent: 14/11/2013 18:39:21 >>Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer >> >>>On 14/11/2013 21:18, Simon Martin wrote: >>>>Hi all, >>>> >>>>I need a periodic timer running at ideally at 125 microseconds and >>>>at least 500 microseconds. I''ve just found the >>>>VCPUOP_set_periodic_timer, however there is a comment saying >>>>"periods less than one millisecond may not be supported". >>>> >>>>I will be running on an x64 machine. Is this supported? If not, is >>>>there any alternate means of generating a fast interrupt? >>>> >>>>Regards. >>> >>>What is the usecase here? 125us is very short indeed. Xen certainly >>>cant guarantee anything more accurate than 50us. Unless the affected >>>vcpu is running uncontested on the hardware, there is very little >>>chance that the vcpu will indeed be woken up again in 125us. >>> >>>It sounds as if you are looking for some pseudo realtime system, at >>>which point you might want to consider a different scheduler. >>> >>>~Andrew >>> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 15/11/13 12:02, Simon Martin wrote:> Thanks Andrew, > > I''ve tried to load the arinc653 scheduler, however Xen crashes > whenever I create a CPU pool with this scheduler. I reported this and > it was confirmed by Seeing as I''ve been fighting to get my little port > up, running and communicating via shared pages with a Windows domU > I''ve kind of ignored this. However before I go and invest more time > and effort with this I just wanted to confirm that I''m not barking up > the wrong tree. > > Regards.The cpupool issue with the arinc653 scheduler is now fixed (or at least believed to be) in xen-unstable. ~Andrew --------------040403040603030405080706 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=UTF-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 15/11/13 12:02, Simon Martin wrote:<br> </div> <blockquote cite="mid:em36c3b116-28b1-4625-86ac-96904d507e7c@smartin-alien" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <style id="eMClientCss">blockquote.cite { margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc } blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding-top: 0px; } .plain pre, .plain tt { font-family: monospace; font-size: 100%; font-weight: normal; font-style: normal; } body {font-family: Courier New;font-size: 10pt;} .plain pre, .plain tt {font-family: Courier New;font-size: 10pt;} </style> <style>#a382538dcf4043d1ae9358df74ea6961 BLOCKQUOTE.cite2 {PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; MARGIN-TOP: 3px; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px} #a382538dcf4043d1ae9358df74ea6961 .plain PRE, #a382538dcf4043d1ae9358df74ea6961 .plain TT {FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal} #a382538dcf4043d1ae9358df74ea6961, #a382538dcf4043d1ae9358df74ea6961 .plain PRE, #a382538dcf4043d1ae9358df74ea6961 .plain TT {FONT-SIZE: 10pt; FONT-FAMILY: Courier New} #a382538dcf4043d1ae9358df74ea6961 #61e401be226d447880f34a151a1f4708 .plain PRE, #a382538dcf4043d1ae9358df74ea6961 #61e401be226d447880f34a151a1f4708 .plain TT {FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal} #a382538dcf4043d1ae9358df74ea6961 #61e401be226d447880f34a151a1f4708, #a382538dcf4043d1ae9358df74ea6961 #61e401be226d447880f34a151a1f4708 .plain PRE, #a382538dcf4043d1ae9358df74ea6961 #61e401be226d447880f34a151a1f4708 .plain TT {FONT-SIZE: 10pt; FONT-FAMILY: Courier New} </style> <div>Thanks Andrew,</div> <div> </div> <div>I''ve tried to load the arinc653 scheduler, however Xen crashes whenever I create a CPU pool with this scheduler. I reported this and it was confirmed by Seeing as I''ve been fighting to get my little port up, running and communicating via shared pages with a Windows domU I''ve kind of ignored this. However before I go and invest more time and effort with this I just wanted to confirm that I''m not barking up the wrong tree.</div> <div> </div> <div>Regards.</div> </blockquote> <br> The cpupool issue with the arinc653 scheduler is now fixed (or at least believed to be) in xen-unstable.<br> <br> ~Andrew<br> <br> </body> </html> --------------040403040603030405080706-- --===============3735914663010010382=Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============3735914663010010382==--
Latency isn''t a big problem. Jitter could be. CPU offline shouldn''t be a problem as the system is simple and static, at power up I create 2 CPU pools, one for the realtime machine domU running on a single core, and one for dom0 and a Windows machine running on the remaining cores. Any idea on the timings I can expect? For example, assuming there are no CPU contention issues, if I request a 125 microsecond single shot timer interrupt, will I get 125 microseconds +/- 5 microseconds (perfectly acceptable) or 500 microseconds (because that is the minimum granularity) +/- 50 microseconds (totally unacceptable)? ------ Original Message ------ From: "Keir Fraser" <keir.xen@gmail.com> To: "Simon Martin" <smartin@milliways.cl>; "Andrew Cooper" <andrew.cooper3@citrix.com> Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org> Sent: 15/11/2013 08:56:18 Subject: Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer>If you have dedicated a core to the vcpu, so you can discount >scheduling crosstalk, you shouldn’t miss deadlines by more than a >microsecond or two. Just enough time to wake up the CPU, handle the >timer interrupt, and wake up the vcpu through the scheduler. There are >infrequent operations that can take out all CPUs for a period: taking a >CPU offline for example. You could avoid doing such things in your >setup perhaps. Ultimately you will need to measure and confirm >deadline-to-notification latency distribution if it is critical to you. > > -- Keir_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 11/15/2013 7:17 AM, Andrew Cooper wrote:> > The cpupool issue with the arinc653 scheduler is now fixed (or at least believed > to be) in xen-unstable.The crashes are fixed in unstable, but pools still do not work in xen-unstable unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus parameter is set to 1, and cpu0 is never removed from Pool-0. I have pools working locally without these restrictions, but I was unable to get a patchset around before the feature freeze was declared, so I believe it is too late to try and get it into unstable. (George, correct me if I am wrong.) However, I am more than happy to provide Simon with my patches to get arinc653 cpu-pools completely working with the understanding that they may change before they make it into xen-unstable. Also, the arinc653 scheduler currently only supports a scheduling resolution of 1ms, but it is a trivial task to change it some other resolution.> > ~Andrew > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
On 15/11/13 12:46, Nate Studer wrote:> On 11/15/2013 7:17 AM, Andrew Cooper wrote: >> The cpupool issue with the arinc653 scheduler is now fixed (or at least believed >> to be) in xen-unstable. > The crashes are fixed in unstable, but pools still do not work in xen-unstable > unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus > parameter is set to 1, and cpu0 is never removed from Pool-0. > > I have pools working locally without these restrictions, but I was unable to get > a patchset around before the feature freeze was declared, so I believe it is too > late to try and get it into unstable. (George, correct me if I am wrong.) > > However, I am more than happy to provide Simon with my patches to get arinc653 > cpu-pools completely working with the understanding that they may change before > they make it into xen-unstable. > > Also, the arinc653 scheduler currently only supports a scheduling resolution of > 1ms, but it is a trivial task to change it some other resolution.Absolutely post the patches. The worst that can happen is that they are labelled as differed, but at least they are available for people to use. It is George''s call, but if they are small and fairly contained then they might be fine to go in. ~Andrew
On 15/11/13 12:46, Nate Studer wrote:> On 11/15/2013 7:17 AM, Andrew Cooper wrote: >> The cpupool issue with the arinc653 scheduler is now fixed (or at least believed >> to be) in xen-unstable. > The crashes are fixed in unstable, but pools still do not work in xen-unstable > unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus > parameter is set to 1, and cpu0 is never removed from Pool-0. > > I have pools working locally without these restrictions, but I was unable to get > a patchset around before the feature freeze was declared, so I believe it is too > late to try and get it into unstable. (George, correct me if I am wrong.)Bug fixes are not features. :-) Bug fixes are accepted very late in the release process. The only reason they might ever be rejected is if we think there may be a risk that the fix will break other working functionality -- particularly if we think that breakage may not be discovered until after the release. If the fixes involve generic scheduling code, I imagine they would be accepted well into RC3 if they were not terribly complicated; if they were limited to the arinc653 scheduler itself, they could probably be accepted until just before the release. -George
You¹ll have to measure to confirm, but if the core is dedicated to your vcpu then there is no reason there should be 50us jitter. Also there is not much point playing with different schedulers if there are no other vcpus schedulable on the cpu, the scheduler can¹t have multiple runnable vcpus to choose between. -- Keir On 15/11/2013 12:37, "Simon Martin" <smartin@milliways.cl> wrote:> Latency isn''t a big problem. Jitter could be. CPU offline shouldn''t be a > problem as the system is simple and static, at power up I create 2 CPU pools, > one for the realtime machine domU running on a single core, and one for dom0 > and a Windows machine running on the remaining cores. > > Any idea on the timings I can expect? For example, assuming there are no CPU > contention issues, if I request a 125 microsecond single shot timer interrupt, > will I get 125 microseconds +/- 5 microseconds (perfectly acceptable) or 500 > microseconds (because that is the minimum granularity) +/- 50 microseconds > (totally unacceptable)? > > > ------ Original Message ------ > From: "Keir Fraser" <keir.xen@gmail.com> > To: "Simon Martin" <smartin@milliways.cl>; "Andrew Cooper" > <andrew.cooper3@citrix.com> > Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org> > Sent: 15/11/2013 08:56:18 > Subject: Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer > >> If you have dedicated a core to the vcpu, so you can discount scheduling >> crosstalk, you shouldn¹t miss deadlines by more than a microsecond or two. >> Just enough time to wake up the CPU, handle the timer interrupt, and wake up >> the vcpu through the scheduler. There are infrequent operations that can take >> out all CPUs for a period: taking a CPU offline for example. You could avoid >> doing such things in your setup perhaps. Ultimately you will need to measure >> and confirm deadline-to-notification latency distribution if it is critical >> to you. >> >> -- Keir >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 15/11/13 13:10, Keir Fraser wrote:> Re: Re[4]: [Xen-devel] VCPUOP_set_periodic_timer You''ll have to > measure to confirm, but if the core is dedicated to your vcpu then > there is no reason there should be 50us jitter. Also there is not much > point playing with different schedulers --- if there are no other > vcpus schedulable on the cpu, the scheduler can''t have multiple > runnable vcpus to choose between. > > -- KeirThe deadline timer code as up to 50us of slop when calculating when to wake up and when timers expire. It is somewhat poor, and is certainly an area in need of improvement. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 15/11/2013 13:13, "Andrew Cooper" <andrew.cooper3@citrix.com> wrote:> > On 15/11/13 13:10, Keir Fraser wrote: > > >> Re: Re[4]: [Xen-devel] VCPUOP_set_periodic_timer You¹ll have to measure to >> confirm, but if the core is dedicated to your vcpu then there is no reason >> there should be 50us jitter. Also there is not much point playing with >> different schedulers if there are no other vcpus schedulable on the cpu, >> the scheduler can¹t have multiple runnable vcpus to choose between. >> >> -- Keir >> > > The deadline timer code as up to 50us of slop when calculating when to wake > up and when timers expire. It is somewhat poor, and is certainly an area in > need of improvement.Forgot about that. For real-time applications, timer_slop=0 on the Xen command line will turn it off. But there could certainly be other gotchas lurking; actual performance needs to be measured. -- Keir> ~Andrew >
On ven, 2013-11-15 at 11:41 +0000, Andrew Cooper wrote:> On 15/11/13 11:24, Simon Martin wrote: > > The operating system I am trying to port to Xen is an industrial > > servo controller. We are currently running at 125 microsecond servo > > update rate on our bespoke hardware (at the moment MIPS64 and ARM) > > (hence the 125 microsecond ideal interrupt period). We will be > > driving EtherCAT servo drives that need to be updated at 500 > > microseconds (hence at least 500 microsecond interrupt period). If > > we can achieve this on a single core ARM11 clocked at 532 MHz, it > > must be achievable on PC hardware. As I said before the idea is to > > dedicate a CPU core to this with all other functions running on the > > other cores. > > > > Luckily we can accept a reasonable amount of jitter on the EtherCAT > > network as we can hand over clock synchronisation to a slave. This > > gives us a little leeway. > > > > Regards. > > > > You should investigate using the arinc653 scheduler in Xen, which is a > realtime scheduler, and might be more appropriate for your usecase. >Right, or even SEDF, if only it were functional! :-/ That''s another area where we really need to do better. Luckily enough, we''ve got Roland, Joshua and Drek (from GVSU and HsH) working on trying to ''resurrect'' the SEDF scheduler. Coming to Simon''s situation, I think that, as it has already been said, if you can partition the system in such a way that the real-time VM/OS gets one full core, without any other interference (either via cpupool or pinning), there is few point in changing scheduler or scheduling parameters, and that''s probably the setup that would guarantee the most reliable performances. If that is not the case, you sure should check arinc, which is probably a rather "static" algorithm, but I''m quite ignorante (sigh) about how it actually works, so I can''t say anything about it (and I see Nathan is on it already, so you''re in good hands :-)). If you need something more advanced, while waiting for SEDF to be ''restored'', you perhaps can check at RT-Xen (Sisu, one of the main developers, is also in Cc). Basically, with that, you can enable some SEDF-alike schedulers, with even more complex and advanced functionalities. https://sites.google.com/site/realtimexen/ These schedulers allow you to specify, for each vcpu, a period and a certain amount of cpu time it will get every period (right Sisu?). This may turn out to be handy in your case, because you then will have the guarantee that the vcpu will get a chance to run at least once every <period> time units. At this point coming up with a period that guarantees the time granularity that you need shouldn''t be too hard, especially is the system is after all pretty simple, as it looks like. Of course, it remains to be verified whether the scheduling parameters can tolerate being set at the small values that your workload requires... But you cannot get along with a RT-ish system without doing some measurements, right? :-) Regards, Dario> ~Andrew > > > > ------ Original Message ------ > > From: "Andrew Cooper" <andrew.cooper3@citrix.com> > > To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org" > > <xen-devel@lists.xen.org> > > Sent: 14/11/2013 18:39:21 > > Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer > > > > > On 14/11/2013 21:18, Simon Martin wrote: > > > > > > > Hi all, > > > > > > > > I need a periodic timer running at ideally at 125 microseconds > > > > and at least 500 microseconds. I''ve just found the > > > > VCPUOP_set_periodic_timer, however there is a comment saying > > > > "periods less than one millisecond may not be supported". > > > > > > > > I will be running on an x64 machine. Is this supported? If not, > > > > is there any alternate means of generating a fast interrupt? > > > > > > > > Regards. > > > > > > What is the usecase here? 125us is very short indeed. Xen > > > certainly cant guarantee anything more accurate than 50us. Unless > > > the affected vcpu is running uncontested on the hardware, there is > > > very little chance that the vcpu will indeed be woken up again in > > > 125us. > > > > > > It sounds as if you are looking for some pseudo realtime system, > > > at which point you might want to consider a different scheduler. > > > > > > ~Andrew > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel-- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>Coming to Simon''s situation, I think that, as it has already been said, >if you can partition the system in such a way that the real-time VM/OS >gets one full core, without any other interference (either via cpupool >or pinning), there is few point in changing scheduler or scheduling >parameters, and that''s probably the setup that would guarantee the most >reliable performances.This is the configuration I am working on. The system is reasonably simple, a single flat memory space with it''s own simple scheduler, so I can''t see why it shouldn''t work, as long as I can get the granularity out of the underlying system.> >If that is not the case, you sure should check arinc, which is probably >a rather "static" algorithm, but I''m quite ignorante (sigh) about how >it >actually works, so I can''t say anything about it (and I see Nathan is >on >it already, so you''re in good hands ).In this kind of environment, a static algorithm is good. It is much better to be predictable than flexible. In an industrial setting where this is controlling machinery that can potentially kill the operators the principle of least surprise is what you need.> >If you need something more advanced, while waiting for SEDF to be >''restored'', you perhaps can check at RT-Xen (Sisu, one of the main >developers, is also in Cc). Basically, with that, you can enable some >SEDF-alike schedulers, with even more complex and advanced >functionalities.Before I embarked on this I looked at RealTime Xen. Very interesting project, however from my understanding of the Xen architecture I thought that the vanilla Xen would be sufficient. This is still my fallback position.> >Of course, it remains to be verified whether the scheduling parameters >can tolerate being set at the small values that your workload >requires... But you cannot get along with a RT-ish system without doing >some measurements, right?This is what is missing. There is a steep learning curve on how to write a PV guest. The mini-os is a good starting point, and so is David Chisnall''s book, however in many ways the mini-os is overkill for what I want as a proof of concept where I want to do the bare minimum to interface with Xen and just work on the fixed flat memory space that it initialises with. Once I have this working smoothly I''ll look at timings, however I really needed to know whether what I am asking is feasible or not. From all the answers I think that it is more than feasible, not only due to the Xen infrastructure, but also because of the good will of the people involved. Thanks very much. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On sab, 2013-11-16 at 20:37 +0000, Simon Martin wrote:> This is the configuration I am working on. The system is reasonably > simple, a single flat memory space with it''s own simple scheduler, so > I can''t see why it shouldn''t work, as long as I can get the > granularity out of the underlying system. >It indeed should work. It''s the fact that the "underlying system" is an hypervisor that makes things fun! :-) There is quite some interest in enabling at least a certain level of real-time behavior in/on top of Xen these days (mostly thanks to the ARM port), which means a lot of people will be looking forward of what you''ll be able to achieve and how.> Before I embarked on this I looked at RealTime Xen. Very interesting > project, however from my understanding of the Xen architecture I > thought that the vanilla Xen would be sufficient. This is still my > fallback position. >Yep, it''s good to have alternatives. Long term, I think we should aim at both fixing SEDF and having something like RT-Xen merged upstream (people are working on both these projects already).> Once I have this working smoothly I''ll look at timings, however I > really needed to know whether what I am asking is feasible or not. > From all the answers I think that it is more than feasible, not only > due to the Xen infrastructure, but also because of the good will of > the people involved. >HeHe... Nice to hear you feel comfortable with us. :-) Keep us posted on how it goes. Thanks and Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi all, I have finally got my little PV guest running. My critical error was not setting __XEN_INTERFACE_VERSION__. After tearing to pieces the code and put it back together a few times I decided to debug my Makefile script comparing it to Mini-OS. I am now starting to do some simple timings and the numbers are so bad I''m wondering what could be wrong with my test. As mentioned previously I have created a CPU pool with one CPU and one domU. I am using the standard credit scheduler. I set timer_slop=0 on the Xen command line. I initialise console, traps, events, and TSC clock in my PV and then start a periodic operation running. I then calculate latency as (clock time - deadline) and period as (clock time - previous deadline).At the moment I''m just displaying min/max values. I also increment a tick count on every cycle. Looking at the statistics I see that I get the expected number of ticks per second, however I get latencies in the range [-1ms, +25us] and the periods in the range of [3us, 1.02ms]. The upper bounds look OK, but the lower bounds are all over the place. This makes me think that I''m not the only thing using VIRQ_TIMER. I seem to remember that this is called nominally every 10ms by the Hypervisor. Is there any way to recognize the origin of the timer? Can anyone suggest other things to try? Regards. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 26/11/2013 14:50, "Simon Martin" <smartin@milliways.cl> wrote:> Hi all, > > I have finally got my little PV guest running. My critical error was not > setting __XEN_INTERFACE_VERSION__. After tearing to pieces the code and put it > back together a few times I decided to debug my Makefile script comparing it > to Mini-OS. > > I am now starting to do some simple timings and the numbers are so bad I''m > wondering what could be wrong with my test. > > As mentioned previously I have created a CPU pool with one CPU and one domU. I > am using the standard credit scheduler. I set timer_slop=0 on the Xen command > line. > > I initialise console, traps, events, and TSC clock in my PV and then start a > periodic operation running. I then calculate latency as (clock time - > deadline) and period as (clock time - previous deadline).At the moment I''m > just displaying min/max values. I also increment a tick count on every cycle. > > Looking at the statistics I see that I get the expected number of ticks per > second, however I get latencies in the range [-1ms, +25us] and the periods in > the range of [3us, 1.02ms]. The upper bounds look OK, but the lower bounds are > all over the place. > > This makes me think that I''m not the only thing using VIRQ_TIMER. I seem to > remember that this is called nominally every 10ms by the Hypervisor. > > Is there any way to recognize the origin of the timer? > > Can anyone suggest other things to try?Are you using the single-shot timer or the periodic timer? How are you calculating clock time? -- Keir> Regards. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Hi Keir,>Are you using the single-shot timer or the periodic timer?I''m using the single-shot timer.> >How are you calculating clock time?Copied code in from Mini-OS. To make sure I don''t get drift between deadlines as this is a single shot timer I read monotonic_clock at startup and then calculate every deadline by incrementing this register by the period on each timer call. As an additional comment. I decided to filter out the samples with negative latency. This seems to happen 100 times a second, i.e. the 10 ms Hypervisor timer event. This improves the timings considerably. So it looks like I''m on the right track looking for ways to distinguish between timer events. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On mar, 2013-11-26 at 14:50 +0000, Simon Martin wrote:> Hi all, >Hi Simon!> I have finally got my little PV guest running. My critical error was > not setting __XEN_INTERFACE_VERSION__. After tearing to pieces the > code and put it back together a few times I decided to debug my > Makefile script comparing it to Mini-OS. >Cool! :-)> I am now starting to do some simple timings and the numbers are so bad > I''m wondering what could be wrong with my test. >Not so cool! :-(> As mentioned previously I have created a CPU pool with one CPU and one > domU. I am using the standard credit scheduler. I set timer_slop=0 on > the Xen command line. > > I initialise console, traps, events, and TSC clock in my PV and then > start a periodic operation running. I then calculate latency as (clock > time - deadline) and period as (clock time - previous deadline).At the > moment I''m just displaying min/max values. I also increment a tick > count on every cycle. > > Looking at the statistics I see that I get the expected number of > ticks per second, however I get latencies in the range [-1ms, +25us] > and the periods in the range of [3us, 1.02ms]. The upper bounds look > OK, but the lower bounds are all over the place. >Ok. Well, I''m not super expert in that area (yet), but even for the ones that are, I think it''s pretty hard to guess what could be going on without seeing the actual code. Is there any chance that you can share/show it? At least the relevant chunks... Some questions: - "looking at the statistics", what statistics? How do you collect them? - you say you get latencies in some range and periods in some other range. It may be my fault, but I''m not sure I understand what you mean with these two terms in this context, could you clarify?> This makes me think that I''m not the only thing using VIRQ_TIMER. I > seem to remember that this is called nominally every 10ms by the > Hypervisor. > > Is there any way to recognize the origin of the timer? >I''m not sure if fits your exact needs, and you probably know it already, but, just in case, have you looked at xentrace and xenalyze? There''s a blog post that could be a nice introduction to them: http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/ If the exact information you''re after is not among the already existing tracepoints, it shouldn''t be to hard to add one or more of those, in the spots where you need them. But again, it''s very hard to judge without seeing the code... Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote:> As an additional comment. I decided to filter out the samples with > negative latency. This seems to happen 100 times a second, i.e. the 10 > ms Hypervisor timer event. >Again, what''s "negative latency" ? Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote: > > >> As an additional comment. I decided to filter out the samples with >> negative latency. This seems to happen 100 times a second, i.e. the >>10 >> ms Hypervisor timer event. >> >Again, what''s "negative latency" ?Latency is the time between the deadline and the handler being called. Negative latency means the handler was called before the next deadline. Regards _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>> >> Looking at the statistics I see that I get the expected number of >> ticks per second, however I get latencies in the range [-1ms, +25us] >> and the periods in the range of [3us, 1.02ms]. The upper bounds look >> OK, but the lower bounds are all over the place. >> >Ok. Well, I''m not super expert in that area (yet), but even for the >ones >that are, I think it''s pretty hard to guess what could be going on >without seeing the actual code. > >Is there any chance that you can share/show it? At least the relevant >chunks...No problems, I can share the complete code it you want, there is nothing proprietary in there yet. What''s the best way to do it? Put it on github? Send a tarball?>Some questions: > - "looking at the statistics", what statistics? How do you collect > them?At every cycle I compare the current time with the expected time and calculate latency (time from when I expect the event to when it actually happens) and period.> - you say you get latencies in some range and periods in some other > range. It may be my fault, but I''m not sure I understand what you > mean with these two terms in this context, could you clarify? >Latency = difference between expected time and actual time of the event. Period = time between two consecutive events.>> This makes me think that I''m not the only thing using VIRQ_TIMER. I >> seem to remember that this is called nominally every 10ms by the >> Hypervisor. >> >> Is there any way to recognize the origin of the timer? >> >I''m not sure if fits your exact needs, and you probably know it >already, >but, just in case, have you looked at xentrace and xenalyze? There''s a >blog post that could be a nice introduction to them: >http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/ >I hadn''t read that. I''ll look at it tomorrow (it''s almost midnight here).>If the exact information you''re after is not among the already existing >tracepoints, it shouldn''t be to hard to add one or more of those, in >the >spots where you need them. > >But again, it''s very hard to judge without seeing the code...I totally understand. Thanks for the effort.> >Regards, >Dario > >-- ><<This happens because I choose it to happen!>> (Raistlin Majere) >----------------------------------------------------------------- >Dario Faggioli, Ph.D, http://about.me/dario.faggioli >Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On mer, 2013-11-27 at 02:32 +0000, Simon Martin wrote:> > > On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote: > > > > > > > As an additional comment. I decided to filter out the samples > > > with > > > negative latency. This seems to happen 100 times a second, i.e. > > > the 10 > > > ms Hypervisor timer event. > > > > > Again, what''s "negative latency" ? > > Latency is the time between the deadline and the handler being called. >Ok, that''s a reasonable definition of latency. However, if you don''t mind another question, ''deadline'' here is basically the deadline of the periodic instance x-1, and you expect a timer firing at that time in order to activate instance x? Or is that a proper deadline (and the above is just called period) and you want to have some handler running when it''s missed?> Negative latency means the handler was called before the next > deadline. >Oh, wow... I see. So you''re saying that you have timers firing early than their expiry time? Is that expected? Why is that happening? :-O Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On mer, 2013-11-27 at 02:36 +0000, Simon Martin wrote:> > > > Is there any chance that you can share/show it? At least the > > relevant > > chunks... > No problems, I can share the complete code it you want, there is > nothing proprietary in there yet. What''s the best way to do it? Put it > on github? Send a tarball? >Well, I think a repo somewhere would be preferrable over a tarball, espacially if you have it in a (although private) repo already, so that we gt to see the history, the commit changelogs, etc. Also, if you go for github, or whatever similar service, people would be able to both checkout your code, or browse it on-line, depending on what they like most. If you don''t, I still personally prefer repositories, but, really, whatever you''re more comfortable with would do.> > Some questions: > > - "looking at the statistics", what statistics? How do you collect > > them? > At every cycle I compare the current time with the expected time and > calculate latency (time from when I expect the event to when it > actually happens) and period. >Sure, that part I got. I was more asking how you get these numbers out, i.e., you print them online, store somewhere and print later/on request, etc. (I guess I could have specified this more clearly, sorry about that :-) )> > - you say you get latencies in some range and periods in some > > other > > range. It may be my fault, but I''m not sure I understand what > > you > > mean with these two terms in this context, could you clarify? > > > Latency = difference between expected time and actual time of the > event. > > Period = time between two consecutive events. >Ok, I think I see now. Just to confirm that I do, this means that the values/ranges you get by measuring both are related, i.e., not completely independent, right? Again, I''m not saying it''s not useful to have both, just trying to double check I understood what you''re doing.> > I''m not sure if fits your exact needs, and you probably know it > > already, > > but, just in case, have you looked at xentrace and xenalyze? There''s > > a > > blog post that could be a nice introduction to them: > > http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/ > > > I hadn''t read that. I''ll look at it tomorrow (it''s almost midnight > here). >Ask again if you need anything. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 26/11/2013 15:38, Simon Martin wrote:> Hi Keir, > >> Are you using the single-shot timer or the periodic timer? > I''m using the single-shot timer. > >> >> How are you calculating clock time? > Copied code in from Mini-OS. To make sure I don''t get drift between > deadlines as this is a single shot timer I read monotonic_clock at > startup and then calculate every deadline by incrementing this register > by the period on each timer call. > > As an additional comment. I decided to filter out the samples with > negative latency. This seems to happen 100 times a second, i.e. the 10 > ms Hypervisor timer event. This improves the timings considerably. So it > looks like I''m on the right track looking for ways to distinguish > between timer events.You should disable the periodic timer if you don''t need it. VCPUOP_stop_periodic_timer David
>> Latency is the time between the deadline and the handler being >>called. >> >Ok, that''s a reasonable definition of latency. However, if you don''t >mind another question, ''deadline'' here is basically the deadline of the >periodic instance x-1, and you expect a timer firing at that time in >order to activate instance x? Or is that a proper deadline (and the >above is just called period) and you want to have some handler running >when it''s missed?Each time the timer handler is called it calculates the deadline for the next handler and calls VCPUOP_set_singleshot_timer specifying the deadline. The deadline is calculated as the "previous deadline + the period". So handler instance ''n'' will set the deadline for handler instance ''n+1'' as "deadline n + period". I''m not sure if that answers the question, but that is what is happening.> >> Negative latency means the handler was called before the next >> deadline. >> >Oh, wow... I see. So you''re saying that you have timers firing early >than their expiry time? Is that expected? Why is that happening?Yes. As I said I think this is the 100 Hz Hypervisor periodic timer. I need to be able to distinguish between the different timer interrupt sources. Regards. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
------ Original Message ------ From: "Dario Faggioli" <dario.faggioli@citrix.com> To: "Simon Martin" <smartin@milliways.cl> Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>; "Andrew Cooper" <andrew.cooper3@citrix.com>; "Roland Heusser" <heusserr@mail.gvsu.edu>; "Sisu Xi" <xisisu@gmail.com>; "Joshua Whitehead" <whitehej@mail.gvsu.edu>; "Drek Darkover" <wackerei@gmail.com>; "Nate Studer" <nate.studer@dornerworks.com> Sent: 27/11/2013 05:56:26 Subject: Re: Re[2]: [Xen-devel] PV guest timings>On mer, 2013-11-27 at 02:36 +0000, Simon Martin wrote: > >> > >> > Is there any chance that you can share/show it? At least the >> > relevant >> > chunks... >> No problems, I can share the complete code it you want, there is >> nothing proprietary in there yet. What''s the best way to do it? Put >>it >> on github? Send a tarball? >> >Well, I think a repo somewhere would be preferrable over a tarball, >espacially if you have it in a (although private) repo already, so that >we gt to see the history, the commit changelogs, etc.Here''s the URL. https://github.com/FurryFuttock/micro-pv. There is no commit history as I mostly use SVN. I just created this and will keep it up to date as I carry on playing with it.>> > Some questions: >> > - "looking at the statistics", what statistics? How do you collect >> > them? >> At every cycle I compare the current time with the expected time and >> calculate latency (time from when I expect the event to when it >> actually happens) and period. >> >Sure, that part I got. I was more asking how you get these numbers out, >i.e., you print them online, store somewhere and print later/on >request, >etc. (I guess I could have specified this more clearly, sorry about >that )Here''s an example. There are 2 lines per sample. Line 0 has format <time> <ticks per second>, line 1 has <latency min/max>,<period min/max> timeofday=12:56:41 - 1000 1043,47287,955121,2001800 timeofday=12:56:42 - 1000 1024,30989,970082,2000885 timeofday=12:56:43 - 1000 1042,30888,970167,2000947 timeofday=12:56:44 - 1000 1042,30910,970661,2001063 timeofday=12:56:45 - 1000 1043,30909,970162,2000057 timeofday=12:56:46 - 1000 1043,30883,970183,2000049 timeofday=12:56:47 - 1000 1066,30923,970160,2001268 timeofday=12:56:48 - 1000 1048,30922,970140,2000031 timeofday=12:56:49 - 1000 1041,30864,970211,2001090 timeofday=12:56:50 - 1000 1043,30898,970162,2000880 timeofday=12:56:51 - 1000 1042,30948,970106,2000947 timeofday=12:56:52 - 1000 1044,30777,970752,2000058> >> > - you say you get latencies in some range and periods in some >> > other >> > range. It may be my fault, but I''m not sure I understand what >> > you >> > mean with these two terms in this context, could you clarify? >> > >> Latency = difference between expected time and actual time of the >> event. >> >> Period = time between two consecutive events. >> >Ok, I think I see now. Just to confirm that I do, this means that the >values/ranges you get by measuring both are related, i.e., not >completely independent, right? Again, I''m not saying it''s not useful to >have both, just trying to double check I understood what you''re doing.They''re not really related. Period is time between events, Latency is time from event to handler.> >> > I''m not sure if fits your exact needs, and you probably know it >> > already, >> > but, just in case, have you looked at xentrace and xenalyze? >>There''s >> > a >> > blog post that could be a nice introduction to them: >> > >>http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/ >> > >> I hadn''t read that. I''ll look at it tomorrow (it''s almost midnight >> here). >> >Ask again if you need anything.I will ! Thanks. Regards. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>You should disable the periodic timer if you don''t need it. > >VCPUOP_stop_periodic_timer > >DavidPerfect. Thanks David, that got rid of my strange counts. I now only have one VIRQ_TIMER source. Regards. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On mer, 2013-11-27 at 13:00 +0000, Simon Martin wrote: ------ Original Message ------> From: "Dario Faggioli" <dario.faggioli@citrix.com>> > > > > Well, I think a repo somewhere would be preferrable over a tarball, > > espacially if you have it in a (although private) repo already, so > > that > > we gt to see the history, the commit changelogs, etc. > Here''s the URL. https://github.com/FurryFuttock/micro-pv. There is no > commit history as I mostly use SVN. I just created this and will keep > it up to date as I carry on playing with it. >Once more, thanks for this. It looks really interesting and, if you ask me, I''ll sure it will be useful! I took a quick look at it, and added to my TODO list to properly look into it. Hopefully, it won''t take that long. :-)> > Sure, that part I got. I was more asking how you get these numbers > > out, > > i.e., you print them online, store somewhere and print later/on > > request, > > etc. (I guess I could have specified this more clearly, sorry about > > that :-) ) > Here''s an example. There are 2 lines per sample. Line 0 has format > <time> <ticks per second>, line 1 has <latency min/max>,<period > min/max> > > timeofday=12:56:41 - 1000 > 1043,47287,955121,2001800 > timeofday=12:56:42 - 1000 > 1024,30989,970082,2000885 > timeofday=12:56:43 - 1000 > 1042,30888,970167,2000947 > ... >Ok, I was wondering whether, e.g., the printing could interfere with the measurements, and stuff like that, but I checked this out in your code, and it doesn''t look like it is happening.> > Ok, I think I see now. Just to confirm that I do, this means that > > the > > values/ranges you get by measuring both are related, i.e., not > > completely independent, right? Again, I''m not saying it''s not useful > > to > > have both, just trying to double check I understood what you''re > > doing. > They''re not really related. Period is time between events, Latency is > time from event to handler. >"Period is time between events", what events? I was thinking the events to be two subsequent invocation of the handler. In an ideal world, that would be exactly equal to the PERIOD that you choose, which also mean latency would be 0. OTOH, if you get some latency in invoking the handler, you''ll see different values for the actual time difference between different invocations... Otherwise, how could period vary over time? I mean, if at t0 you set the next event to occur at t1=t0+T, still in the ideal world, you''ll get the event both triggered and handled exactly at t1, i.e., period: p1=t0+T-t0=T, latency: l1=t1-t1=0. If, OTOH, you get some latency, i.e., you are processing the event in the handler at t1''=t0+T+l1, that means latency: t1''-t1=t0+T+l1-t0-T=l1, and that also makes the distance between the two handling event p1''=t1''-t0=t0+T+l1-t0=T+l1. Going further on, you get t2=t1+T=t0+T+T=t0+2T t2''=t2+l2 and thus p2''=t2''-t1''=t0+2T+l2-t0-T-l1=T+l2-l1 That''s what I meant with "related". Checking the code (but very quickly, so bear with me if I''m getting something wrong), _BUT_, I totally understand that this relationship is only useful if you perform the calculation a bit differently, and the way you''re doing them allows you to monitor both latency and jitter good enough anyway, so I''m cool with it. :-) Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>Once more, thanks for this. It looks really interesting and, if you ask >me, I''ll sure it will be useful! > >I took a quick look at it, and added to my TODO list to properly look >into it. Hopefully, it won''t take that long.Thanks. All feedback is welcome.> >> >> timeofday=12:56:41 - 1000 >> 1043,47287,955121,2001800 >> timeofday=12:56:42 - 1000 >> 1024,30989,970082,2000885 >> timeofday=12:56:43 - 1000 >> 1042,30888,970167,2000947 >> ... >> >Ok, I was wondering whether, e.g., the printing could interfere with >the >measurements, and stuff like that, but I checked this out in your code, >and it doesn''t look like it is happening.It shouldn''t. I calculate numbers in the periodic interrupt and then print them from the normal run loop.> >> > Ok, I think I see now. Just to confirm that I do, this means that >> > the >> > values/ranges you get by measuring both are related, i.e., not >> > completely independent, right? Again, I''m not saying it''s not >>useful >> > to >> > have both, just trying to double check I understood what you''re >> > doing. >> They''re not really related. Period is time between events, Latency is >> time from event to handler. >> >"Period is time between events", what events?As you say, events in this context are subsequent invocations of the handler.>In an ideal world, that >would be exactly equal to the PERIOD that you choose, which also mean >latency would be 0. OTOH, if you get some latency in invoking the >handler, you''ll see different values for the actual time difference >between different invocations... Otherwise, how could period vary over >time?Not quite. Let us say that latency is always 5 µs, no jitter. That means that the measured period will always be exactly the same, just slightly delayed in time. This is fine. However there IS jitter in the latency (+/- 30 µs) so the period varies likewise. One event comes in late, the next comes in on time, period is less than expected. One comes in on time the next comes in late, period is longer than expected.> >I mean, if at t0 you set the next event to occur at > > t1=t0+T, > >still in the ideal world, you''ll get the event both triggered and >handled exactly at t1, i.e., > > period: p1=t0+T-t0=T, > latency: l1=t1-t1=0. > >If, OTOH, you get some latency, i.e., you are processing the event in >the handler at > > t1''=t0+T+l1, > >that means > > latency: t1''-t1=t0+T+l1-t0-T=l1, > >and that also makes the distance between the two handling event > > p1''=t1''-t0=t0+T+l1-t0=T+l1. > >Going further on, you get > > t2=t1+T=t0+T+T=t0+2T > t2''=t2+l2 > >and thus > > p2''=t2''-t1''=t0+2T+l2-t0-T-l1=T+l2-l1 > >That''s what I meant with "related".I agree, it is not a direct relationship, but yes they are related.> >Checking the code (but very quickly, so bear with me if I''m getting >something wrong), _BUT_, I totally understand that this relationship is >only useful if you perform the calculation a bit differently, and the >way you''re doing them allows you to monitor both latency and jitter >good >enough anyway, so I''m cool with it.I''m glad I haven''t got it too wrong at the first attempt. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel