Keir: When running vConsolidation on top of Xen in a 4-core platform, we noticed the I/O introduced scheduling per CPU is ~3K Hz, which seems to be too frequent and cause frequent involve of domain 0 / Qemu, which may polute cache of the guest and thus increase CPI (cycle per instruction). We are thinking if we can reduce the domin switch here, and think the output of I/O can be buffered and return immediately. The buffered I/O can be flushed out at next IN emulation (or any Hypervisor emulated I/O) or timeout such as 10 or 100 us to guarantee minimal response. Ideally it can cover both PIO & MMIO, but we can start from PIO. How do you think of that? Thx, Eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Oct-12 06:38 UTC
Re: [Xen-devel] Reducing I/O introduced domain scheduling
On 12/10/2010 02:12, "Dong, Eddie" <eddie.dong@intel.com> wrote:> Keir: > When running vConsolidation on top of Xen in a 4-core platform, we noticed > the I/O introduced scheduling per CPU is ~3K Hz, which seems to be too > frequent and cause frequent involve of domain 0 / Qemu, which may polute cache > of the guest and thus increase CPI (cycle per instruction). > > We are thinking if we can reduce the domin switch here, and think the output > of I/O can be buffered and return immediately. The buffered I/O can be flushed > out at next IN emulation (or any Hypervisor emulated I/O) or timeout such as > 10 or 100 us to guarantee minimal response. > > Ideally it can cover both PIO & MMIO, but we can start from PIO. > > How do you think of that?First of all have you tested with PV drivers? Performance tests with no PV drivers are not that interesting. Apart from that this sounds like a generalisation of the buffered-i/o mechanism we already have for svga emulation. I suppose it might be plausible. -- Keir> Thx, Eddie > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2010-Oct-12 07:19 UTC
RE: [Xen-devel] Reducing I/O introduced domain scheduling
Keir Fraser wrote:> On 12/10/2010 02:12, "Dong, Eddie" <eddie.dong@intel.com> wrote: > >> Keir: >> When running vConsolidation on top of Xen in a 4-core platform, we >> noticed the I/O introduced scheduling per CPU is ~3K Hz, which seems >> to be too frequent and cause frequent involve of domain 0 / Qemu, >> which may polute cache of the guest and thus increase CPI (cycle per >> instruction). >> >> We are thinking if we can reduce the domin switch here, and think >> the output of I/O can be buffered and return immediately. The >> buffered I/O can be flushed out at next IN emulation (or any >> Hypervisor emulated I/O) or timeout such as 10 or 100 us to >> guarantee minimal response. >> >> Ideally it can cover both PIO & MMIO, but we can start from PIO. >> >> How do you think of that? > > First of all have you tested with PV drivers? Performance tests with > no PV drivers are not that interesting.We used PV driver as more as possible, however the C disk in Windows guest doesn''t support PV yet. (A task in BIOS side or Win HAL side?)> > Apart from that this sounds like a generalisation of the buffered-i/o > mechanism we already have for svga emulation. I suppose it might be > plausible.It is generic, similar to SVGA, but the timeout in generic I/O may be much smaller than display stuff, if a guest is waiting for an interrupt after several OUT instruction.> > -- Keir >Thx, Eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Oct-12 08:47 UTC
Re: [Xen-devel] Reducing I/O introduced domain scheduling
On 12/10/2010 08:19, "Dong, Eddie" <eddie.dong@intel.com> wrote:>> First of all have you tested with PV drivers? Performance tests with >> no PV drivers are not that interesting. > > We used PV driver as more as possible, however the C disk in Windows guest > doesn''t support PV yet. (A task in BIOS side or Win HAL side?)Running the system disk in PV mode certainly works okay with the Citrix PV drivers. Might this simply be a current limitation of the GPLPV drivers? If so, I definitely think you are better off putting effort into removing that limitation, rather than implementing your proposed optimisation. You will definitely get a bigger perf boost that way, and no suspicious hypervisor hacks needed. Cc''ing James Harper, who will best know what the current GPLPV drivers can/can''t do. Also Paul Durrant, who can likely help with advice on how to PV up the Windows system disk. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
James Harper
2010-Oct-12 08:55 UTC
Re: [Xen-devel] Reducing I/O introduced domain scheduling
Gplpv has supported all disks pv since the beginning. Recent versions turn off pv for the CDROM as qemu is more flexible (real CDROM works properly etc) but I think only under xen 3.4.1 or newer Sent from my iPhone On 12/10/2010, at 19:48, "Keir Fraser" <keir@xen.org> wrote:> On 12/10/2010 08:19, "Dong, Eddie" <eddie.dong@intel.com> wrote: > >>> First of all have you tested with PV drivers? Performance tests with >>> no PV drivers are not that interesting. >> >> We used PV driver as more as possible, however the C disk in Windows guest >> doesn''t support PV yet. (A task in BIOS side or Win HAL side?) > > Running the system disk in PV mode certainly works okay with the Citrix PV > drivers. Might this simply be a current limitation of the GPLPV drivers? If > so, I definitely think you are better off putting effort into removing that > limitation, rather than implementing your proposed optimisation. You will > definitely get a bigger perf boost that way, and no suspicious hypervisor > hacks needed. > > Cc''ing James Harper, who will best know what the current GPLPV drivers > can/can''t do. Also Paul Durrant, who can likely help with advice on how to > PV up the Windows system disk. > > -- Keir > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Paul Durrant
2010-Oct-12 09:15 UTC
[Xen-devel] RE: Reducing I/O introduced domain scheduling
Just wondering... does Xen/can Xen take VM exits on fences? If not then I don''t see you could safely buffer MMIO writes. Paul> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of Dong, Eddie > Sent: 12 October 2010 02:12 > To: Keir Fraser > Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao > Subject: [Xen-devel] Reducing I/O introduced domain scheduling > > Keir: > When running vConsolidation on top of Xen in a 4-core > platform, we noticed the I/O introduced scheduling per CPU is ~3K > Hz, which seems to be too frequent and cause frequent involve of > domain 0 / Qemu, which may polute cache of the guest and thus > increase CPI (cycle per instruction). > > We are thinking if we can reduce the domin switch here, and > think the output of I/O can be buffered and return immediately. The > buffered I/O can be flushed out at next IN emulation (or any > Hypervisor emulated I/O) or timeout such as 10 or 100 us to > guarantee minimal response. > > Ideally it can cover both PIO & MMIO, but we can start from > PIO. > > How do you think of that? > > Thx, Eddie > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Oct-12 09:18 UTC
Re: [Xen-devel] RE: Reducing I/O introduced domain scheduling
No, you can''t vmexit on a fence. I don''t know whether that matters, so long as buffered writes get flushed before the guest can observe their effects (presumably via some kind of I/O read). Agree that generalising the buffered I/O concept feels a bit dodgy however. -- Keir On 12/10/2010 10:15, "Paul Durrant" <Paul.Durrant@citrix.com> wrote:> Just wondering... does Xen/can Xen take VM exits on fences? If not then I > don''t see you could safely buffer MMIO writes. > > Paul > >> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >> bounces@lists.xensource.com] On Behalf Of Dong, Eddie >> Sent: 12 October 2010 02:12 >> To: Keir Fraser >> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao >> Subject: [Xen-devel] Reducing I/O introduced domain scheduling >> >> Keir: >> When running vConsolidation on top of Xen in a 4-core >> platform, we noticed the I/O introduced scheduling per CPU is ~3K >> Hz, which seems to be too frequent and cause frequent involve of >> domain 0 / Qemu, which may polute cache of the guest and thus >> increase CPI (cycle per instruction). >> >> We are thinking if we can reduce the domin switch here, and >> think the output of I/O can be buffered and return immediately. The >> buffered I/O can be flushed out at next IN emulation (or any >> Hypervisor emulated I/O) or timeout such as 10 or 100 us to >> guarantee minimal response. >> >> Ideally it can cover both PIO & MMIO, but we can start from >> PIO. >> >> How do you think of that? >> >> Thx, Eddie >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Paul Durrant
2010-Oct-12 09:30 UTC
RE: [Xen-devel] RE: Reducing I/O introduced domain scheduling
My concern is a read from a non-MMIO page following a write to an MMIO page. Paul> -----Original Message----- > From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir > Fraser > Sent: 12 October 2010 10:19 > To: Paul Durrant; Dong, Eddie > Cc: xen-devel@lists.xensource.com; Zhang, Xiantao > Subject: Re: [Xen-devel] RE: Reducing I/O introduced domain > scheduling > > No, you can''t vmexit on a fence. I don''t know whether that matters, > so long > as buffered writes get flushed before the guest can observe their > effects > (presumably via some kind of I/O read). Agree that generalising the > buffered > I/O concept feels a bit dodgy however. > > -- Keir > > On 12/10/2010 10:15, "Paul Durrant" <Paul.Durrant@citrix.com> wrote: > > > Just wondering... does Xen/can Xen take VM exits on fences? If not > then I > > don''t see you could safely buffer MMIO writes. > > > > Paul > > > >> -----Original Message----- > >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > >> bounces@lists.xensource.com] On Behalf Of Dong, Eddie > >> Sent: 12 October 2010 02:12 > >> To: Keir Fraser > >> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao > >> Subject: [Xen-devel] Reducing I/O introduced domain scheduling > >> > >> Keir: > >> When running vConsolidation on top of Xen in a 4-core > >> platform, we noticed the I/O introduced scheduling per CPU is ~3K > >> Hz, which seems to be too frequent and cause frequent involve of > >> domain 0 / Qemu, which may polute cache of the guest and thus > >> increase CPI (cycle per instruction). > >> > >> We are thinking if we can reduce the domin switch here, and > >> think the output of I/O can be buffered and return immediately. > The > >> buffered I/O can be flushed out at next IN emulation (or any > >> Hypervisor emulated I/O) or timeout such as 10 or 100 us to > >> guarantee minimal response. > >> > >> Ideally it can cover both PIO & MMIO, but we can start from > >> PIO. > >> > >> How do you think of that? > >> > >> Thx, Eddie > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Oct-12 09:44 UTC
Re: [Xen-devel] RE: Reducing I/O introduced domain scheduling
Are there any such scenarios that could reveal not-allowed-by-hardware-spec buffering? If so then that would make the optimisation invalid. But anyway, it seems that the Intel guys should just fix their test setup to get their system disk using PV mode. -- Keir On 12/10/2010 10:30, "Paul Durrant" <Paul.Durrant@citrix.com> wrote:> My concern is a read from a non-MMIO page following a write to an MMIO page. > > Paul > >> -----Original Message----- >> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir >> Fraser >> Sent: 12 October 2010 10:19 >> To: Paul Durrant; Dong, Eddie >> Cc: xen-devel@lists.xensource.com; Zhang, Xiantao >> Subject: Re: [Xen-devel] RE: Reducing I/O introduced domain >> scheduling >> >> No, you can''t vmexit on a fence. I don''t know whether that matters, >> so long >> as buffered writes get flushed before the guest can observe their >> effects >> (presumably via some kind of I/O read). Agree that generalising the >> buffered >> I/O concept feels a bit dodgy however. >> >> -- Keir >> >> On 12/10/2010 10:15, "Paul Durrant" <Paul.Durrant@citrix.com> wrote: >> >>> Just wondering... does Xen/can Xen take VM exits on fences? If not >> then I >>> don''t see you could safely buffer MMIO writes. >>> >>> Paul >>> >>>> -----Original Message----- >>>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- >>>> bounces@lists.xensource.com] On Behalf Of Dong, Eddie >>>> Sent: 12 October 2010 02:12 >>>> To: Keir Fraser >>>> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao >>>> Subject: [Xen-devel] Reducing I/O introduced domain scheduling >>>> >>>> Keir: >>>> When running vConsolidation on top of Xen in a 4-core >>>> platform, we noticed the I/O introduced scheduling per CPU is ~3K >>>> Hz, which seems to be too frequent and cause frequent involve of >>>> domain 0 / Qemu, which may polute cache of the guest and thus >>>> increase CPI (cycle per instruction). >>>> >>>> We are thinking if we can reduce the domin switch here, and >>>> think the output of I/O can be buffered and return immediately. >> The >>>> buffered I/O can be flushed out at next IN emulation (or any >>>> Hypervisor emulated I/O) or timeout such as 10 or 100 us to >>>> guarantee minimal response. >>>> >>>> Ideally it can cover both PIO & MMIO, but we can start from >>>> PIO. >>>> >>>> How do you think of that? >>>> >>>> Thx, Eddie >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2010-Oct-12 09:46 UTC
Re: [Xen-devel] RE: Reducing I/O introduced domain scheduling
At 10:30 +0100 on 12 Oct (1286879457), Paul Durrant wrote:> My concern is a read from a non-MMIO page following a write to an MMIO page.Yes, buffering MMIO in the general case is totally unsafe. That''s why the existing buffered-MMIO ring is only used for VGA. Tim.> Paul > > > -----Original Message----- > > From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir > > Fraser > > Sent: 12 October 2010 10:19 > > To: Paul Durrant; Dong, Eddie > > Cc: xen-devel@lists.xensource.com; Zhang, Xiantao > > Subject: Re: [Xen-devel] RE: Reducing I/O introduced domain > > scheduling > > > > No, you can''t vmexit on a fence. I don''t know whether that matters, > > so long > > as buffered writes get flushed before the guest can observe their > > effects > > (presumably via some kind of I/O read). Agree that generalising the > > buffered > > I/O concept feels a bit dodgy however. > > > > -- Keir > > > > On 12/10/2010 10:15, "Paul Durrant" <Paul.Durrant@citrix.com> wrote: > > > > > Just wondering... does Xen/can Xen take VM exits on fences? If not > > then I > > > don''t see you could safely buffer MMIO writes. > > > > > > Paul > > > > > >> -----Original Message----- > > >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > > >> bounces@lists.xensource.com] On Behalf Of Dong, Eddie > > >> Sent: 12 October 2010 02:12 > > >> To: Keir Fraser > > >> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao > > >> Subject: [Xen-devel] Reducing I/O introduced domain scheduling > > >> > > >> Keir: > > >> When running vConsolidation on top of Xen in a 4-core > > >> platform, we noticed the I/O introduced scheduling per CPU is ~3K > > >> Hz, which seems to be too frequent and cause frequent involve of > > >> domain 0 / Qemu, which may polute cache of the guest and thus > > >> increase CPI (cycle per instruction). > > >> > > >> We are thinking if we can reduce the domin switch here, and > > >> think the output of I/O can be buffered and return immediately. > > The > > >> buffered I/O can be flushed out at next IN emulation (or any > > >> Hypervisor emulated I/O) or timeout such as 10 or 100 us to > > >> guarantee minimal response. > > >> > > >> Ideally it can cover both PIO & MMIO, but we can start from > > >> PIO. > > >> > > >> How do you think of that? > > >> > > >> Thx, Eddie > > >> _______________________________________________ > > >> Xen-devel mailing list > > >> Xen-devel@lists.xensource.com > > >> http://lists.xensource.com/xen-devel > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, XenServer Engineering Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel