Hi, As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ, and guest does EOI (provided conditions) the maintenance interrupt is not needed. In xen arch/arm/gic.c always while setting an LR using gic_set_lr , the maintenance_int is enabled. Can some one clear the doubt on why it is done a) is this because EOI by guest would result in control back to hypervisor to replenish the LR for inflight, as there is otherwise no way to know in hypervisor that the LR is available. or b) some other reason ?
Ian Campbell
2013-Oct-31 15:43 UTC
Re: [query] gic_set_lr always uses maintenance Interrupt
On Wed, 2013-10-30 at 15:08 +0530, Mj Embd wrote:> Hi, > > As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If > hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ,Is it? You mean LR.HW is set? AIUI this means the IRQ is an actual hardware IRQ which is passed through, whereas the VIRQs are entirely fictional virtual interrupts. (or maybe you mean some other sort of VIRQ?)> and guest does EOI (provided conditions) the maintenance interrupt is > not needed. > > In xen arch/arm/gic.c always while setting an LR using gic_set_lr , > the maintenance_int is enabled. > > Can some one clear the doubt on why it is done > a) is this because EOI by guest would result in control back to > hypervisor to replenish the LR for inflight, as there is otherwise > no way to know in hypervisor that the LR is available. > or > b) some other reason ?CCing Stefano, he understands this stuff and IIRC there was some reason your a) didn''t work in practice. Ian.
On Thu, Oct 31, 2013 at 9:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Wed, 2013-10-30 at 15:08 +0530, Mj Embd wrote: >> Hi, >> >> As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If >> hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ, > > Is it? You mean LR.HW is set? AIUI this means the IRQ is an actual > hardware IRQ which is passed through, whereas the VIRQs are entirely > fictional virtual interrupts. (or maybe you mean some other sort of > VIRQ?) >If say a timer HW interrupt is injected into guest. And if guest does EOI, the HW interrupt is deactivated. This is an optimisation as per ARM which saves a trap to hypervisor which is in form of a maintenance interrupt. I see in the code gic_set_lr always chooses an option for maintenance interrupt. So it is not using ARM''s optimisation technique. The reason I thought why this is done is bcase problem I see with this is that hypervisor still has its LR used and has no way of determining when it can replenish LR for further interrupts to be injected in guest. But I could be wrong, why would ARM provide a way an keep a loophole>> and guest does EOI (provided conditions) the maintenance interrupt is >> not needed. >> >> In xen arch/arm/gic.c always while setting an LR using gic_set_lr , >> the maintenance_int is enabled. >> >> Can some one clear the doubt on why it is done >> a) is this because EOI by guest would result in control back to >> hypervisor to replenish the LR for inflight, as there is otherwise >> no way to know in hypervisor that the LR is available. >> or >> b) some other reason ? > > CCing Stefano, he understands this stuff and IIRC there was some reason > your a) didn''t work in practice. > > Ian. >-- -mj
Stefano Stabellini
2013-Nov-06 18:24 UTC
Re: [query] gic_set_lr always uses maintenance Interrupt
On Thu, 31 Oct 2013, Mj Embd wrote:> On Thu, Oct 31, 2013 at 9:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Wed, 2013-10-30 at 15:08 +0530, Mj Embd wrote: > >> Hi, > >> > >> As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If > >> hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ, > > > > Is it? You mean LR.HW is set? AIUI this means the IRQ is an actual > > hardware IRQ which is passed through, whereas the VIRQs are entirely > > fictional virtual interrupts. (or maybe you mean some other sort of > > VIRQ?) > > > > If say a timer HW interrupt is injected into guest. And if guest does > EOI, the HW interrupt is deactivated. > This is an optimisation as per ARM which saves a trap to hypervisor > which is in form of a maintenance interrupt. > > I see in the code gic_set_lr always chooses an option for maintenance > interrupt. So it is not using ARM''s optimisation technique. > > The reason I thought why this is done is bcase problem I see with this > is that hypervisor still has its LR used and has no way of determining > when it can replenish LR for further interrupts to be injected in > guest. But I could be wrong, why would ARM provide a way an keep a > loopholeYour analysis is correct. Even if the hardware interrupt is deactivated how can Xen know exactly which LRs need to be cleared and when? If you can figure out the answer to these questions we might be able to start avoiding maintenance interrupts on hardware interrupts :)
A few thoughts are circling around my mind, don''t know how much interrupt latency would it have. Rather than the hypervisor entry when guest does EOI, a late / lazy checkin on LR''s can be done on next hypervisor entry by a) guest doing something and trapping to hypervisor b) schedular timer in hypervisor What do you think on this... On Wed, Nov 6, 2013 at 11:54 PM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Thu, 31 Oct 2013, Mj Embd wrote: >> On Thu, Oct 31, 2013 at 9:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> > On Wed, 2013-10-30 at 15:08 +0530, Mj Embd wrote: >> >> Hi, >> >> >> >> As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If >> >> hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ, >> > >> > Is it? You mean LR.HW is set? AIUI this means the IRQ is an actual >> > hardware IRQ which is passed through, whereas the VIRQs are entirely >> > fictional virtual interrupts. (or maybe you mean some other sort of >> > VIRQ?) >> > >> >> If say a timer HW interrupt is injected into guest. And if guest does >> EOI, the HW interrupt is deactivated. >> This is an optimisation as per ARM which saves a trap to hypervisor >> which is in form of a maintenance interrupt. >> >> I see in the code gic_set_lr always chooses an option for maintenance >> interrupt. So it is not using ARM''s optimisation technique. >> >> The reason I thought why this is done is bcase problem I see with this >> is that hypervisor still has its LR used and has no way of determining >> when it can replenish LR for further interrupts to be injected in >> guest. But I could be wrong, why would ARM provide a way an keep a >> loophole > > Your analysis is correct. Even if the hardware interrupt is deactivated > how can Xen know exactly which LRs need to be cleared and when? > > If you can figure out the answer to these questions we might be able to > start avoiding maintenance interrupts on hardware interrupts :)-- -mj
Ian Campbell
2013-Nov-07 11:29 UTC
Re: [query] gic_set_lr always uses maintenance Interrupt
(please don''t top post) On Thu, 2013-11-07 at 05:29 +0530, Mj Embd wrote:> A few thoughts are circling around my mind, don''t know how much > interrupt latency would it have. > > Rather than the hypervisor entry when guest does EOI, a late / lazy > checkin on LR''s can be done > on next hypervisor entry by > a) guest doing something and trapping to hypervisor > b) schedular timer in hypervisor > > What do you think on this...If you have e.g. 5 pending interrupts and 4 LR registers then ideally you would want to inject the 5 highest priorities into the LRs immediately and then inject the fifth as soon as a slot became available. Or maybe it could be deferred until all of the original 4 are acked, since it is lower priority all four would need to be handled before it came up. Ian.
Stefano Stabellini
2013-Nov-07 11:37 UTC
Re: [query] gic_set_lr always uses maintenance Interrupt
Please don''t top post as it makes it harder to follow the conversation. On Thu, 7 Nov 2013, Mj Embd wrote:> A few thoughts are circling around my mind, don''t know how much > interrupt latency would it have. > > Rather than the hypervisor entry when guest does EOI, a late / lazy > checkin on LR''s can be done > on next hypervisor entry by > a) guest doing something and trapping to hypervisor > b) schedular timer in hypervisor > > What do you think on this...It might work. One key issue is how to identify that the guest EOIed a particular irq and henceforth the corresponding LR can be reused. I hope that the status bits in the LR register reflect this condition. Maybe the status becomes 00 invalid after the guest does EOI? Otherwise I can''t really see how we could make it work.
On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> Please don''t top post as it makes it harder to follow the conversation. > > On Thu, 7 Nov 2013, Mj Embd wrote: >> A few thoughts are circling around my mind, don''t know how much >> interrupt latency would it have. >> >> Rather than the hypervisor entry when guest does EOI, a late / lazy >> checkin on LR''s can be done >> on next hypervisor entry by >> a) guest doing something and trapping to hypervisor >> b) schedular timer in hypervisor >> >> What do you think on this... > > It might work. > One key issue is how to identify that the guest EOIed a particular irq > and henceforth the corresponding LR can be reused.[mj] I believe that GICH_ELSR0/1 can be read anytime to get the status.> I hope that the status bits in the LR register reflect this condition. > Maybe the status becomes 00 invalid after the guest does EOI? Otherwise[mj] The state in LR is marked invalid by Virtual CPU interface.> I can''t really see how we could make it work.I am planning to stop using gmail, it makes you top post as it hides everything. -- -mj
Ian Campbell
2013-Nov-07 16:46 UTC
Re: [query] gic_set_lr always uses maintenance Interrupt
On Thu, 2013-11-07 at 22:14 +0530, Mj Embd wrote:> On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > Please don''t top post as it makes it harder to follow the conversation. > > > > On Thu, 7 Nov 2013, Mj Embd wrote: > >> A few thoughts are circling around my mind, don''t know how much > >> interrupt latency would it have. > >> > >> Rather than the hypervisor entry when guest does EOI, a late / lazy > >> checkin on LR''s can be done > >> on next hypervisor entry by > >> a) guest doing something and trapping to hypervisor > >> b) schedular timer in hypervisor > >> > >> What do you think on this... > > > > It might work. > > One key issue is how to identify that the guest EOIed a particular irq > > and henceforth the corresponding LR can be reused. > [mj] I believe that GICH_ELSR0/1 can be read anytime to get the status. > > > I hope that the status bits in the LR register reflect this condition. > > Maybe the status becomes 00 invalid after the guest does EOI? Otherwise > [mj] The state in LR is marked invalid by Virtual CPU interface.If you can make it work then we''d be interested in seeing the patches. Ian.
Stefano Stabellini
2013-Nov-07 16:48 UTC
Re: [query] gic_set_lr always uses maintenance Interrupt
On Thu, 7 Nov 2013, Mj Embd wrote:> On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > Please don''t top post as it makes it harder to follow the conversation. > > > > On Thu, 7 Nov 2013, Mj Embd wrote: > >> A few thoughts are circling around my mind, don''t know how much > >> interrupt latency would it have. > >> > >> Rather than the hypervisor entry when guest does EOI, a late / lazy > >> checkin on LR''s can be done > >> on next hypervisor entry by > >> a) guest doing something and trapping to hypervisor > >> b) schedular timer in hypervisor > >> > >> What do you think on this... > > > > It might work. > > One key issue is how to identify that the guest EOIed a particular irq > > and henceforth the corresponding LR can be reused. > [mj] I believe that GICH_ELSR0/1 can be read anytime to get the status. > > > I hope that the status bits in the LR register reflect this condition. > > Maybe the status becomes 00 invalid after the guest does EOI? Otherwise > [mj] The state in LR is marked invalid by Virtual CPU interface.Right. In that case the lazy LR clearance should work.> > I can''t really see how we could make it work. > > I am planning to stop using gmail, it makes you top post as it hides > everything.Yes, I use gmail for my personal email and I find it very inconvenient for anything but simple messages. Thank you.
On Thu, Nov 7, 2013 at 10:18 PM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Thu, 7 Nov 2013, Mj Embd wrote: >> On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini >> <stefano.stabellini@eu.citrix.com> wrote: >> > Please don''t top post as it makes it harder to follow the conversation. >> > >> > On Thu, 7 Nov 2013, Mj Embd wrote: >> >> A few thoughts are circling around my mind, don''t know how much >> >> interrupt latency would it have. >> >> >> >> Rather than the hypervisor entry when guest does EOI, a late / lazy >> >> checkin on LR''s can be done >> >> on next hypervisor entry by >> >> a) guest doing something and trapping to hypervisor >> >> b) schedular timer in hypervisor >> >> >> >> What do you think on this... >> > >> > It might work. >> > One key issue is how to identify that the guest EOIed a particular irq >> > and henceforth the corresponding LR can be reused. >> [mj] I believe that GICH_ELSR0/1 can be read anytime to get the status. >> >> > I hope that the status bits in the LR register reflect this condition. >> > Maybe the status becomes 00 invalid after the guest does EOI? Otherwise >> [mj] The state in LR is marked invalid by Virtual CPU interface. > > Right. In that case the lazy LR clearance should work. >[mj] I have started analysing it, can i send a patch in near future>> > I can''t really see how we could make it work. >> >> I am planning to stop using gmail, it makes you top post as it hides >> everything. > > Yes, I use gmail for my personal email and I find it very inconvenient > for anything but simple messages. Thank you.-- -mj