Yijing Wang
2014-Aug-04 06:43 UTC
[RFC PATCH 00/11] Refactor MSI to support Non-PCI device
On 2014/8/1 21:52, Arnd Bergmann wrote:> On Wednesday 30 July 2014, Yijing Wang wrote: >> On 2014/7/29 22:08, Arnd Bergmann wrote: >>> On Saturday 26 July 2014 11:08:37 Yijing Wang wrote: >>>> >>>> The new data struct for generic MSI driver. >>>> struct msi_irqs { >>>> u8 msi_enabled:1; /* Enable flag */ >>>> u8 msix_enabled:1; >>>> struct list_head msi_list; /* MSI desc list */ >>>> void *data; /* help to find the MSI device */ >>>> struct msi_ops *ops; /* MSI device specific hook */ >>>> }; >>>> struct msi_irqs is used to manage MSI related informations. Every device supports >>>> MSI should contain this data struct and allocate it. >>> >>> I think you should have a stronger association with the 'struct >>> device' here. Can you replace the 'void *data' with 'struct device *dev'? >> >> Actually, I used the struct device *dev in my first draft, finally, I replaced >> it with void *data, because some MSI devices don't have a struct device *dev, >> like the existing hpet device, dmar msi device, and OF device, like the ARM consolidator. >> >> Of course, we can make the MSI devices have their own struct device, and register to >> device tree, eg, add a class device named MSI_DEV. But I'm not sure whether it is appropriate. > > It doesn't have to be in the (OF) device tree, but I think it absolutely makes > sense to use the 'struct device' infrastructure here, as almost everything uses > a device, and the ones that don't do that today can be easily changed.I will try to use "struct device" infrastructure, thanks for your suggestion. :)> >>> The other part I'm not completely sure about is how you want to >>> have MSIs map into normal IRQ descriptors. At the moment, all >>> MSI users are based on IRQ numbers, but this has known scalability problems. >> >> Hmmm, I still use the IRQ number to map the MSIs to IRQ description. >> I'm sorry, I don't understand you meaning. >> What are the scalability problems you mentioned ? >> For device drivers, they always process interrupt in two steps. >> If irq is the legacy interrupt, drivers will first >> use the irq_of_parse_and_map() or pci_enable_device() to parse and get the IRQ number. >> Then drivers will call the request_irq() to register the interrupt handler. >> If irq is MSIs, first call pci_enable_msi/x() to get the IRQ number and then call >> request_irq() to register interrupt handler. > > The method you describe here makes sense for PCI devices that are required to support > legacy interrupts and may or may not support MSI on a given system, but not so much > for platform devices for which we know exactly whether we want to use MSI > or legacy interrupts. > > In particular if you have a device that can only do MSI, the entire pci_enable_msi > step is pointless: all we need to do is program the correct MSI target address/message > pair into the device and register the handler.Yes, I almost agree if we won't change the existing hundreds of drivers, what I worried about is some drivers may want to know the IRQ numbers, and use the IRQ number to process different things, as I mentioned in another reply. But we can also provide the interface which integrate MSI configuration and request_irq(), if most drivers don't care the IRQ number.> >>> I wonder if we can do the interface in a way that >>> hides the interrupt number from generic device drivers and just >>> passes a 'struct irq_desc'. Note that there are long-term plans to >>> get rid of IRQ numbers entirely, but those plans have existed for >>> a long time already without anybody seriously addressing the device >>> driver interfaces so far, so it might never really happen. >>> >> >> Maybe this is a huge work, now hundreds drivers use the IRQ number, so maybe we can consider >> this in a separate title. > > Sorry for being unclear here: I did suggest changing all drivers now. What I meant > is that we use a different API for non-PCI devices that works without IRQ numbers. > I don't think we should touch the PCI interfaces at this point.OK, I got it.>>> What I'd envision as the API from the device driver perspective is something >>> as simple like this: >>> >>> struct msi_desc *msi_request(struct msi_chip *chip, irq_handler_t handler, >>> unsigned long flags, const char *name, struct device *dev); >>> >>> which would get an msi descriptor that is valid for this device (dev) >>> connected to a particular msi_chip, and associate a handler function >>> with it. The device driver can call that function and retrieve the >>> address/message pair from the msi_desc in order to store it in its own >>> device specific registers. The request_irq() can be handled internally >>> to msi_request(). >> >> This is a huge change for device drivers, and some device drivers don't know which msi_chip >> their MSI irq deliver to. I'm reworking the msi_chip, and try to use msi_chip to eliminate >> all arch_msi_xxx() under every arch in kernel. And the important point is how to create the >> binding for the MSI device to the target msi_chip. > > Which drivers are you thinking of? Again, I wouldn't expect to change any PCI drivers, > but only platform drivers that do native MSI, so we only have to change drivers that > do not support any MSI at all yet and that need to be changed anyway in order to add > support.I mean platform device drivers, because we can find the target msi_chip by some platform interfaces(like the existing of_pci_find_msi_chip_by_node()). So we no need to explicitly provide the msi_chip as the function argument.> >> For PCI device, some arm platform already bound the msi_chip to the pci hostbridge, then all >> pci devices under the pci hostbridge deliver their MSI irqs to the target msi_chip. >> And other platform create the binding in DTS file, then the MSI device can find their msi_chip >> by device_node. >> I don't know whether there are other situations, we should provide a generic interface that >> every MSI device under every platform can use it to find its msi_chip exactly. > > We have introduced the "msi-parent" property to mirror the "interrupt-parent" property. > For the PCI case, this property is only needed in the PCI host controller, and there > can be a system-wide default, by putting the "msi-parent" property into the root device > node or the node of a bus that is parent to all devices supporting MSI. > > For non-PCI devices, it should be possible to override the "msi-parent" property per > device, but those can also use the global property. > > The main use case that I see are PCI host controllers that have their own MSI catcher > included, so meaning that any PCI device can either send its MSIs there, or to a > system-wide GICv3 instance, and we need a way to select which one.Yes, agree.> > A degenerate case of this would be a system where a PCI device sends its MSI into > the host controller, that generates a legacy interrupt and that in turn gets > sent to an irqchip which turns it back into an MSI for the GICv3. This would of > course be very inefficient, but I think we should be able to express this with > both the binding and the in-kernel framework just to be on the safe side.Yes, the best way to tell the kernel which msi_chip should deliver to is describe the binding in DTS file. If a real degenerate case found, we can update the platform interface which is responsible for getting the match msi_chip in future.> > > Arnd > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > > . >-- Thanks! Yijing
Arnd Bergmann
2014-Aug-04 14:59 UTC
[RFC PATCH 00/11] Refactor MSI to support Non-PCI device
On Monday 04 August 2014, Yijing Wang wrote:> On 2014/8/1 21:52, Arnd Bergmann wrote: > > On Wednesday 30 July 2014, Yijing Wang wrote: > >> On 2014/7/29 22:08, Arnd Bergmann wrote: > >>> The other part I'm not completely sure about is how you want to > >>> have MSIs map into normal IRQ descriptors. At the moment, all > >>> MSI users are based on IRQ numbers, but this has known scalability problems. > >> > >> Hmmm, I still use the IRQ number to map the MSIs to IRQ description. > >> I'm sorry, I don't understand you meaning. > >> What are the scalability problems you mentioned ? > >> For device drivers, they always process interrupt in two steps. > >> If irq is the legacy interrupt, drivers will first > >> use the irq_of_parse_and_map() or pci_enable_device() to parse and get the IRQ number. > >> Then drivers will call the request_irq() to register the interrupt handler. > >> If irq is MSIs, first call pci_enable_msi/x() to get the IRQ number and then call > >> request_irq() to register interrupt handler. > > > > The method you describe here makes sense for PCI devices that are required to support > > legacy interrupts and may or may not support MSI on a given system, but not so much > > for platform devices for which we know exactly whether we want to use MSI > > or legacy interrupts. > > > > In particular if you have a device that can only do MSI, the entire pci_enable_msi > > step is pointless: all we need to do is program the correct MSI target address/message > > pair into the device and register the handler. > > Yes, I almost agree if we won't change the existing hundreds of drivers, what > I worried about is some drivers may want to know the IRQ numbers, and use the IRQ > number to process different things, as I mentioned in another reply. > But we can also provide the interface which integrate MSI configuration and request_irq(), > if most drivers don't care the IRQ number.The driver would still have the option of getting the IRQ number for now: With the interface I imagine, you would get a 'struct msi_desc' pointer, from which you can look up the 'struct irq_desc' pointer (either embedded in msi_desc, or using a pointer from a member of msi_desc), and you can already get the interrupt number from the irq_desc. My point was that a well-written driver already does not care about the interrupt number: the only information a driver needs in the interrupt handler is a pointer to its own context, which we already derive from the irq_desc. The main interface that currently requires the irq number is free_irq(), but I would argue that we can just add a wrapper that takes the msi_desc pointer as its first argument so the driver does not have to worry about it. We can add additional wrappers like that as needed.> >>> What I'd envision as the API from the device driver perspective is something > >>> as simple like this: > >>> > >>> struct msi_desc *msi_request(struct msi_chip *chip, irq_handler_t handler, > >>> unsigned long flags, const char *name, struct device *dev); > >>> > >>> which would get an msi descriptor that is valid for this device (dev) > >>> connected to a particular msi_chip, and associate a handler function > >>> with it. The device driver can call that function and retrieve the > >>> address/message pair from the msi_desc in order to store it in its own > >>> device specific registers. The request_irq() can be handled internally > >>> to msi_request(). > >> > >> This is a huge change for device drivers, and some device drivers don't know which msi_chip > >> their MSI irq deliver to. I'm reworking the msi_chip, and try to use msi_chip to eliminate > >> all arch_msi_xxx() under every arch in kernel. And the important point is how to create the > >> binding for the MSI device to the target msi_chip. > > > > Which drivers are you thinking of? Again, I wouldn't expect to change any PCI drivers, > > but only platform drivers that do native MSI, so we only have to change drivers that > > do not support any MSI at all yet and that need to be changed anyway in order to add > > support. > > I mean platform device drivers, because we can find the target msi_chip by some platform > interfaces(like the existing of_pci_find_msi_chip_by_node()). So we no need to explicitly > provide the msi_chip as the function argument.Right, that works too. I was thinking we might need an interface that allows us to pick a particular msi_chip if there are several alternatives (e.g. one in the GIC and one in the PCI host), but you are right: we should normally be able to hardwire that information in DT or elsewhere, and just need the 'struct device pointer' which should probably be the first argument here. As you pointed out, it's common to have multiple MSIs for a single device, so we also need a context to pass around, so my suggestion would become something like: struct msi_desc *msi_request(struct device *dev, irq_handler_t handler, unsigned long flags, const char *name, void *data); It's possible that we have to add one or two more arguments here.> > A degenerate case of this would be a system where a PCI device sends its MSI into > > the host controller, that generates a legacy interrupt and that in turn gets > > sent to an irqchip which turns it back into an MSI for the GICv3. This would of > > course be very inefficient, but I think we should be able to express this with > > both the binding and the in-kernel framework just to be on the safe side. > > Yes, the best way to tell the kernel which msi_chip should deliver to is describe > the binding in DTS file. If a real degenerate case found, we can update the platform > interface which is responsible for getting the match msi_chip in future.Ok. Arnd
Yijing Wang
2014-Aug-05 02:12 UTC
[RFC PATCH 00/11] Refactor MSI to support Non-PCI device
>>> The method you describe here makes sense for PCI devices that are required to support >>> legacy interrupts and may or may not support MSI on a given system, but not so much >>> for platform devices for which we know exactly whether we want to use MSI >>> or legacy interrupts. >>> >>> In particular if you have a device that can only do MSI, the entire pci_enable_msi >>> step is pointless: all we need to do is program the correct MSI target address/message >>> pair into the device and register the handler. >> >> Yes, I almost agree if we won't change the existing hundreds of drivers, what >> I worried about is some drivers may want to know the IRQ numbers, and use the IRQ >> number to process different things, as I mentioned in another reply. >> But we can also provide the interface which integrate MSI configuration and request_irq(), >> if most drivers don't care the IRQ number. > > The driver would still have the option of getting the IRQ number for now: With > the interface I imagine, you would get a 'struct msi_desc' pointer, from which > you can look up the 'struct irq_desc' pointer (either embedded in msi_desc, > or using a pointer from a member of msi_desc), and you can already get the > interrupt number from the irq_desc. > > My point was that a well-written driver already does not care about the interrupt > number: the only information a driver needs in the interrupt handler is a pointer > to its own context, which we already derive from the irq_desc.Agree, I will try to introduce this similar interface in next version, thanks!> > The main interface that currently requires the irq number is free_irq(), but > I would argue that we can just add a wrapper that takes the msi_desc pointer > as its first argument so the driver does not have to worry about it. > > We can add additional wrappers like that as needed.OK>>>> This is a huge change for device drivers, and some device drivers don't know which msi_chip >>>> their MSI irq deliver to. I'm reworking the msi_chip, and try to use msi_chip to eliminate >>>> all arch_msi_xxx() under every arch in kernel. And the important point is how to create the >>>> binding for the MSI device to the target msi_chip. >>> >>> Which drivers are you thinking of? Again, I wouldn't expect to change any PCI drivers, >>> but only platform drivers that do native MSI, so we only have to change drivers that >>> do not support any MSI at all yet and that need to be changed anyway in order to add >>> support. >> >> I mean platform device drivers, because we can find the target msi_chip by some platform >> interfaces(like the existing of_pci_find_msi_chip_by_node()). So we no need to explicitly >> provide the msi_chip as the function argument. > > Right, that works too. I was thinking we might need an interface that allows us to > pick a particular msi_chip if there are several alternatives (e.g. one in the GIC > and one in the PCI host), but you are right: we should normally be able to hardwire > that information in DT or elsewhere, and just need the 'struct device pointer' which > should probably be the first argument here. > > As you pointed out, it's common to have multiple MSIs for a single device, so we > also need a context to pass around, so my suggestion would become something like: > > struct msi_desc *msi_request(struct device *dev, irq_handler_t handler, > unsigned long flags, const char *name, void *data); > > It's possible that we have to add one or two more arguments here.Good suggestion, thanks!> >>> A degenerate case of this would be a system where a PCI device sends its MSI into >>> the host controller, that generates a legacy interrupt and that in turn gets >>> sent to an irqchip which turns it back into an MSI for the GICv3. This would of >>> course be very inefficient, but I think we should be able to express this with >>> both the binding and the in-kernel framework just to be on the safe side. >> >> Yes, the best way to tell the kernel which msi_chip should deliver to is describe >> the binding in DTS file. If a real degenerate case found, we can update the platform >> interface which is responsible for getting the match msi_chip in future. > > Ok. > > Arnd > > . >-- Thanks! Yijing
Reasonably Related Threads
- [RFC PATCH 00/11] Refactor MSI to support Non-PCI device
- [RFC PATCH 00/11] Refactor MSI to support Non-PCI device
- [RFC PATCH 00/11] Refactor MSI to support Non-PCI device
- [RFC PATCH 00/11] Refactor MSI to support Non-PCI device
- [RFC PATCH 00/11] Refactor MSI to support Non-PCI device