Hello, I''m trying to get to work InfiniBand card in DomU via exporting PCI device. How does the export works internally? I guess that pci frontend forwards stuff like pci_read_config_dword and pci_write_config_dword to pci backend that actually does it. Am I right? So if the driver of InfiniBand writes some config values to PCI config space and these values are not applied actually, then the problem is in pci front and back ends. Right? I.e., the PCI backend does not write the values to the device. It is currious that PCI frontend returns correct values. Could it be caused by some caching that the frontend may do? I''ve reported some troubles with InfiniBand cards in Xen but nobody cares. Nor the developers of the IB drivers nor people in the xen-dev list. :( -- Lukáš Hejtmánek _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m trying to get to work InfiniBand card in DomU via exporting PCI device.OK, cool.> How does the export works internally? > > I guess that pci frontend forwards stuff like pci_read_config_dword and > pci_write_config_dword to pci backend that actually does it. Am I right?IIRC, yes that is what is done. However the backend has various policies over what it allows. For instance, I think it doesn''t by default allow writes to BARs or to parts of config space not owned by this device.> So if the driver of InfiniBand writes some config values to PCI config > space and these values are not applied actually, then the problem is in pci > front and back ends. Right? I.e., the PCI backend does not write the values > to the device.That sounds right, yes. I think there''s a permissive mode for the backend where it lets things through, and there''s a means of listing devices that require more permissive behaviour on a per-device basis.> It is currious that PCI frontend returns correct values. > Could it be caused by some caching that the frontend may do?Uhhhh, possibly, I guess. Maybe it''s updating it''s own internal data structures but the changes aren''t getting applied? Have you checked if error messages are appearing in dmesg in either the domU or the dom0?> I''ve reported some troubles with InfiniBand cards in Xen but nobody cares. > Nor the developers of the IB drivers nor people in the xen-dev list. :(I care! IB on Xen should be really useful. There was a project (which I assume is still ongoing...?) to support virtualisation of IB devices in Xen, so that you can give several (PV initially) domains direct access to an IB device. This would leverage the IB device''s safe user accessibility feature to make this passthrough safe for untrusted domains whilst eliminating dom0 from the IO path. AFAIK those changes haven''t been applied to the mainline Xen tree. There''s another tree where they were being developed, but I don''t know a) how well it works b) how up-to-date it is or c) whether it''s currently being worked on. Hope that helps you some! Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Jun 29, 2007 at 01:00:46PM +0100, Mark Williamson wrote:> That sounds right, yes. I think there''s a permissive mode for the backend > where it lets things through, and there''s a means of listing devices that > require more permissive behaviour on a per-device basis.Yes, I use permissive mode for the device. However, I noticed that even in permissive mode, not all values are allowed. Forcing the pcibackend to allow direct writes into the device it solves the problem (partly). Using the force, the device is able to recovery from the device reset but it does not receive IRQ. I found out that it may be caused by wrong reading the IRQ from the device instead of using quirk and reading it from dev->irq structure. Should such quirk help? I will test it next week as I have some huge problems with the DomU - the kernel crashes too much, I guess I corrupted the root with XFS file system.> Have you checked if error messages are appearing in dmesg in either the domU > or the dom0?Yeah, nothing special there.> > I''ve reported some troubles with InfiniBand cards in Xen but nobody cares. > > Nor the developers of the IB drivers nor people in the xen-dev list. :( > > I care! IB on Xen should be really useful.OK, great!> There was a project (which I assume is still ongoing...?) to support > virtualisation of IB devices in Xen, so that you can give several (PV > initially) domains direct access to an IB device. This would leverage the IB > device''s safe user accessibility feature to make this passthrough safe for > untrusted domains whilst eliminating dom0 from the IO path.Yes, I know there is an reborn effort for IB support in DomU via Novell and OpenFabrics. I did not test it yes because they have no support for Xen 3.1 and 2.6.18 kernel now. -- Lukáš Hejtmánek _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel