Hi Ryan, I remember you showing some interest when I suggested rebinding PCI devices automatically. I''ve hacked together a quick prototype of "what I meant". I put a bundle at: http://www.cambridge.intel-research.net/~mwilli2/pci_auto_rebind.hg You can fetch my changes into a repository using: hg unbundle <url> The behaviour is: * If the device specified in a domain config file is already bound to another driver in dom0, fail creation with an error - as before. * If the device specified in the config is not currently bound to a driver, add it to the pciback driver and create the domain. * If the device specificed in the config is bound *and* pci-force-rebind is set, then unbind it from the existing driver and rebind it to pciback, then start the domain. pci-force-rebind allows users who want total automation in the rebinding process to have it, whilst by default preventing the automatic rebinding from stealing devices dom0 is relying on. I''d be interested to see what you think about it - in particular, whether there are potential failures in the rebinding code that I ought to be catching (e.g. if drivers won''t unbind from a device - can that happen?). Any objections to the general functionality? Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Apr 04, 2006 at 05:43:32PM +0100, Mark Williamson wrote:> The behaviour is: > * If the device specified in a domain config file is already bound to another > driver in dom0, fail creation with an error - as before. > * If the device specified in the config is not currently bound to a driver, > add it to the pciback driver and create the domain.That sounds very reasonable.> * If the device specificed in the config is bound *and* pci-force-rebind is > set, then unbind it from the existing driver and rebind it to pciback, then > start the domain.That sounds awkward. What''s the use case you see in mind from stealing devices from a domain while it''s running? For what it''s worth, we''ll be looking soon at creating IOMMU IO spaces for devices automatically when they''re granted to domains. Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> * If the device specificed in the config is bound *and* >> pci-force-rebind is set, then unbind it from the existing driver and >> rebind it to pciback, then start the domain. > >That sounds awkward. What''s the use case you see in mind from stealing >devices from a domain while it''s running?If a driver in domain 0 has grabbed a device but isn''t actually doing anything useful with it I can''t start a domU without explicitly unbinding. Rather than having to do the unbind (or add it to rc.local, or whatever) it''s somewhat convenient to have Xend trust my domU device allocations as being correct and override this binding. It won''t steal from another domU, however. I see switching force-rebind off by default as being a "safety catch": it''ll stop new users for PCI passthrough from shooting themselves in the foot. When force-rebind is on, the behaviour is suitable either for an experienced administrator with a known working config (I''d think they''re not going to change PCI device allocations often) or for config files generated by some other management tool (which can do safety checks when allocating the device). In either of these cases, the user is confident that rebinding is the correct thing to do, and can let Xend do it automatically.>For what it''s worth, we''ll be looking soon at creating IOMMU IO spaces >for devices automatically when they''re granted to domains.That would be way cool. :-) Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > * If the device specificed in the config is bound *and* pci-force-rebind > > is set, then unbind it from the existing driver and rebind it to pciback, > > then start the domain. > > That sounds awkward. What''s the use case you see in mind from stealing > devices from a domain while it''s running?To clarify my thinking on this a little more: I think it would be good to be able to move PCI devices between domains in a reasonably straightforward way. This would be something like the PCI hotplug for IBM''s dynamic LPAR, which is already supported by Linux. The eventual goal being that you can reassign PCI devices "hotplug-style" (it would look like a hotplug to the pcifront kernel) at runtime, with the user interface providing appropriate safety checks / "do you really want this?" dialogs, according to the level of confidence of the user. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Apr 05, 2006 at 04:18:16PM +0100, Mark Williamson wrote:> To clarify my thinking on this a little more: I think it would be good to be > able to move PCI devices between domains in a reasonably straightforward way. > This would be something like the PCI hotplug for IBM''s dynamic LPAR, which is > already supported by Linux. > > The eventual goal being that you can reassign PCI devices "hotplug-style" (it > would look like a hotplug to the pcifront kernel) at runtime, with the user > interface providing appropriate safety checks / "do you really want this?" > dialogs, according to the level of confidence of the user.Thank for the clarification; sounds good to me! Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2006-04-04 at 17:43 +0100, Mark Williamson wrote:> Hi Ryan, > > I remember you showing some interest when I suggested rebinding PCI devices > automatically. I''ve hacked together a quick prototype of "what I meant". > > I put a bundle at: > http://www.cambridge.intel-research.net/~mwilli2/pci_auto_rebind.hg > > You can fetch my changes into a repository using: > hg unbundle <url> > > The behaviour is: > * If the device specified in a domain config file is already bound to another > driver in dom0, fail creation with an error - as before. > * If the device specified in the config is not currently bound to a driver, > add it to the pciback driver and create the domain. > * If the device specificed in the config is bound *and* pci-force-rebind is > set, then unbind it from the existing driver and rebind it to pciback, then > start the domain. > > pci-force-rebind allows users who want total automation in the rebinding > process to have it, whilst by default preventing the automatic rebinding from > stealing devices dom0 is relying on. > > I''d be interested to see what you think about it - in particular, whether > there are potential failures in the rebinding code that I ought to be > catching (e.g. if drivers won''t unbind from a device - can that happen?). > Any objections to the general functionality? > > Cheers, > Mark >Hi Mark, Sorry for the delay in getting back to you. I finally had an opportunity to look at your code. FWIW, I think it''s a good idea that can only make things easier for the Xen user. I definitely agree that it''s better to default the rebinding by force feature to off. A driver can''t do anything to stop the unbinding of a device. At least not with the current code in Linux. The driver can be notified (if it implements the remove callback), but it can''t stop it. A driver can only prevent binding. That said, it''s probably a good idea to check after binding that the device really did bind to the pciback driver (there will be a symlink in the pciback driver directory with the name of the device if binding occurred successfully) just in case the Linux behavior changes or some strange error occurs. You may want to also add a test to check if the pciback driver is even loaded (and possibly try a modprobe if it''s not). On a stylistic note (which is strictly my personal opinion, it doesn''t really matter how it''s done), I think self.driver should be equal to None if there is no driver already bound to a device (instead of the empty string). However, I think my code (the sysfs and proc parts) assumed there would always be a driver (which was an incorrect assumption) so this change would require a couple of other changes in this file to work. So, no objections here. :) It''s a great idea to make the binding happen transparently. Ryan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel