Hello, I have been debugging kexec interaction problems with XenServer and found that the problem lies in how Xen tares down the computer in a crash. The Xen kexec path does not touch IOMMU at all, which leaves the kexec native kernel with interrupt remapping enabled without realizing it. This leads to the kexec kernel failing to understand why its interrupts aren''t working. As a debugging measure, I have put iommu_ops->suspend() and iommu_disable_IR() on the kexec path and this ''fixes'' the problem, although it is far from safe. From a correctness point of view, Xen really does need to shutdown all IOMMU remapping before it jumps to the crash kernel. I know that kdump is a "seat of the pants best effort" in the best case, but there is more which Xen needs to do to help it along. I was considering adding a crash_shutdown function to iommu_ops which goes and twiddles the relevant disable bits, without saving state. However, disabling DMA remapping while transfers are still ongoing is likely asking for trouble. Seeing as people on here are likely to know far more than me on this subject: 1) Is there a systematic way to find and disable active DMA transfers, or indeed a systematic way to shut down PCI (etc) devices which is safe for the kexec path. 2) Are there any other PC subsystems which could do with being shut down in a sensible manor to make life easier for the kdump kernel? Thanks in advance, ~Andrew -- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I believe Jan was involved with adding kexec support in iommu code. As for systematic way to disable active DMA, isn''t this similar to OS shutdown case when all the drivers are unloaded? Does kexec unload all of the device drivers? Once all the drivers are unloaded, there shouldn''t be any DMA transactions going on. I don''t know much about kexec flow but it sound like the high level flow is: 1) dom0 kernel shutdown all of the device dirver and then 2) call iommu_ops->suspend() or crash_shutdown() to disable all of the iommu hardware. Allen -----Original Message----- From: Andrew Cooper [mailto:andrew.cooper3@citrix.com] Sent: Wednesday, May 11, 2011 5:47 AM To: xen-devel@lists.xensource.com Cc: Kay, Allen M; Wei Wang Subject: Crashdump and IOMMU problems Hello, I have been debugging kexec interaction problems with XenServer and found that the problem lies in how Xen tares down the computer in a crash. The Xen kexec path does not touch IOMMU at all, which leaves the kexec native kernel with interrupt remapping enabled without realizing it. This leads to the kexec kernel failing to understand why its interrupts aren''t working. As a debugging measure, I have put iommu_ops->suspend() and iommu_disable_IR() on the kexec path and this ''fixes'' the problem, although it is far from safe. From a correctness point of view, Xen really does need to shutdown all IOMMU remapping before it jumps to the crash kernel. I know that kdump is a "seat of the pants best effort" in the best case, but there is more which Xen needs to do to help it along. I was considering adding a crash_shutdown function to iommu_ops which goes and twiddles the relevant disable bits, without saving state. However, disabling DMA remapping while transfers are still ongoing is likely asking for trouble. Seeing as people on here are likely to know far more than me on this subject: 1) Is there a systematic way to find and disable active DMA transfers, or indeed a systematic way to shut down PCI (etc) devices which is safe for the kexec path. 2) Are there any other PC subsystems which could do with being shut down in a sensible manor to make life easier for the kdump kernel? Thanks in advance, ~Andrew -- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 23:11 +0100 on 11 May (1305155493), Kay, Allen M wrote:> I believe Jan was involved with adding kexec support in iommu code. > > As for systematic way to disable active DMA, isn''t this similar to OS shutdown case when all the drivers are unloaded? Does kexec unload all of the device drivers? Once all the drivers are unloaded, there shouldn''t be any DMA transactions going on. > > I don''t know much about kexec flow but it sound like the high level flow is: 1) dom0 kernel shutdown all of the device dirver and then 2) call iommu_ops->suspend() or crash_shutdown() to disable all of the iommu hardware. >In Xen we might have to kexec on crash without knowing even which VM is driving the devices, so we can''t do a clean unload. It might be worthwhile just walking the PCI busses disabling everything, though. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Xen Platform Team Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> "Kay, Allen M" <allen.m.kay@intel.com> 12.05.11 00:11 >>> >I believe Jan was involved with adding kexec support in iommu code.No, I definitely wasn''t (nevertheless I''m interested in the subject). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel