Hi, With xen unstable I have the problem that the network stops working after a while. Heavy network traffic seems to make it more likely, it triggers almost every time I try to untar a kernel source tree, where the tarball comes from a NFS-mounted volume. The network card doesn''t receive interrupts any more according to /proc/interrupts. It works fine in native linux and also with xen 2.0. It''s a tulip card. Anyone has an idea what this might be or where to look? Gerd /proc/interrupts in native linux: CPU0 0: 613024 XT-PIC timer 1: 673 XT-PIC i8042 2: 0 XT-PIC cascade 5: 3460 XT-PIC sym53c8xx 7: 1 XT-PIC parport0 8: 2 XT-PIC rtc 9: 0 XT-PIC acpi, uhci_hcd 10: 0 XT-PIC Ensoniq AudioPCI 11: 1131 XT-PIC eth0 12: 446 XT-PIC i8042 14: 313 XT-PIC ide0 15: 9492 XT-PIC ide1 NMI: 0 LOC: 0 ERR: 0 MIS: 0 /proc/interrupts in domain0, unstable: CPU0 1: 113 Phys-irq i8042 5: 2521 Phys-irq sym53c8xx 9: 0 Phys-irq acpi 11: 779 Phys-irq hw-eth0 256: 0 Dynamic-irq ctrl-if 257: 15736 Dynamic-irq timer0 258: 0 Dynamic-irq console 259: 0 Dynamic-irq net-be-dbg NMI: 0 LOC: 0 ERR: 0 MIS: 0 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30 May 2005, at 13:52, Gerd Knorr wrote:> With xen unstable I have the problem that the network stops > working after a while. Heavy network traffic seems to make it > more likely, it triggers almost every time I try to untar a > kernel source tree, where the tarball comes from a NFS-mounted > volume. > > The network card doesn''t receive interrupts any more according > to /proc/interrupts. It works fine in native linux and also > with xen 2.0. It''s a tulip card. Anyone has an idea what this > might be or where to look?The new IRQ code in unstable is not so well tested on non-ACPI and non-IOAPIC systems (most of our boxes are SMP servers). May be worth looking at boot-time output of Xen vs. native Linux. Also, particularly if you have a null modem cable attached, add tracing on a debug key to see what Xen thinks the status of each IRQ line is (disabled, in_progress, etc.). Managing IRQ lines via legacy PIC is pretty straightforward so this can''t really be that hard a bug to fix I think (hope). :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> looking at boot-time output of Xen vs. native Linux. Also, particularly > if you have a null modem cable attached, add tracing on a debug key to > see what Xen thinks the status of each IRQ line is (disabled, > in_progress, etc.). Managing IRQ lines via legacy PIC is pretty > straightforward so this can''t really be that hard a bug to fix I think > (hope). :-)(XEN) IRQ states: (XEN) 0: (XEN) 1: guest (XEN) 2: (XEN) 4: inprogress (XEN) 5: guest (XEN) 9: guest (XEN) 11: guest No change before and after network freeze (#11 is the nic). Hmm. Gerd -- -mm seems unusually stable at present. -- akpm about 2.6.12-rc3-mm3 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31 May 2005, at 11:18, Gerd Knorr wrote:> No change before and after network freeze (#11 is the nic). > Hmm.Also interesting, for the guest irqs, is to print out integer: ((irq_guest_action_t *)desc->action)->in_flight And for all irqs, to print out string: desc->handler->typename Maybe the interrupt is getting stuck in flight, and therefore never unmasked at the IOAPIC... -- Keir PS. I''m looking at your Xen PAE patches now, so hopefully I''ll get them checked in later today. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Also interesting, for the guest irqs, is to print out integer: > ((irq_guest_action_t *)desc->action)->in_flight > > And for all irqs, to print out string: > desc->handler->typename(XEN) IRQ states: (XEN) 0: 45508 XT-PIC (XEN) 1: 1105 XT-PIC guest(0) (XEN) 2: 0 XT-PIC (XEN) 4: 7 XT-PIC inprogress (XEN) 5: 3140 XT-PIC guest(0) (XEN) 9: 0 XT-PIC guest(0) (XEN) 11: 24131 XT-PIC guest(0) ^^^^^ irq counter Nothing unusual, also the irq count is identical with what linux thinks about it in /proc/interrupts ... I''ve also noticed meanwhile that I can make the network work again by setting the interface down and up again. That seems to re-initialize something. Dones''t look like a xen bug to me. I suspect it''s a bug in the tulip driver, probably triggered by the timings being a bit different when running under xen.> PS. I''m looking at your Xen PAE patches now, so hopefully I''ll get them > checked in later today.Thanks. At least the "pae-support" patches being merged would be really nice as it touches many places and breaks all the time due to some updates in xen. The hypercall interface changes probably need some more work in the tools, I''m trying to get domU''s up with the changed hypercall interface. Well, I would if I wouldn''t debug the network issues at the moment ... cheers, Gerd -- -mm seems unusually stable at present. -- akpm about 2.6.12-rc3-mm3 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31 May 2005, at 12:36, Gerd Knorr wrote:> Thanks. At least the "pae-support" patches being merged would > be really nice as it touches many places and breaks all the time > due to some updates in xen. The hypercall interface changes > probably need some more work in the tools, I''m trying to get > domU''s up with the changed hypercall interface.Gerd, Having looked at the PAE patch and thought about the issue some more, I''m actually inclined not to split pfn and flags at the hypercall interface (e.g., mmu_update_t, do_update_va_mapping()). IIRC I think you wanted not to do this in the first place. :-) I now think that this is a rather arbitrary interface to impose on guests. Also it means that we increase the size of mmu_update_t on non-PAE, and change the interface unnecessarily for non-PAE. The thing that really changed my mind is that the no-execute bit is bit 63 of a page-table entry -- so we would need to ''cook'' that down into a 32-bit flags value to pass across the hypercall interface. Another arbitrary and frankly gross thing to have to do. I think changing val to u64 in mmu_update_t (for 32-bit PAE builds), and changing the val to u64 in update_va_mapping(), is probably the way to go. Of course, the machine address of the PTE to be modified will also need to be a u64. :-) The downside of this approach is that the C declarations of mmu_update_t, update_va_mapping, etc are different on 32-bit PAE builds. But only low-level guest code will touch those interfaces anyway, and there is unlikely to be code sharing between PAE and non-PAE at that level. What do you think? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31 May 2005, at 19:21, Keir Fraser wrote:> The downside of this approach is that the C declarations of > mmu_update_t, update_va_mapping, etc are different on 32-bit PAE > builds. But only low-level guest code will touch those interfaces > anyway, and there is unlikely to be code sharing between PAE and > non-PAE at that level. > > What do you think?Actually, we could export intpte_t and physaddr_t at the guest interface and declare mmu_update_t and friends in terms of those typedefs. This would also avoid needing different wrapper implementations of those hypercalls within Xen itself. Neat. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jonathan S. Shapiro
2005-May-31 18:32 UTC
Re: [Xen-devel] Hypercall interface changes for PAE
On Tue, 2005-05-31 at 19:21 +0100, Keir Fraser wrote:> The downside of this approach is that the C declarations of > mmu_update_t, update_va_mapping, etc are different on 32-bit PAE > builds....The rest of this idea I like, but the statement above seems like the wrong thing to do. The right thing to do is to preserve a uniform interface, breaking compatibility. This is a change being made in an unreleased major version -- an interface change that requires a recompile is perfectly okay. A further advantage of this change is that the structure is likely to be more compatible across different architectures, which will reduce source code changes in the hypervisor and also (in some cases) in certain commonly used bits of guest kernel code. shap _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31 May 2005, at 19:32, Jonathan S. Shapiro wrote:> The rest of this idea I like, but the statement above seems like the > wrong thing to do. The right thing to do is to preserve a uniform > interface, breaking compatibility. This is a change being made in an > unreleased major version -- an interface change that requires a > recompile is perfectly okay.I think my followup suggestion of hiding the interface difference behind typedefs is best (keeps the interface as uniform as possible at the source level; and non-PAE will continue to work even without recompilation). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, May 31, 2005 at 07:26:51PM +0100, Keir Fraser wrote:> > On 31 May 2005, at 19:21, Keir Fraser wrote: > > >The downside of this approach is that the C declarations of > >mmu_update_t, update_va_mapping, etc are different on 32-bit PAE > >builds. But only low-level guest code will touch those interfaces > >anyway, and there is unlikely to be code sharing between PAE and > >non-PAE at that level. > > > >What do you think? > > Actually, we could export intpte_t and physaddr_t at the guest > interface and declare mmu_update_t and friends in terms of those > typedefs. This would also avoid needing different wrapper > implementations of those hypercalls within Xen itself. Neat. :-)That certainly would be the way to go if we want to have different interfaces for PAE and non-PAE. I''m not sure it is a good idea to have different hypercall interfaces for PAE and non-PAE cases in the first place. What does this mean for the tools? Would these also be either PAE or non-PAE then? What about the option to maybe run non-PAE guests in PAE-xen in some translated shadow mode? That wouldn''t work then. I don''t think this would be a big problem though ... Gerd -- -mm seems unusually stable at present. -- akpm about 2.6.12-rc3-mm3 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31 May 2005, at 20:02, Gerd Knorr wrote:> That certainly would be the way to go if we want to have > different interfaces for PAE and non-PAE. I''m not sure it > is a good idea to have different hypercall interfaces for > PAE and non-PAE cases in the first place. > > What does this mean for the tools? Would these also be > either PAE or non-PAE then?At least some parts of the tools (e.g., libxc) will need re-building for PAE as they know about the structure of pagetables (2-level vs. 3-level and so on). Either that or we need to compile both cases into the library and auto-switch between implementations of some functions at run time. Either way, this problem isn''t solved by making the mmu hypercalls ''binary compatible'' across pae/non-pae. If we really do care about compatibility across pae/non-pae, I would do this by making the pte_val a u64 in all cases rather than splitting pfn from flags. Then everyone would just ignore the upper 32 bits on non-pae systems. This would waste no more space than 32-bit pfn + 32-bit flags. I''m fairly neutral on this: if we''re happy to make definitions of things like pte''s and physaddr''s be u64 even on non-pae then that may help binary compatibility in future at probably very little cost here and now. OTOH we can''t be fully binary compatible without shadow pagetables anyway, because the pagetable structure differs, so is the effort worth it?> What about the option to maybe run non-PAE guests in PAE-xen > in some translated shadow mode? That wouldn''t work then. > I don''t think this would be a big problem though ...I think we can live without this. But even if not then we haven''t burnt our bridges -- we could redirect those few hypercalls to a very slim compatibility layer. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, May 31, 2005 at 08:23:38PM +0100, Keir Fraser wrote:> > On 31 May 2005, at 20:02, Gerd Knorr wrote: > > >That certainly would be the way to go if we want to have > >different interfaces for PAE and non-PAE. I''m not sure it > >is a good idea to have different hypercall interfaces for > >PAE and non-PAE cases in the first place. > > > >What does this mean for the tools? Would these also be > >either PAE or non-PAE then? > > At least some parts of the tools (e.g., libxc) will need re-building > for PAE as they know about the structure of pagetables (2-level vs. > 3-level and so on). Either that or we need to compile both cases into > the library and auto-switch between implementations of some functions > at run time. Either way, this problem isn''t solved by making the mmu > hypercalls ''binary compatible'' across pae/non-pae.Disclaimer: Havn''t even looked at the tools code yet. I''d expect with a identical interface we''ll only have to add a PAE-enabled domain builder to boot domU (there are multiple already anyway, right?), whereas with different interfaces the split goes down to the shared lib which provides the hypercall interface to the tools. Point is I expect it being much easier to switch at runtime between pae and non-pae in the tools when the hypercall interface is identical. I might be wrong on this though.> If we really do care about compatibility across pae/non-pae, I would do > this by making the pte_val a u64 in all cases rather than splitting pfn > from flags.Agreed.> >What about the option to maybe run non-PAE guests in PAE-xen > >in some translated shadow mode? That wouldn''t work then. > >I don''t think this would be a big problem though ... > > I think we can live without this. But even if not then we haven''t burnt > our bridges -- we could redirect those few hypercalls to a very slim > compatibility layer.I''d prefeare 64-bit everythere over a compatibility layer. What about VT/Pacifica btw? As far I know the x86_64 guys plan to allow 32-bit vmx guests, right? I think to make that work xen must be able to translate non-PAE page table entries into x86_64 tables (which are very simliar to PAE). If this translation code is present anyway it''s probably only a small step to allow non-PAE guests in PAE mode as well. If we get this almost for free we might just implement it, although I see this more as a "nice-to-have" feature that something really important. Gerd -- -mm seems unusually stable at present. -- akpm about 2.6.12-rc3-mm3 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31 May 2005, at 21:38, Gerd Knorr wrote:> I''d expect with a identical interface we''ll only have to add a > PAE-enabled domain builder to boot domU (there are multiple > already anyway, right?), whereas with different interfaces the > split goes down to the shared lib which provides the hypercall > interface to the tools.I don''t think we want pae/non-pae builder made visible to higher-level tools. We can hide the auto-switch between builders within libxc itself. The same goes for domain save/restore code as well (although there is potentially scope for more binary-code sharing between pae/non-pae in this case).> Point is I expect it being much easier to switch at runtime > between pae and non-pae in the tools when the hypercall > interface is identical. I might be wrong on this though.The code that would be affected by an interface difference is precisely that code which manipulates page tables, and so is already broken by the different pagetable format. The only way unmodified non-pae code could possibly be made to work is by using shadow page tables. In that case we would hook off the call to e.g., do_mmu_update() very early anyway (off into shadow code). Hardly different really from jumping in the first place at a different hypercall function with different prototype: this latter would arguably be cleaner and less cluttered, as well as easily allowing us to support non-pae hypercall interface within pae xen. I really think that creating this interface inconsistency is not something to be worried about. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31 May 2005, at 22:18, Keir Fraser wrote:> The code that would be affected by an interface difference is > precisely that code which manipulates page tables, and so is already > broken by the different pagetable format. The only way unmodified > non-pae code could possibly be made to work is by using shadow page > tables. In that case we would hook off the call to e.g., > do_mmu_update() very early anyway (off into shadow code). Hardly > different really from jumping in the first place at a different > hypercall function with different prototype: this latter would > arguably be cleaner and less cluttered, as well as easily allowing us > to support non-pae hypercall interface within pae xen. > > I really think that creating this interface inconsistency is not > something to be worried about.I''ll repeat again, though, that I''m neutral on this. A larger mmu_update_t and extra register argument to update_va_mapping is not going to noticeably slow down non-pae i386. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> On 31 May 2005, at 20:02, Gerd Knorr wrote: > >> That certainly would be the way to go if we want to have >> different interfaces for PAE and non-PAE. I''m not sure it >> is a good idea to have different hypercall interfaces for >> PAE and non-PAE cases in the first place. >> >> What does this mean for the tools? Would these also be >> either PAE or non-PAE then? > > At least some parts of the tools (e.g., libxc) will need re-building for > PAE as they know about the structure of pagetables (2-level vs. 3-level > and so on). Either that or we need to compile both cases into the > library and auto-switch between implementations of some functions at run > time. Either way, this problem isn''t solved by making the mmu hypercalls > ''binary compatible'' across pae/non-pae.But it is greatly simplified, IIUC. If the hypercalls are binary compatible then you have only one set of hypercall interface functions and types, and switching between two *implementations* of pagetable-related stuff (only the things that actually need to be different) is quite straightforward. If you are trying to auto-switch between two hypervisor interfaces that use different types, then I can think of several ways of doing that, but they are all significantly more complicated: a) use preprocessor hackery to declare the interface headers twice with different symbol prefixes, b) use different hypercalls where needed (i.e. not just the same hypercall with different types) for PAE and non-PAE c) compile libxc twice and switch between the versions using dynamic linking.> If we really do care about compatibility across pae/non-pae, I would do > this by making the pte_val a u64 in all cases rather than splitting pfn > from flags. Then everyone would just ignore the upper 32 bits on non-pae > systems. This would waste no more space than 32-bit pfn + 32-bit flags.Right, that makes sense.> I''m fairly neutral on this: if we''re happy to make definitions of things > like pte''s and physaddr''s be u64 even on non-pae then that may help > binary compatibility in future at probably very little cost here and > now. OTOH we can''t be fully binary compatible without shadow pagetables > anyway, because the pagetable structure differs, so is the effort worth it?Yes, I think so: the code for the tools will end up being much simpler if PAE and non-PAE are as similar as possible. -- David Hopwood <david.nospam.hopwood@blueyonder.co.uk> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel