The subject of interrupt forwarding just came up on lkml. How is this dealt with in Xen? I''m not currently a Xen user so I may be misunderstanding how it works. In Xen I believe you can assign a piece of hardware for exclusive use from a domain. How does this work for shared interrupts, what if the domain receiving the interrupt dies and can''t acknowledge it? Here''s my comment from lkml.... On Sat, 12 Mar 2005 10:55:21 -0500, Jon Smirl <jonsmirl@gmail.com> wrote:> I''ve tried implementing this before (on UML) and could not get around the > interrupt problem. Most interrupts on the x86 architecture are shared. > Disabling the IRQ at the PIC blocks all of the shared IRQs. This works > (hope your userspace handler is last on the shared handler list) until > you have a problem in userspace. > > Once you have a problem in userspace there is no way to acknowledge > the interrupt anymore. I tried to address that by maintaining a timer > and suspending the hardware through the D0 state to reset it. That had > some success. Not acknowledging the interrupt results in an interrupt > loop and reboot. > > The problem can be mitigated by choosing what slot your hardware to > put your hardware in. This can reduce the number of shared interrupts. > If you can get exclusive use of the interrupt this method will work. > > If I were designing a new bus I would make interrupt acknowledge part > of PCI config space in order to allow a single piece of code to > acknowledge them. Since we can''t change the bus the only safe way to > do this is to build a hardware specific driver for each device to > acknowledge the interrupt. > > Bottom line is that I could find no reliable solution for handing interrupts.-- Jon Smirl jonsmirl@gmail.com ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> The subject of interrupt forwarding just came up on lkml. How is this > dealt with in Xen? I''m not currently a Xen user so I may be > misunderstanding how it works. In Xen I believe you can assign a piece > of hardware for exclusive use from a domain. How does this work for > shared interrupts, what if the domain receiving the interrupt dies and > can''t acknowledge it?The paper on IO on the Architecture section of the web site is probably reference (other than the code). Shared interrupts aren''t good news for performance, but they work OK. If a driver domain locks up we can kill and restart it -- for safety we don''t rely on it to ack the interrupt. Ian> Here''s my comment from lkml.... > > On Sat, 12 Mar 2005 10:55:21 -0500, Jon Smirl > <jonsmirl@gmail.com> wrote: > > I''ve tried implementing this before (on UML) and could not > get around the > > interrupt problem. Most interrupts on the x86 architecture > are shared. > > Disabling the IRQ at the PIC blocks all of the shared IRQs. > This works > > (hope your userspace handler is last on the shared handler > list) until > > you have a problem in userspace. > > > > Once you have a problem in userspace there is no way to acknowledge > > the interrupt anymore. I tried to address that by > maintaining a timer > > and suspending the hardware through the D0 state to reset > it. That had > > some success. Not acknowledging the interrupt results in an > interrupt > > loop and reboot. > > > > The problem can be mitigated by choosing what slot your hardware to > > put your hardware in. This can reduce the number of shared > interrupts. > > If you can get exclusive use of the interrupt this method will work. > > > > If I were designing a new bus I would make interrupt > acknowledge part > > of PCI config space in order to allow a single piece of code to > > acknowledge them. Since we can''t change the bus the only safe way to > > do this is to build a hardware specific driver for each device to > > acknowledge the interrupt. > > > > Bottom line is that I could find no reliable solution for > handing interrupts. > > > -- > Jon Smirl > jonsmirl@gmail.com > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from > real users. > Discover which products truly live up to the hype. Start reading now. > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel >------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > If I were designing a new bus I would make interrupt acknowledge part > > of PCI config space in order to allow a single piece of code to > > acknowledge them. Since we can''t change the bus the only safe way to > > do this is to build a hardware specific driver for each device to > > acknowledge the interrupt.I believe this problem is eliminated under PCI express, since interrupts are message-based. Cheers, Mark ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On 12 Mar 2005, at 18:11, Ian Pratt wrote:> Shared interrupts aren''t good news for performance, but they work OK. > > If a driver domain locks up we can kill and restart it -- for safety we > don''t rely on it to ack the interrupt.Shared interrupts are slightly worse because the irq won''t get unmasked until all receivers say their work is done. If one lock sup it starves the rest -- until this is detected and that domain gets blown away. Fortunately the days of shared interrupts are numbered with MSI and PCI-Express. -- Keir ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, 12 Mar 2005 18:21:19 +0000, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > On 12 Mar 2005, at 18:11, Ian Pratt wrote: > > > Shared interrupts aren''t good news for performance, but they work OK. > > > > If a driver domain locks up we can kill and restart it -- for safety we > > don''t rely on it to ack the interrupt. > > Shared interrupts are slightly worse because the irq won''t get unmasked > until all receivers say their work is done. If one lock sup it starves > the rest -- until this is detected and that domain gets blown away.After you blow away the domain how do you acknowledge the interrupt? Is all hardware required to have a tiny driver in the supervisor to handling acking in this case? If you don''t ack it, it is going to keep interrupting.> > Fortunately the days of shared interrupts are numbered with MSI and > PCI-Express. > > -- Keir > >-- Jon Smirl jonsmirl@gmail.com ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On 12 Mar 2005, at 18:33, Jon Smirl wrote:>> Shared interrupts are slightly worse because the irq won''t get >> unmasked >> until all receivers say their work is done. If one lock sup it starves >> the rest -- until this is detected and that domain gets blown away. > > After you blow away the domain how do you acknowledge the interrupt? > Is all hardware required to have a tiny driver in the supervisor to > handling acking in this case? If you don''t ack it, it is going to keep > interrupting.We don''t go that far. A sensible approach would be to require the driver to be restarted, and to reset the hardware device, before unmasking. Or to rate limit each interrupt line to an administrator-configurable ''reasonable'' number of IRQs per second -- this might also catch bugs where drivers are not properly acking devices for other reasons. -- Keir ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, 12 Mar 2005 18:48:53 +0000, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > On 12 Mar 2005, at 18:33, Jon Smirl wrote: > > >> Shared interrupts are slightly worse because the irq won''t get > >> unmasked > >> until all receivers say their work is done. If one lock sup it starves > >> the rest -- until this is detected and that domain gets blown away. > > > > After you blow away the domain how do you acknowledge the interrupt? > > Is all hardware required to have a tiny driver in the supervisor to > > handling acking in this case? If you don''t ack it, it is going to keep > > interrupting. > > We don''t go that far. A sensible approach would be to require the > driver to be restarted, and to reset the hardware device, before > unmasking. Or to rate limit each interrupt line to an > administrator-configurable ''reasonable'' number of IRQs per second -- > this might also catch bugs where drivers are not properly acking > devices for other reasons.In x86 boxes almost everything is shared. Leaving the interrupt masked off will probably disable 20% of the hardware in the box. -- Jon Smirl jonsmirl@gmail.com ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, 12 Mar 2005 18:48:53 +0000, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> We don''t go that far. A sensible approach would be to require the > driver to be restarted, and to reset the hardware device, before > unmasking. Or to rate limit each interrupt line to anHave you tried cycling the cards to suspend S0 (power off) as a way of clearing the interrupt? -- Jon Smirl jonsmirl@gmail.com ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > We don''t go that far. A sensible approach would be to require the > > driver to be restarted, and to reset the hardware device, before > > unmasking. Or to rate limit each interrupt line to an > > administrator-configurable ''reasonable'' number of IRQs per second -- > > this might also catch bugs where drivers are not properly acking > > devices for other reasons. > > In x86 boxes almost everything is shared. Leaving the interrupt masked > off will probably disable 20% of the hardware in the box.That''s not my experience, certainly on modern server hardware. On systems with an IOAPIC its rare to see a shared interrupt, and even if you do you can typically avoid it just by swapping a couple of PCI cards around. Ian ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, 12 Mar 2005 19:59:15 -0000, Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> wrote:> > > > > We don''t go that far. A sensible approach would be to require the > > > driver to be restarted, and to reset the hardware device, before > > > unmasking. Or to rate limit each interrupt line to an > > > administrator-configurable ''reasonable'' number of IRQs per second -- > > > this might also catch bugs where drivers are not properly acking > > > devices for other reasons. > > > > In x86 boxes almost everything is shared. Leaving the interrupt masked > > off will probably disable 20% of the hardware in the box. > > That''s not my experience, certainly on modern server hardware. On > systems with an IOAPIC its rare to see a shared interrupt, and even if > you do you can typically avoid it just by swapping a couple of PCI cards > around. > > Ian >If I load video drivers two more drivers will share interrupts. 169 with have 4 devices on it instead of three. [jonsmirl@jonsmirl proc]$ cat interrupts CPU0 CPU1 0: 484259 448188 IO-APIC-edge timer 8: 0 0 IO-APIC-edge rtc 9: 0 0 IO-APIC-level acpi 14: 776 762 IO-APIC-edge ide0 15: 8040 7726 IO-APIC-edge ide1 169: 37509 0 IO-APIC-level libata, uhci_hcd:usb4, eth0 185: 5 0 IO-APIC-level ehci_hcd:usb1 193: 548 272 IO-APIC-level uhci_hcd:usb2, uhci_hcd:usb5 201: 0 0 IO-APIC-level uhci_hcd:usb3 209: 3 0 IO-APIC-level ohci1394 NMI: 0 0 LOC: 932357 932306 ERR: 0 MIS: 0 [jonsmirl@jonsmirl proc]$ -- Jon Smirl jonsmirl@gmail.com ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On 12 Mar 2005, at 20:07, Jon Smirl wrote:> [jonsmirl@jonsmirl proc]$ cat interrupts > CPU0 CPU1 > 0: 484259 448188 IO-APIC-edge timer > 8: 0 0 IO-APIC-edge rtc > 9: 0 0 IO-APIC-level acpi > 14: 776 762 IO-APIC-edge ide0 > 15: 8040 7726 IO-APIC-edge ide1 > 169: 37509 0 IO-APIC-level libata, uhci_hcd:usb4, eth0 > 185: 5 0 IO-APIC-level ehci_hcd:usb1 > 193: 548 272 IO-APIC-level uhci_hcd:usb2, > uhci_hcd:usb5 > 201: 0 0 IO-APIC-level uhci_hcd:usb3 > 209: 3 0 IO-APIC-level ohci1394 > NMI: 0 0 > LOC: 932357 932306 > ERR: 0 > MIS: 0Bizarre. Not only routed through an I/O APIC, but the IRQ numbers are large enough to probably be MSI vectors. Perhaps the mobo layout has legacy PCI slots sharing IRQ lines before reaching MSI logic in the chipset? Putting network on the same line as anything else is very sucky. Is this really a server-class board? -- Keir ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sun, 13 Mar 2005 10:01:42 +0000, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > On 12 Mar 2005, at 20:07, Jon Smirl wrote: > > > [jonsmirl@jonsmirl proc]$ cat interrupts > > CPU0 CPU1 > > 0: 484259 448188 IO-APIC-edge timer > > 8: 0 0 IO-APIC-edge rtc > > 9: 0 0 IO-APIC-level acpi > > 14: 776 762 IO-APIC-edge ide0 > > 15: 8040 7726 IO-APIC-edge ide1 > > 169: 37509 0 IO-APIC-level libata, uhci_hcd:usb4, eth0 > > 185: 5 0 IO-APIC-level ehci_hcd:usb1 > > 193: 548 272 IO-APIC-level uhci_hcd:usb2, > > uhci_hcd:usb5 > > 201: 0 0 IO-APIC-level uhci_hcd:usb3 > > 209: 3 0 IO-APIC-level ohci1394 > > NMI: 0 0 > > LOC: 932357 932306 > > ERR: 0 > > MIS: 0 > > Bizarre. Not only routed through an I/O APIC, but the IRQ numbers are > large enough to probably be MSI vectors. Perhaps the mobo layout has > legacy PCI slots sharing IRQ lines before reaching MSI logic in the > chipset? Putting network on the same line as anything else is very > sucky. Is this really a server-class board?It is a Dell Poweredge 400SC server. I don''t like how the net and SATA drives are on the same IRQ. I am running the latest linus bk kernel. The IRQ numbering system in the kernel has been changed. The board is just a standard APIC without MSI. No PCI Express. Xen is planned for desktops too, right? Interrupts are shared all of the time on desktop systems. It is not safe to assume you can get an exclusive one.> > -- Keir > >-- Jon Smirl jonsmirl@gmail.com ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On 13 Mar 2005, at 15:56, Jon Smirl wrote:> It is a Dell Poweredge 400SC server. I don''t like how the net and > SATA drives are on the same IRQ. > > I am running the latest linus bk kernel. The IRQ numbering system in > the kernel has been changed. The board is just a standard APIC without > MSI. No PCI Express. > > Xen is planned for desktops too, right? Interrupts are shared all of > the time on desktop systems. It is not safe to assume you can get an > exclusive one.But Xen does support shared IRQs. They''re not as good as a separate IRQ per device from an isolation pov, but would you really expect otherwise? -- Keir ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sun, 13 Mar 2005 16:01:19 +0000, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> But Xen does support shared IRQs. They''re not as good as a separate IRQ > per device from an isolation pov, but would you really expect > otherwise?Does Xen require the IRQ handler to run in the supervisor or domain? If it is in the domain and the domain dies, what do you do about the interrupt that keeps interrupting because there is no more code to acknowledge it? If you shut it off at the PIC you shut off all devices on that interrupt. A proposal was recently made on lkml that request_irq would also pass in a structure that would enable the kernel to acknowledge the IRQ without the handler being there. Xen could pass info like that from the domain back to the supervisor. Then in the domain dies the supervisor has the info needed to shut off the specific interrupt. On Fri, 11 Mar 2005 19:14:13 +0000, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:> I posted a proposal for this sometime ago because X has some uses for > it. The idea being you''d pass a struct that describes > > 1. What tells you an IRQ occurred on this device > 2. How to clear it > 3. How to enable/disable it. > > Something like > > struct { > u8 type; /* 8, 16, 32 I/O or MMIO */ > u8 bar; /* PCI bar to use */ > u32 offset; /* Into bar */ > u32 mask; /* Bits to touch/compare */ > u32 value; /* Value to check against/set */ > } >It might useful to add this to the main kernel API, and then over time modify all of the drivers to use it. If a driver does this it would be safe to transparently move it to user space like in UML or xen. I''ve been told that PCI Express and MSI does not have this problem.> > -- Keir > >-- Jon Smirl jonsmirl@gmail.com ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, 12 Mar 2005, Mark Williamson wrote:> I believe this problem is eliminated under PCI express, since interrupts > are message-based.I got corrected on this in the openib mailing list :-) the message signalled interrupts are even available on older hardware, it seems; sometimes but not always. That doesn''t help much but it is interesting ... ron ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Monday 14 March 2005 15:39, Ronald G. Minnich wrote:> On Sat, 12 Mar 2005, Mark Williamson wrote: > > I believe this problem is eliminated under PCI express, since interrupts > > are message-based. > > I got corrected on this in the openib mailing list :-) > > the message signalled interrupts are even available on older hardware, it > seems; sometimes but not always.I''d seen they''re available on some non-PCIE hardware. I assumed it was just recent / high end hardware (maybe just on PCI-X). (for instance, I think my workstation at Intel does use MSI interrupts and IIRC has PCI / PCI-X support). Does anyone know to what extent MSI is actually used these days?> That doesn''t help much but it is interesting ...Interesting indeed... Cheers, Mark ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Mon, 14 Mar 2005, Mark Williamson wrote:> Does anyone know to what extent MSI is actually used these days?from what I''ve seen people are really starting to count on it. It was an issue in openib that the AMD 8131 did not support MSE on PCI-X ron ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Mon, 14 Mar 2005, Ronald G. Minnich wrote:> > > On Mon, 14 Mar 2005, Mark Williamson wrote: > > > Does anyone know to what extent MSI is actually used these days? > > from what I''ve seen people are really starting to count on it. It was an > issue in openib that the AMD 8131 did not support MSE on PCI-XMSI sorry ron ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Mark Williamson <maw48@cl.cam.ac.uk> writes:> > I''d seen they''re available on some non-PCIE hardware. I assumed it was just > recent / high end hardware (maybe just on PCI-X). (for instance, I think my > workstation at Intel does use MSI interrupts and IIRC has PCI / PCI-X > support). > > Does anyone know to what extent MSI is actually used these days?Not very widely. The PCI card and the chipset has to support it. Most PCI cards don''t. Non Intel chipsets often don''t support MSI at all (e.g. on most AMD systems). Worst there are even some cards who advertise MSI support, but they lock up when you actually use it. PCI-Express requires MSI support though, but there are hardly any PCI-Express cards already available. MSI-X support is even worse, only a few highend cards seem to support it. Also the MSI support in Linux seems to be not entirely stable yet. -Andi ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel